|
|
|
 |
  |
|
|
|
|
|
ORGAPET should be cited as: Lampkin, N.; Schmid, O.; Dabbert, S.; Michelsen, J. and Zanoli, R. (eds.) (2008) Organic action plan evaluation toolbox (ORGAPET). Final output of the ORGAP research project (www.orgap.org) for the European Commission. Institute of Biological, Environmental and Rural Sciences, Aberystwyth University, UK and Research Institute of Organic Agriculture (FiBL), Frick, CH.
Navigation tips : Copyright : Contacts : Acknowledgements : Disclaimer : Glossary of technical terms (Evalsed)
The Organic Action Plan Evaluation Toolbox (ORGAPET) is a collection of different information/data sources and evaluation tools, including participative techniques, quantitative assessments and methods to identify relevant indicators, which can be used selectively to meet the needs of a particular assessment of national or EU organic action plans.
The toolbox is structured around ‘compartments’ or sections containing ‘tools’ fulfilling different functions. Each section contains an overview document and a series of annexes detailing a range of methodological approaches (including background documents, relevant data sources and other items), as well as examples of how these have been applied in specific cases, for example the evaluations and workshops conducted as part of the ORGAP project. The structure of ORGAPET is summarised below.
ORGAPET is aimed primarily at organic action plan managers/administrators and engaged stakeholders involved in action plan implementation and the commissioning (and possibly conduct) of evaluations. It does not attempt to provide the full methodological guidance that might be necessary for the training of expert evaluators, but expert evaluators should benefit from the specific organic farming policy examples presented and the information on relevant data sources (particularly in the annexes to each section).
The full version of ORGAPET is not aimed at stakeholders involved in overseeing the implementation of action plans or working with the results of evaluations, for example as participants in action plan steering groups. For this purpose, a manual for developing, implementing and evaluating organic action plans has been produced. (If using the CD-ROM version of ORGAPET, the manual referred to is the one accompanying the CD-ROM.) The manual is intended to be a tool for stakeholder involvement in future action plan development and implementation processes at national, regional and EU levels and to provide an introduction to the use of ORGAPET and the interpretation of evaluations.
ORGAPET covers all possible aspects of action plan evaluation - if at first this seems overwhelming, try a small part first!
an introduction to the EU and other organic action plans and to the mechanisms by which policy instruments affect the development of the organic sector,
an outline of the principles behind policy evaluation and the steps to take in planning evaluations,
a guide to the importance of engaging stakeholders at all stages in the policy process, and ways of doing this effectively.
Section A1: Introduction to organic action plans and the ORGAP Project covers:
the policy context for the EU Organic Action Plan and national action plans,
the rationale for evaluating these plans, as well as
the background to the ORGAP project and the organic action plan evaluation toolbox (ORGAPET); supported by
full details of the EU Organic Action Plan, information on its implementation progress, responses to the plan from key stakeholder groups, as well as a comparative assessment of national action plans conducted as part of the ORGAP project.
Section A2: Principles of policy evaluation with reference to organic action plans includes:
an introduction for policy-makers and other stakeholders to the nature and principles of policy evaluation,
examples of European policy evaluation frameworks, including the MEANS/Evalsed approach that is used by the European Commission for the evaluation of socio-economic programmes and that has been used as the basis for ORGAPET,
a review of the special characteristics of organic action plan evaluations and their implications for the design of ORGAPET,
further in-depth information on evaluation principles and organic farming policies from academic and governmental sources.
Section A3: How does policy influence the development of organic food and farming? considers:
how a sound theoretical understanding of the mechanisms by which policy interventions impact on a sector (programme theory) can help make policy interventions more effective; and
what specific programme theories might be applicable in the context of organic farming policy and action plans, supported by examples from previous research on organic farming policy development in Europe.
Section A4: Involving stakeholders in programme design, implementation and evaluation covers:
the role of and need for the inclusion of stakeholders at all stages in the policy process,
issues relating to the identification of appropriate stakeholders,
ways to ensure effective stakeholder involvement, including examples of participatory approaches used in the context of official action plans and in research workshops,
significant additional supporting material relating to official perspectives on stakeholder engagement and examples of stakeholder involvement in practice.
Section A5: Planning an evaluation details:
the issues that need to be considered and the practical steps that need to be undertaken in preparing for and conducting an evaluation,
ways of ensuring the quality of an evaluation, and
a checklist summarising the issues that need to be addressed.
the process of designing and implementing action plans, including the effectiveness of stakeholder engagement, and
the logic, synergies, priorities and failure risks of action plan content.
Section B1: Evaluating the process of programme design and implementation addresses:
the process of policy design and decision-making, including the potential for conflict or collaboration and the need for integration of good governance principles,
the process of policy implementation, including specific issues relating to the implementation of EU policy in member states and the potential for implementation failure; supported by
a checklist summarising the main issues to be considered and extensive references providing academic background to the issues as well as lessons learned from organic policy research.
Section B2: Assessing content and failure risk of action plans covers:
the use of logical analysis to structure programme objectives and assess programme coherence,
the assessment of synergies and conflicts within programmes,
the prioritisation of activities, and
the assessment of the failure risk of individual measures; supported by
a checklist summarising the main issues to be considered and annexes illustrating the application of these techniques in the context of the ORGAP project and the EU action plan.
Section B3: Methods for evaluating the level and nature of stakeholder involvement covers:
stakeholder perspectives and how they influence potential involvement at different stages,
issues to consider in evaluating stakeholder involvement, summarised in a checklist, and
examples of techniques such as network analysis that can be used to support evaluation.
procedures to support the identification and measurement of the effects of organic action plans on the organic sector and on wider policy goals; including
identification of relevant objectives, indicators and criteria for assessing performance,
suggestions for generic indicators with links to data sources and methods, and
the use of expert judgement techniques in situations where data is poor, or cause/effect relationships are unclear.
Section C1: Methods for identifying action plan and policy objectives considers:
how both implicit and explicit objectives can be identified and clarified as a basis for assessing the actual achievements of actions plans, using the logical analysis approach set out in Section B2,
how differentiating between hierarchical levels of objectives can help identify the goals of different stakeholder groups,
possible generic objectives that might be applicable to action plan evaluation, illustrated with reference to the EU Organic Action Plan, the IFOAM principles and the Commission's strategic guidelines for rural development.
Section C2: Methods for defining indicators considers:
the nature and classification of indicators to reflect different types of programme effects,
how appropriate indicators can be defined using impact statements and effects diagrams to make the link between policy actions and objectives,
how to ensure indicator quality, using the EU Organic Action Plan as an example, and
examples of indicators used in other contexts (rural development, environmental impact).
Section C3: Generic indicators provides:
a set of key indicators reflecting different categories of effect and different objectives; with a distinction made between
primary indicators that are likely to be relevant and quantifiable with respect to most action plan evaluations, and
secondary indicators that may be relevant only in specific circumstances or may be more difficult to quantify; supported by
methodology and data factsheets, data sources and examples of relevant indicators used in other contexts.
Section C4: Evaluating policy outcomes using stakeholder feedback and expert judgement covers:
techniques that can be used where indicators are difficult to quantify, or causal relationships between policy actions and the final impacts are difficult to establish, due to the number of intermediate effects or the complexity of interacting elements; including
stakeholder feedback, focus groups and more formal expert judgement methods such as Delphi and Nominal Group Technique; supported by
examples of how the techniques have been applied in organic policy research contexts and guidelines for their application.
techniques for integrating and interpreting results from complex evaluations; and
examples of evaluations of organic action plans that have previously been conducted.
Section D1: Integrating and interpreting results covers:
the range of issues that needs to be addressed when interpreting results, including how to interpret results from multiple objectives, allowing for trade-offs and conflicts and the priorities of different stakeholders;
utilising experts (including appropriate stakeholders) to make judgements based on their direct knowledge and understanding of specific issues;
formal methods, such as multi-criteria analysis and cost-benefit analysis, that can be used to support the synthesis process.
Section D2: Examples of national action plan evaluations covers:
evaluation experiences from Germany, Denmark, Netherlands and England, as well as
an assessment of the lessons to be learned from these evaluations prepared as part of the ORGAP project.
ORGAPET navigation tips
To move between sections of ORGAPET, use the links at the top of the screen.
To move within a section, use the links on the left side.
To return to the home page, click on Overview (Home) in the top panel.
Within a document:
a) links to another part of the same document will take you to the
new point in the document - to go back, use the back button on your
browser or the links on the left side.
b) links to a different section will result in the section being
opened in place of the existing one - to go back, use the back
button on your browser.
c) links to a web page will open in a new window - if using tabbed
browsing, CTRL+SHIFT+Click will open the link in a new tab.
d) links to a document will open the document within the section -
to go back, use the back button on your browser or the links on the
left side.
Where possible, references have been included in pdf format - a direct link to the document is possible where the title is formatted as a hyperlink. Alternatively, the publication series or publisher may be formatted with a link to the publisher's website.
To check for updates to ORGAPET, including new annexes and data sources, click on Updates.
The widespread distribution and use of this product is encouraged free of charge, subject to recognition that the Institute of Biological, Environmental and Rural Sciences, Aberystwyth University, Wales and the Research Institute of Organic Agriculture (FiBL), Frick, Switzerland are the lead organisations responsible for its development and maintenance and retain the copyright for ORGAPET.
Comments on ORGAPET content and enquiries concerning support with the application of ORGAPET are welcome:
Dr. Nic Lampkin, Institute of Biological, Environmental and Rural Sciences, Aberystwyth University, Llanbadarn Campus, Aberystwyth, SY23 3AL, Wales. Tel: +44 (0)1970 622248. Fax: +44 (0)1970 611264
Mr. Otto Schmid, Research Institute of Organic Agriculture (FiBL), Ackerstrasse, Postfach, CH-5070 Frick, Switzerland. Tel: +41 62 865 7272. Fax: +41 62 865-7273.
The editors acknowledge the contribution of the individual authors indicated in each section. For full details of the project partnership responsible for the development of ORGAPET, see www.orgap.org. The input of the project advisory committee, and the participants in the ORGAP workshops, into the design and refinement of ORGAPET is also gratefully acknowledged, as is the input of Ian Jeffreys into the creation of the electronic version of ORGAPET, and the input of Anne-Marie Sherwood into the final copy-editing and checking process.
ORGAPET has been developed as part of the EU project "European Action Plan for organic food and farming - Development of criteria and evaluation procedures for the evaluation of the EU Action Plan for Organic Agriculture” (ORGAP: www.orgap.org, CT-2005-006591). The editors and authors gratefully acknowledge financial support for this project from the Commission of the European Communities under the Sixth Framework Programme for European Research and Technological Development (Priority 8.1 Policy-oriented Research (SSP), Area 8.1.1 Sustainable management of Europe’s natural resources: B. 1.2 Tools and assessment methods for sustainable agriculture.)
The content of ORGAPET does not necessarily reflect the Commission’s views and in no way anticipates the Commission’s future policy in this area. The contents of ORGAPET are the sole responsibility of the editors and the authors indicated in each section. The information contained herein, including any expression of opinion and any projection or forecast, has been obtained from sources believed by the authors to be reliable but is not guaranteed as to accuracy or completeness. The information is supplied without obligation and on the understanding that any person who acts upon it or otherwise changes his/her position in reliance thereon does so entirely at his/her own risk.
