ORGAPET Section A2:
Evaluation Principles and Special Considerations for Evaluating Organic Farming Policies and Action Plans

Nic Lampkin
Aberystwyth University, UK

Johannes Michelsen
University of Southern Denmark

Matthias Stolze, Hanna Stolz
Research Institute of Organic Agriculture (FiBL), CH

Christian Eichert, Stephan Dabbert
University of Hohenheim, DE

 

Version 6, April 2008

A2-1          Introduction

Since the late 1980s, European countries have developed policies for organic farming on a number of levels. These include EC Regulation 2092/91 defining organic production; support for organic production, processing and marketing through agri-environment, rural development and structural measures; support for research and information dissemination measures; the development of national and EU action plans for organic farming; and continuing reforms of the main commodity elements of the Common Agricultural Policy. Increasingly, an integrated approach to the implementation of these policies has been developed in the form of action plans in several countries, and now also at the EU level. The evaluation of these action plans, and organic farming policies in general, provides the rationale for ORGAPET.

EU member states have invested more than €6 billion in policy support for organic farming, with investment in organic farming agri-environmental measures rising from €250 million annually in 1996, to €650 million annually in 2003 (Lampkin et al., 1999; Stolze and Lampkin, 2006), although this is still a tiny fraction of total public expenditure on agricultural policy.

From a governance perspective alone, there is a need to ensure that these resources are being invested appropriately. The impacts and cost effectiveness of these policies is an issue of increasing importance as the size of the organic sector, and the consequent demand for resources, increases. There is, in any case, a formal requirement for ongoing monitoring and evaluation of policies at national and EU level (e.g. current mid-term review of rural development and structural programmes). Competing claims on resources are likely to become louder, and there needs to be clear evidence of benefits to justify their continuing application to organic farming.

However, the evaluation of these impacts is not simple because organic farming works on a number of different levels, with multiple, sometimes conflicting, objectives and impacts. In addition, if the focus is on a single objective (e.g. hedge-planting), the benefits from supporting a systems approach such as organic farming may be less than can be achieved by more targeted measures. However, if the total benefits across all relevant objectives are considered, this may be sufficient to more than justify the costs of organic support compared with single-objective, single-measure schemes. To make a full assessment of the effects of such synergies and conflicts in a complex systems context, current evaluation methodologies need further development.

This document set outs the basic principles of evaluation as well as the specific issues relating to organic farming policies that have helped define the scope and structure of the ORGAP evaluation toolbox.

A2-2          Principles of evaluation

The evaluation approach used in this project builds on the European Commission’s requirements for ongoing monitoring and evaluation of policies at both a national and European level, which feed into the process of reviewing rural development and structural fund programmes. As part of ongoing concerns about the quality of programme monitoring and evaluation and the validity of outputs, DG XVI (Regional Policies and Cohesion) commissioned the MEANS programme (1994-1999) to develop a coherent set of approaches and methods for future evaluations. The results (EC, 1999) provide a framework for evaluation, as well as guidance on developing structures for collecting common indicator sets for monitoring and subsequent evaluation purposes.

However, it is appropriate to look in broader terms at principles of evaluation in order to determine whether the MEANS framework is appropriate and/or adaptations or alternative approaches are needed.

A2-2.1        Definitions

The following definitions of policy programme evaluation are based on the literature cited in each case. Definitions 1 to 4 derive from the context of political science, while definition 5 is specific to EU structural policy and tailored to the socio-economic evaluation of agricultural policy programmes (MEANS framework). The sixth definition is related to programme evaluation.

  1. Careful retrospective assessment of the merit, worth and value of administration, output and outcome of government interventions, which is intended to play a role in the future, practical action situation (Vedung, 1997).

  2. Judgement of the value of a (usually) public intervention with reference to criteria and explicit standards (e.g. its relevance, efficiency, sustainability, equity etc.). The judgement usually concerns the needs which have to be met by the intervention and the effects produced by it. The evaluation is based on information which is specially collected and interpreted to support the judgement. For example: evaluation of the effectiveness of a programme, cost-benefit evaluation of a project, evaluation of the validity of a policy and evaluation of the quality of a service delivered to the public. Certain definitions of evaluation exclude the judgement dimension and limit evaluation to an assessment, description or measurement of the intervention’s effects. Other, more restrictive, definitions limit evaluation to the ex-post estimation of effects. In certain contexts, evaluation focuses not on a public intervention but on a public organisation (e.g. evaluation of a university, hospital or EU institution) (Tavistock Institute, 2003).

  3. Evaluation as qualified monitoring is usually a feature of public decision-making systems. As impact assessment, evaluation is frequently commissioned on specific occasions. Whether permanent or periodic, monitoring or impact-assessing, however, evaluation is performed for either accountability, intervention improvement or basic knowledge advancement (Vedung, 1997).

  4. In a political sense, evaluation is the verification of a programme or an institution by a group composed for this purpose, which generates a certification on the basis of expertise, sometimes including recommendation for cessation, continuation or optimisation of the evaluated scheme (Kromrey, 2001).

  5. The evaluation of public intervention consists of judging its value in relation to explicit criteria and on the basis of information that has been specially gathered and analysed. The purpose of policy programme evaluations is to check the raison d’être of a public intervention, to confirm both reproducible success stories and failures not to be repeated, and to report back to citizens. Whereas private sector organisations have little need to justify or account for their actions since they are judged by the market, public sector organisations are obliged to evaluate their activities. The European Union is using a number of different forms of evaluation: managerial evaluation aimed at improving management, democratic evaluation used for accounting to citizens and pluralistic evaluation that tries to bring agreement between the partners involved in public action. Each of these three concepts are to be found within the experience of evaluating EU policies, with one of them predominating according to whether the evaluation concerned is ex-ante, ex-post or mid-term/collateral (EC, 1999).

  6. Evaluation is the systematic investigation of the merit or worth of an object (programme) for the purpose of reducing uncertainty in decision-making (Mertens, 1998).

In summary, policy evaluation involves the systematic gathering of information and assessment of a programme according to specific criteria in order to make judgements about the value of the programme, thus reducing uncertainty in decision-making about future actions. The assessment of value may relate to the goals of more than one specific interest group, including policy-makers, beneficiaries and third parties, and they may fulfil a range of purposes, from financial control and accountability to intervention improvement and knowledge advancement.

A2-2.2        Paradigms and conceptions of evaluation

Stockmann (2004) identifies two different paradigms of evaluation:

  1. Evaluation as an empiric-scientific process which is based on empirical research methods. Evaluation could therefore be seen as applied social research which has to respect special research conditions and emphasises the practical use of evaluation results.

  2. The second paradigm questions the ability of empirical science to identify reality objectively. Moreover, 'reality' is assumed to be socially constructed from varying and sometimes conflicting perspectives. As stakeholder groups take different power positions, particular interests can be 'over'-emphasised. Therefore, the aim of an evaluation is not in giving a description and assessment as near to reality as possible, but rather in changing reality in favour of the disadvantaged stakeholder groups. Evaluation in this sense is a transformative act.

EC (1999, Vol. 1:19-23) identifies three different conceptual approaches to evaluation: managerial, democratic and pluralistic.

  1. Managerial evaluation emphasises the efficient use (optimisation) of limited public resources between competing demands. In this context, the political dimension is not important – the evaluator operates to objective, scientific standards, guaranteeing the independence of the evaluation.

  2. The democratic concept emphasises the need for public authorities to be accountable to their citizens, involving them directly in the process with judgements made by elected representatives, through forums such as the Danish public consensus conferences and open juries in the UK. Evaluation professionals provide technical advice only.

  3. Pluralistic evaluation (also known as participative or stakeholder evaluation) attempts in an impartial way to integrate the technical components of the managerial approach and the political components of the democtratic approach to problem-solving and the involvement of stakeholders in the evaluation process.

Even though the discussion of paradigms is not yet concluded, in recent years there is increasing consensus that evaluations should consider the views and the needs of stakeholders. The pluralistic approach, to some extent represented by the MEANS framework and not inconsistent with the second of Stockmann’s paradigms, reflects this.

A2-2.3        Aims of programme evaluation

Stockmann (2004) identifies four possible aims of an evaluation (summarised in Figure A2-1):

  1. Provision of information and knowledge to the benefit of both the client and the target group of a programme. Areas of interest are, for example, the way a programme is running, the needs of the target group, whether the measures reach the target group, the acceptance of the programme, whether institutions are able to implement the programme efficiently and effectively, etc. The aim of the collection of knowledge and of the corresponding assessments (evaluation), conducted on the basis of evaluation criteria, is to identify control decisions and to reduce deficits in the programme.

  2. Exertion of direct and indirect monitoring of those who are involved in a programme. Apart from gathering knowledge about the programme, evaluation also discloses whether all actors involved fulfil their duties and whether they are competent and qualified to do so.

  3. Creation of transparency, enabling dialogue between the stakeholders of a programme. Evaluation thus provides transparency about the success and progression of the co-operation between stakeholders and decision-makers, providing a basis for learning processes.

  4. Documentation of achievements over time or legitimisation (sometimes without making use of results). Evaluations allow the verification of programme input/output and achieved impacts over time. With ex-post evaluations, the sustainability of a programme may be verified. Thus, financiers and operational organisations can be informed about resource efficiency and programme impact levels. Evaluation results may be used for legitimising programme operation.

Figure A2-1: Aims of programme evaluation

 

Source: Stockmann (2004)

According to Stockmann (2004), evaluation results may be used for:

Table A2-1 and Figure A2-2 illustrate this in more detail.

Table A2-1: Dimensions of evaluation research

Stages of the
programme process

Analysis
perspective

Perception
interests

Evaluation
concept

formulating the
programme/planning stage

ex-ante

“analysis for policy”

“science for action”

preformative/formative: proactive design, process-orientated, constructive

implementation stage

ongoing

both possible

formative/summative: both possible

impact stage

ex-post

“analysis for policy”

“science for knowledge

summative: summarising, making up the balance, result-orientated

Source: Stockmann (2004)

Evaluation may therefore have a formative (process-orientated, constructive or communication-promoting) design or a summative (result- oriented, concluding, accounting) character.

Except in the ex-ante case, evaluations may permit observation of operational processes and the identification of programme problems. In this context, important questions include: whether the measures are acceptable to stakeholders; which conflicts of interests are occurring and whether enough qualified personnel is provided for the implementation of measures; whether communication and co-ordination with the programme target group works well; and whether the innovations established by the programme are leading to programme objectives etc. (Stockmann, 2004).

Figure A2-2: Profile of tasks in an evaluation

   

Source: Stockmann (2004)

EC (1999, Vol. 1: 24-25) identifies three key aims which correspond, at least in part, with Stockmann’s perspectives:

  1. Verifying that public action responds to uncovered or insufficiently satisfied needs (legitimisation – is there a real problem to be addressed and can public intervention be justified by the failure of the private sector to act? In the EU context, this also refers to the balance between actions at EU level and considerations of national sovereignty.)

  2. Improving interventions, particularly through feedback and modification during the period of the programme.

  3. Accountability (responsibility/liability), reporting to political authorities and citizens on the results obtained and on the sound use of allocated resources. At the EU level, community value-added (relative to actions supported at the national level) is a key element.

Vedung (1997) similarly identifies control, knowledge and change as key potential aims of evaluations, with an emphasis on practical application rather than the pursuit of science.

A2-3          Examples of evaluation approaches

A2-3.1        The Stockmann (CEVAL) approach

Stockmann (2004) describes two relevant aspects of evaluation: the participative and multi-methodological aspects. The first aspect is related to the respective roles of persons concerned in the evaluation, while the second relates to the evaluation methods used.

A2-3.1.1        The participative aspect

According to Stockmann (2004), external evaluators are faced with a range of different stakeholders and actors and, therefore, a participative aspect is advisable. The participative aspect of programme evaluation means that evaluators and evaluatees work together as partners, instead of a relationship consisting of a controlling and controlled group of persons. The intention of the participative aspect is to enhance the validity of evaluation results, because valid assessment of measures and results is improved through the voluntary and active co-operation of all those involved. In Stockmann's view, the most effective method of participative evaluation is achieved where evaluators and evaluatees generate the processes and criteria of evaluation together. He suggests that they decide together which actors are to be involved and adjust their ideas alongside other participants. The substance and implementation of the evaluation should be connected as closely as possible with the interests and needs of all persons involved. According to Stockmann, such a procedure allows for continuous adoption of appropriate evaluation instruments, and enables a flexible reaction to changes of context within the processes of evaluation (Stockmann, 2004).

Based on Stockmann (2004), a participative evaluation consists of three stages. During the first stage, there is emphasis on the methodological knowledge of evaluators. The second stage is principally focused on an investigation of relevant information delivered mainly by the evaluatees. Through continuous intermediation and through workshops with stakeholders, the integration of all participants in the evaluation process can be guaranteed. Following data analysis using empirical social research methodology, the third stage of the evaluation is reached: the review and discussion of findings by both evaluatees and evaluators (and, in some cases, with stakeholders). At this stage, development strategies are worked out which are then implemented by those concerned and their organisations. Through the establishment of monitoring and evaluation systems, the progress of implementation may also be observed.

A2-3.1.2        The multi-methodological aspect

According to Stockmann (2004), another important aspect of the evaluation concept is the choice of suitable methods, as well as the precise development of instruments for data analysis. Lack of time often prevents the use of experimental design, which would usually be necessary for impact assessment, so a combination of qualitative and quantitative instruments is advisable. For the analysis of process-oriented data (e.g. programme steering, programme process, etc.), qualitative instruments are predominantly suitable, while quantitative investigational and analytical methods are chosen for the verification of aim achievements, impacts and causalities. The following methods are used frequently in evaluations:

The question of which methods are chosen depends on the respective task and aims of the evaluation but, generally, qualitative and quantitative research methods should be combined for the benefit of both clients and stakeholders (Stockmann 2004).

A2-3.2        The MEANS approach

The MEANS approach (EC, 1999) was developed for the European Commission to provide a basis for assessing, in particular, the socio-economic impacts of rural development and structural policies. Although the MEANS approach does not formally address the environmental impacts of policies, which have been addressed separately by OECD and EU initiatives (e.g. IRENA agri-environmental policy indicators 1, 2, 3, 4 and 6: EEA, 2005), some of the basic MEANS concepts are applicable also for this purpose.

The MEANS approach is designed to address two key issues which are also common to the evaluation of organic action plans (EC, 1999 Vol. 1:27-30):

  1. The policies to be evaluated are intended to achieve multiple goals, and these goals (and the values attached to them) will differ according to the time frame (short, medium, or long-term), the political level (EU, national or local) and the stakeholders involved (politicians, administrators, beneficiaries or other interest groups). Consequently, evaluations need to consider multiple, often interlinked, objectives that are difficult to observe and analyse in detail.

  2. The policy instruments used can be applied in multiple domains (e.g. fisheries, agriculture, transport, tourism) and a wide range of instruments are available, resulting in complex interactions between multiple objectives, multiple domains, multiple instruments and multiple geographic/institutional levels. This makes it difficult to identify causal links between policies and outcomes.

The MEANS approach, as set out in the six-volume MEANS collection (EC, 1999), proposes a multi-methodological approach. It addresses in detail the:

Since the development of ORGAPET is intended primarily to assist the European Commission in the evaluation of the European action plan for organic food and farming, it makes sense to base the evaluation approach on systems already in use by and familiar to the Commission, and to member states who are required to deliver national/local evaluations to the Commission with suggested modifications to address the specific context of organic farming policy (see below). However, it is already clear that a participative, multi-methodological approach, including quantitative and qualitative tools, will be needed. The different elements of ORGAPET will set out how many of the tools and approaches covered by MEANS can be implemented in the specific context of organic farming. In particular, ORGAPET Section A5 identifies the key issues to be considered in planning an evaluation and elaborates on the issues identified above.

A2-3.3        Evalsed: MEANS updated for 2007-2013

At the end of 2006, the European Commission published an internet-based framework for the evaluation of socio-economic development (Evalsed), with a particular focus on evaluation of regional policy within the 2007-2013 programming period. Evalsed is designed to update the MEANS framework to take account of developments in both the policy environment and in policy evaluation methodology since MEANS was originally published.

Evalsed consists of two main parts: a Guide and three Sourcebooks. The Guide is designed primarily for decision-makers – specifically those who design and manage evaluations in order to enhance decision-making on socio-economic development policies. It defines the role of evaluation in socio-economic development, discusses various ways to develop evaluation capacity and elaborates on evaluation methods as well as providing guidance on how to design and implement evaluations and how to ensure their quality. The Sourcebooks are of particular interest to practitioners and those wishing to impart or acquire evaluation skills relating to the following issues: evaluation approaches for particular themes and policy areas, evaluation methods and techniques and evaluation capacity-building.

Many of the issues discussed under Principles of evaluation above are also well-summarised in the History and purposes of evaluation section of the Evalsed Guide. Particular emphasis is given to evaluation as a learning process in order to improve programmes, not to undertake evaluations for their own sake and the need to integrate all stakeholders, in particular the intended beneficiaries, in the process.

A2-4          Specific considerations relating to organic farming policy evaluation

Organic farming policies have developed progressively in European countries since the 1980s (Lampkin et al., 1999; Dabbert et al., 2004; Stolze and Lampkin, 2006). Initially, policy measures focused on two main issues:

  1. the harmonisation of organic farming standards, including the development of a legal framework to protect the interests of consumers and bona fide producers and provide a basis for the development of the organic food market;

  2. the provision of financial support (typically on a per hectare basis as an agri-environmental measure) to support producers during conversion to organic production and, in many cases, the longer-term continuation of organic production, in recognition of the environmental and other societal benefits but also as a means to encourage growth in the sector.

These policies have contributed to the rapid growth in the organic food and farming sectors since the mid-1990s although, as might be expected with a small and rapidly developing sector, supply and demand have not grown in tandem. The increase in supply resulting from area-based support and other supply-push policies has led in several cases to market disruptions, with over-supply leading to falling prices. As a result, policy-makers have increasingly turned to demand-pull measures including promotion and public procurement, as well as marketing and processing projects, supported to a large extent through EU rural development and structural programmes. These policies are often integrated in an action plan framework at national or local (regional) level, in order to address specific local development bottlenecks as well as to achieve better complementarity between measures.

The development of the EU action plan (EC, 2004) was envisaged to provide an overarching vision and framework for development of policies at national and regional levels. It recognises the dual role of organic farming – responding to consumer demands for quality food and providing environmental and other societal benefits. It also looks at developing opportunities for promotion and research, but the primary focus is on improvements to the organic regulation and the implementation of control systems.

The policy mix which is now in place for organic farming in Europe means that the evaluation of organic farming policies and action plans faces the same challenges identified with respect to rural and structural programmes in general (and which themselves provide the basis for most organic farming policy measures).

However, there is another aspect to organic food and farming which is critical in the context of both policy development and evaluation, and that is the role of stakeholders, in particular producers and consumers. Unlike many agri-environmental and rural development policy measures, the organic farming concept was not developed by policy-makers and technical experts responding to a specific policy need. It has evolved since the early 20th century as a global movement for agricultural change, with committed individuals and producers working together to define standards and develop systems and practice, in order to achieve the movement’s goals of environmental protection, animal welfare, food quality and health, and social justice. At the same time, producers have had to find solutions to the challenge of maintaining financial viability while limiting the use of specific output-enhancing technologies and practices in pursuit of the broader goals. In the absence of direct policy support during most of the 20th century, organic producers turned to the market place and the willingness of committed consumers to pay a premium price, in order to maintain their financial viability.

As the goals of policy-makers and the organic movement have come closer together, the opportunities to provide policy support for common societal goals have been recognised, but the challenge of doing so, consistent with the market focus of the sector, has not always been met. With increasing policy support, the incorporation of organic standards in legal frameworks and the involvement of major retailers and food multinationals in the organic market, there has been increasing concern about the ‘institutionalisation’ of the organic movement and its ‘takeover’ by public and private corporate interests.

It is thus an issue of evaluations to consider the extent to which policy interventions change the dynamics of the original organic food and farming sector and to consider whether the effects are positive or negative. These question can hardly be answered without a systematic involvement of stakeholders associated with the organic movement – on national, regional or European level.

There have been a number of recent efforts to focus specifically on evaluation of organic farming policies at the European level. The EU-funded research project OFCAP (Organic Farming and CAP Reform (FAIR3-CT96-1794) looked at policies implemented in the period 1993-1997, with the results reported in the Organic Farming in Europe – Economics and Policy series (Volumes 1-10) published by the University of Hohenheim. Of particular relevance is the overview of policies implemented (Lampkin et al., 1999) and the policy impact assessment of Häring (2003), with Dabbert et al. (2004) providing an overview of the whole project. More recently, Häring et al. (2004) (funded by DG-ENVIRO) provided a first evaluation of the impacts of the Agenda 2000 reforms, in particular the main commodity measures and the rural development programme, on organic farming. From 2003 to 2007, the EU-funded EU-CEE-OFP project (Further development of European organic farming policies, with particular emphasis on EU enlargement (QLK5-2002-00917) analysed the effectiveness of organic farming policies (OFPs) in the old EU member states and Switzerland (EU15/CH); the regional and spatial impacts of existing and potential OFPs on farm structures and production in EU15/CH; the development of organic farming and the policy and regulatory environment in the new EU member states from central and eastern Europe; the development and implementation of organic farming regulations and markets in the CEE countries; the farm level economic impacts of OFPs, Agenda 2000 implementation and EU enlargement in selected countries; policy networks for developing OFPs in selected countries; and the involvement of policy-makers and stakeholders in identifying parameters for further development of European OFPs.

In developing approaches to the evaluation of the EU organic action plan and other organic farming policies, the ORGAP project needs to build on the experiences and knowledge developed in these earlier studies, in particular with respect to the application of evaluation methods, but also with respect to stakeholder involvement. ORGAPET Section A3 goes into more detail on the programme theory – i.e. the mechanisms by which policy interventions impact on the development of the organic sector – to provide a sound technical basis for the conduct of evaluations.

A2-5          Implications for the Organic Action Plan Evaluation Toolbox (ORGAPET)

The perspectives outlined above lead to the conclusion that the ORGAP evaluation toolbox should enable the careful (i.e. systematic) assessment of policy-driven organic action plans and other implemented policies, reflecting the interests not only of policy-makers but also other stakeholders. Evaluations should not be used primarily to advance scientific knowledge and understanding (although this may be a by-product) but should aim to find practical solutions to the further development of action plans and organic agriculture.

A2-5.1        Objectives of ORGAPET

The ORGAP evaluation toolbox is intended for use by the European Commission and EU member state governments and relevant stakeholders to develop and evaluate the European and national organic action plans. As many of the administrators and members of the action plan stakeholder groups involved are not expert evaluators or researchers, ORGAPET must be easy and practical to apply and parsimonious regarding the number of variables (objectives, measures and indicators) included in the toolbox. ORGAPET must include a limited number of evaluation procedures, serving different goals and focusing on different stakeholders.

In order to meet the different information requirements of ORGAPET, three levels of information are envisaged:

  1. A practical manual aimed at policy-makers and stakeholders engaged in policy development and in utilising the results of evaluation. At this level, information on evaluation methods is restricted to that necessary for the interpretation of results.

  2. A more detailed overview of each of the main areas of evaluation aimed at non-expert administrators and stakeholders who are involved in commissioning or carrying out evaluations. For each section of ORGAPET, a separate document is provided which aims to set out the key issues that need to be considered. Some of this information will already be well-known to professional evaluators, but the overview should still be useful for putting the specific organic action plan issues that need to be considered in context.

  3. The overviews in each section of the toolbox are supported by detailed annexes which fulfil different supporting roles: they may be examples of applications of specific techniques including working guidelines and results, or they may be information or data resources that might be valuable in interpreting results and making comparisons with other situations.

The objectives of the toolbox can be defined in more detail by addressing 8 questions suggested by Vedung (1997) as a basis for planning an evaluation:

  1. What is the aim of the evaluation? (e.g. control/accountability, improved knowledge, programme modification/change etc.)

  2. How is the evaluation organised? (Who commissions and who carries out the evaluation, and how are they interrelated?)

  3. What is the programme to be evaluated? (There needs to be a clear description to set the boundaries for what is to be evaluated)

  4. What is the public management process between input and output? (How has the decision been implemented, which agency was involved, what was the content, have target groups responded the way they promised, is there evidence of implementation failure?)

    5.

  5. What are the results (outputs/outcomes) of the programme (The focus should not only be on intended outcomes – are there other outputs, e.g. national action plans resulting from the EU action plan. What further outcomes could be included?)

  6. What are the factors explaining the results (programme/others)? (What are the effects/impacts of the policy; what are the causalities involved; how are these influenced by the general context of society?)

  7. What are the evaluation criteria and standards used for assessment? (What value statements are involved; what are the goals; what did the stakeholders expect?)

  8. How and by whom is the evaluation to be used? (Intention and reality – possible purposes include instrumental, political, internal reflection).

Taking each of these in turn, we can identify both the scope of ORGAPET and the sections of the toolbox that address them in greater detail.

  1. the aims of organic action plan evaluations for which ORGAPET is developed are to document policy actions and their impact, to make suggestions for changes in policy and to promote transparency in the development of action plans, in order to meet the multiple goals of both policy-makers and organic sector stakeholders (ORGAPET Section A2 (this document) and Section C1 address this);

  2. organisation: action plan evaluations will be commissioned by the European Commission and/or national/regional governments, with organic stakeholders as participants and either internal administrators or external researchers/consultants acting as evaluators. To achieve this ORGAPET needs to have an open and relatively simplistic approach to evaluation (ORGAPET Section A5 addresses the planning of evaluations).

  3. programme: a typology of European and/or national action plans is needed regarding action plan objectives, programmes and organisation, including? descriptions of implicit or explicit programme theories (i.e. how and why are distinct goals transformed to distinct outputs and how are outputs expected to influence outcomes within given contexts?). A first suggestion is to distinguish between action plans where programme theories focus a) mainly on market drivers, b) mainly on public good drivers or c) balancing market and public good approaches. This distinction will need further refinement on the basis of the variation among national action plans, but the number of categories should be limited. The distinction must be based on a characterisation of all objectives and measures within an action plan in relation to programme theory; this is the basis for defining outputs, outcome and impacts as well as evaluation criteria (see ORGAPET Section A3).

  4. public management: the focus is on the process of policy development and implementation, in particular the description of actors participating and their activities. The policy network analysis conducted as part of the EUCEEOFP project is relevant in this context, focusing on institutional change involving the farming community, agricultural policy, the food market and a potential institutional setting across the three, including a distinction between organic and non-organic actors (see ORGAPET Section B1 (process) and Section B2 (synergy and coherence)).

  5. results: data (qualitative or quantitative) with respect to a limited number of relevant indicators selected as basis for the evaluation (see ORGAPET Sections C1, C2, C3 and C4). Results should cover:

a.      output = direct actions taken by public agencies and/or stakeholders in terms of support, grants etc.;

b.      outcome = effect on organic sector in terms of its objectives;

c.      impact = effect of changes in the organic sector on the (higher level) goals of policy-makers

        Similar outputs and outcomes may be evaluated differently on the basis of the different objectives and values of stakeholders.

  1. explanations: did the action plan lead to the expected results (i.e. is the programme theory supported, rejected or influenced by intervening factors? This is important for conclusions regarding future actions and would require insights and examples from research and other sources (see ORGAPET Section D1).

  2. assessment: do the evaluation criteria and standards reflect the diversity of views held among public agencies and organic stakeholders? On the basis of the action plans and evaluations collected in WP3 and the WP2 workshops, a limited number of themes regarding action plan objectives, corresponding evaluation criteria and how they could be assessed from various viewpoints should be formulated. The emphasis on formulating relatively few but different evaluation criteria implies acceptance that they may be interpreted differently by various stakeholders and public agencies. Strategies for dealing with different values among stakeholders are required (see ORGAPET Section D1).

  3. the use of evaluations will be influenced by the choice of formative or summative evaluations: formative evaluation implies that evaluators both analyse the development and implementation of an action plan and intervene in it, while summative evaluation implies a clear separation of analysis and use of evaluation in terms of the timing of the evaluation and the involvement of evaluators in decision-making. Another aspect of use is the position held by policy makers and stakeholders on the evaluation as such, as well as on the recommendations. Evaluation may thus be part of a strategy for using the results to involve stakeholders as much as possible, whether in specific parts or all parts of the evaluation (which is one of the main reasons for developing participative evaluation).

A2-5.2        Integration of stakeholders

This topic has been identified as particularly important in the context of organic action plans. Systematic assessment requires a scientific basis (including relevant objective criteria and sound evidence) but, at the same time, needs to take account of subjective values of stakeholders. These values may be determined with reference to the published aims of the intervention but, in the case of action plans, many different values or goals may be represented and they are not necessarily prioritised, and they may reflect the values of governmental institutions more strongly than they do the direct beneficiaries or other stakeholders. In the absence of specific value statements, it becomes ambiguous whose values are being promoted. There is therefore a need to include consideration of these issues in any evaluation.

The approach chosen in this project is to involve stakeholders and their values, but stakeholders have different values that may depend on position in the production chain, material position, culture and many other aspects. Hence the toolbox to be developed needs to be able to grasp different sets of values – and any tool may be interpreted differently by different stakeholders. Vedung (1997) mentions two different ways that stakeholders might be integrated in evaluations:

  1. the north American way (Vedung, 1997: 69) – the evaluator decides which stakeholders to involve and in what capacity – this leads to a strongly differentiated evaluation, where the evaluator decides how to weigh the views of each stakeholder. This might involve evaluations where different stakeholders receive different answers and hence receive different support by means of the evaluation;

  2. the Swedish/Scandinavian way (Vedung, 1997: 76) – the relevant stakeholders are selected by the promoter of the evaluation and they are all involved directly in the execution of the evaluation – servicing with manpower and data or just overseeing its implementation. The aim is to serve the whole spectrum of relevant political and material interests and to reach a consensus.

The importance of values in any evaluation, the necessary inclusion of stakeholders in evaluating organic action plans and the various ways of integrating stakeholders are one important aspect that ORGAPET needs to address. The role and potential for integration of stakeholders is developed further in ORGAPET Section A4 while Section B3 addresses the evaluation of these processes.

A2-6    Conclusions

(with grateful acknowledgement to the Evalsed Golden Rules!)

Policy evaluation involves the systematic gathering of information and assessment of a programme according to specific criteria in order to make judgements about the value of the programme, reducing uncertainty in decision-making concerning future actions. The assessment of value may relate to the goals of more than one specific interest group, including policy-makers, beneficiaries and third parties, and they may fulfil a range of purposes, from financial control and accountability to intervention improvement and knowledge advancement. Evaluation takes place in order to improve programmes, not to undertake evaluations for their own sake.

Aligning the time cycles of evaluations with the time cycles of programmes and policies can help ensure evaluations make their maximum contribution. It is better to deliver an incomplete or imperfect evaluation on time than to achieve a 10% improvement in evaluation quality and miss the window of opportunity, when policy-makers and programme managers can use evaluation results.

Different stakeholders (e.g. policy-makers, professionals, managers and citizens) have different expectations of evaluation. If a major stakeholder interest is ignored, this is likely to weaken an evaluation, either because it will be poorly designed and/or because its results will lack credibility. Involving policy-makers and those responsible for programmes will ensure they take results seriously.

Evaluations must be fully integrated into programme planning and management. Policy makers should consider expressing their goals in terms of something measurable and to ensure availability of such measures. Programme managers need to think of evaluation as a resource: a source of feedback, a tool for improving performance, an early warning of problems (and solutions) and a way of systematising knowledge. Evaluation is not simply an external imposition. Equally, evaluators need to take on board the concerns of programme managers (and their partnerships) and try to take seriously their need for answers to difficult questions.

Getting good work from the diverse groups which make up the contemporary evaluation professional community needs bridge-building and team-building. Bridges need to be built at national, regional and European levels between the different traditions of evaluators, social scientists, economists, policy analysts and management consultants. At a micro-level, the priority is integration and the combination of different skills and competences within evaluation teams.

Evaluation is not only about looking back to rate success or failure and allocate blame. It has a contribution to make at every stage in the programme cycle. In particular, evaluation can strengthen programmes at the earliest stage by helping to unpick intervention logics and reveal weaknesses in programme design, allowing remedial action to be taken early.

It is not appropriate to gather large quantities of data in the belief that these will eventually provide answers to all evaluation questions. Data dredging is nearly always inefficient. This does not mean that data systems are not essential: they must be put in place at an early stage. However, by being clear about assumptions, by drawing on available theory and being clear about the type of evaluation that is needed, evaluations can be more focused and offer a higher yield for the resources expended.

The policy context is an important framework within which evaluations need to be located. Of course, policies change or are being constantly restated in different terms and with subtly changing priorities. However, it is always necessary to keep one eye on policy debates and decisions in order to ensure that evaluations are sensitised to policy priorities. The broader criteria that need to be designed into evaluations usually derive from the wider policy framework.

While all stakeholders are important, particular prominence needs to be given to one important and often neglected group: the intended beneficiaries of the programme interventions. Incorporating the voice of these intended beneficiaries in evaluations implies more than asking their opinions. It also implies incorporating their criteria and judgements into an evaluation and accepting that their experience and benefits are the justification for programme interventions. This is consistent with the logic of bottom-up, participative and decentralised approaches that are so common now in socio-economic development. It is also why responsive and participatory methods have become such an important part of the evaluators toolkit.

Organic action plan and organic farming policy evaluations need to take account also of the complex systems and multiple objective nature of the organic approach, with due attention to synergies and conflicts between objectives, and the different emphases that will be placed on these by different stakeholders.

Finally, we live in an imperfect world where resources are limited, administrators are not always efficient, co-ordination is imperfect, knowledge is patchy and data is often not available. It is nonetheless worth taking small steps, working with what is available and increasing, even marginally, the efficiency and legitimacy of organic action plans and policies. Even modest outputs can make a big difference especially when this is set within a longer-term vision to build capacity and allow for more ambitious evaluations in the future.

A2-7          References

Dabbert, S.; A. M. Häring and R. Zanoli (2004) Organic Farming: Policies and Prospects. Zed Books, London.

EC (1999) Evaluating Socio-economic Programmes. MEANS Collection Vols. 1-6. Office for Official Publications of the European Communities, Luxembourg.

EC (2004) European Action Plan for Organic Food and Farming, COM(2004)415 final, Commission of the European Communities, Brussels.

EEA (2005) IRENA: Indicator Fact Sheets and Data Sheets. CIRCA v3.2 [on-line], European Environment Agency, Copenhagen.

Häring, A. M. (2003) An Interactive Approach to Policy Impact Assessment for Organic Farms in Europe. Organic Farming in Europe: Economics and Policy, Vol. 10, University of Hohenheim, Stuttgart.

Häring, A. M., S. Dabbert, J. Aurbacher, B. Bichler, C. Eichert, D. Gambelli, N. Lampkin, F. Offermann, S. Olmos, J. Tuson and R. Zanoli (2004) Organic Farming and Measures of European Agricultural Policy. Organic Farming in Europe: Economics and Policy, Vol. 11, University of Hohenheim, Stuttgart.

Kromrey, H. (2001) Evaluation - einvielschichtiges Konzept. Begriffe und Methodik von Evaluierung und Evaluationsforschung. Empfehlungen fuer die Praxis. In: SuB Sozialwissenschaften und Berufspraxis. 24(2):105-132.

Lampkin, N., C. Foster, S. Padel and P. Midmore (1999) The Policy and Regulatory Environment for Organic Farming in Europe. Organic Farming in Europe: Economics and Policy, Vols. 1 and 2, University of Hohenheim, Stuttgart.

Mertens, D. (1998) Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. Sage, Thousand Oaks.

Stolze, M. and N. Lampkin (2006) European organic farming policies: an overview. In: Proceedings of the European Joint Organic Congress, 30-31 May 2006. (Andreasen CB et al. (eds.)), Danish Research Centre for Organic Food and Farming, Foulum.

Stockmann, R. (2004) Was ist eine gute Evaluation? CEVAL, Arbeitspapier Nr. 9. Centrum für Evaluation, Saarbrücken.

Tavistock Institute with GHK and IRS (2003) The Evaluation of Socio-Economic Development: The Guide, Tavistock Institute, London [on-line], http://www.evalsed.com/ accessed 21/08/2006. (NB. The original website accessed is now superseded by the EU Commission's Evalsed).

Vedung, E. (1997) Public Policy and Program Evaluation. Transaction Publishers, New Brunswick, New Jersey.

A2-8          Annexes

Annex A2-1  Impact Assessment Guidelines (EU Commission) (SEC(2005) 791)