Nic Lampkin and Phillipa Nicholas
Aberystwyth University, UK
Version 6, April 2008
A fundamental part of any evaluation is to determine the objectives of the programme or individual policy measure, whether they have been achieved and how well they have been achieved. This applies in both a formative context in which the information might be used to modify the measures or reallocate resources where there is failure to achieve an objective or to achieve it to a high enough standard, and in a summative context where the outputs, results and impacts relative to use of resources may be the main focus.
Evaluation of the achievement of objectives may be on the basis of:
pass/fail (either the objective has or has not been achieved);
grade (the level of achievement – e.g. a satisfactory to a high level); or
incremental change (the situation is better or worse than previously).
To assess the achievement or otherwise of objectives, there is a need to define relevant indicators which represent the core issue in the objective (see ORGAPET Sections C2 and C3), as well as criteria or benchmarks against which the indicator values can be assessed.
Well-defined objectives may already embody the key information needed for their assessment, particularly if they are defined using SMART (specific, measurable, achievable/accepted, relevant/realistic, time-bound/dependant) criteria. An example might be:
To convert 100 farms or 10000 ha to organic management by 2005.
This is specific because of the focus on conversion of holdings, measurable and time-bound because of the targets and dates stated, relevant to the broader policy goals (see below) and, depending on context, achievable provided that the necessary resources (e.g. funds to pay support payments) are made available. Contained within the objective are the indicators (no. of farms, area of land) and the benchmark criteria (100 holdings, 10,000 ha and date) that can be used to assess whether or how well the objective has been achieved. In practice, action plan or policy goals are rarely defined in this way, except perhaps the overall target, but sometimes these targets are more aspirational than achievable.
|
Box C1-1: SMART Objectives The term SMART is widely used in
the context of objective setting in management and evaluation literature.
There is no clear original source, and different
sources use different terms, but the basic ideas are similar. The SMART criteria as defined by
EC
(2001) are: Measurable: Objectives should define a desired future state in measurable terms, so that it is possible to verify whether the objective has been achieved or not. Such objectives are either quantified or based on a combination of description and scoring scales. Accepted*: If objectives and target levels are to influence behaviour, they must be accepted, understood and interpreted similarly by all of those who are expected to take responsibility for achieving them. Realistic*: Objectives and target levels should be ambitious – setting an objective that only reflects the current level of achievement is not useful – but they should also be realistic so that those responsible see them as meaningful. Time-dependent: Objectives and target levels remain vague if they are not related to a fixed date or time period. *Achievable
is also sometimes used, but this is more closely linked to Realistic, in
which case |
While objectives may be expressed in a verifiable way which specifies clearly the expected achievements, they may also be expressed in a more political or rhetorical way, making it difficult to connect them to a precise achievement. The evaluation process may uncover implicit objectives, i.e. cause-and-effect assumptions that indirectly explain how an objective may be achieved. Identifying and clarifying objectives is therefore a necessary step towards reconstructing the intervention logic, judging its internal coherence (see also ORGAPET Section B2) and asking questions and defining evaluation criteria. To do this, the evaluation should set out for each objective:
a statement of the objective
possible ambiguities in some of the terms and clarifications proposed
means proposed for checking whether the objective has been achieved
date at which the objective has to be achieved.
Objectives can be identified in the official documents that instituted the intervention and in the documents framing it at a higher political or strategic level. When the statement of an objective contains ambiguities, clarifications can be proposed based on interviews with key informants, recognising that there is a continuum between more or less explicit objectives, which may include:
an objective clearly stated in the official documents
an ambiguous objective clarified with the stakeholders' help
an intermediate effect that has to be obtained if an objective is to be met (implicit objective)
an effect expected by the stakeholders but not initially anticipated, even implicitly.
All these elements warrant explanation in the report but the status of each one must be specified.
If the intervention is framed by numerous documents at a political or strategic level, or if these documents also contain many objectives, the evaluation team may be faced with an over-abundance of objectives. Rather than presenting them all, the evaluation team should underline this fact and concentrate on the most important ones in the context of the evaluated intervention.
From previous research work on organic farming policies, it is clear that there are two levels in the hierarchy of objectives that need to be considered in particular:
organic sectoral-level objectives which focus on the development (growth and improvement) of the organic sector, and
societal-level objectives which focus on broader policy goals, where the expectation is that growth and improvement of the organic sector will make a positive contribution.
From a policy-maker’s perspective, the development of the organic sector is more a means to an end in pursuit of societal-level objectives, not an end in itself, whereas organic sector stakeholders are more likely (but not exclusively) to see the development of the organic sector as an end in itself.
These contrasting organic sector and societal-level goals are reflected in the distinctions made between different types of indicators in existing evaluation frameworks (see ORGAPET Section C2), but they also highlight the importance of considering the values, goals and priorities of different groups of stakeholders, not only the ‘owners’ of the top-level objectives (see also ORGAPET Section B3).
With these issues in mind, it is clear that differentiated analyses are needed which reflect the type of evaluation (as defined in ORGAPET Section A5), the hierarchy of objectives and the interests of different stakeholder groups.
As indicated above, the starting point for identifying relevant objectives is published sources, in particular official documents, including:
The EU action plan goals and action points (discussed in more detail below)
The enabling regulations (e.g. rural development 2007-2013: 1698/2005)
The rural development strategic priorities at EU and member state levels (see also Annex C1-2 concerning organic farming’s contribution to these priorities)
The national action plans goals and action points (see Annex A1-8 for overview)
National rural development plans (refer to national sources but cross-refer to EU documents)
Specific support scheme objectives as set out in the scheme documents (refer to national sources but cross-refer to EU documents)
The new and original EU regulations defining organic agriculture
The IFOAM baseline standards (norms) and principles
Other organic farming literature
As the EU Action Plan for Organic Food and Farming is the main focus for the ORGAP project, we use the vision and the action points as a starting point for identifying the key objectives for evaluation.
In general terms, organic action plans exist to promote organic food and farming. The EU action plan sees a dual purpose for this:
In designing a global policy concept for organic farming, the dual societal role of organic farming should be recognised.
1. Organic food marketing, responding to the concerns of some consumers, should therefore be rewarded by the markets and hence be financed by the consumers. The development of organic farming will, in this respect, be governed by market rules.
2. Organic land management is known to deliver public goods, primarily environmental, but also rural development benefits and, in certain respects, it may also result in improved animal welfare. Seen from this angle, the development of organic farming should be driven by society.
In order to ensure stable market development there needs to be a balance between supply and demand.
This dual role ‘vision’ for the action plan embodies some key policy objectives (market-focus, economic development, provision of public goods) although they are not clearly stated as objectives, and there is a need to make these more explicit for evaluation purposes (see below). However, the individual action points are more explicit and these can provide a starting point:
Develop an information and promotion campaign by amending Reg. 2826/2000 (internal market promotion), launching a multi-annual EU-wide information and promotion campaign to inform consumers, public institutions canteens, schools and other key actors in the food chain about the merits of organic farming, especially its environmental benefits, and to increase consumer awareness and the recognition of organic products, including recognition of the EU logo, in co-ordination with member states and professional organisations.
Establish and maintain an Internet database listing the various private and national standards (including international standards and national standards in main export markets) compared to the Community standard.
Improve the collection of statistical data on both production and marketing of organic products.
Allow member states to top-up with aids the EU support devoted to fruit and vegetable sector for producer organisations involved in organic food.
Develop a web-based menu listing all EU measures that can be used by the organic sector in relation to production, marketing and information.
Strongly recommend member states to make full use within their rural development programmes of the instruments available to support organic farming, for example by developing national or regional action plans focusing on:
· quality schemes to stimulate demand;
· actions to benefit the environment;
· incentives to encourage whole farm conversion;
· investment support for organic as non-organic farmers;
· incentives for producers to facilitate distribution and marketing and supply chain integration;
· support for extension services; training and education covering production, processing and marketing;
· organic farming as the preferred management option in environmentally sensitive areas.
Strengthen research on organic agriculture and production methods.
Make the regulation more transparent by defining the basic principles of organic agriculture.
Ensure the integrity of organic agriculture by reinforcing the standards and maintaining the foreseen end dates of the transitional periods.
Complete and further harmonise the standards for organic agriculture
Establish an independent expert panel for technical advice.
Set thresholds for adventitious presence of GMOs and clarify labelling provisions relating to GMOs in Reg. 2092/91.
Improve the performance of the inspection bodies and authorities by introducing a risk-based approach targeting operators presenting the highest risk in terms of fraudulent practices, and by requiring cross-inspections under Reg. 2092/91.
Continue the ongoing work in the JRC to develop sampling and analytical methods which can be used in organic farming.
Member states should look at using CAP management land parcel identification for the location and monitoring of the land under organic farming.
Ensure better coordination among inspection bodies and between the inspection bodies and the enforcement authorities under Reg. 2092/91.
Develop a specific accreditation system for inspection bodies under Reg. 2092/91.
Publish an annual report from the member states on the supervision of approved inspection bodies including statistics on type and number of breaches.
Improve procedures for establishing technical equivalency and inclusion of third countries, including replacing national derogations with a single, permanent list of recognised inspection bodies operating in third countries, taking account of the different climate and farming conditions and the stage of development of organic farming in each country and offering all imported products access to the EU logo.
Compare EU, Codex Alimentarius and IFOAM standards and increase efforts towards global harmonisation and development of multilateral equivalency. Support capacity-building in developing countries under EU development policy by facilitating information on using general support instruments for organic agriculture and other measures.
Reinforce recognition of EU organic farming standards and inspection systems in third countries by obtaining a negotiation mandate from the Council.
A first stage in a summative evaluation of the plan would need to involve the assessment of what has been delivered on each of the action points, by when, and with what resources. From a formative perspective, issues of process, programme coherence and stakeholder engagement with respect to each action point would be a key focus.
This section addresses the definition of key objectives applicable to both the EU action plan for organic food and farming and to national organic action plans/policies, as a necessary first step to the definition of relevant indicators (see ORGAPET Section C2). In terms of the concept of a hierarchy of objectives, the focus here is on intermediate level objectives which justify (determine the relevance of) the individual action points (operational objectives). This assumes that global objectives are captured by the dual public good and market vision set out in the EU organic food and farming action plan (see above).
From the published documents and stakeholder workshops as described above, it may be possible to identify a large variety of objectives but impossible to evaluate each individually, given resource constraints. It will therefore be necessary to identify a small number of key (higher level) objectives which can form the focus for evaluation. This can be achieved by informal or formalised processes. The informal process would be to revisit the hierarchy of objectives and to answer the question “why?” for each objective. This should achieve the result that several sub-objectives could be grouped under one main objective which could be the focus for evaluation (see above). The formal approach would be to follow the procedures set out in ORGAPET Section C2 to identify impact statements and, through clustering procedures, identify a limited range of indicators that relate to individual clusters of impact statements and associated objectives.
As part of the ORGAP national workshops held in April 2006, participating stakeholders were asked to comment on the relevance of a set of generic objectives intended to provide the basis for the specification of generic indicators (ORGAPET Section C3). The detailed results of the workshop discussions are presented in Annex C3-5 – the main relevant outcomes only are summarised here. The development of the proposed set of generic objectives was based on experiences from previous policy evaluation work in the OFCAP and EU-CEE-OFP projects, classified according to sectoral or societal relevance.
1. Maintaining and enhancing the technical and financial performance of organic farms and related food-sector businesses
2. Increasing the scale of the organic sector
3. Meeting consumer demands for choice and quality, safe, affordable food
4. Better regulation, i.e. improving transparency of organic farming regulation and reducing bureaucracy
5. Maintaining and enhancing the integrity of organic principles and organic food
6. Promoting understanding of the concept and potential of organic farming in society
7. Promoting the sustainable use of natural resources
8. Maintaining and enhancing the environment (including biodiversity, pollution and climate change issues)
9. Maintaining and enhancing animal health and welfare
10. Maintaining and enhancing the social and economic wellbeing of rural communities
11. Maintaining and enhancing the competitiveness of European agriculture
12. Promoting public health and food security
Participants were asked to consider the list of actions and policy measures identified in the European action plan and their national action plan/organic farming policy context, to determine whether the list of generic objectives was appropriate and whether any key objectives had been omitted. The main results from the workshop were:
1. Maintaining and enhancing the technical and financial performance of organic farms and related food-sector businesses
In the English (ENG) workshop, it was pointed out that ”financial performance” should actually read financial viability and in the Czech (CZ) and Dutch (NL) workshops, it was felt that the emphasis seemed to be on production and that processing and marketing could be specifically mentioned instead of “related food-sector businesses”. The German (DE) group discussed the objective and concluded that it encompasses two separate aims. The participants therefore decided to split the objective into “agricultural production” (objective 1A) and “processing and marketing” (objective 1B).2. Increasing the scale of the organic sector
In Andalusia (AND), it was suggested that “developing the domestic market” was a crucial part of increasing the scale of the organic sector, therefore this was added as a qualifying statement onto this objective.3. Meeting consumer demands for choice and quality, safe, affordable food
In ENG it was felt that objective 3 (as a sectoral objective) should include something about extending the market into areas such as fibre and fuel, but should also link with the rural development plan to include tourism. It should also be modified to suggest expansion of availability, rather than just meeting existing consumer demand – i.e. increasing the availability and meeting consumer demands for choice and quality, safe, affordable food, fibre and other products. Participants in the DE workshop felt that objective 3 included different aims that were partly in conflict with one another.4. Better regulation, i.e. improving transparency of organic farming regulation and reducing bureaucracy
In the NL, it was suggested that the objective be worded as “Better regulation, i.e. improving transparency of organic production, better market level playing field (harmonisation of the European market) and reducing bureaucracy. In AND, it was suggested that “integrating social standards” be included at the end of this objective.5. Maintaining and enhancing the integrity of organic principles and organic food
It was felt in the NL that a good definition of “integrity” is missing and therefore this word should be left out of the objective. In DE, a new objective was suggested by the certification participant: ““Harmonisation of the implementation of the guidelines, for example with harmonised and effective implementations” and it was decided by the group that this could be incorporated into objective 5.6. Promoting understanding of the concept and potential of organic farming in society
In ENG, improving the understanding of the impacts of organic farming was put forward as a new objective. It was suggested that this could be incorporated into objective 6 by stating that the promotion and development of understanding of the concept and potential of organic food and farming should be based on sound evidence. Also in ENG, consumer education was thought to be missing from list. The suggestion was made that this was covered in objective 6 which includes “promoting understanding…”. It was also suggested in ENG that food be included in the objective as well. The final objective 6 read: Promoting and developing understanding (using evidence) of the concept and potential of organic food and farming in society. Participants in the DE workshop felt that objective 6 mixes both the method and the aim. The research participant in DE suggested a new objective “Acknowledgement of organic farming as an example for the whole farming sector” and it was decided by the group that this could be included within the scope of objective 6.7. Promoting the sustainable use of natural resources
8. Maintaining and enhancing the environment (including biodiversity, pollution and climate change issues)
In the DE and Italian (IT) workshops, the participants could not distinguish between the objectives of 7 and 8 – the question arose as to whether a difference exists between resources and environment. Their conclusion was to join the two objectives into one “promoting the sustainable use of natural resources and maintaining and enhancing the environment (including biodiversity, pollution and climate change issues)”.9. Maintaining and enhancing animal health and welfare
In ENG, it was suggested that animal health is implicit in welfare rather than separate and that “high levels” of welfare should be incorporated into the objective. A re-worded objective was: “maintaining and enhancing high levels of animal welfare, including health”.10. Maintaining and enhancing the social and economic wellbeing of rural communities
It was suggested in ENG that objective 10 should be expanded to include reference to the regeneration of disadvantaged communities. In IT, it was felt that labour issues were not addressed specifically in any of the objectives, therefore “(including employment)” was added at the end of this objective.11. Maintaining and enhancing the competitiveness of European agriculture
In the NL, it was suggested that the objective be amended to “Maintaining and enhancing the competitiveness of the European food sector (by a demand driven chain approach).12. Promoting public health and food security
The CZ group thought that the objective could be reworded to “Public health improvement” and the NL group thought the term 'food safety' should replace 'food security', as food security has an entirely different meaning to what they felt the objective was aiming for.
The following list of new objectives was generated from the discussion. The numbers in the brackets indicate where the authors could potentially see these new objectives being incorporated into the existing list of 12 objectives.
1. Shorter, more producer-controlled supply chains (ENG) (1)
2. Stimulation of research and search for improvements on all stages of organic supply (SI) (1)
3. Contribution to organic education (CZ) (6)
4. Public health improvement (CZ) (12)
5. Organic food quality (CZ) (3, 12)
6. Alternative to traditional/conventional farming (DK)
7. Protect the identity and the contextual knowledge of the territory (IT)
8. Maintaining and enhancing consumer awareness and trust in organic food (NL) (3, 5, 6)
9. Reinforcing internal organisation of the organic sector (AND) (1, 4)
10. Better informed public of the benefits and externalities of organic farming (AND) (6)
11. Protecting and assessing handmade and traditional production systems and the local culture associated (in extinction danger) (AND)
Objective 2 on the above list could potentially form part of the sectoral-level objective “Maintaining and enhancing the technical and financial performance of organic farms and related food-sector businesses” and they both refer to technical development. Objective 4 above is very similar to the societal-level objective of “Promoting public health and food security”. There are three objectives in the above list (6, 7 and 11) which deal with traditional/cultural aspects of agriculture and the issue of regional/territorial origin of food. Objectives 7 and 11 highlight the role organic farming might play with respect to the maintenance of traditional farming and regional products, whilst objective 6 offers organic farming as an alternative to traditional farming. Objective 8 above could potentially be incorporated into the sectoral-level objectives of “Meeting consumer demands for choice and quality, safe, affordable food, “Maintaining and enhancing the integrity of organic principles and organic food” and “Promoting understanding of the concept and potential of organic farming in society”.
Greater emphasis was placed by stakeholders on sectoral-level rather than societal-level objectives in the national workshops. The societal objectives that were the exception to this were: 7) Promoting the sustainable use of natural resources and 8) Maintaining and enhancing the environment (including biodiversity, pollution and climate change issues), both of which dealt broadly with the environmental objectives of the action plans and which, in some countries, (e.g. DE and IT) were taken as fulfilling the same objective. Interestingly, many of the newly-developed objectives could be classified as societal-level, which perhaps indicates the need to reflect regional customs and traditional practices with at least some, specifically designed, societal objectives. In contrast, the sector-level objectives appeared to be more broadly accepted by stakeholders across all countries.
On the basis of the workshop outcomes, the following list of generic intermediate objectives are proposed as a starting point for action plan evaluations:
Objective 1a: Maintaining and enhancing the technical performance and financial viability of organic farms
Objective 1b: Maintaining and enhancing the technical performance and financial viability of organic processing, marketing and related food-sector businesses
Objective 2: Increasing the scale of the organic sector (land under organic management, number of businesses and quantity of products available and sold in the market place)
Objective 3a: Meeting consumer demands for choice and quality, safe, affordable food, fibre and other agricultural products
Objective 3b: Maintaining and enhancing consumer awareness and trust in organic food fibre and other agricultural products
Objective 4 Better regulation, i.e. improving transparency of organic farming regulation, ensure market level playing field (harmonisation), integrating ‘public good’ standards (social, environmental etc.) and reducing bureaucracy
Objective 5 Maintaining and enhancing the integrity of organic principles and organic food
Objective 6 Promoting and developing understanding of the concept and potential of organic food and farming in society based on sound evidence
Objective 7 Promoting the sustainable use of natural resources
Objective 8 Maintaining and enhancing the environment (including biodiversity, pollution and climate change issues)
Objective 9 Maintaining and enhancing animal health and welfare
Objective 10a Maintaining and enhancing the social, employment and economic wellbeing of rural communities
Objective 10b Preserving threatened traditional and authentic craft skills and food production and processing systems with their associated local cultures
Objective 11 Maintaining and enhancing the competitiveness of European agriculture
Objective 12 Promoting public health, food safety and food security
The objectives diagram for each organic action plan evaluation will look different in each individual case. However, it is proposed that the starting point for structuring the objectives should be:
Level 1 (global): The two components of the dual vision of the EU plan: public goods (met through organic land management) and consumer demands (met through the organic food market).
Level 2 (intermediate): The 12 generic objectives set out in sub-section 3.3 above, supplemented by published aims of specific action plans if appropriate.
Level 3 (operational): The action points in the action plan to be evaluated.
In the process of structuring these and identifying cause and effect relationships, it may well be that additional intermediate and operational objectives are identified, so that the generic or published objectives should not be seen as exclusive.
The checklist for this section represents the first step in developing indicators in Sections C2 and C3, but the definition and structuring of objectives considered here is also closely related to the logic and coherence issues discussed in Section B2, so the two checklists should be considered together.
Do the proposed global (top level) objectives correspond to the action plan to be evaluated? If not, which should be added or deleted?
Do the proposed generic intermediate objectives, supplemented by published action plan aims, reflect the implicit as well as explicit objectives of the action plan? If not, which should be added or deleted?
What are the specific operational objectives (action points – EU example) of the plan to be evaluated?
In the context of the action plan and specific regional, national or
international situation to be evaluated, to what extent (e.g. not at all,
partly, highly, completely) are the defined objectives
SMART, i.e.:
a) specific
b) measurable
c) achievable (or acceptable)
d) relevant (or realistic)
e) time-bound.
EC (2001) Ex Ante Evaluation: A Practical Guide for Preparing Proposals for Expenditure Programmes. European Commission, Brussels.
Annex C1-1: Community Strategic Guidelines for Rural Development (Council Decision)
Annex C1-2: Organic farming’s contribution to EU strategic guidelines (Lampkin, 2006)
Annex C1-3: Principles of organic agriculture (International Federation of Organic Agriculture Movements, 2005)