Christian Eichert and Stephan
Dabbert
University of Hohenheim, DE
Nic Lampkin
Aberystwyth University, UK
Version 6, April 2008
One of the tasks of the ORGAP project was to learn from the experience of evaluations of organic action plans that have already been carried out. This section of ORGAPET summarises the results of this assessment which focused on evaluation quality assurance issues as outlined in ORGAPET Section A5, as well as on examples of existing action plan evaluations.
The results of four differing evaluations of national action plans (Germany, Denmark, Netherlands and England) were analysed using a meta-evaluation approach in order to:
assess the feasibility and appropriateness of such evaluations and determine the potential contribution of the results and available data to the application of ORGAPET at the European level, and
draw preliminary conclusions concerning issues affecting the success or failure of these plans as input into the ORGAP manual.
It was not the intention, and would not have been possible with the available resources, to conduct complete (re-)evaluations of these action plans or to collect new primary data for testing ORGAPET.
The meta-evaluation approach adopted involved the systematic analysis of the selected studies according to a defined set of criteria, drawing conclusions about each evaluation methodology by comparing the studies in order to gather information on each particular methodology. As a result of the meta-evaluation, studies can be differentiated according to scientific quality, practical relevance, utilisation and the effects of a study or a set of studies (Widmer, 1996: p4). In order to generate scientifically justifiable results, it was necessary to review each complete evaluation study. Thus, the studies had to be available in either English or German. The Dutch and Danish evaluations were therefore translated into English prior to their analysis.
The German evaluation study (Annex D2-3) focuses on the Bundesprogramm Oekologischer Landbau (Federal Organic Farming Scheme - Annex D2-2), which is technically not a 'pure' action plan. The FOFS can be seen as a preliminary stage on the way to a German organic action plan, because of its focus on a subset of measures relating primarily to information. Other measures, including financial support for organic production and the German organic food logo, were implemented outside the FOFS framework.
The evaluation was planned from the outset. An evaluation concept with specific and relevant questions for the evaluation was defined in the preparatory phase of the plan (2001). Then a public call for tender for conducting the evaluation was prepared and published. According to the call for tender the evaluation should answer the following questions:
Is the combination of measures used by the Federal Organic Farming Scheme adequate for achieving the scheme's specific objectives?
Have the scheme's objectives been achieved?
Which measures should be continued (in modified form if need be)?
Was the (relative and absolute) amount of funding adequate for the measures?
Did the Office for the Federal Organic Farming Scheme implement the scheme (tender and award of contract) effectively?
The evaluation was not only planned as a way of providing direct input to the programme managers during the implementation phase, but also the financial component was taken into account from the beginning: the funds for the evaluation were budgeted and fixed from the outset as an important part of the financial planning of the programme.
The purpose of the German evaluation study was to analyse and review the conduct and impacts of selected programme measures and to review the programme and process management as a whole. From this analysis, the evaluation makes recommendations for improving the execution and impact of the future FOFS. One main question was to assess the achievement of objectives of the measures. The evaluation also assessed the effectiveness and efficiency of the FOFS measures in order to detect potentially unexploited capabilities. The main methods of the German evaluation study were a combination of qualitative and quantitative methods and semi-structured interviews (via phone, postal, on-line, face-to-face).
The approach and goal of the German evaluation study was to gain insight into the functioning of the management process of a newly established programme. Accordingly, the study can be seen as an on-going (mid-term) formative evaluation. As a consequence, a final judgement on the quality and achievement of objectives of every single measure was not the intention and not possible with the chosen evaluation design. The evaluation was divided into two sub-studies: the topic 'consumer information' was evaluated separately by another company. All results were included and presented in a final report (which is the main basis of this study). The methods used were mainly standardised surveys of indicators in the direct sphere of influence of the measures that were evaluated, partly supplemented by qualitative data. Accordingly, the resulting statements move directly into the assessment of short-term programme effects.
Considering the circumstances of the evaluation (e.g. time pressures, newly-established programme with newly-involved programme officers) the study was undertaken in a courageous manner. The evaluation team surveyed plenty of (relevant) information and data on the topic and interpreted them very well. The evaluation design was constructed precisely and adequately, consistent with the underlying topic. Most of the achieved statements were valid and comprehensible.
One of the strengths of the German study is the fact that the whole evaluation was guided by a steering committee. Stakeholders were involved comprehensively in the evaluation and were able to discuss the first results during a reflective workshop, in order to ensure a feedback process and to avoid misinterpretation of the conclusions drawn. A relatively heterogeneous expert (stakeholder) group was invited (in addition to the steering committee, the accompanying board and the project group of FOFS), including participants from the whole organic food chain, the Ministries of the Federal States, scientists and contractors for the different FOFS measures. The main goals of the workshop were:
an exchange of experiences between experts and stakeholders concerning reached and unfulfilled expectations of the scheme;
feedback of information (condensed results) from the evaluation work to experts and stakeholders;
discussion of consequences for the different measures; and
discussion about the future design of the scheme.
The workshop produced plenty of stimuli, suggestions and background information for the assessment of the evaluation results.
The methods used were relevant and all data were interpreted well. Overall, the evaluation team did a good job, following systematically a consistent and well-built evaluation design.
The Danish Law no. 363 of 10.06.1987 is the first ever example of an organic action plan, although it was not described as such at the outset. Instead, it can be seen as a forefather of the two Danish action plans, including all relevant points and references. Evaluation of the Danish Law, or Organic Farming Act, was included in the meta-evaluation process in order to develop a deeper understanding of evaluation methodology in the context of organic farming policy in more general terms.
The Organic Farming Act was implemented through Orders of the Agricultural Department 830: 15.12.1987 (organic farming) and 4: 7.01.1988 (support for promotion of organic farming). The objective of the law was to promote the development of organic farming and the idea of organic farming by:
introducing support for conversion to organic farming;
providing support for development projects;
introducing public certification for organic (and biodynamic) products through the Plant Production Inspectorate;
defining the principles of organic farming;
establishing the Council of
Organic Agriculture under the Ministry of Agriculture. The Council
should:
- follow and assess the development possibilities for Danish organic
production;
- develop proposals for supporting the promotion of organic farming;
- assess extension and experimental work;
- make suggestions for further activities; and
-
comment on certification
rules, marketing, storing, transport,
labelling, distribution, and retail sales.
The Council consists of one member from each of the following
organisations: the Ministry of Agriculture, the Ministry of
Environment, the Biodynamic Association, the Danish Association of
Organic Agriculture, the Co-ordination Council for the Organic and
Biodynamic Agriculture, the Danish Farmers’ Union, the Danish Small
Farmers Association and the Danish Consumer Council.
The Danish evaluation study of this law (Annex D2-4) was conducted by a private consultancy and included specific recommendations for future development areas, which served as a model for the two subsequent action plans. The same firm of consultants was appointed by the Minister to carry out practical tasks in preparing the two action plans with the involvement of the Organic Food Council. The purpose of the Danish evaluation study was therefore to assess whether or not the Organic Farming Act had promoted the development of organic farming, and to serve as a strategic basis for decision-making in relation to future initiatives within the organic field. The key evaluative questions were:
What effects do the development projects initiated under the Act have?
What importance does the support for conversion have for the development of organic farming?
Assessment of the administration of the development programmes and the interplay between the Directorate for Farming, the Council for Organic Farming and the programme stakeholders.
Assessment of the labelling system for organic products.
Assessment of the functioning of the whole organic food chain (including monitoring and marketing).
The main methods of the Danish evaluation study were phone interviews, face-to-face interviews and document analysis.
The evaluation report starts with a sound status quo analysis of the Danish organic farming sector. To this end, the evaluation team collected an abundance of information via phone interviews with farmers receiving conversion support, project leaders participating in the development programme and conventional farmers. The overall coverage of these interviews can be considered to be very representative (17% of farmers receiving conversion support and 91% of project leaders). The face-to-face interviews were conducted with members of the Council for Organic Farming, representatives from the Directorate for Farming, representatives from the Directorate for Plants and the Veterinary Directorate, representatives from organisations and associations with interest in organic farming, and researchers. In addition, a document analysis (e.g. summaries from meetings of the Council for Organic Farming) was undertaken. The basic data obtained on the Organic Farming Act was further analysed in relation to three different aspects:
Management of the Organic Farming Act (assessment of the efficiency of the actors and institutions involved)
Quantitative effects of the Organic Farming Act (assessment via turnover, export, investments and employment)
Qualitative effects of the Organic Farming Act (assessment of the influence on professionalisation, organisational development, information and counselling, research, labelling and monitoring, reduction of barriers).
The results describe the overall strategic perspective of the Organic Farming Act on production and markets. The assessment of each subchapter is made in a neutral and logical manner. The final recommendations are listed in a separate (overview) chapter for ease of understanding.
Overall, the study was conducted systematically, following a defined path. It describes the status quo of the topic well, the intended targets and the measures and criteria applied, in addition to giving a number of recommendations for the future design of the researched policy field. The apparent strengths of the study are a well-founded analysis of the Danish organic sector, a detailed analysis of the current market situation and an assessment of future expansion opportunities for the Danish sector (e.g. potential for Danish exports). A multitude of interviews with broad coverage and, accordingly, good statistical value (eased by the small size of the sector under study) is provided. The recommendations for improvement of administration and monitoring functions is a worthwhile aspect not seen in comparable studies, and is particularly useful with regard to the complex competency of the organic sector at European level. An apparent weakness is that quantitative data are, at least in part, not presented and analysed well in the final document. In the explanation of data, it is desirable that the validity, reliability and, consequently, informational value of the study be clearly demonstrated.
The Dutch evaluation (Annex D2-6) focused on the Dutch organic action plan for 2001-2004 (Annex D2-5). The objectives of the Dutch evaluation study were: to verify whether the societal aims of the action plan had been realised; to provide justification to parliament for the costs of the programmes supporting organic farming (how well has the money been used?); and to learn from experience for the sake of future policies and support schemes. The key evaluative question of the Dutch study was final evaluation of the 2001-2004 action plan (i.e. what went well? what should be continued? what should change?) with respect to:
professionalising the demand-oriented supply-chains,
optimising the transparency and closing of the supply-chains,
knowledge development and dissemination,
stimulating organic primary production,
other measures (e.g. fiscal measures).
The research questions were formulated by using a causality (effects) diagram. For each main policy area, two causality schemes were defined: one in accordance with the policy document and one in accordance with the actual policy implementation. As a result, four main evaluation questions were formulated:
Degree to which the targets are achieved:
a. Growth of land area: Will the target of 10% organic farming area in 2010 be realised? Will the intended growth of organic area be realised?
b. Consumer expenditure: Will the Dutch consumer, at the end of 2010, be among the top users of organic products in Europe?
c. Competitive strength: Will the sector compete internationally by the end of 2010?
Effectiveness of the policy: Has the policy contributed enough to the growth of the organic sector and the growth of consumer expenditure on organic products? (Followed by sub-questions)
Effectiveness of the implementation: Could implementation of the policy (product and services) of national government have been undertaken with lesser means, without damaging the quality?
Assessment of 'Platform Biologica' (market support stage): To what degree has 'Biologica' contributed to the objectives of the policy document (ex-post)? What will 'Biologica' do if the yearly fixed contribution of LNV is removed?
As principal methodology, the Dutch evaluation used existing research results and data on organic farming in the Netherlands, the collection of additional information (substantial amount of interviews and written information provided by project leaders/co-ordinators), quantitative information, the collection of information on project activity (visible outputs, results, effects of that output, causality) and qualitative information: interviews designed to research difficulties, critical factors, general picture, ideas, relative effectiveness, appreciation, etc.
After a mid-term evaluation in 2002, the Dutch study constitutes the final evaluation of the organic action plan 2001-2004, and was finalised in July 2004. The study started with a clearly-structured plan, with the main topic (documenting policy on organic farming) being structured into comprehensible subchapters which were assessed separately. The study followed the prescribed plan and formal requirements very strictly, and provides supporting documents and sources in an accurate manner. The statements made are neutral, well-founded and justified. The evaluation team appears clearly distanced from the evaluation topic and, therefore, can be seen as neutral and fair in their judgements. External factors (e.g. the development of the organic market in other EU countries) and unwanted side effects of the relevant support schemes were observed and assessed. In the annex, the evaluation team presents a detailed overview of the evaluation questions.
Of notable value in the study is the detailed analysis of stakeholder integration in the political process: the Dutch study provides a very detailed assessment of how stakeholders were integrated via the different existing or newly-established bodies or boards (e.g. the newly-established 'Task force', building a network and institutionalising the dialogue between policy, market actors and other interested stakeholders).
“Changing the focus of the supply oriented policy to a demand oriented policy has proved to be a good choice. The policy instrument used, being a covenant, based on consensus with relevant stakeholders, supported by a Task Force with financial means for stimulation of activities, has proved to be both innovative and effective, even thought the final objectives have not been met.”
“A very important role of the Task Force, according to people involved, was improving the cooperation between the different stakeholders. The image formation about each other has changed creating more understanding for each others opinions and instead of the original ‘everybody for his own interest creates more common interest.” (Quotes from the Dutch evaluation)
The Dutch evaluation report is available as an internet document to all interested parties, which helps ensure that its results are used. The report is written in an open and impartial way, allowing and encouraging all interested stakeholders to draw their own conclusions, and is valuable with regard to the stakeholder integration aspect. The study follows the intended evaluation cycle in a systematic manner, starting with a status quo analysis, followed by the fixing of targets, fixing of measures and criteria for the assessment, a mid-term evaluation, a final census and recommendations as to how to design future policies.
The apparent strengths of the study are the feedback process with the steering committee scheduled into the evaluation design, as well as the evidence collected (where relevant and essential). Also, there is clear differentiation between the depiction of facts and parts of the study allowing interpretation, such as those interpreted by one of the stakeholders. The evaluation team has chosen a competent and realistic approach in preparing the plan and time schedule for the evaluation study. An apparent weakness of the study is that too few data and statistical sources are specifically referred to in interpreting the different programme parts.
The English action plan (Annex D2-7) was launched in 2002 and completed in 2007. A limited mid-term evaluation (Annex D2-8) was conducted in 2004, relying primarily on an extended workshop involving stakeholders as members of the action plan steering group. Although a proposal for evaluation was developed at the outset (by external stakeholder members of the original working group), this was not implemented. The evaluation therefore did not meet the criteria for inclusion in the meta-evaluation but is included here as an additional example.
The four action plan evaluations covered here represent examples of the different types of evaluation discussed in ORGAPET Section A5. The German and English evaluations can be considered to be formative mid-term evaluations designed to influence the continuing implementation process, while the Dutch and Danish evaluations were summative, ex-post evaluations intended to assess the achievements of the plans, but still with a view to learning lessons for future policy development.
All four evaluations made use of stakeholder input in some form, with the German and English evaluations utilising stakeholder workshops, while the Dutch and Danish evaluations made extensive use of stakeholder interviews. The Dutch, German and Danish plans also included significant quantitative data collection to support assessment of key questions and objectives.
Only the German evaluation was explicitly planned for from the outset, which affected both the resources and the types of data that could be collected and evaluated.
The need to develop a specific indicator framework to assist in the evaluation of available data was identified as an area for improvement in the evaluations conducted, but this needs to be done in the context of general quality standards for evaluation to avoid elementary errors.
Widmer, T. (1996) Meta-Evaluation: Kriterien zur Bewertung von Evaluationen. Haupt, Bern.
PLS-Consult (1992) Evaluation of Law no 363 10 June 1987 on Organic Farming by PLS-Consult. Ministry of Agriculture - Directorate of Agriculture, Copenhagen.
GIB (2004) Abschlussbericht der Evaluation des Bundesprogramms Oekologischer Landbau. Gesellschaft für Innovationsforschung und Beratung mbH.
Ministerie van Landbouw, Natuur en Voedselkwaliteit (2004) Biologisch meer gangbaar. Evaluatie-onderzoek Nota Biologische Landbouw 2001-2004. Onderdeel extern onderzoek.
For information on other national organic action plans, see also the ORGAP project action plan library.
Annex D2-1: ORGAP Meta-evaluation report
Annex D2-2: German Federal Organic Farming Scheme
Annex D2-3: German Federal Organic Farming Scheme evaluation (GIB 2004)
Annex D2-4: Danish Law on Organic Farming (1987) evaluation (English translation of PLS-Consult (1992))
Annex D2-5: Dutch organic action plan 2001-2004
Annex D2-6: Dutch organic action plan 2001-2004 evaluation (English translation of Ministerie (2004))
Annex D2-7: English organic action plan (2002)
Annex D2-8: English organic action plan interim evaluation (2004)