ORGAPET Section A5:
Planning an Evaluation

Christian Eichert, Stephan Dabbert
University of Hohenheim, DE

Version 6, April 2008

A5-1    Introduction

This section addresses the practical steps that need to be taken in setting up and managing an evaluation.  It builds upon Section A2 which addressed the principles and specific characteristics of organic policy evaluations which have influenced the design of ORGAPET and are important considerations in the planning of evaluations.  This section of ORGAPET is closely related to Volume 1 of the MEANS collection (Evaluation Design and Management, EC, 1999) as well as the Designing and Implementing Evaluation section of Evalsed. This section contains three main parts focusing on a) the planning stages of the evaluation process, b) the specific steps that should be undertaken in the evaluation and c) quality assurance.

A5-2    Issues to consider in planning an evaluation

From the literature, various approaches to planning evaluations can be found.

Vedung (1997) poses eight key questions to be considered when planning an evaluation:

1. What is the aim of the evaluation? (e.g. control/accountability, improved knowledge, programme modification/change etc.)

2. How is the evaluation organised? (Who commissions and who carries out the evaluation, and how are they interrelated?)

3. What is the programme to be evaluated? (There needs to be a clear description to set the boundaries for what is to be evaluated)

4. What is the public management process between input and output? (How has the decision been implemented and which agency was involved; what was the content; have target groups responded the way they promised; is there evidence of implementation failure?)

5. What are the results (outputs/outcomes) of the programme? (The focus should not only be on intended outcomes – are there other outputs, e.g. national action plans resulting from the EU action plan. What further outcomes could be included?)

6. What are the factors explaining the results (programme/others)? (What are the effects/impacts of the policy; what are the causalities involved; how are these influenced by the general context of society?)

7. What are the evaluation criteria and standards used for assessment? (What value statements are involved; what are the goals; what did stakeholders expect?)

8. How and by whom is the evaluation to be used? (Intention and reality – possible purposes include instrumental, political, internal reflection).

Patton and Sawicki (1993) suggest a problem-solving approach to planning evaluations which includes six steps:

1. Verifying, definition of and detailing the problem (determine the magnitude and extent of the problem; continually re-define the problem in the light of what is possible; eliminate irrelevant material; question the accepted thinking about the problem; question initial formulations of the problem; say it with data; locate similar policy analyses; locate relevant sources of data; eliminate ambiguity; clarify objectives; resolve conflicting goals; focus on the central, critical factors: is it important? is it unusual? can it be solved?; identify who is involved, and why? what power do involved parties have?; list resources required to deal with the problem)

2. Establishment of evaluation criteria (what are the important policy goals and how will they be measured?; identify criteria relevant to the stakeholders and to the problem; clarify goals, values and objectives; identify desirable and undesirable outcomes; is there a rank order of importance among the criteria?; what will be the rules for comparing alternatives?; administrative ease; costs and benefits; effectiveness; equity, legality and political acceptability)

3. Identification of alternative policy (consider a wide range of options; consider the status quo, or no-action alternative; consult with experts; brainstorming, Delphi, scenario writing; redefine the problem if necessary)  

4. Assessment of alternative policy (select appropriate methods and apply them correctly; estimate expected outcomes, effects and impacts of each policy alternative; do the predicted outcomes meet the desired goals?; can some alternatives be quickly discarded?; continue in-depth analysis of alternatives that make the first cut)

5. Displaying and distinction (choose a format for display; show strengths and weaknesses of each alternative; describe the best and worst case scenario for each alternative; use matrices, reports, lists, charts, scenarios, arguments)

6. Implementation, monitoring and evaluation of the policy (draw up a plan for implementation; design monitoring system; suggest design for policy evaluation; was the policy properly implemented?; did the policy have the intended effect(s)?)

Weiss (1998) proposes a key question approach which includes seven key questions:

1. Deciding whether to use quantitative or qualitative method, or a combination of the two methods.

2. Developing measures and techniques to answer the key questions.

3. Figuring out how to collect the necessary data to carry out the measures.

4. Planning an appropriate research design (with attention to the kinds of comparisons that will be drawn and the timing of data collection, for example).

5. Collecting and analysing the data.

6. Writing and disseminating the report(s) of study results.

7. Promoting appropriate use of results.

For the planning of an evaluation, further questions have to be answered (Weiss 1998):

• The right time for an evaluation: A major reason for undertaking evaluation is to improve programme operation; with foresight, managers can think through the possible uses (and their intentions) of evaluation data.

• Types of evaluation questions: The following questions help identify what information has to be collected. In quantitative studies, data need to be coded, checked for reliability, entered into the computer and then statistically analysed, hypothesised about, tested, modelled and reanalysed. With qualitative data, narratives have to be read, meanings and themes identified, hypotheses developed and tested, and interpretations subjected to disconfirmation and finally reported.

1. Programme Process: Questions about programme process essentially ask what is going on in the programme. The issue here is the fidelity of the programme with respect to the intention of its designer.

2. Programme Outcomes: An emphasis on programme outcome would direct the evaluation to the consequences of the intervention for its clients. The focus is the change in situations; but goals sometimes change during the course of programme operation. Another question of outcomes would be the hunches of programme staff, clients, observers or others about what the real results are, regardless of the rhetoric of intent.

3. Attributing Outcomes to the Programme: These questions are aimed at finding out whether any changes that are observed over time are due to the programme. The evaluation may want to devote some questions to understanding the extent to which the programme was responsible for the changes.

4. Links between Programme and Process: With information about process, the question of whether particular features of the programme are related to better or poorer outcomes can be analysed.

5. Explanations: Evaluation is not only aimed at finding out what happened but also how and why it happened. This entails a search for explanations. For the improvement of a programme, the reasons for achievements and shortfalls have to be indicated.

In developing the set of questions to be answered, the following should also be considered (Weiss 1998):

• The decision timetable

• The relative influence of interested parties

• Preferences of stakeholders

• Uncertainties in the knowledge base

• Practicalities;

• Assumptions of programme theory

• Potential for use of the findings

• The evaluators’ professional judgement

In the MEANS collection Volume 1 (EC, 1999), three factors that affect the form of an evaluation are identified:

1. the stage of the policy cycle;

2. the level of decision making;

3. the scope of the evaluation.

Evaluations can be undertaken at three stages within the policy cycle: ex-ante, mid-term or intermediate and/or ex-post.

• Ex-ante – before the implementation of the policy or programme, outcomes of this formative evaluation, sometimes referred to as a status quo analysis, should be included in the policy or programme.  An evaluation could be undertaken to choose between policy options, identifying that which is best aligned to stakeholder goals and objectives.  At this stage, policy shortcomings can be identified and addressed.  This analysis may draw on ex-post evaluations of previous policy.

• Mid-term – a formative evaluation that is used to improve a subsequent stage of the policy or programme.  This assesses the ongoing relevance of the policy or programme and highlights changes in the general economic and social context affecting the policy. It draws on the ex-ante evaluation of the policy and ex-post evaluations of previous policy.

• Ex-post  – these evaluations inform future policy and programme development.  They are undertaken some time after the policy has finished allowing for the full ramifications to become evident. An ex-post evaluation may inform the ex-ante evaluation of the next policy cycle.

• Ongoing evaluations – the series of ex-ante – mid-term – ex-post evaluations in successive policy cycles should, if managed well, be integrated and minimise overlap and repetition of evaluation and data collection. New programmes and policy should be developed considering the performance of policies and programmes from the recent past.

This ongoing process of evaluation runs parallel to that of policy development and reflects and assesses the impact of the policy on the observed direct and indirect policy results. 

An evaluation must be undertaken for each level of instrument (policy, programme or project). It can not be assumed that the performance of a programme will be the sum or the synergy of its component projects and there will be other effects that must be considered. When evaluating projects and programmes, the objectives of higher level programme or policy should also be taken into account. In evaluating policy, the wider goals of economic, social and environmental factors must be taken into consideration.

Some of the questions identified above are addressed in the remainder of this section of ORGAPET; the rest are addressed in ORGAPET Part B and Part C.

A5-3    Steps in preparing the evaluation

A5-3.1        Defining the purpose of the evaluation

Vedung (1997) poses a critical question: Why is the evaluation being conducted? This is clearly linked to Vedung’s second question: Who is organising the evaluation? As discussed in ORGAPET Section A2, Stockmann (2004) asks whether evaluations are primarily oriented towards information, monitoring, learning or legitimisation. Stockmann identifies some key differences in possible answers (Table A5-1) and concludes that evaluations can be categorised as either formative (process-orientated, constructive or communication-promoting designs) or summative (result-oriented, concluding, accounting) in character.

Table A5-1:   Dimensions of evaluation research

Source: Stockmann (2004)

This is consistent with the MEANS approach outlined in the preceding section and with the Evalsed guidance on defining the object of an evaluation. Both the scope and purpose of the evaluation should be defined.  While the purpose may be summative or formative in nature, the scope defines the programme and the evaluation within set institutional, temporal, sectoral and geographical limits.  Institutional limits may be set at an EU, national or local government scale; temporal limits to a given period of time; sectoral limits to an industry, society or environment (e.g. rural or urban environment); and geographical to a country, region or town. However, it might also be worth considering:

Evalsed advises keeping a narrow focus to evaluations, particularly ex ante ones, by precisely defining the scope, so that the really central issues are addressed.

In ORGAPET Section A2, we concluded that:

• the aims of organic action plan evaluations for which ORGAPET is developed are to document policy actions and their impact, to make suggestions for changes in policy and to promote transparency in the development of action plans, in order to meet the multiple goals of both policy-makers and organic sector stakeholders (ORGAPET Section A2, Section A5 (this document) and Section C1 address this);

•  action plan evaluations will typically be commissioned by the European Commission and/or national/regional governments, with organic stakeholders as participants, and either internal administrators or external researchers/consultants acting as evaluators. To achieve this, ORGAPET needs to have an open and relatively simplistic approach to evaluation.

Taking these perspectives into account, it is possible to categorise four different types of evaluations which will place different weights on the different elements of ORGAPET (Table A5-2).

From the perspective of organic action plans, the two types of formative evaluation should focus on:

• Ex-ante: identifying the current situation and development needs of the organic sector from the perspective of stakeholders (public and private, organic and non-organic), as well as the barriers to desired development and policy options to address these.

• Mid-term: reviewing programme design/coherence, resourcing, uptake and implementation problems, particularly from the perspective of target stakeholders (both beneficiaries and implementing agencies), with a view to refining and adapting the plan if necessary.

The summative evaluations should focus on:

• Cost-effectiveness of measures: needs to take account of the overall effects from the whole action plan (i.e. synergy), not just from individual actions.

• Process: from design to implementation and evaluation, including integration of stakeholders at all levels (see ORGAPET Part B).

• Goal attainment: in particular the outputs/results for target beneficiaries and the organic sector, but also the broader policy impacts of relevance to policy-makers/funding agencies at regional, national and EU levels (see ORGAPET Part C).

Table A5-2: Relevance of ORGAPET Sections to different types of evaluation

A5-3.2        Structuring and scheduling the evaluation

An evaluation can be undertaken in three distinct phases:

1. Deciding on the evaluation;

2. Drawing up terms of reference;

3. Launching the evaluation.

The tasks that need to be completed at each stage depend on the type and timing (ex-ante, mid-term, ex-post) of the evaluation. Table A5-3 displays examples of the tasks that should be completed at these different stages.

The timing and frequency of evaluations needs to be considered in terms of stage in the policy cycle as discussed above, but also in administrative terms. If carried out too frequently (more than once every three-five years?), the resources required could be substantially greater than the benefits derived, and it could detract from policy implementation activities. If the evaluation is carried out too late, or the evaluation process takes too long, decisions on the next stage of policy development could already have been taken, and the results are then only of historical or academic research interest.

It is also important is that there is enough content in the plan and clear targets to evaluate. Indicators (see Sections C2 and C3) need to be determined and monitored from the outset, and therefore evaluation should form part of the planning of the action plan - if there is no data, the plan cannot be evaluated.

Table A5-3: Steps in preparing an evaluation (adapted from MEANS)

Source: EC (1999) modified

Evalsed provides further guidance on drawing up terms of reference, which should serve as the contractual basis between the commissioner and the team conducting the evaluation. Typically the terms of reference will cover the regulatory framework, the scope of the evaluation, the main users and stakeholders of the study, the evaluative and research questions, the available knowledge, the main methods and techniques to be used, the schedule, the indicative budget, the expertise required, and administrative requirements for a proposal.

A5-4    Performing an evaluation

The key elements of performing an evaluation are covered in detail in ORGAPET Parts B, C and D. Table A5-4 summarises the key steps in performing an evaluation and indicates in each case where further information can be found in ORGAPET.

Table A5-4: Steps in performing an evaluation (adapted from MEANS)

Source: EC (1999) modified

Evalsed includes further guidance on the use of evaluations, identifying three different ways in which evaluation work is used:

• Individual evaluations may be used directly or in an instrumental manner whereby the results, findings, conclusions and recommendations are taken up. In practice this is unusual and where it does occur it tends to take place only partially.

• More often, several evaluations or individual evaluations combined with other evidence and opinion are used cumulatively to inform debates and influence decision-making. Evaluation work thus stimulates the process of debate, challenge and counter challenge to evidence and its interpretation.

• Even where evaluation results are not used the process of evaluation initiation and reflection can be useful by offering opportunities to exchange information, clarify thinking and develop frameworks.

The extent to which the results of an evaluation can be used effectively will depend on timeliness of the evaluation, dissemination and follow-up activities, the institutional arrangements and engagement of senior staff, as well as the engagement of stakeholders and the quality of the evaluation.

A5-5    Ensuring the quality of an evaluation

A key part of effective policy evaluation is to ensure that the quality of the evaluation is sound, otherwise the conclusions drawn may not be robust. Evaluation standards have been developed in many countries and institutions in order to contribute to evaluation quality. Here, the evaluation standards of SEVAL, the Swiss Evaluation Society (Widmer et al., 2000), are presented briefly, as well as the MEANS and Evalsed guidelines.

A5-5.1        SEVAL evaluation standards

The SEVAL standards define the demands of evaluations but do not specify the instruments to be used within the evaluation (Widmer et al., 2000). The standards were formulated in a way that is suitable for all kinds of evaluations, except those of personnel. They are based on the Programme Evaluation Standards of the Joint Committee on Standards for Educational Evaluation. The SEVAL standards, as is the case with most of the evaluation standards, distinguish four sub-groups (Widmer et al., 2000):

• utility standards guarantee that an evaluation is oriented towards the information needs of intended users of the evaluation.

• feasibility standards call for evaluation systems that are as easy as possible to implement, efficient in their use of time and resources, adequately funded and viable from a number of other standpoints.

• propriety standards require that evaluations are conducted legally, ethically and with due regard for the welfare of evaluates and clients of the evaluations.

• accuracy standards require that the obtained information is technically accurate and that conclusions are linked logically to the data.

A5-5.1.1       Utility standards

The utility standards guarantee that an evaluation is oriented towards the information needs of the intended users of the evaluation. The utility standards include:

• Identification of stakeholders, i.e. those persons who are participating in, and affected by, an evaluation, in order that their interests and needs can be taken into account.

• Clarifying the objectives of the evaluation: all who are involved in an evaluation should ensure that the objectives of the evaluation are clear to all stakeholders.

• Credibility: those who conduct the evaluations should be both competent and trustworthy; this will help to ensure that the results achieved by an evaluation are accorded the highest possible degree of acceptance and credibility.

• The scope and selection of information that has been collected should make it possible to ask pertinent questions about the object of the evaluation. Scope and selection also takes into account the interests and needs of the parties commissioning the evaluation, as well as other stakeholders.

• Transparency of value judgements: the underlying reasoning and points of view upon which an interpretation of evaluation results is based  should be described in such a manner that the bases for the value judgements are clear.

• Comprehensiveness and clarity in reporting: evaluation reports should describe the object of the evaluation, including its context, goals, questions posed and procedures used, as well as the findings, in such a manner that the most pertinent information is available and readily comprehensible.

• Timely reporting: significant interim results, as well as final reports, are made available to the intended users such that they can be used in a timely manner.

• Evaluation impact: the planning, execution and presentation of an evaluation should encourage stakeholders both to follow the evaluation process and to use the evaluation.

A5-5.1.2       Feasibility standards

The feasibility standards call for evaluation systems that are as easy to implement as possible, efficient in their use of time and resources, adequately funded, and viable from a number of other standpoints. The feasibility standards include:

• Practical procedures should be designed such that the required information is collected without undue disruption to the object of the evaluation or the evaluation itself.

• Anticipating political viability: the various positions of the different interests involved should be taken into account in planning and carrying out an evaluation in order to win their co-operation and discourage possible efforts by one or another group to limit evaluation activities or distort or misuse the results.

• Cost effectiveness: an evaluation should produce information which is of a value that justifies the cost of the process.

A5-5.1.3       Propriety standards

The propriety standards require that evaluations be conducted legally, ethically and with due regard for the welfare of evaluatees and clients of the evaluations. The propriety standards include:

• Formal written agreement: the duties of the parties who agree to conduct an evaluation should be set out in a written agreement, in order to oblige contracting parties to fulfil all agreements about conditions or, if not, to renegotiate the agreement.

• Ensuring individual rights and well-being: evaluations should be planned and executed in such a manner as to protect and respect the rights and well-being of individuals.

• Respecting human dignity: evaluations should be structured in such a manner that the contacts between participants are marked by mutual respect.

• Complete and balanced assessment: evaluations are complete and balanced when they assess and present the strength and weaknesses that exist in the object being evaluated, in a manner that strength can be built upon and problem areas addressed.

• Making findings available: the contracting parties in an evaluation should ensure that its results are made available to all persons who are potentially affected, as well as to all others who have a legitimate claim to receive them.

• Declaring conflicts of interest: conflicts of interest should be addressed openly and honestly so that they compromise the evaluation process and conclusions as little as possible.

A5-5.1.4       Accuracy standards

The accuracy standards require that the obtained information be technically accurate and that conclusions be linked logically to the data. The accuracy standards include:

• Precise description, documentation and unambiguous identification of the object of evaluation.

• Analysing the context so that its influences on the object of evaluation are identified.

• Precise description of goals, questions and procedures: the goals pursued, questions asked and procedures used in the evaluation should be sufficiently precisely described and documented so that they can be identified as well as assessed.

• Trustworthy sources of information: the sources of information used in an evaluation should be sufficiently precisely described for their adequacy to be assessed.

• Valid and reliable information: to ensure the validity and reliability of the interpretation, it is necessary to select, develop and employ procedures for that given purpose.

• Systematic checking for errors: the information collected, analysed and presented in an evaluation should be systematically checked for errors.

• Qualitative and quantitative information should be systematically and appropriately analysed in an evaluation, in such a manner that the questions posed by the evaluation can actually be answered.

• Substantiated conclusions: the conclusions reached in an evaluation should be explicitly substantiated in such a manner that stakeholders can comprehend and judge them.

• Neutral reporting: free from distortion arising from the personal feelings or preferences of any party to the evaluation. Evaluation reports should present conclusions in a neutral manner.

A5-5.2        Assessing quality of an evaluation in the MEANS and Evalsed frameworks

The MEANS framework (EC, 1999, Vol. 1:169 ff.) and the Evalsed section on quality assurance and quality control identify eight quality assessment criteria that should be addressed to identify whether an evaluation report is unacceptable, acceptable, good or excellent:

• Meeting needs: Does the evaluation adequately address the requests for information formulated by the commissioners and does it correspond to the terms of reference?

• Relevant scope: Have the rationale of the programme, its outputs, results, impacts, interactions with other policies and unexpected effects been carefully studied?

• Defensible design: Is the design of the evaluation appropriate and adequate for obtaining the results (with their limits of validity) needed to answer the main evaluative questions?

• Reliable data: Are the collected or selected primary and secondary data suitable? Are they sufficiently reliable compared with expected use?

• Sound analysis: Are qualitative and quantitative data analysed in accordance with established rules, and are they complete and appropriate for answering the evaluative questions correctly?

• Credible results: Are the results logical and justified by the data analysis and by interpretations based on carefully presented explanatory hypotheses?

• Impartial conclusions: Are the conclusions just and non-biased by personal or partisan considerations, and are they detailed enough to be firmly implemented?

• Clear report: Does the report describe the context and goal, as well as the organisation and results of the evaluated programme, in such a way that the information provided is easily understood.

In reaching an overall assessment of the evaluation, account should be taken of constraints weighing on the evaluation and the team which performed it, in particular whether the terms of reference and the time and resources allocated to the evaluation were realistic and whether the necessary data could be obtained.  In the same section, Evalsed provides further guidance on quality assurance with respect to the evaluation process.

Quality assessments can be undertaken by various groups with various aims. In Table A5-5, possible tasks for each group are presented.

The MEANS framework also defines how to assess the quality of individual indicators or systems of indicators (see Section C2).

Table A5-5: Who assesses the quality of an evaluation?

Source: EC (1999) modified

A5-6    Conclusions

When preparing an evaluation, it is necessary to be clear about the purpose and scope of the evaluation related to the type of evaluation (formative or summative) and the stage in the policy cycle (ex ante, mid-term, ex post) in which it is to be carried out. Clear terms of reference and quality assurance procedures are required to ensure optimal results. The detailed steps to be followed in the preparation and conduct of evaluations are set out in Tables A5-3 and A5-4. The evaluation team and programme managers should be supported by a steering group including relevant stakeholders to assist with the planning, interpretation and final evaluative judgement of the results. A clear dissemination/communication strategy is also required to inform relevant stakeholders about the process and the outcomes of the evaluation, and to ensure that follow-up actions take place.

A5-7    Checklist

The checklists in each section are intended to provide a structured listing of the key issues covered in the text that should be addressed as part of an action plan evaluation. The following questions relate to the material covered in this section of ORGAPET and should be answered at the planning stage of an evaluation. Normally this will be from the perspective of the person/institution organising the evaluation, but it can also be undertaken by others with an interest in the evaluation process.

1. What is the purpose (aims, objectives, desired outcomes) of the evaluation (in your own words)?

2. What is to be evaluated (define the scope – e.g. national action plan from 2000-2005 – be as specific as possible)?

3. Was an evaluation planned for from the outset, with an appropriate monitoring programme and baseline data in place?

4. When (at what stage of the policy cycle) is the evaluation to be carried out (ex-ante, mid-term, ex-post)?

5. What type of evaluation is needed (formative = to assist future planning; summative = to evaluate past actions; or both)?

6. With reference to Tables A5-1 and A5-2, classify the evaluation type as A, B, C or D.

7. Who (which agency/organisation) will commission the evaluation?

8. Who will conduct the evaluation (consultants, stakeholders, others)?

9. What is the timescale (schedule) over which the evaluation should be conducted?

9. How and by whom can the results of the evaluation be used (dissemination and decision making)?

10. Have any relevant evaluations or reviews been conducted previously (the results will be relevant in Part B and Part C)?

11. Will the evaluation meet (or has it met) the quality assurance guidelines (specify whether SEVAL or MEANS or other)?

12. Have clear terms of reference for the evaluation been defined?

A5-8    References

A5-9    Annexes

No annexes are currently included in this section.