ORGAPET Section C2:
Methods for Defining Indicators

Ian Jeffreys and Nic Lampkin
Aberystwyth University, UK

Version 6, March 2008

C2-1      Introduction

Methods for defining indicators are presented in this section. These methods will be used to develop indicators informed by the stakeholder objectives that are defined using methods presented in ORGAPET Section C1. The discussions presented in this section will, in turn, inform the selection of the generic indicators presented in ORGAPET Section C3.

The methods proposed in this section draw heavily on the indicator work in the MEANS collection (EC, 1999) and the Evalsed updated guidelines on creating indicators and indicator systems, as well as the OECD indicator publications and other European and international studies with respect to socio-economic and agri-environmental objectives. The choice of indicators requires the programme objectives to have been defined and the impact of a policy on those objectives to be understood (programme theory). Examples of potentially relevant indicators from European programmes are given in Annexes C2-1 (EU rural development 2000-2006), C2-2 (EU rural development 2007-2013) and C2-3 (IRENA).

The MEANS and Evalsed frameworks differentiate between context, resource, output, result and impact indicators reflecting different hierarchical levels of objectives, which are also relevant in this context. These will be discussed in this section with reference to other indicator frameworks.

C2-1.1    What makes a useful indicator?

Indicators are an important part of assessing the effects of policies or action plans (programmes). Indicators are often chosen to represent complex issues in a simple way that can be easily quantified or described in qualitative terms. The European Commission (EC, 1999) states “A good indicator must provide simple information that both the supplier and the user can easily communicate and understand.” The OECD defines an indicator as “a summary measure combining raw data of something identified as important to OECD policy-makers” (OECD, 2001), and specifies that they should be policy-relevant, analytically sound, measurable and easy to interpret.

However, their choice is also influenced by the costs of obtaining data relative to the benefits that the data will yield. An indicator that is exact and close to a problem (e.g. soil analyses to measure actual nitrate leaching) may be very expensive for the purpose of assessing pollution from agriculture on an EU-wide scale, whereas an alternative indicator (such as the area of organically-managed land, based on the assumption that organic farming leads to reduced pollution) might be much easier to obtain, but also much less precise in terms of cause and effect relationships.

In addition to questions of data availability, precision and cost/benefit relationships, indicators need to be relevant to be effective, i.e. clearly linked to specific objectives on the basis of a clear concept of the effect that a particular policy or action will have on that objective (i.e. an impact statement derived from programme theory). There should also be a clear understanding of how an indicator result can be used to indicate performance with respect to the objective – performance criteria are required to determine whether the indicators show a) positive or negative changes over time, b) success or failure or c) degree of success (a grading or scoring system). In this context, the SMART criteria for defining objectives may be relevant.

The precise definition and quantification of indicators may not be possible in all or many cases, and a judgement-based evaluation may need to be undertaken, which may involve the judgement of an individual assessor or a group of assessors. In these situations, a process for eliciting expert opinion needs to be considered (see ORGAPET Section C4).

C2-1.2    Classification of indicators

Indicators may be classified according to the nature and the extent of the phenomenon they are measuring, as well as the level in the hierarchy of objectives to which they are linked.

C2-1.2.1    The MEANS/Evalsed approach

The MEANS framework (EC, 1999; see also its application to rural development monitoring in Annex C2-1) and the Evalsed guidelines on creating indicators and indicator systems provide a structure in which indicators can be defined. These indicators are defined in terms of their nature and relevance at different scales and regarding differing goals and objectives. In the MEANS approach, five types of indicator are defined, namely resource, programme, output, result and impact indicators.

As these five types of indicators are relevant to different scales, they will also be more or less useful for decision-making and evaluations at the different scales. The resource indicators provide the basic information about programme implementation and are important to assess the efficiency of the application of the programme. They also provide background information for other indicators and other scales. Output and result indicators can be used to evaluate the ability of organic farming policy to develop the organic sector. Impact indicators represent the ability of organic sector development to meet rural development and agri-environmental objectives.

C2-1.2.2    The OECD DSR approach

The OECD (2001) Driving Force-State-Response (DSR) framework for environmental indicators for agriculture identifies three types of indicators for assessing the bio-physical effects of agriculture on the farmed environment:

C2-1.2.3    The EEA/IRENA approach

The European Environment Agency (EEA) identifies four types of indicators, again with  a strong focus on bio-physical effects of human activity (Smeets and Weterings, 1999):

The IRENA indicators for reporting the integration of environmental concerns into agricultural policy (Annex C2-3) use an extended version of the OECD indicator classification: Driving forces - Pressures - State - Impact – Responses (DPSIR).

C2-1.2.4    Towards an ORGAPET classification

These three different definitions of possible indicator groups have many similarities in the types of indicators they present. As the MEANS framework is the dominant framework for evaluating EU policy, this will be used as the basic framework. The other indicator types will be placed in this framework and used to extend it. In addition, indicators reflecting the process and stakeholder issues considered in ORGAPET Section B1 and B3 should be included, and the relevance to evaluation types A-D (formative/summative, ex-ante/mid-term/ex-post), defined in ORGAPET Section A5, should be identified where possible.

  1. Design process indicators (see ORGAPET Sections B1 and B3) provide information on the nature of the design process, including the degree and quality of stakeholder involvement and the relevance (nearness) of the process to the target beneficiaries.

  2. Programme or context indicators provide information on the business, social and environmental characteristics of the targeted sector, including the EEA descriptive indicators that report the current state of environmental and bio-physical stocks considered to be of concern or potentially at risk. This should provide a baseline set of data against which change would be assessed and other indicators and indices would be calculated and interpreted, in the context of the general agricultural and policy situation, as well as the historical development of the organic sector.

  3. Resource and implementation process indicators provide information on the resources used by operators in implementing a policy or programme. In the MEANS framework, these include financial, human, material, organisational and regulatory resources, but should also include issues such as stakeholder involvement in the implementation process (see Section B3).

  4. Output indicators, as in MEANS, would represent the direct effect of the programme on the immediate beneficiaries, for example the number of hectares supported or the number of farmers participating in a scheme, or other measures of uptake with respect to specific actions in an action plan.

  5. Result indicators from the MEANS framework provide information about the product of an operator’s activity and the activity of a whole sector. They represent the immediate advantage for direct beneficiaries of the programme but are indirectly a result of programme activity, for example the increase in farm incomes or market share – these are most likely to relate to the sectoral-level objectives identified in ORGAPET Section C1. They are similar to the EEA performance indicators, as this group is focused on the performance of a sector against a defined set of goals and the programme’s beneficiaries.

  6. Impact indicators, as in MEANS, represent the effects of the changes made by beneficiaries as a result of the programme on wider public policy goals, for example environmental protection or animal welfare goals – these are most likely to relate to the societal-level objectives in ORGAPET Section C1. This group would therefore also include the state and response indicators defined by the OECD and IRENA, where the state indicators measure impacts on the environment and the response indicators seek to measure changes in policy in response to changes in the environment.

A key to adapting this classification for organic action plan evaluation lies in the two main levels of analysis, reflecting the different hierarchical levels of objectives, that is:

It is possible that stakeholders will have more interest in the output and result indicators, as these reflect the effect of the organic action plans on the organic community, while policy-makers may see this only as a means to an end, having more interest in the impact indicators. This does have the potential for generating conflicting policy evaluation results, but the needs of all the different groups interested in the outcomes of evaluations should be considered in the choice of indicators. Where we are interested in process evaluation, then the design process and resource indicators are most likely to be relevant, providing the most pertinent information on stakeholder involvement.

Table C2-1 indicates where particular types of indicators may be relevant in the different types of evaluation defined in ORGAPET Section A5.

Table C2-1: Linking indicator and evaluation types

Type

A

B

C

D

Nature

Formative

Summative

Timing with respect to action plan implementation

Before
(ex-ante)

Mid-term

Mid-term

After
(ex-post)

Design process

Yes

Yes

 

Yes

Programme/context

Yes

Yes

Yes

Yes

Resources and implementation process

Budgets and planned procedures

Compare budgets with actual and revise

 

Relate outputs achieved to available resources

Outputs

Predicted, baseline

Key to review implementation progress

Yes

Yes

Results

Predicted, baseline

 

Preliminary assessment

Final assessment

Impacts

Predicted, baseline

 

Preliminary assessment

Final assessment

C2-2       Defining indicators and impact statements

The MEANS approach (EC, 1999: Vol. 2, Section III) and the Evalsed update (though less detailed) contain useful suggestions on methods of producing and using indicators. In particular, they highlight that a system of indicators has more chance of functioning when the suppliers and the users of the information have been involved in its creation, suggesting that a closed group of specialists will be tempted to construct an expensive, technically ideal system which may never be operational. To solve this problem, it is suggested that a steering group including data suppliers and users should be established, which should take responsibility for defining the indicators. In the ORGAPET context, we include as users not only policy-makers but also organic sector stakeholders with an interest in the outcome of action plans or specific policies. This group may be very similar to the group that might be responsible for conducting the evaluation. Broader public or stakeholder involvement could be achieved through a series of focus groups to provide input into the steering group discussions.

In the first series of ORGAP National Workshops (Annex C3-5), stakeholders were asked to focus on three priority objectives and to suggest indicators relevant to their perspectives. This process was necessarily constrained by the resources available to run the workshops, but the experience may be useful for future reference. The specific indicators suggested in the workshops are considered further in ORGAPET Section C3.

However, the experience of selecting objectives and indicators, using stakeholder workshops in both the EU-CEE-OFP and ORGAP projects, is that a very large number of indicators may be selected by participants. The number of indicators needs to be manageable – the literature suggests that an individual decision-maker cannot take into account more than about ten indicators at one time. Too many indicators will result in information overload but different actors need different indicators for their specific purposes, so it may be better to have a series of narrowly-defined, targeted evaluations rather than one all encompassing one. In ORGAPET Section C3, a sub-set of indicators is proposed for each of the evaluation types A-D (see Table C2-1).

In complex, multi-objective/multi-policy programmes, the temptation is to measure everything, including the output and results for each action, but if an organic action plan has 20 or more actions, the number of indicators would quickly grow out of control. Simplification can be achieved by distinguishing between the needs of operators monitoring delivery, and the indicators needed for programme evaluation, which may require the input of some monitoring data collected by operators. (Monitoring may be defined as the observation system for project managers to verify whether project activities are happening as planned and resources are being used efficiently – the main focus would be on the resource and output indicators in the above classification; evaluation focuses more on the benefits to the sector and society, i.e. the result and impact indicators, and is concerned about learning from experience, identifying best practice and optimising the overall programme outcomes.) Action plan actions (operational objectives) may also be grouped by focus on the main beneficiaries (e.g. producers, consumers etc.) or the intermediate objectives in each case.

In the context of a specific evaluation, it should be possible to allocate more time to the indicator identification and rationalisation process, but the best situation would be one in which the evaluation framework had been determined from the outset. A significant part of the evaluation may be addressed using generic indicators such as those proposed in ORGAPET Section C3, but it is likely that there will be specific objectives and evaluation needs which are not covered by the generic indicators, requiring a customised approach. In this case, the following five step process is recommended:

C2-2.1    Step 1: Specifying objectives

It is assumed that a short, prioritised list of objectives or a small number of objective groups has been developed as an output of the methods presented in ORGAPET Section C1. The objectives or groups of objectives will then be used to define impact statements in Step 2.

C2-2.2    Step 2: Defining impact statements

As the second step in identifying indicators, it is important to consider the effect that the organic farming actions/policy measures have on a particular objective. Evaluations must take into account the effect system (interactions between outputs, results and impacts) linked to the programme or the strategy to be assessed, and organise it. The information on expected effects may be contained in the action plan documents, may be known from research or, in the absence of direct evidence, may be predicted either from a sound theoretical perspective or based on expert judgement (see below and ORGAPET Section C4).

Key challenges for this step are to be:

  1. clear about cause and effect relationships, particularly if several different actions have similar effects (programme theory may help) and

  2. able to specify the scale and direction of the effect.

Defining a cause and effect relationship between the programme and the effect or measured phenomenon is highly difficult. The logical nature of the links between objectives and effects (impact statements) are based essentially on experience/research and programme theories derived from the experience/research. However, research does not always result in the same conclusions and may not be universally applicable due to the dependency of the objectives/effect on various poorly-understood factors, including local market conditions, as well as exogenous factors, such as food scares, foot and mouth disease and BSE. The impact statements may also be affected by disagreement about the definition of intermediate objectives and, in particular, operational objectives.

In some cases, identifying cause and effect may be relatively easy. For example, if the objective is to increase the area of land under organic management and the relevant policy measure is area-based agri-environmental payments, then the impact could be defined (and quantified) in terms of the area of land supported by the policy. This does not however address the counter-factual issue of what would have happened in the absence of the policy. Were there enough other incentives (personal, market etc.) in existence that would have resulted in the land being converted anyway? Impact statements should take account of the counter-factual where possible.

In contrast, the effect of ‘demand-pull’ measures, such as promotion campaigns on consumer demand, may be very difficult to disentangle from the wide range of other media and commercial influences on consumer behaviour.

There is also a need to structure the different effects identified. An effects diagram can be used to describe the theoretical organisation of the effect system which leads to the overall intended impact. The effects diagram displays the classification of the results, outcomes and impacts of what is intended from the implementation of the objectives system. It connects the actual activities which have been planned, and the outputs which should produce direct results, to the medium-term intermediate impacts and the long-term global impacts. As a consequence, the systems of results, outcomes and effects can have the same degree of complexity as the objectives system and can be illustrated by similar diagrams (see, for example, the objectives diagram in ORGAPET Section B2).

Figure C2-1: Effects diagram

Source: Europeaid

The objectives and effects illustrated in the diagrams are related to each other with horizontal or vertical links. These links are called 'logical' when expressing an inference relation or impact statement (induction or deduction) which has been validated by experience. They highlight the fact that:

The logical links between objectives/effects can simultaneously be checked with:

These two groups of actors may validate the logical nature of the diagram unanimously or individually.

C2-2.3    Step 3: Clustering impact statements

The idea of grouping or clustering impact statements is to allow the assessment of many individual actions to be narrowed down to a few key impacts and a small number of indicators. For example, an action plan may contain separate actions (objectives) relating to research, advice, training and standards, all of which have a common impact in terms of reducing nitrate leaching. On this basis, one indicator can be selected to represent the common impact, rather than a separate indicator for each action. At the same time, the research, advice, training and standards actions may also have impacts in other areas, such as profitability, so this allows multiple effects to be assessed.

Experience has shown that the process of identifying objectives and impact statements can result in very large numbers of impact statements, needing to be grouped. In addition to the effects diagrams and logical frameworks discussed above, there are a range of supporting techniques available that can be applied, including impact mapping, Metaplan and statistical clustering techniques (see also EC, 1999: Vol. 3 and Evalsed). Impact matrices, illustrated in ORGAPET Section B2, can also be used.

Impact mapping is well-suited to stakeholder participation (see for example NEF, 2005), in that it is based on a group approach to a) defining and validating the list of impacts, b) weighting the impact statements in terms of their strategic importance to the programme, and c) grouping them in terms of their conceptual proximity. Statistical software can be used to generate an impact map, taking account of the weightings and groupings prepared by each stakeholder, with impact statements placed in similar groups by several stakeholders being shown close together, while those not sorted similarly will be shown further apart. This process is formalised in the Metaplan concept (a brand name registered by the consultancy firm Metaplan Thomas Schnelle).

C2-2.4    Step 4: Moving from impact statements to indicators

At this stage, the idea of an indicator as a simple representation of a complex situation needs to be remembered – indicators are, by their nature, an abstraction from the complex system, not a precise representation of it. Data availability and cost implications are relevant considerations, particularly where no baseline data exist and new systems need to be established. The following examples illustrate how an indicator may be defined:

Action (Objective) 1: Provide direct financial support for organic land management

Action 2: Provide financial support for conversion-related advice

Impact statement 1: The provision of direct financial support will increase/increased the area of land under organic management (by ?? hectares).

Impact statement 2: The support for conversion advice will increase/increased the area of land under organic management (could also consider the quality of that management).

Indicator (combining linked impact statements): Area under organic management

These impact statements and indicators could be linked, in turn, to higher-level objectives (aims/goals):

Aim (top-level objective): Maintaining and enhancing the environment

Impact statement: Research shows that organic management generally has a positive (how big?) impact on the environment, so that an increase in land area under organic management benefits the environment.

Indicator (as for lower-level objectives): Area under organic management

Using this approach, area under organic management can serve as an indicator for several objectives and is a relatively easy indicator to quantify at reasonable cost. However, it could be argued that the area under organic management indicator is too imprecise to measure the environmental impact, and that a more closely related indicator, such as nutrient balances, would be preferable, provided that the data can be obtained/estimated at reasonable cost. This needs to be considered in the context of individual evaluations, depending both on priorities and the resources (expertise as well as financial) available locally.

Adequate coverage for all objectives is required to ensure all important stakeholder considerations are measured in the evaluation. However, despite the techniques for developing objectives/effects diagrams and clustering impact statements described above, addressing many and detailed objectives in complex policy programmes like action plans may still lead to a very large number of indicators. This is not desirable for two reasons. Firstly, many indicators are likely to confuse the message as they will be difficult to synthesise into an overall measure of performance and, secondly, large numbers of indicators will increase the amount of data that needs to be collected or found from existing data sources. Good evaluation practice suggests that data mining ('let's collect anything/everything we can') should be avoided and that a parsimonious list of indicators should be aimed for. A firm figure on the desirable number of indicators is difficult to define but, arguably, it should rest within the 10-20 range. Where the policy programme to be evaluated is more complex, it may be desirable to create a hierarchy of indicators, with a short list of essential primary indicators and a longer list of optional/desirable secondary indicators. The core indicators represent the essential qualities of a programme – these being the most important qualities as defined by the relevant stakeholders – whilst the desirable indicators represent the other qualities of a programme, being of lesser importance to the relevant stakeholders.

Indicators can be prioritised by further stakeholder consultation or by a further application of the coding and clustering analysis undertaken in developing the objectives and impact statements. This may involve an expert assessment against a set of criteria or using data from scoping or previous studies regarding the performance of programmes.  It would then be possible to use the indicator deemed to be of highest importance by the stakeholders and use it as a proxy for the other indicators in the group.

The idea of 'double dipping' is when two or more indicators measure similar or the same effect, with the result that some of the indicators are effectively redundant. Clustering can be used to identify cases of potential double dipping; the less important indicators can then be removed or placed as low priority in the list. Another solution is to create a composite indicator, comprised of the two related indicators.

C2-2.5    Step 5: Quantifying and describing indicators – sources of data

The data needed to quantify or describe the selected indicators may be derived from a range of sources, including documents relating to the programme, interviews with key stakeholders, administrative (monitoring) data, specially commissioned surveys and research. General statistics or research literature will also provide contextual data. Examples of data sources for specific indicators are given in ORGAPET Section C3.

However, it is important that, where possible, the indicators and systems for data collection are established from the outset, so that baseline data can be obtained to allow changes to be assessed. Ex-post evaluation, in the absence of such advance planning, will be limited to only variables for which data happens to be available and may therefore be seriously limited in scope. New indicators may require new or innovative data collection methods, so that indicators are not limited to data that is collected currently or historically.

Issues of scale need to be considered. Scale includes the geographic and temporal scale at which the indicator has been measured and at which it will be used for decision-making and evaluation. For each indicator, the scale of measurement must be compared with the scale at which the evaluation or decision is being made. For example, is the aggregation of many observations made at farm or field scale relevant for measuring the quality of resource at a regional scale? Can an observation made at a lower scale be aggregated to measure an affect at a greater scale? Temporal scales need to be considered similarly. Is an indicator measuring a long-term trend and change in condition or a short- term variation or an anomalous event?

Issues of scale are linked to relevance to decision-making and the objectives and scale of influence of the decision-maker. The objectives and influence of policy-makers will be different to an individual landholder and the data requirements to inform their decision-making will be different. However, in many situations, it will be the individual landholder who will be asked to collect and report the data required to quantify performance against a given indicator, an indicator that may have little or no relevance to them.

C2-2.6    Step 6: Defining performance criteria

There should also be a clear understanding of how an indicator result can be used to indicate performance with respect to the objective. Performance criteria are required to determine whether the indicator shows:

  1. positive or negative changes over time,

  2. success or failure, or

  3. degree of success (a grading or scoring system)

For example, if the land area under organic management is a relevant indicator, has a target (e.g. 10% by 2010) been set which can be used as a performance criterion? Alternatively, is an increase in area a sufficient basis to judge performance?

Clearly there is an interaction between the availability of data, the establishment of performance criteria for specific indicators and the inclusion of specific targets in objectives that illustrates why it is important to see evaluation as integral to the process of developing action plans and other policy programmes, not just something that happens afterwards.

C2-2.7    Quality assurance

ORGAPET Section A5 identifies the need for quality assurance procedures for policy evaluations and this also applies to the quality of individual indicators and indicator systems that underpin the evaluations.

The quality of individual indicators can be assessed using the MEANS (EC, 1999: Vol 1, 193-195)/Evalsed indicator quality criteria:

  1. Availability: has the indicator ever been quantified or is data potentially available? If no data is, or ever can be, available, the value of the indicator is reduced.

  2. Freshness: how soon after the relevant time period will the data become available? If the process of statistical data collection, analysis and reporting is slow, the data may not be available at an appropriate time to be useful for evaluation or subsequent policy decision-making.

  3. Sensitivity: how clearly does the indicator respond to any effects that might be generated by a programme? If the effect is small, for example because the indicator is submerged by other, less relevant factors, then it may not show any effect.

  4. Reliability: are the results trusted by programme actors, i.e. will the results when measured by different people/methods be the same and credible to those who have to work with the results?

  5. Comparability: how comparable are the results across programme measures (actions), or across regions or time periods?

  6. Normativity: how well does the indicator enable judgements about whether the outcome is satisfactory? This is directly linked to the issue of performance criteria outlined above.

  7. Meaning: can the indicator be understood without ambiguity by everyone who has to use it? The indicator must accurately reflect the concept to be measured so that decision-makers, the public and programme managers all understand it in the same way.

These quality criteria can be scored using a scale 0 to 3 (where 0 is no, 1 is low and 3 is high quality with respect to the criterion). It may also be relevant to combine the scores for these individual quality criteria in order to provide an overall quality score for the indicator. However, there are potentially significant difficulties in arriving at a suitable weighting system for the individual elements so overall scores should be treated with caution.

An example of this is the evaluation of the proposed ORGAPET generic indicators carried out in the context of the EU organic action plan as part of the ORGAP project (Annex C2-4). It should be noted that in some cases, indicators may turn out to be high quality with respect to a specific action point within a programme, but not to the programme as a whole.

OECD (1999) uses similar criteria for measuring indicator and index performance, namely:

  1. Policy relevance: The criterion of policy relevance relates to those agri-environmental issues identified in the DSR framework as being of importance to policy-makers. The indicator should be able to quantify the components and issues described in the DSR framework, and agriculture should be a significant component in relation to the issue. The indicator should also be relevant to an environmental issue in agriculture which policies can potentially address and should contribute to the understanding and interpretation of these issues.

  2. Analytical soundness: The criterion of analytical soundness concerns, in particular, the extent to which the indicator can establish links between agriculture activities and environmental conditions.  It should also be possible for the indicator to explain a link between agriculture and an environmental issue which is easy to interpret and applicable to a wide set of farming systems. The indicator should also be able to show trends and ranges of values over time, which might be complemented by nationally defined targets and thresholds where these exist.

  3. Measurability: The criterion of measurability relates to the appropriate data available to measure the indicator. The indicator should be developed from established national or sub-national data, preferably using a long time series where this is available, given the lengthy time period for many environmental effects to become apparent.

  4. Level of aggregation: The criterion of the level of aggregation seeks to determine at which level (e.g. farm, sectoral, regional, national), the indicator can be meaningfully applied for policy purposes, without losing too much useful information through aggregation. This criterion highlights the issue of encapsulating the spatial and temporal diversity of the environment and the geographical scale of different environmental issues, ranging from the farm through to the global scale.

An adaptation of the SMART principles for objectives may be relevant to the assessment of indicator quality: Simple, Measurable, Accessible, Relevant, Timely (WUR, 2006), which include both relevance and analytical soundness from the OECD (1999) criteria. For assessing organic action plan indicators, a modified set of SMART criteria, integrating these various ideas, could be used:

Given the importance of stakeholder involvement in organic action plans, the SPICED criteria proposed by Mayoux (2002) may be helpful for evaluating indicators for use in participatory processes:

Finally, the IRENA methodology and data fact sheets (see for example Annex C3-3) include a meta data table which can be used for assessing the quality of individual indicators in a similar approach to Step 5 above:

  1. Technical information

    1.    Data source

    2.    Description of data

    3.    Geographical coverage

    4.    Temporal coverage

    5.    Methodology and frequency of data collection

    6.    Methodology of data manipulation

     

  2. Quality information

    7.    Strength and weakness (at data level)

    8.    Reliability, accuracy, robustness, uncertainty (at data level)

    9.    Overall scoring (give 1 to 3 points: 1=no major problems, 3=major reservations)
            i. Relevancy
           ii. Accuracy
           iii. Comparability over time

           iv. Comparability over space

As well as assessing the quality of individual indicators, it is also necessary to assess the quality of whole indicator systems.  EC (1999: Vol. 2, Appendix 4) sets out a grid for assessing the quality of a system of indicators, the key elements of which are:

  1. Defining the system to be assessed (e.g. the action plan).

  2. Dividing the system into homogeneous components. A component is homogeneous if it is possible to apply the same output indicator to all the projects (actions) and all the projects have the same target public.

  3. Identify and count indicators in order to calculate the density (number of indicators divided by total planned expenditure) for each homogeneous component.

  4. Classify the indicators by process, resource, output, result and impact category and calculate the percentage of components that have at least one result or impact indicator for the entire programme.

  5. For each indicator, apply a quality score, from 0 to 3 (zero, low, moderate, high), for the applicable quality criteria: availability, freshness, sensitivity, reliability, comparability, normativity and meaning.

  6. For each component, calculate the average score for each quality criterion (step 5) and for each indicator category (step 4) to get an average score for the indicators relating to that component.

Using this approach, a high score will be achieved by components with few but high quality indicators compared to those with many, but lower quality indicators. A total score for the whole indicator system can be obtained by calculating an average of the individual components, weighted by the budgetary expenditure on the different components.

C2-3    Conclusions

The choice of indicators requires careful consideration. Indicators must be appropriate for decision-making and evaluations for each particular situation. The separation of indicators into Resource, Output, Result, and Impact indicators will aid this process. In each case, thought must be given to the temporal, geographical and organisational scale at which the indicators will be measured and evaluations will be made. Indicators must be relevant for decision-makers and should be linked strongly to the goals and objectives of stakeholders. For the indicators to be accepted widely, stakeholders must include a wide range of persons, those who have power to affect policy development and those who will be affected by the policy. Wherever possible, the indicator should measure a phenomenon that is directly affected by the programme, and the influence of exogenous events should be limited. Finally, a clear selection process should be used in identifying indicators such as the SMART criteria.

C2-4    Checklist

The checklist for this section represents the second step in developing indicators started in Section C1 and developed further in Section C3.

  1. Have the objectives for the programme been clarified (i.e. as described in ORGAPET Section C1)?

  2. Define the impact statements and structure them in an effects diagram.

  3. Cluster the impact statements to reduce the potential number of indicators.

  4. Using the clustered impact statements and the generic indicator list in ORGAPET Section C3 as a guide, identify a parsimonious list of indicators relevant to the specific programme to be evaluated.

  5. Identify appropriate sources of data (see examples in ORGAPET Section C3) and/or potential data problems that may require additional research/data collection or new monitoring systems.

  6. Conduct a quality assessment of the individual indicators and the indicator system.

C2-5    References

EC (1999) The MEANS Collection: “Evaluating Socio-Economic Programmes”. Office for Official Publications of the European Communities Luxembourg.

Mayoux, L. (2002) What do we want to know? Selecting Indicators.  Enterprise Development Impact Assessment Information Service.

NEF (2005) Proving and improving: a quality and impact toolkit for social enterprise. Section: Impact mapping exercise. New Economics Foundation, London.

OECD (1999) Environmental Indicators for Agriculture: Concepts and Framework. Volume 1,  Organisation for Economic Co-operation and Development, Paris.

OECD (2001) Environmental Indicators for Agriculture: Methods and Results. Volume 3, Organisation for Economic Co-operation and Development, Paris.

Smeets, E. and R. Wetering (1999) Environmental Indicators: Typology and Overview. European Environment Agency, Copenhagen.

WUR (2006) Multi-Stakeholder Processes and Social Learning (MSP) Resource Portal. Section: Learning and Adapting. Wageningen University and Research Centre, Wageningen.

C2-6    Annexes

Annex C2-1:    Agenda 2000 rural development indicators

Annex C2-2:    2007-2013 rural development indicators (for further information see: http://ec.europa.eu/agriculture/agrista/rurdev2006/index_en.htm)

Annex C2-3:    IRENA Indicators

Annex C2-4:    Quality assessment of ORGAPET generic indicators