Johannes
Michelsen
University of Southern Denmark (USD)
Christian
Eichert
University of Hohenheim (UHO), DE
Otto
Schmid
Research Institute of Organic Agriculture (FiBL), CH
Nic Lampkin
Aberystwyth University, UK
Version 6, April 2008
ORGAPET Section A4 illustrates that increasing numbers of stakeholder groups have become involved in agricultural policy-making, implementation and evaluations. The purpose of this section is to consider methods for evaluating the level and nature of stakeholder involvement in policy-making, implementation and evaluation of organic action plans. In particular, mid-term and ex-post evaluations will reflect what actually comes out of concrete attempts to involve stakeholders and may use the approaches to stakeholder involvement described in Section A4 as evaluation criteria. However, the introduction to Section A4 is repeated here to provide context to the evaluation considerations that follow.
Stakeholders include all those individuals, groups and organisations with an interest in the evaluated policy/programme/action plan. They include public agencies, private organisations (whether non-profit or profit-oriented), public and private enterprises and private individuals (EC,1999: Vol. 6: 37). One or more stakeholders are thus by definition involved in all stages of a policy process, starting with initiation of a policy (often involving private organisations and public agencies), via decision-making (involving public agencies assisting politicians and perhaps hearings of private organisations and private enterprises) and implementation (the interaction between public agencies and addressees of the policy (individuals or enterprises), to eventual finalisation (public agencies assisting politicians).
Policy-making, implementation and evaluation can be seen as separate stages in one joint policy cycle as illustrated in Figure B1-1 or in successive policy cycles as described in Figure B3-1 below. The distinction made between stages may be helpful for analytical purposes but could be difficult to use when analysing real policy processes because, in reality, the different stages may not be as separate as they are in theory – for instance, if actions are implemented before they are decided upon, and then afterwards become part of an action plan (Peters and Pierre, 2006). It is thus worth keeping the full policy cycle or policy process in mind when considering the actual involvement of stakeholders in public policy-making on organic action plans. In what follows, the policy cycle is condensed into four main stages:
programme design (which includes agenda-setting and policy formulation);
decision-making;
implementation; and
evaluation.
In addition, Figure B3-1 illustrates that, in theory, an evaluation may trigger a feedback process by which a policy either reappears on the agenda of the original policy process, or may be moved to another policy process characterised by another context and content. It should be remembered, however, that it is not necessarily the case that an evaluation will trigger a new policy process, whether original or new. Active efforts from one or more stakeholders are needed if an evaluation is to feed back into the policy process, and these efforts might not be successful. In addition, a new policy process whereby an existing programme is changed within a different policy context, may take place whether or not a formal evaluation has taken place. Stakeholder involvement thus takes place through time and conditions for involvement may change with changes in content and context.
Figure B3-1: Stages in the policy process, within larger policy cycles – a theoretical model
Source: Eichert
In what follows, the focus is on how stakeholder involvement has actually taken place, and to illustrate how this kind of evaluation may be undertaken using specific examples. The ways in which stakeholders can actually be involved in the four main stages of the policy cycle are considered in ORGAPET Section A4.
ORGAPET Section A4 outlines the need for stakeholders in the policy process, and how to involve stakeholders in the stages of programme design, decision-making, implementation and evaluation, with particular reference to:
the different, relevant perspectives/views which stakeholders represent and which are particularly relevant for a specific programme and stage in the policy cycle, and
the degree of involvement in the policy process (central or peripheral actors in the process – see also Section B1).
When preparing an evaluation it is important to gather sufficient information about the different actors (see below). In which way are they stakeholders? Which interests do they represent and which views and perspectives do they bring forward?
With regard to policies for organic food and farming, three key perspectives should be taken into account in the different stages of the policy cycle:
the organic values perspective (definition of principles of organic food and farming);
the market perspective; and
the public good perspective.
These perspectives are described in ORGAPET Section A4. Within each perspective many stakeholders, stakeholder groups and organisations representing stakeholder interests may be present. The interests of the various stakeholders may be more or less common or may be in opposition to each other. From the outset, any evaluation needs to be open to any type of interest or stakeholder. This may imply the inclusion of clearly opposite interests in the evaluation and thus cause many difficulties in the evaluation process. The only valid reason for in- or excluding stakeholders, however, is their relevance to the policy under evaluation.
Although some stakeholder or stakeholder group representatives claim to represent different perspectives, it might be necessary to determine their main priorities and goals in the process. There are several possible ways of determining the type of perspectives promoted by individuals or organisations/institutions, e.g. by analysing:
their documented aims/objectives;
by whom they are financed; and
whether or not their political activities benefit others than themselves.
If the analysis includes many stakeholders, this knowledge may serve as basis for grouping stakeholders as specified below.
Another important factor is the degree of involvement. This could be expressed with a picture: which actors ‘hold the stake’ all the time (central actors), which ones just 'touch' the stake from time to time (interlinked actors) and which ones just facilitate the way the stake is held (supportive environment or peripheral actors)?
As outlined in ORGAPET Section A4, the organisational background – i.e. being in a mainly ('pure') organic-oriented organisation or institution or in a mixed (organic and non-organic) operation or institution (Figure A4-2) – might have an influence on:
how much the stakeholder representative or individual holds relevant information and expert knowledge;
how much the stakeholder himself or the stakeholder group might be affected by a measure.
The higher the level of involvement of an actor may generally be seen as leading to higher legitimacy and stronger involvement in the different stages of the policy process. However, as some strongly involved and engaged stakeholders might be biased by their personal legitimate interests, it is important that these interests are made transparent in the process. This may be done by asking stakeholders to outline, in a consultation process address, which interest of which stakeholder group they represent and for whom they are mandated to speak or to write. This could be documented in a standardised format for the interested public.
The starting point for any evaluation of stakeholder involvement is an analysis of the historical context of the policy programme in question (e.g. an organic action plan). The historical background is one of the main explanatory factors in process evaluation (Vedung, 1997). The analysis of the historical background must focus on identifying the stakeholders involved in all four main stages, in order to look for change and continuity in the composition of stakeholders throughout the policy cycle which may contribute to understanding and explaining actual outcomes or policy impacts of an organic action plan. The experience of implementation research is that administrative processes cannot be expected to run in exactly the way expected when programming the policy. Hence, it is necessary to investigate the dynamics of the stakeholders involved in implementation, as this may also help explain the degree of conformity between policy intentions and policy outcomes.
The analysis of the history of the organic action plan will help to identify which particular interests were at stake in each stage of the policy process. In the words of the policy process, it is necessary to ask which stakeholders were involved in which stages.
The systematic starting point is agenda-setting:
why did an organic action plan come on to the political agenda;
which stakeholders promoted which ideas for policy action and for what reasons; and
which stakeholders reacted on others’ initiatives and on what grounds?
Stakeholders may include private organisations and individuals, as well as political parties and individual politicians in addition to public agencies. The follow-up question is:
how and in which ways were the stakeholders involved or influenced by the potential political decision of an organic action plan; and
which resources relevant for the realisation of the action plan did they control?
This analysis can be undertaken on the basis of the considerations concerning stakeholder perspectives above and in ORGAPET Section A4. Answers to these questions will help to identify the role of different stakeholders in initiating policy decisions. Some stakeholders may have been pro-active in the process while others may have been reactive, and some stakeholders might promote policy decisions while others attempt to hinder or postpone policy decisions.
In the other phases of the policy process, similar questions must be answered:
which stakeholders participated in which ways in policy formulation, implementation or evaluation;
how did they become involved in (or excluded from) the process; and
what was the fate of their suggestions?
In this way, it becomes possible to differentiate between stakeholder participation in:
the creation of a political process/programme;
in the decision-making process;
implementation of a policy programme;
the accomplishment of an evaluation.
It might be that one small group of stakeholders is involved in all stages of the policy cycle – or it might be that the involved stakeholders shift from stage to stage in the policy cycle. Whatever the case, it is important to know and understand the reasons for (non-) involvement of stakeholders in each phase, as a precondition for understanding the content of the action plan and its eventual impact on target group behaviour. The methods for this part of the analysis may include analyses of documents, and interviews with key participants and with actors who are expected to be much influenced by the action plan.
Other (more resource demanding) methods used to analyse organic agriculture policy include network analysis (Moschitz and Stolze, 2007) or institutional change (Michelsen et al., 2001). Network analysis (Annex B3-1) is capable of documenting how far policy networks have been established and the way in which they work. It can help in exploring which actors make up the organic food and farming policy network, which are in the centre/periphery of the network, and how close relations are both within the organic sector and between the organic and the mainstream agriculture sector. Computing and visualising the various network measures can be carried out with UCINET and Visone software (see Annex B3-2 for details).
A second step is to analyse the congruence between stakeholders involved in policy-making and in the interests, and stakeholders actually affected by the implementation of the action plan as a whole and of major elements of the action plan. This involves an analysis of the policy instruments included in the process of making the organic action plan and their direct impact on various groups of stakeholders, and asking whether these groups of stakeholders did actually participate in the creation of these instruments and their implementation. The working of any organic action plan might be problematic if influential stakeholders keep away from it – and very profitable if they involve themselves enthusiastically. Similarly, it might be problematic if the action plan strongly influences the situation of stakeholders who happen to have no or only little political and administrative influence – and profitable if they are involved.
Having identified the most influential stakeholders regarding participation in policy-making and those most heavily influenced by the impact of policy, it is important to know these stakeholders’ orientation on the three main issues involved in implementation (Vedung 1997). They include stakeholders’ comprehension of the intervention (central or peripheral to their main activity), their available capability (financial and otherwise) relevant to the intervention, and their willingness to act in support of, or in opposition to, organic action plans as such or any concrete element of an action plan. Performance with regard to each of these properties may be different from that which might be attached to any stakeholder in advance. Some purely organic stakeholders may oppose political intervention if they feel it to be a threat to their current positions, while purely non-organic stakeholders may see an action plan as an opportunity to become involved in a business which they find is in accordance with their business strategy. On this basis, the main stakeholders may be divided into main groupings to ease the analysis. One suggestion is to distinguish between opponents and proponents in terms of measuring a level of conflict, for instance between organic and mainstream actors. Another suggestion is to distinguish between main opponents in the market place: producers and consumers with distributors coming in-between – or, as might be the case in some instances of organic food and farming, primary producers and consumers on the one side and processors and distributors on the other. A third potential relevant distinction is between actors working within four domains: agricultural policy, farming community, food market and the institutional setting of organic action plans (Michelsen et al., 2001).
Regarding stakeholder participation in the evaluation of an action plan (see also ORGAPET Section C4), it is of special importance to investigate whether there are stakeholders involved other than those identified as the most important stakeholders, in the creation and/or implementation phases. In addition, it is important to identify how stakeholders are involved in evaluation. Are they involved in defining the themes of the evaluation and in making use of the evaluation reports or are they mere objects for study by the evaluator? In both instances, it might help in understanding the purpose of the evaluation, and it may influence what is actually included or excluded in the evaluation.
A separate point of analysis is to identify the dynamics of the interaction between stakeholders regarding implementation (see also ORGAPET Section C1). One issue is the interplay between public agencies, as studied within implementation research. Here, the increasing complexity of agricultural policy in general and the special complexity involved in administering organic food and farming policy in particular, implies that organic action plans are expected to involve several public agencies originating in various ministries and/or administrative sectors within the same ministry. This kind of interrelationship is analysed in implementation research (Winter, 2003) and, with respect to the implementation of EU policies, in Europeanisation research (Featherstone and Radaelli, 2003). The common focus is on policy output, i.e. the extent to which public agencies approach target groups (whether farmers, business firms or citizens) in expected ways. The main idea behind the research is an attempt to explain implementation failure, i.e. why policies have not been properly implemented.
One issue is to identify the administrative setting of organic food and farming policies: which public agencies are involved in which parts of the policy, where are they located in the national administrative landscape, and how are they able to interact. A simple theory based on implementation research is that the fewer the number of veto points (i.e. the number of agencies and bureaucrats involved and the number of administrative boundaries between them), the easier the implementation and, hence, the realisation of policy impact (Pressman and Wildawsky, 1973). Another theory developed within both implementation and Europeanisation research is that successful implementation may be the result of joint social learning within public administration (e.g. Olsen, 1996; Börzel and Risse, 2003). They try to identify actors and institutions facilitating learning within the public sector. Specifically adapted for analyses of EU policies on organic food and farming, Michelsen (2008) suggests a combination and simplification of the two approaches by identifying the level of conflict involved in public administration of policies in support of organic food and farming. The main issue is that the development of organic food and farming is expected to release some level of conflict with the existing agricultural policy system because of the overt opposition to mainstream agricultural production systems. However, this conflict can be dealt with in various ways within the public sector – as low and medium levels of conflict among and within public agencies may help promote organic food and farming, while either a high level of conflict or a low level signifying ignorance with organic food and farming may hamper organic food and farming development.
It is a general experience of implementation and Europeanisation research that it is difficult to establish causality between policy output and policy impact, because policy impact often depends very much on (national or regional) contexts that are out of reach of public policy agencies such as market forces, or external shocks such as economic depression or prosperity. In addition, implementation of public policies takes time – especially when the policy involves some kind of major change or reform.
Having dealt with the issue of the dynamics of public agencies, it is also relevant to assess the character and extent of interrelationships between the two types of stakeholders, private and public. One issue is whether the interest groups of those affected by the policy are involved in policy administration. Another issue is the type of interaction with addressees of the policy in terms of promoting stakeholder comprehension, willingness and use of capabilities in accordance with policy intentions. To achieve the political intentions, changes in individual behaviour and in the working of institutions may be needed. This may take the form of a trial and error process, involving changes or adaptations to the institutional set up within a policy field. Analysing the growth patterns of organic food and farming in various types of EU member states, Michelsen et al. (2001) and Michelsen (2008) found that organic food and farming prospered in EU member states where changes in public and private stakeholders’ organisations took place relatively simultaneously, as this improved conditions for interaction between all types of stakeholders and beneficiaries in adapting to changing economic and political conditions. Hence, an analysis of the extent to which simultaneous changes in organisations have taken place may serve as an indicator for the level of dynamics between public and private stakeholders in support of the development of organic food and farming. It would be interesting to make an analysis of the learning process over several years, linking it to the way in which the stakeholders were involved in the various stages of the policy process. One key element is communication between the mainstream and the organic food sector. Was it possible for both sides to reach to a common understanding concerning goals or the way problems were analysed and solutions proposed?
Stakeholder involvement is given high importance in the EU action plan for organic food and farming. It is therefore necessary to make an assessment of stakeholder involvement regardless of the type of evaluation. The above-mentioned analyses can be input into any type of evaluation but they should be adapted to the main idea of the evaluation. One distinction is between evaluations where stakeholders are in charge of the evaluation and evaluations where stakeholders are mainly (or only) objects of the evaluation. Stakeholders in charge of the evaluation must influence the design of the evaluation and, thereby, questions regarding the activities of other stakeholders. Hence, it is part of the evaluation to select those stakeholders to be involved in the design and the ones to be left out.
One solution to the problem of stakeholder involvement is to leave it to a third party to define the basis for the evaluation, and to an independent body, without any connection to the organic food and farming sector, to perform it and to select the stakeholders involved. Another solution is to cope with the problem by involving the main stakeholders in the evaluation as much as possible – whether in terms of a formative or a summative evaluation. A third solution is to include a targeted selection of few stakeholders in the group in charge of the evaluation – for instance, interests intended to be promoted by the actions taken in response to the evaluation. A separate issue is choice of methods of obtaining reliable information from stakeholders, via postal surveys, self evaluation schemes and third party interviews, or through the collection of public data on performance.
The aim of the checklist in this section is to assist the evaluation of the involvement of stakeholders at all stages of the policy cycle. Some of the questions relate to the generic indicators identified in ORGAPET Section C3.
For EACH relevant stage of the policy cycle (agenda-setting, policy formulation/decision-making, implementation and evaluation), the following questions should be addressed, where necessary taking account of the perspectives of different informants (e.g. government, organic sector, research):
1. With respect to ALL relevant stakeholders (whether involved or not), identify the following information for each stakeholder in tabular form:
a. The
identity and type of the relevant
stakeholders/stakeholder groups
(e.g. public,
non-government, private and business organisations and individuals directly and
indirectly affecting, and affected by, a policy)
b. Their specific areas of interest
(e.g. agriculture, food, environment, consumer, health,
animal welfare)
c. Their level of expertise with respect to the policy
area
(high, moderate, limited, none)
d. Their priorities, objectives and
‘authority’ (i.e. power to make decisions/take action)
(either
documented, or implicit from financial backing or as potential beneficiaries of
policy actions)
e. Their involvement
in the policy process
(direct, indirect (e.g. represented by others),
voluntarily not involved or deliberately excluded)
- where appropriate give reasons
f. The degree of impact
(e.g. increase/reduction in operational activity, financial benefit) of the proposed policy on the
different stakeholders
(high, moderate, limited,
none)
g. Their orientation to the policy (e.g. central or peripheral, supportive or opposed, engaged or disengaged)
h. Their influence on the policy
(high, moderate, limited, none)
- explain why (e.g. political perspectives,
membership, expertise and knowledge, access to financial, labour, land resources)
Based on this tabulated information and other relevant sources, consider:
2. The overall result concerning involvement of stakeholders:
a. What priority was given to the involvement of
stakeholders?
(high, moderate, limited, none)
b. How well, and by which stakeholders
were the main stakeholder perspectives
(organic values/principles, market/ business and public goods (environment etc.))
covered?
(very well, adequately, hardly, not at all)
c. To
what extent can
the process of stakeholder involvement be considered to be
unbiased?
(unbiased, slightly biased,
highly biased)
d. How well did the process balance desirable inclusion, engagement, legitimation and knowledge on the one hand, against undesirable promotion of personal, business or institutional interests on the other?
(successfully, partly, unsuccessfully)
3. The actual engagement by stakeholders in the policy process and with each other:
a. What influence (effect) did the decision to implement the policy process have on stakeholders
b. Which stakeholders promoted which ideas?
c. What was the fate of their suggestions and for what reasons?
(positive/negative reactions, adoption/rejection)d. Who supported/opposed them and for what reasons?
e. What conflicts, if any, arose between different types of stakeholders?
(e.g. organic v. non-organic; pure v. mixed; between competitors; producers v. consumers; between other specific interest groups)f. What was the degree of conflict?
(e.g. low due to ignorance/disregard of organics by main stakeholders; high due to overt disagreement on all/most issues; medium due to agreement on some but disagreement on other issues; low due to consensus/agreement on all or nearly all issues)?
4. The level of communication and understanding of the policy process and outcomes
a. What mechanisms were used to promote two-way communication with stakeholders (see ORGAPET Sections A4 and C4)
(e.g. board/advisory group; consultation exercises; conferences/ workshops; ongoing or time-limited)?b. How were the plans and activities for involving stakeholders documented and communicated?
c. How well did stakeholders understand the policy and have the capability and willingness to act and affect policy change?
(very well, partly, hardly, not at all)d. To what extent did involvement in the policy process lead to learning and understanding by stakeholders?
(significant, moderate, limited, none)
Börzel, T. A. and T. Risse (2003) Conceptualizing the Domestic Impact of Europe. In: K. Featherstone and C.M. Radaelli (Eds.). The Politics of Europeanization. Oxford University Press, Oxford. pp. 57-82.
EC (1999) Evaluating Socio-economic Programmes. MEANS Collection Vols. 1-6. Office for Official Publications of the European Communities, Luxembourg.
Featherstone, K and C. M. Radaelli (eds.) (2003) The Politics of Europeanization. Oxford University Press, Oxford.
Michelsen, J., K. Lynggaard, S. Padel and C. Foster (2001) Organic farming development and agricultural institutions: a study of six countries. Organic Farming in Europe: Economics and Policy, Vol. 9, University of Hohenheim, Stuttgart.
Michelsen, J. (2008) A Europeanization deficit? The impact of EU organic agriculture regulations on new member states. Journal of European Public Policy 15/1.
Moschitz, H. and M. Stolze (2007) Policy networks of organic farming in Europe. Organic Farming in Europe: Economics and Policy Vol. 12. University of Hohenheim, Stuttgart.
Peters, B. G. and J. Pierre (2006) Handbook of Public Policy. SAGE Publications, London.
Pressman, J. L. and A. Wildawsky (1973) Implementation. How Great Expectations in Washington Are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes. The Oakland Project. University of California Press, Berkeley.
Olsen, J. P. (1996) Europeanization and Nation-State Dynamics. In: S. Gustavsson and L. Lewin (eds.), The Future of the Nation-State, Routledge, London, pp. 245-285.
Vedung, E. (1997) Public Policy and Program Evaluation, Transaction Publishers, London.
Winter, S. C. (2003) Implementation Perspectives: Status and Reconsideration. In: B. G. Peters and J. Pierre (eds.), Handbook of Public Administration, Sage Publications, London, pp. 212-222.
Annex B3-1: Network analysis introduction
Annex B3-2: Network analysis example