Measuring the effects of

environmental policies:
an issue of methodology
and legitimacy

EPA Evaluation, Washington DC, June

2006
 Point of departure

ƒ Quantitative methods are only one type of

method to evaluate policy
ƒ They work well under certain assumptions and
ƒ For certain purposes
ƒ Policy analysis often requires qualitative
approach in addition to ‘data’/numbers
oriented approaches
ƒ The issue is not quantitative or qualitative, but
which method, when and how?
EPA Evaluation, Washington DC,
June 2006
 Policy is ...

A complex, multi-faceted proces including

multiple actors and institutions, multiple fases,
multiple measures and instruments, multiple
secondary processes,...
In other words, there is no such thing as a
simple, straightforward policy evaluation from
the perspective of the social policy
scientist/evaluator
EPA Evaluation, Washington DC,
June 2006
 Policy evaluation: basic questions

ƒ Why evaluate?
ƒ What are the results of policy making? (judgement)
ƒ Legitimacy through results (political)
ƒ Pragmatic improvements (learning)
ƒ Many evaluations answer this question in a very
restricted way. This obscures the debate about
policy, about the environment and also about the
value of the discipline of evaluation

EPA Evaluation, Washington DC,

June 2006
 Plan evaluation

ƒ Mostly ex ante: will this policy/plan work,

why, how, with what effects?
ƒ Importance of the policy theory
ƒ Instrument choice
ƒ Context elements
ƒ Increasingly done in multiple policy fields
ƒ Strong demand/activity from civil society
actors
EPA Evaluation, Washington DC,
June 2006
 Process evaluation

ƒ Process: the interactions between actors,

institutions with the goal of environmental policy
making and implementation
ƒ Relevant questions:
ƒ who was involved, why and how?
ƒ status of diff. elements of the process,
legitimacy of the process
ƒ link between the process and the outcomes
EPA Evaluation, Washington DC,
June 2006
 Instrument evaluation

ƒ Instruments: broad definition

ƒ Steering instruments: carrot, stick, preech or legal,
economic, social
ƒ Direct instrument: actions by the government: ex:
infrastructure
ƒ Relevant questions:
ƒ does the instrument work?
ƒ circumstances: context, prerequisites
ƒ instrument mix

EPA Evaluation, Washington DC,

June 2006
 Shift towards effectiveness
evaluation
ƒ Why?
ƒ At the end of the day, the results are what matters
ƒ link with legitimacy!
ƒ State of policy making vs. state of the environment
ƒ Attention from:
ƒ historically: environmentalists
ƒ scholars studying environmental policy
ƒ policy makers: EU, OECD
EPA Evaluation, Washington DC,
June 2006
 Effects evaluation: basics

ƒ Effectiveness
ƒ Causal relationship between the policy and goal
attainment
ƒ Side effects evaluation
ƒ Causal relationship between the policy and side effects
(=unintended effects)
ƒ Effects evaluation
ƒ Combined appraisal of effectiveness and side effects
evaluation
EPA Evaluation, Washington DC,
June 2006
 Societal effectiveness

NEEDS MEANS - SOCIETAL OUTCOME IMPACT

INSTRUMENTS

OBJECTIVES OUTPUT

MEANS – POLICY PROCESS

INPUTS

Institutional effectiveness

Target group effectiveness

ImpactEPA
effectiveness
Evaluation, Washington DC,
June 2006
 Why include side-effects?

ƒ More complete cost/benefit analysis of

environmental policies
ƒ Attention for the intrinsic interconnectedness of
policy goals and outcomes in different policy
domains
ƒ Important in the public discourse and
appreciation
ƒ Legitimacy of the policy
EPA Evaluation, Washington DC,
June 2006
 Methodology (1/2)
ƒ ‘Modus Narrandi’ (cf. Modus Operandi)
ƒ 4 steps
A. Reconstruction of the effect spectrum
ƒ Reconstruction of the policy theory
ƒ Reconstruction of the blind spot
ƒ How? Documents, interviews, literature, experience
B.Valueing effects
ƒ Observed changes: linked to goals and side-effects
ƒ How? Data, if available; proxi-variables; qualitative
data
EPA Evaluation, Washington DC,
June 2006
 Methodology (2/2)
C. Establishing causality
ƒ Causal chains: Logical linkages between output, outcome,
impact
ƒ Causal proximity
ƒ Causal contribution
ƒ How? Analytical models, interviews, expert judgements
D. Causal narrative story:
ƒ ‘Putting things together’
ƒ How: logic, triangulation, ‘craftsmanship’
ƒ Shifting the burden of proof
EPA Evaluation, Washington DC,
June 2006
 Modus Narrandi
Reconstruction Establi shing
of the effects the effects
spectrum
E stablishin g th e
un derla ying principles
to achie ve th e goal

Reconstruction
of the causal and
impact model

Judging the merit

of the policy theory

Valuing of
effects

Reconstruction
of the ca usal Em pirical da ta
narrati ve sto ry

Triangulation

Causality
Pol icy the ory

Illu strate cau sal

links

Ca usal
proxim ity

Ca usal
con tribution

Conclude and make

statement on (side) effectiveness