Assessing Policy Impact Through Partnerships

ILRI Process and Partnership for
Pro-Poor Policy Change
Methods for Assessing

Policy Impact
Process and Partnerships for
Pro-Poor Policy Change,
Project Initiation Workshop 1
ILRI, 21st February 2005

Workshop Outline
Introduction to the RAPID Framework and
ILRI/ODI Project
Case Study Approach
Episode Study Approach
Outcome Mapping Approach
RAPID Outcome Assessment (ROA)
Approach
Lunch
Practical Sessions
Workshop Purpose & Objectives

Purpose:
To familiarise the participants with the general approach and
specific methods to be used in the SDP case study
Objectives
By the end of the workshop, participants will:
understand the Process and Partnership for Pro-Poor
Policy Change projects purpose and general approach
have the opportunity to contribute their own suggestions to
improve the project;
understand, and have had the chance to try out the three
key methods which will be used in the project;
assess the usefulness of the approaches in their own work.
An introduction to the
RAPID Framework and
ILRI/ODI Project
Definitions
Research: any systematic effort to increase the
stock of knowledge
Policy: a purposive course of action followed by an
actor or set of actors
Agendas / policy horizons
Official statements documents
Patterns of spending
Implementation processes
Activities on the ground
Policy Processes
- Identify a policy problem
- Commission research
- Assess the results
- Select the best policy
- Establish the policy framework
- Implement the policy
- The problem is solved
Reality
The whole life of policy is a chaos of purposes
and accidents. It is not at all a matter of the rational
implementation of the so-called decisions through
selected strategies 1
Most policy research on African agriculture is
irrelevant to agricultural and overall economic
policy in Africa2
1 - Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in
Agricultural and Rural Development, Heineman Educational Books, London
2 Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges,
International Service for National Agricultural Research (ISNAR) Research Report No 21

Existing theory
1. Linear model 18. Linear model of communication, Shannon
2. Percolation model, Weiss 19. Interactive model,
3. Tipping point model, Gladwell 20. Simple and surprising stories,
4. Context, evidence, links framework, ODI Communication Theory
5. Policy narratives, Roe 21. Provide solutions, Marketing Theory I
6. Systems model (NSI) 22. Find the right packaging, Marketing II
7. External forces, Lindquist 23. Elicit a response, Kottler
8. Room for manoeuvre, Clay & Schaffer 24. Translation of technology, Volkow
9. Street level bureaucrats, Lipsky 25. Epistemic communities
10. Policy as social experiments, Rondinelli 26. Policy communities
11. Policy Streams & Windows, Kingdon 27. Advocacy coalitions etc, Pross
12. Disjointed incrementalism, Lindquist 28. Negotiation through networks, Sebattier
13. The tipping point, Gladwell 29. Shadow networks, Klickert
14. Crisis model, Kuhn 30. Chains of accountability, Fine
15. Framework of possible thought, 31. Communication for social change,
Chomsky Rockefeller
16. Variables for Credibility, Beach 32. Wheels and webs, Chapman & Fisher
17. The source is as important as content,
Gladwell
www.odi.org.uk/rapid/lessons/theory
Existing theory a short list

Policy narratives, Roe
Systems of Innovation Model, (NSI)
Room for manoeuvre, Clay & Schaffer
Street level bureaucrats, Lipsky
Policy as social experiments, Rondene
Policy streams and policy windows, Kingdon
Disjointed Incrementalism, Lindblom
Social Epidemics, Gladwell
The RAPID Framework
An Analytical Framework
External Influences The political context
Socio-economic and political and economic structures
cultural influences, and processes, culture, institutional
donor policies etc pressures, incremental vs radical
change etc.
The links between policy The evidence credibility, the

and research communities degree it challenges received
networks, relationships, power, wisdom, research approaches
competing discourses, trust, and methodology, simplicity of
knowledge etc. the message, how it is packaged
etc
Case Studies
Sustainable Livelihoods: The
Evolution of DFID Policy
The PRSP Initiative: Research in
Multilateral Policy Change
The adoption of Ethical
Principles in Humanitarian Aid
post Rwanda
Animal Health Care in Kenya:
Evidence fails to influence Policy
A Practical Framework
External Influences political context
Politics and
Campaigning, Policymaking
Policy analysis, &
Lobbying
research
Media,
Advocacy,
Scientific Networking Research,
information learning &
exchange & thinking
validation
links evidence
What you need to know

The external environment: Who are the key actors?
What is their agenda? How do they influence the political
context?
The political context: Is there political interest in
change? Is there room for manoeuvre? How do they
perceive the problem?
The evidence: Is it there? Is it relevant? Is it practically
useful? Are the concepts familiar or new? Does it need re-
packaging?
Links: Who are the key individuals? Are there existing
networks to use? How best to transfer the information? The
media? Campaigns?
What researchers need to do

What researchers What researchers How to do it
need to know need to do
Political Context: Get to know the policymakers. Work with them seek
Who are the policymakers? Identify friends and foes. commissions
Is there demand for ideas? Prepare for policy Strategic opportunism
What is the policy process? opportunities. prepare for known events
Look out for policy windows. + resources for others
Establish credibility Build a reputation
Evidence Provide practical solutions Action-research
What is the current theory?
Establish legitimacy. Pilot projects to generate
What are the narratives?
Present clear options legitimacy
How divergent is it?
Use familiar narratives. Good communication
Get to know the others Build partnerships.
Links
Who are the stakeholders? Identify key networkers,
Work through existing
What networks exist? networks. mavens and salesmen.
Who are the connectors, Build coalitions. Use informal contacts
mavens and salesmen? Build new policy networks.
Policy entrepreneurs
Storytellers Networkers
Engineers Fixers
Practical Tools
Overarching Tools
- The RAPID Framework Context Assessment Tools
- Using the Framework - Stakeholder Analysis
- The Entrepreneurship - Forcefield Analysis
Questionnaire - Writeshops
- Policy Mapping
Communication Tools - Political Context Mapping
- Communications Strategy
- SWOT analysis
- Message Design
- Making use of the media Research Tools
- Case Studies
Policy Influence Tools - Episode Studies
- Influence Mapping & Power Mapping - Surveys
- Lobbying and Advocacy - Bibliometric Analysis
- Campaigning: A Simple Guide - Focus Group Discussion
- Competency self-assessment
Practical Application
Within ODI
Workshops for researchers, policy makers
and activists.
Advice to a DFID forest/ground water
research project in India:
Less research
More communication
Developing champions in regional and national
government
Local, Regional & National advocacy campaign
Further Information / Resources

ODI Working Papers
Bridging Research and
Policy Book
Meeting series
Monograph
Tools for Policy Impact
RAPID Briefing Paper
www.odi.org.uk/rapid
Can ILRI do it?

Yes, but:
It this its role?
Global Public Good Research vs Policy Advocacy
Probably needs to do both:
How?
Understand the political context
Get the evidence & package it well
Strategic networking / lobbying / campaigning
Collaboration.
ILRI
ILRI
Process and Partnership for
International Livestock Research Institute
Process and partnership for

pro-poor policy change
The New DfID funded Project

ILRI
ILRI
Project Leaders: ODI / ILRI

Key collaborators: ECAPAPA
Case study collaborators in Kenya:
MoLFD / KARI
Range of NGOs & other SDP partners
ILRI
ILRI
Why would I be interested?

Not all research is expected or intended to lead

to policy change, but there may be;
Specific cases where research is expected to;
provide evidence for policy change
identify potential policies (or impact of)
influence the policy making process
(advocacy)
Cases where speculative research becomes
relevant because of changes in circumstance
ILRI
ILRI
The project
Ideas for methods and approaches

Lessons learnt from earlier activities
Identification of appropriate communication
tools
ILRI
ILRI
What will we be doing?

Three case studies in three DIFFERENT

countries
A project considered to have influenced
policy change
A stream of research addressing a particular
policy area
A clear policy change;
New policy statement
New law
Irrefutable change in way something is done
ILRI
ILRI
What will we be doing?

Three case studies

SDP and impact on changed view of
informal milk trade
????
????
ILRI
ILRI
Discussion:
Can ILRI hope to influence pro-poor policy
through research?
Any good case studies?
Case Study Approach
What is a Case Study?

Definition:
" A systematic inquiry into an event or a set of
related events which aims to describe and
explain the phenomenon of interest" Bromley
(1990)
Why is it useful?
Goal :
to describe as accurately as possible the
fullest, most complete description of the case.
An ideal methodology when a holistic, in-

depth investigation is needed
Designed to bring out the details from the
viewpoint of the participants by using
multiple sources of data
Types of Case Study

Types of case studies:
Exploratory,
Explanatory,
Descriptive (Yin, 1993)
Stake (1995) included three others:
Intrinsic - when the researcher has an interest in
the case;
Instrumental - when the case is used to
understand more than what is obvious to the
observer;
Collective - when a group of cases is studied.
Issues
The unit of analysis is a critical factor
Typically a system of action rather than an
individual or group of individuals
Tend to be selective, focusing on one or two
issues that are fundamental to understanding the
system being examined
Case studies are multi-perspectival analyses
The researcher considers not just the voice and
perspective of the actors, but also of the relevant
groups of actors and the interaction between them
They give a voice to the powerless and voiceless.
Triangulation
Data source triangulation, when the researcher looks
for the data to remain the same in different contexts;
Investigator triangulation, when several investigators
examine the same phenomenon;
Theory triangulation, when investigators with
different view points interpret the same results; and
Methodological triangulation, when one approach is
followed by another, to increase confidence in the
interpretation.
Applications
To explain complex causal links between
research and policy
To describe the real-life context in which
policy has been influenced by research
To describe the policy influencing process
itself
To explore those situations in which the
policy intervention being evaluated has no
clear set of outcomes.
Process
1. Design the case study protocol:
determine the required skills
develop and review the protocol
2. Conduct the case study:
prepare for data collection
distribute questionnaire
conduct interviews
3. Analyze case study evidence:
analytic strategy
4. Develop conclusions, recommendations, and
implications based on the evidence
Episode Study
Approach
What is an Episode Study

A study that focuses on a clear policy change
and tracks back to assess what impact
research had among the variety of issues that
led to the policy change.
What is the purpose?

Tracking backwards from policy change to
any particular research which influence policy
an excellent way of investigating the
influence of research on policy
Can focus on a single episode or
comparative episodes.
Advantage
The process of working backwards in time
gives a more realistic view of the broad
range of factors other than research that
influence policy
Tracking forward probably overemphasizes
the importance of research
Issues
Policy processes are complex, multi-layered and
change over time
Often difficult to isolate the impact of research
from other factors
Actors may re-write history
Important to seek the views of a wide range of
informed stakeholders
The process of preparing an episode study is
iterative
Key facts and / or inconsistencies need to be
cross-checked with key informants
Process
1. Identify a clear policy change.
2. Identify key Research Questions (draw on
RAPID framework)
3. Explore how and why those policy
decisions and practices took place
4. Assess the relative role of research in that
process by drawing on the framework.
Apply the RAPID Framework

External Influences political context
Politics and
Campaigning, Policymaking
Policy analysis, &
Lobbying
research
Media,
Advocacy,
Scientific Networking Research,
information learning &
exchange & thinking
validation
links evidence
Key Questions
The external environment: Who are the key actors?
What is their agenda? How do they influence the political
context?
The political context: Is there political interest in
change? Is there room for manoeuvre? How do they
perceive the problem?
The evidence: Is it there? Is it relevant? Is it practically
useful? Are the concepts familiar or new? Does it need re-
packaging?
Links: Who are the key individuals? Are there existing
networks to use? How best to transfer the information? The
media? Campaigns?
Methods
Steps 3 and 4 can be done through a variety
of methods:
review of the literature;
interviews with key actors;
capturing the authors own experience; and
discussions at workshops.
Episode Study Examples
Paravets in Kenya
1970s - Professionalisation of Public Services.Research
International
- Structural Adjustment Collapse
- Paravet projects emerge.
1980s - ITDG projects
projects. collaborative research.
- Privatisation
Privatisation.
- ITDG Paravet network
network.and change of DVS.
1990s
- Rapid spread in North. The Hubl Study
Dr Kajume
- KVB letter (January 1998).
- Multistakeholder WSs new policies.
2000s
- Still not approved / passed!
PRSPs Political Context

Widespread awareness of a problem with
international development policy in late 90s
Failure of SAPs (and Asian financial crisis)
Mounting public pressure for debt relief
Stagnation of Comprehensive Development
Framework idea
Diverging agendas (UK Poverty, US
Governance)
WB/IMF Annual General Meeting, Sept 1999
PRSPs Evidence
Long-term academic research informing new
focus on poverty, participation, ownership, aid
effectiveness etc
Applied policy research:
ESAF reviews
HIPC review
SPA Working Groups
NGO research on debt
Ugandas PEAP
PRSPs Links
WB, IMF, SPA, Bilaterals, NGOs all involved
Formal an informal networks
None of the players was more than two
handshakes away from any of the others
Outcome Mapping
What is it?
an integrated PM&E tool
a system to think holistically & strategically about
how we intend to achieve result
an approach that focuses on changes in the
behaviour, relationships or actions of partners (as
outcomes)
a methodology that characterizes and assesses the
programs contributions to the achievement of
outcomes
an approach for designing in relation to the broader
development context but assessing within your
sphere of influence
Focus: On Behavioural Change
The Focus of Outcome Mapping
Inputs Activities Outputs Outcomes Impacts
Behavioural Changes 25
How can it be used?

For a program to tell its performance story in
outcome terms by:
articulating its goals and designing its activities
designing a monitoring system for assessing
internal performance and outcomes of partners
setting a use-oriented evaluation plan
Why use it?

Focussing on changes in partners behaviour,
relationships, or actions allows a program to:
measure results within its sphere of influence
obtain feedback about its efforts in order to
improve its performance
take credit for its contributions to the achievement
of outcomes
show progress towards outcomes
Terminology
Outcomes: changes in behaviours, relationships,
activities and/or actions of the people, groups and
organisations with whom we work
Vision: the broad human, social and environmental
betterment we desire
Mission: how we intend to contribute towards the
achievement of the vision
Boundary partners: individuals, groups and
organisations with whom we interact directly to
effect change
Outcome challenges: changes behaviours of the
boundary partners as identified by the vision
The Three Stages

Intentional Study Design
Why ? How ? Who ? What ?

Mission
Vision Statement Strategy Maps Boundary Partners Outcome Challenges
Organizational Progress Markers
Practices
Performance Monitoring
Provides a framework for a continuous
monitoring of the initiative as a tool to
achieve its outcomes.
The program uses progress markers, a set
of graduated indicators of behavioural
change, identified in the intentional design
stage to clarify direction with its primary
partners and to monitor outcomes
Evaluation Planning
Helps identify the evaluation priorities
assessing the strategy at greater depth than
the performance monitoring stage
Main Elements
RAPID Outcome
Assessment
What is it?
A Visual Tool
Combines the outcome mapping concept within a
case study & episode study approach
Systematic approach to collecting information about
changes in behaviour of key project partners that
contributed to the policy change
Assessment of the contribution of the project
(programme, strategy, etc.) to observed changes in
behaviour and ultimately to the policy change
Approach
1. Describe policy environment at end
2. Describe policy environment at the beginning
3. Identify the key policy actors
4. Identify key boundary partners
5. Describe boundary partner behaviour at end
6. Describe boundary partner behaviour at beginning
7. Describe changes in BP behaviour
8. Describe changes in project (strategic/opportunistic)
9. Describe external influences
10. Determine level of impact of changes in project
11. Determine level of impact of external influences
12. Check through external interviews
13. Write report
Sources & Outputs

1. Literature review
- Project background, progress, (published) achievements
2. Participatory workshop with staff (and BP)
Gather detailed information
Identify issues for further investigation
3. Interviews with key informants to:
Triangulate the result of the workshop,
Fill the gaps of information
Clarify causality
4. Report Writing
Visual and Narrative
ROA Terms and Definitions

Boundary partners: individuals, groups and
organisations with whom we interact directly
to effect change.
Outcomes: changes in behaviours,
relationships, activities and/or actions of the
people, groups and organisations with whom
we work.
Behaviours: the way we or our boundary
partners do or think about things.
The key steps of the ROA framework

1. Describe the policy environment at the end
of the project
BP 0 1 2 3 4 5 6 7 8 9 10
Policy Environment
Policy Change
Project
EE

2. Describe the policy environment at the
beginning of the project
BP 0 1 2 3 4 5 6 7 8 9 10
Policy Environment
Policy Change
Project
EE

3/4. Identify key policy actors and boundary
partners (that were influential at end)
Before year/month Today
BP 0 1 2 3 4 5 6 7 8 9 10
BP1
BP2
Policy Environment
BP3
Policy Change
BP4
BP5
BP6
BP7
Project
EE

5. Describe the behaviours of the boundary
partners that contributed to the change in
the policy environment or policy
BP
Before
0 1 2 3 4
year/month
5 6 7 8 9 10
Today
BP1 5
BP2 4
Policy Environment
BP3 8
Policy Change
BP4 7
BP5 4
BP6 9
BP7 3
Project
EE

6. Describe the behaviours of the boundary partners
at the beginning of the project
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 5
BP2 0 4
Policy Environment
BP3 0 8
Policy Change
BP4 0 7
BP5 0 4
BP6 0 9
BP7 0 3
Project
EE

7. Map the key changes in behaviour for each
boundary partner from the start of the project
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project
EE

8. Map the key changes in the project including
organisational changes, outputs and changes in
behaviour during the same period.
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project 0 1 2,3 4,5 6 7 8 9 10
EE

9. Map the external influences including the actions f
strategic partners and other exogenous factors
during the same period
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project 0 1 2,3 4,5 6 7 8 9 10
EE 0 1 2 3,4 5 6 7 8 9 10 11

10. Determine the level of impact/influence of the
project on the changes in behaviour of the
boundary partners
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project 0 1 2,3 4,5 6 7 8 9 10
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence

11. Determine the level of impact/influence of external
influences on the changes in behaviour of the
boundary partners and the project
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project 0 1 2,3 4,5 6 7 8 9 10
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence

12. Refine conclusions with in-depth interviews and
assess the real contribution of the project on the
policy change
BP 0 1 2 3 4 5 6 7 8 9 10
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP6 0 1,2 3,4,5 6,7,8 9
BP7 0 1, 2 3
Project 0 1 2,3 4,5 6 7 8 9 10 11
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence

Assessing Policy Impact Through Partnerships

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Assessing Policy Impact Through Partnerships

Transféré par

Droits d'auteur :

Formats disponibles

ILRI Process and Partnership for

Pro-Poor Policy Change

Methods for Assessing

ILRI, 21st February 2005

Workshop Purpose & Objectives

International Service for National Agricultural Research (ISNAR) Research Report No 21

Existing theory a short list

The links between policy The evidence credibility, the

What you need to know

What researchers need to do

Further Information / Resources

Can ILRI do it?

International Livestock Research Institute

Process and partnership for

The New DfID funded Project

International Livestock Research Institute

Project Leaders: ODI / ILRI

Why would I be interested?

International Livestock Research Institute

Not all research is expected or intended to lead

International Livestock Research Institute

Ideas for methods and approaches

What will we be doing?

International Livestock Research Institute

Three case studies in three DIFFERENT

What will we be doing?

International Livestock Research Institute

Three case studies

International Livestock Research Institute

What is a Case Study?

An ideal methodology when a holistic, in-

Types of Case Study

What is an Episode Study

What is the purpose?

Apply the RAPID Framework

PRSPs Political Context

Focus: On Behavioural Change

The Focus of Outcome Mapping

Inputs Activities Outputs Outcomes Impacts

How can it be used?

Why use it?

The Three Stages

Intentional Study Design

Why ? How ? Who ? What ?

Sources & Outputs

ROA Terms and Definitions

The key steps of the ROA framework

The key steps of the ROA framework

The key steps of the ROA framework

The key steps of the ROA framework

The key steps of the ROA framework

The key steps of the ROA framework

BP6 0 1,2 3,4,5 6,7,8 9

The key steps of the ROA framework

BP6 0 1,2 3,4,5 6,7,8 9

Project 0 1 2,3 4,5 6 7 8 9 10

The key steps of the ROA framework

BP6 0 1,2 3,4,5 6,7,8 9

Project 0 1 2,3 4,5 6 7 8 9 10

The key steps of the ROA framework

BP6 0 1,2 3,4,5 6,7,8 9

Project 0 1 2,3 4,5 6 7 8 9 10