Oxfam GB

Impact Measurement for NGOs: Experiences from India and Sri Lanka Author(s): Linda Kelly, Patrick Kilby, Nalini Kasynathan Reviewed work(s): Source: Development in Practice, Vol. 14, No. 5 (Aug., 2004), pp. 696-702 Published by: Taylor & Francis, Ltd. on behalf of Oxfam GB Stable URL: http://www.jstor.org/stable/4029898 . Accessed: 09/01/2012 18:16
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Taylor & Francis, Ltd. and Oxfam GB are collaborating with JSTOR to digitize, preserve and extend access to Development in Practice.

http://www.jstor.org

5, 14, Development Practice,Volume Number August2004 in

FrPubishing CTaylof&

Impactmeasurement for NGOs: experiences from Indiaand Sri Lanka

Linda Kelly, Patrick Kilby,and Nalini Kasynathan

Introduction
In modem developmentdiscourse,one areathathas gained considerablecurrencyhas been the impactdeliveredby NGOs, mainlybecause theirwork is understoodto have directandobvious people. Despite these high expectations,a number effects on the lives of poor andmarginalised of studies of NGO work point to a perceived lack of 'evidence' from which to establish measurableimpact.These studiesinclude, among others,ODI (1996), the Danish NGO Impact Study (Oakley 1999), andthe AusAID NGO EffectivenessReview (1996). Thereis also a view thatthe NGO sectoras a whole tendsto exaggeratethe impactof its workto supportthe case for increaseddonorsupport(Roche 1999; Riddell 1999; Kenall and Knapp 1999). But perhapsthe most relevantchallengeto NGOs comes frompoor people themselves: ... poor people give NGOs mixed ratings. Given the scale of poverty, NGOs touch relativelyfew lives .. . some NGOs are largely irrelevant,self-serving, limited in their outreach and corrupt, although to a much lesser extent than the State ... (Narayan
1999)

It has been difficult for NGOs to demonstratethe worth and relevance of their work in a manneracceptableto scepticaloutsiders.Most of the studies cited above tend to reportthatthe evidence and frameworksfor demonstrating impactare lacking in the examples of NGO work, in work has no impact as such. Furthermore, developmentwork ratherthan suggest that their there are neither agreed definitions of what impact is, nor agreed methods for measuringit. It was within this context that Oxfam CommunityAid Abroad(CAA) decided to undertake an 'impact project' in 2000 aimed at developing a frameworkand process for the ongoing measurementof the impact of the organisation'swork. The impact project, which ran into 2001, was not only in response to the wider critiquesabout the failure to establish the impact of developmentworkbut was also a recognitionby OxfamCAA itself thatits own externaland more clearly what the impact of its work internalaudiences were interestedin understanding was and how that impact may be measured. The project ran over an 18-monthperiod. It startedwith several debates and discussions about how the agency might define and measure impact, and then two specific activities in measuringimpact in different countries where Oxfam CAA worked-India and Sri Lanka. The final stage brought the findings of this research together with the agency's needs and This report issues, to look at how it might develop an ongoing model of impact measurement. is a summaryof the researchundertakenand some of the conclusions about Oxfam CAA's impact that were reached throughthis project.

and impact Defining measuring

first what changes had been experiencedby people, The impact project sought to understand and, second, how the work of Oxfam CAA had or had not contributedto those changes. The
696
ISSN 0961-4524 print/ISSN1364-9213 online 050696-14 C)2004 OxfamGB DOI: 10.1080/0961452042000239841 Carfax Publishing

for Impact measurement NGOs

organisationrequired an understandingof impact that allowed it to focus on experiences interventionand make linkages to the beyond the more immediateoutcomes of any particular broadercontexts in which developmentactivities were occurring.The key questions that were being investigated were 'had peoples' lives improved?' and 'had we contributed to this improvement?' Moving from definition to measurementcreated furtherdifficulties, the main one being whose perspective should be taken into account when measuringimpact. Roche (1999) has to identifiedtwo broadapproaches deal with the issue of perspectivein impactassessment.The first is the 'projectout' approach.It startswith the aim of the projector programmeand then finds ways of measuringthe outcomes of that aim or objective from a variety of perspectives. The second is the 'context in' approach.It looks at the changes that are happeningin people's lives, what is significantaboutthese changes, and then assesses the usefulness or effect of any interventionin relation to these changes. Like the 'project out' approach,the 'context in' approachis resourceintensive and ultimatelyvalue laden and perspectivedependent.But the difference in this approach, which presents a significant step forward in addressing the relevance of development interventions, is that the starting point for measurementis the change in the conditions of a particularcontext, ratherthan the interventionitself. one. For example,in can These two approaches be appliedat a very basic level or at a broader anempowerment wouldlook at the intervention the changes and project,a 'projectout' approach whether thatoccurredin domesticor local powerrelationsas a consequenceof the intervention, it be microfinance,education,or health.A 'context in' approachwould look at the causes of disempowermentthat an individual or community is facing (the context) and relate the interventionto thatcontext.At the broaderor macro level, these two approachesto measuring such as nationwideadvocacycampaigns. impactcould be appliedto programmes in At a practicallevel, Oxfam CAA adopteda blend of both approaches its study of impact.It was clear thatinvestigatingonly the changes thathad occurredfor people, in the absence of an had of appreciation the changesthe respectiveprogrammes been tryingto achieve, would be too artificial.Partof the tension with the 'context in' approachis that staff and constituentsboth useful. It value whattheyhave been doing andhave valid reasonsfor consideringan intervention is difficultto assumethatwe startwith a blankslate andthatall thatimpactassessmentwould be aboutis looking afreshat whatpeople have experiencedandhow theirlives have changed.Part of the value of the impact enquiry is to assess what organisationswere trying to achieve in comparison to the changes experienced by people, and understandwhy the two might be different. Oxfam CAA was also looking to understand change from both the perspectiveof the people concerned,and from other perspectives, such as broaderlocal and regional changes that had more indirectandlonger-term impactson the people. This needed to be done in a fresh way that the significanceof such changefromthe perspectiveof the beneficiariesthemselves. openedup A furtherstep was then to test the relevanceof specific interventions againstthese changes. Withinthis framework,Oxfam CAA undertooktwo researchstudies,one in Indiaand one in to Sri Lanka,each designed to try differentapproaches defining andmeasuringimpactfor poor of and marginalisedpeople. The overall aim was to furtherthe organisationalunderstanding whathad been achieved,to considerthe implicationsof this learningfor ongoing programming, and also to assess how to measuresuch changemore effectively in the future.

The Indian study

in A study was undertaken India that looked at programmesof interventionover long periods (up to ten years) with long-standing Oxfam CAA partners, which had as their goal the Developmentin Practice, Volume14, Number5, August 2004 697

Linda Kelly, Patrick Kilby, and Nalini Kasynathan

empowermentof women. For the purposes of the study, empowermentwas defined as an expansion of choice and a person's enhancedcapacity or opportunityto act on those choices (Kabeer 1999; Hindess 1996). A total of 15 NGOs were studied, of which half were Oxfam CAA partners.From each of these NGOs a sample of self-help groups (the constituentgroup of around20 women that NGOs generallyfoster and work with) was chosen and a total of 77 women's groups were interviewed. A process for measuringempowerment was adoptedbased upon the experiencesof poor and marginalised women and how they answereda series of open-endedquestionsrelatingto (any) changes in their lives (Hines 1993). The approachthen set out to test the degree to which the work of the partnerNGOs had facilitated this change, and whether the change was in fact empowering, i.e. correspondingto an increased range of choices and action in the lives of women. Care was taken to ask open-endedquestions that did not suggest certain answers, and on sensitive issues questionswere asked in the 'thirdperson' so thatrespondentsdid not feel they were being personally 'put on the spot'. Similarly,the researcherwas not identified with a donor agency, but ratherwith a researchinstitution.These approachesenabled the women to make an assessment of the perceived changes that had occurredin their lives and gave them the opportunityto attributethese changes to the work of the NGO and explain how that occurred. Other questions were asked about village life and women's participationin it to provide some idea of the broadercontext and the disempoweringinfluences women generally faced. The resultsof the answersto the questionson the change that women had experiencedwere ranked.The broadrankingsthat emergedfrom the women themselves rangedfrom being able to go out of the house and engage with people in authoritysuch as bank managers and the police, throughto engaging with the local political processes. The empowermentchanges were then statistically tested in relation to several factors such as caste, education, land, village social capital, etc., and the range of informal, semi-formal, and formal accountability mechanisms the NGO had established with respect to its partnerorganisations.The results showed thatthreefactorswere statisticallysignificantin determiningempowermentoutcomes: 'downward'NGO accountabilityto local groups;how long the group had been meeting; and the leadershipof the group. The finding that formal accountabilityof the NGO to its constituency is related to strong outcomeshas implicationsfor how Oxfam CAA field partners empowerment engage with their constituency in terms of the level of control they give their constituency in determiningthe direction of their work. It also has implications for how Oxfam CAA develops and fosters partnerships,and how it manages its development programmes.It points to a much greater focus on local control at all levels. The second point is that empowermentof the most disadvantagedtakes time and requires to a long-termfocus. This finding has implicationsfor the 'project'approach developmentand tends to favour more strategic longer-term interventions. It also has implications for developing and supportingpartnersfor the long term, and recognises that seeking short-term outcomes may not lead to empowerment. The issue of the natureof the leadershipof groups appearedto be very important(an area often poorly covered in NGO practice). Those groups that had a rotated or collaborative leadershipapproachshowed strongerempowermentresults than those groups who depended on a single leader or even on the NGO to provide leadership.Again, these findings can give some insight into how Oxfam CAA develops and supportsit partners. Related to these specific findings was the broad observationthat even the most successful community development programmes should be aware of the context in which they are 698 Developmentin Practice, Volume14, Number5, August 2004

for Impact measurement NGOs

situated. This may require some integration with advocacy to ensure that there is a supportivelocal economic, political, and social environmentat the broaderlevel to ensure the sustainabilityof empowermentoutcomes. For example, a group of waste-pickerwomen who had enormous success in empowermentthat led to significant changes to their working and living conditions were having their livelihoods threatenedby a proposedprivatisationof waste management in their city. This change in urban waste management policy had the potential of underminingthe strong empowermentresults that had been achieved. A broader advocacy campaign on waste managementmay be requiredto preserve the rights of these women.

The Sri Lankanstudy

The Oxfam CAA programme in Sri Lanka works with poor and marginalised people, especially women, living in the areas subjectto civil war between the governmentand Tamil separatist groups. The methodology developed for the study had two parts that roughly to correspond the notions of 'projectout' and 'context in' approachesto impactassessment.As in the Indian study, people were asked 'What is going on in your lives? What have been significant changes in your lives? What are the significant problems?' Second, the study sought information against a set of impact indicators or 'change dimensions', developed externallyby Oxfam CAA staff, that focused upon the long-termchanges that the programme was trying to achieve. Data were sought from various sources and involved a large number of meetings with includingcommunitygroups,staff, the centralcommittee,and subcommitteesof stakeholders, each of the partnerorganisations.OtherNGOs, governmentofficials, externalexperts, and the like were also interviewed. Finally, a study on the national and regional changes that influenced developmentin Sri Lanka was undertaken,so that change could be understoodat several differentlevels beyond that of the communitiesthemselves. For the changes sought by the programme,the 'project out' assessment, each 'change dimension' was consideredfor the seven partnerNGOs studied, and either an assessment of change over time or an overall evaluation of progress was made. The dimensions that were examined were the levels of: * * * * . . * communityparticipation; social awareness; building; empowerment/capacity genderjustice; provision of services and benefits; inclusion of the most marginalised; institutionalstrengthening.

The issue-based assessment also identified other key factors that were outside the direct questioning but clearly related to the possible impact of the programme.These included the effects of the war and issues related to incompatibility and competition among donors. While the findings were generallypositive and the changes experiencedwere relatedto the work of Oxfam CAA, the relevance of the programme in terms of the overall impact experiencedby the poorest and most marginalisedpeople was not as great as expected. The study suggested that the CAA programmein Sri Lankahad been highly relevant to some of
Development in Practice, Volume 14, Number 5, August 2004 699

Linda Kelly, Patrick Kilby,and Nalini Kasynathan

poor problemsfaced by people-human rightsviolations, genderdiscrimination, the particular The interventionstended, however, to be limited services, and lack of civil society structures. to responding to the local expression of these problems and were unable to address the problemsof war,humanrightsviolations, caste, genderand ethnic discrimination, fundamental economic disadvantage,and the failure of governance. People were more able to take advantageof situationsof change (for example, they know more about their rights and how to organise), but the changes were not happeningat a level that ensured sustained difference. An example of this limited level of change came from looking at the degree to which women were empoweredby the programme. The staff knew what constituted women's empowerment and what the outcomes were, and they felt that the programmeswere moving in the right direction.The strategiesadopted to improve effective participationof women such as policies of positive discrimination, constitutionalchanges, regular training, and awareness raising were all tested and proven. However, the findings of the study clearly indicatedthat Oxfam CAA had a long way to go. While women reported a decrease in domestic violence and an increase in household income, many of the important decisions were still being made by men and, in some instances, women were willingly handing over decision-making power to the men. What became evident is that the empowerment of women in a complex war-zone situation requiresmore sophisticatedinterventionsthan those used in other settings. This finding led to changes in programming. New activities and approaches were introduced by programme staff and communities, following discussion and analysis of the research findings. The direction of the programme was improvedso that the objectives and indicatorsmore closely matchedthe areas of change requiredfor real empowerment.Second was a revision of the use of training as a tool for communitydevelopment.The content and approachesto the trainingoffered by Oxfam CAA were revised with a greaterfocus on group and issues-based training. The relationship between Oxfam CAA and the community-based organisations they supported was also revised, with Oxfam CAA establishing more participatory and arrangement-i.e. processes to bring the relationshipinto more of a partnership transparent promoting 'downward' accountabilitymechanisms, as in the case of Oxfam CAA's work in India detailed above. Finally, the findings indicated the need for strongerlinkages between development programmes at the local level and advocacy efforts at a broader national level.

learning Organisational
The two case studies identifiedissues thathave relevance for Oxfam CAA as an organisation. First is that a process of 'downward' accountability to partner organisations and their constituenciesinvolves mutual learning and the ability to listen to them. In this respect it is mechanismsbuilt into the design of the to important have formalor semi-formalaccountability including how local people can assert their rights vis-'a-visthe NGO. project, Second, action needs to take place at more than one 'level' of interventionfor any impact to be sustainable. That is, advocacy and policy work should be integrated with field programmesand capacity building. Action for change occurs at local, national,regional, and in some cases intemationallevels. Third,it appearsthat interventionsare dynamic and should change with a changing context and over time, and be monitoredto account for the changes While it is possible andensurethese changes arerelatedto the overall goals of the programme. in field programmessuch as Oxfam CAA's programmesin India to measureshort-termgains and Sri Lanka,it is clear that any impacttakes time and the variouscomponentsof change that 700 Developmentin Practice, Volume14, Number5, August 2004

Impact measurement NGOs for

need to occur will take place in differentways and at differenttimes. Managingthis process requiresregularattentionto measurementof change and difference. Finally,both studies indicatedthat people know what impact they want. Poor people know what is helpful to them and what is not. Unless the poor participatein defining the impact being sought and their perspective is recognised as central to assessing impact, NGOs (or indeed any aid-delivery mechanism) run the real risk of being considered 'largely irrelevant'.

Conclusions
The final lesson from the impact project was that the process of trying to measure impact taught Oxfam CAA more about the meaning and process of impact than many of the earlier discussions and debates had. While it seems that there is no one approachto measuring or defining impact, it would be a mistake to allow this to stop an organisationfrom attempting to learnmore aboutthe contribution is makingto sustainedand positive changes in people's it lives. One problemwith these types of field studies is the difficulty of drawingfirm conclusions about attribution. However, the attemptto learn about the impact of interventions,ratherthan just test the attainmentof objectives, was a powerful challenge to Oxfam CAA and its staff. In many ways the struggle to develop and test an appropriate methodology was as important as the specific conclusions.As one staff membernoted, 'Perhapsif we keep trying to measure what's really important,we might end up doing it.'

References
AusAID (1996) A Review of the Effectiveness of AusAID Support to NGO Programs, AusAID. Canberra: Hindess, B. (1996) Discourses of Power: From Hobbes to Foucault, Oxford:Blackwell. Hines, A. M. (1993) 'Linkingqualitativeand quantitativemethods in cross-culturalresearch: techniques from cognitive science', American Journal of Community Psychology 21(3):729-746. Kabeer, N. (1999) 'Resources, agency, achievements:reflections on measures of women's empowerment',Developmentand Change 30(3):435-464. Kenall, J. and M. Knapp (1999) 'Evaluationand the voluntarysector: emerging issues', in Action: Reshaping the ThirdSector, D. Lewis (ed.) InternationalPerspectives on Voluntary London:Earthscan. Narayan, D. (1999) 'Can anyone hear us? Voices from 47 countries', in D. Narayanwith Raj Patel, Kai Schafft, Anne Rademacher,and Sarah Koch-Schulte, Voices of the Poor: Can Anyone Hear us?, New York, NY: OUP (for the WorldBank). Oakley, P. (1999) Danish NGO Impact Study: A Review of Danish NGO Activities in Developing Countries-Synthesis Report, Oxford and Copenhagen: INTRAC/BECH Distribution. ODI (1996) 'The Impact of NGO Development Projects', Briefing Paper No. 2, London: ODI. Riddell, R. C. (1999) 'Evaluating NGO development interventions', in D. Lewis (ed.) International Perspectives on VoluntaryAction: Reshaping the Third Sector, London: Earthscan. Roche, C. (1999) ImpactAssessmentfor DevelopmentAgencies, Oxford: Oxfam GB. Developmentin Practice, Volume14, Number5, August 2004 701

James L. Garrett

The authors
Linda Kelly is an independent practitionerwho specialises in community development, monitoringand evaluation,and gender and aid management.Contactdetails: 37 North Valley Road, Park Orchards,3114, Australia. <paulnichols@onaustralia.com.au>.Patrick Kilby lectures in programmemanagementand empowermentand rights-baseddevelopment at the Australian National University. Contact details: Asia Pacific School of Economics and Government, Crawford Building, Australian National University, ACT 0200, Australia. <patrick.kilby@anu.edu.au>.Nalini Kasynathanhas taught at University Peredeniyain Sri Lankaand is currentlyresponsiblefor managingOxfam CAA programmeswithin South and East Asia. Contactdetails: Oxfam CommunityAid Abroad, 156 George St, Fitzroy,Vic 3065, Australia. <Nalinik@caa.org.au>.

Bridginggaps: collaborationbetween research and operational organisations

James L. Garrett

The potential of collaboration

Informationis essential to improving organisationaleffectiveness. The potential for benefits from collaborationbetween a researchorganisationand an operationalNGO seems large. The NGO can tap into the latest knowledge and learnhow to improveits own survey and analytical methods.This can in turnstrengthenits show of impact and innovationto donors.By working with NGOs, researcherscan get a bettersense of criticalpolicy and programmequestions and shape their work to demand,therebyincreasingthe probabilitythat others will actively make use of their findings. This Practical Note builds on the insights of Laura Roper's 2002 article on 'Achieving researchcollaborations'by reviewing one example of such successful academic-practitioner collaboration, a partnershipbetween CARE and the InternationalFood Policy Research Institute(IFPRI). Since 1997, CARE and IFPRI have collaboratedon increasingknowledge abouturbanlivelihoods that will be of use to programmedevelopment.The two have worked in a numberof countries,including livelihood assessments in Tanzania,issue-based research in Bangladesh,technical assistance in Mozambique,and programmeassessment in Peru and Ethiopia.Examiningtheir collaborativeefforts, this paper provides a concrete illustrationof how to build bridges and profit from synergies between two such organisations while highlightingpotentialbumps to expect along the way and what to do about them.

The gap
Despite the apparent benefits, explicit collaborations between research and operational perceptionsthrowup barriersto workingtogether. organisationsare not common. Institutional cultureand intellectual The roots of the problemmay be primarilydifferencesin organisational 702 Developmentin Practice, Volume14, Number5, August 2004