Evaluation of the FAO Response to the Pakistan Earthquake

May July 2008

Final Report
February 2009

John Watt, Olivier Cossee, Javid Ahmed, Naeem Khalid, Mohammed Riaz, Paigham Shah

Table of Content Executive Summary .......................................................................................................6 Part I Background .....................................................................................................15 1. 2. 3. Introduction......................................................................................................15 Purpose of the Evaluation and Methodology...................................................18 Overview of the FAO Programme...................................................................21 3.1. Damage and Needs Assessments .............................................................21 3.2. The Relief Programme: Saving Livelihoods to Save Lives.....................24 3.3. The Rehabilitation Programme ................................................................24

Part 2 Systemic Functions.........................................................................................28 1. 2. 3. Strategic and Programme Planning..................................................................28 Logistics, Administration and Human Resource .............................................31 Monitoring Performance ..................................................................................35

Part 3 Sectoral Intervention Areas ............................................................................36 1. Farming Inputs .................................................................................................36 1.1 Description of deliverables to date ..........................................................36 1.2 Relevance .................................................................................................37 1.3 Efficiency and timeliness .........................................................................38 1.4 Effectiveness and quality .........................................................................39 1.5 Targeting and equity ................................................................................39 1.6 Impact and sustainability .........................................................................41 Livestock Inputs...............................................................................................45 2.1 Description of deliverables to date ..........................................................45 2.2 Relevance .................................................................................................45 2.3 Efficiency and timeliness .........................................................................45 2.4 Effectiveness and quality .........................................................................47 2.5 Targeting and equity ................................................................................47 2.6 Impact and sustainability .........................................................................48 Extension, Training and Farmer Field Schools................................................49 Rehabilitation of Community Physical Infrastructures....................................51 4.1 Description of deliverables to date ..........................................................51 4.1 Relevance .................................................................................................52 4.2 Efficiency and timeliness .........................................................................53 4.3 Effectiveness and quality .........................................................................54 4.4 Impact and sustainability .........................................................................55 Watershed Management...................................................................................56 5.1 Description of deliverables to date ..........................................................56 5.2 Relevance .................................................................................................57 5.3 Efficiency and timeliness .........................................................................58 5.4 Effectiveness and quality .........................................................................59 5.5 Impact and sustainability .........................................................................60 2

2.

3. 4.

5.

5.6

Environmental impacts of reconstruction ................................................61

Part 4 Working with Partners....................................................................................62 1. Community Mobilization and Participation - Between Rhetoric and Practice 62 1.1 Context.....................................................................................................62 1.2 Roles in Community Participation...........................................................62 1.3 The CBLRP Experience...........................................................................63 1.4 The Case of the ADB-funded Project ......................................................65 1.5 The Case of the SIDA/FAO Project.........................................................66 1.6. Conclusions..............................................................................................67 Coordination and the Cluster Approach ..........................................................69

2.

Part IV: Conclusions, Lessons and Recommendations................................................72 1. 2. 3. 4. 5. 6. 7. 8. A Relief Phase of Good Quality ......................................................................72 A Poor Maize Seed Distribution in 2006.........................................................73 Rehabilitation as a Balancing Act....................................................................74 Insufficient Monitoring Capacity within the ERCU........................................76 Forestry and Watershed Management .............................................................77 Coordination: a Major Role .............................................................................78 Fostering Successful Partnerships....................................................................79 Building Back Better? ..................................................................................80

Annex 1 Annex 2 Annex 3 Annex 4

Terms of Reference Persons Met Mission Itinerary Cost-effectiveness of the 2006 Wheat Seed and Fertilizer Distribution

List of Acronyms
3W ADB AFSP AGAH AGAP AGNP AGPC AGPS AJK AKDN AKRSP AUD CBLRP CCA CLRP CO CPI CRS CTA DAP DDF DfID DOA DRU EDO ERCU ERRA EU FAO FAOR FFS FGD FOMC GOP HFSA HH ICIMOD ICRC IFAD IFRC ILO IP LEGA LOA LWG MDG MINFAL MOU MT Who does what where Asian Development Bank Procurement Service, FAO Animal Health Service, FAO Animal Production Service, FAO Nutrition Programmes Service, FAO Crop and Grassland Service, FAO Seed and Plant Genetic Resources Service, FAO Azad Jammu and Kashmir Aga Khan Development Network Aga Khan Rural Support Programme Office of the Inspector General, FAO Community Based Livelihoods Recovery Programme Common Country Assessment (UN) Community Livelihoods Rehabilitation Plan Community Organization Community Physical Infrastructure Catholic Relief Services Chief Technical Advisor Di-Ammonium Phosphate Dosti Development Foundation UK Department for International Development Department of Agriculture District Reconstruction Unit Executive District Officer Emergency and Rehabilitation Coordination Unit Earthquake Relief and Rehabilitation Authority European Union Food and Agriculture Organization of the United Nations FAO Representative Farmer Field School Focus Group Discussion Forest Conservation Service, FAO Government of Pakistan Holistic Food Security Assessment Household International Centre for Integrated Mountain Development International Committee of the Red Cross International Fund for Agricultural Development International Federation of Red Cross and Red Crescent Societies International Labour Organization Implementing Partner General Legal Affairs Service, FAO Letter of Agreement Livelihoods Working Group Millennium Development Goal Ministry of Food, Agriculture and Livestock Memorandum of Understanding Metric Tonnes 4

MTR NARC NDMA NGO NRCB NRSP NWFP OCD OCHA OFDA OFWM PAK PERRA PKR PO QIA RC SERRA SFERA SIDA SLA SRSP SUPARCO TCAP TCE TCEO TCER TCOM TCP TOR UC UK UN UNDAF UNDP UNDSS UNHCR UNICEF UNIDO UNRC USA USAID WB WFP

Mid-Term Review National Agricultural Research Council National Disaster Management Authority Non Governmental Organization Bioenergy Group, FAO National Rural Support Programme (NGO) North West Frontier Province Office for Coordination and Decentralization, FAO Office for the Coordination of Humanitarian Affairs (UN) Office of U.S. Foreign Disaster Assistance On-Farm Water Management Pakistan-administered Kashmir Provincial Earthquake Recovery and Rehabilitation Authority Pakistani Rupees Purchase Order Qualitative Impact Assessment Resident Coordinator (UN) State Earthquake Recovery and Rehabilitation Authority Special Fund for Emergency and Rehabilitation Activities (FAO) Swedish International Development Agency Sustainable Livelihoods Approach Sarhad Rural Support Program (NGO) Pakistan Space and Upper Atmosphere Research Commission Field Programme Development Service, FAO Emergency Operations and Rehabilitation Division, FAO Emergency Operations Service, FAO Rehabilitation and Humanitarian Policies Unit, FAO Field Programme Monitoring and Coordination Service, FAO Technical Cooperation Programme Terms of Reference Union Council United Kingdom United Nations United Nations Development Assistance Framework United Nations Development Programme United Nations Department of Safety and Security United Nations High Commissioner for Refugees United Nations Children's Fund United Nations Industrial Development Organization United Nations Resident Coordinator United States of America United States Agency for International Development World Bank World Food Programme

EXECUTIVE SUMMARY
Introduction
On 8 October 2005, an earthquake of magnitude 7.6 on the Richter scale struck northern Pakistan. It flattened entire towns and villages across the Northwest Frontier Province (NWFP) and Pakistan-administered Kashmir (PAK), killing more than 73,000 people and seriously injuring some 70,000. Around 3.5 million people were rendered homeless. The quake also extolled massive losses on assets supporting local livelihoods, notably in the agriculture sector with damage and loss estimated at US$409 million. Crop losses ranged from 30 to 75 percent. Food, tools and seed stocks were buried under collapsed buildings. About half of the traditional irrigation infrastructure was deeply damaged, rendering it inoperative. Livestock died under collapsed buildings and in landslides. The earthquake also magnified the impact of erosion and environmental degradation in this Himalayan region, causing several major landslides and thousands of smaller landslips. A massive relief effort was mounted involving the Government of Pakistan (GOP), the army and relief agencies. People from every corner of Pakistan and overseas volunteered to help or donated money, clothes and food. The GOP put in place a fund to support the reconstruction of earthquake-resistant homes and set up to reconstruct public buildings and other infrastructure through the newly established Earthquake Relief and Rehabilitation Authority (ERRA).

The FAO Programme

The response was funded through nine projects totalling US$22 million and staged in two phases: a relief programme of five projects implemented in 2006, all called Saving livelihoods to save lives and aiming at the prompt resumption of agricultural and animal production activities; and a rehabilitation programme implemented from the end of 2006 to date and composed of three larger projects. Each rehabilitation project had its own specific objectives and operational set up, as follows: a) The Community Based Livelihoods Recovery Programme (CBLRP) funded by the EC involved four United Nations agencies: UNDP in charge of overall programme management and community mobilisation, FAO providing assistance in the agriculture and forestry sectors and ILO and UNIDO in charge of training and income-generating activities. b) The Immediate Support to Poor and Vulnerable Households in the 2005 Earthquake was funded by the Asian Development Bank (ADB) and provided communities residing at altitudes higher than 1500 meters above sea level with various agricultural and livestock inputs, the rehabilitation of Community Physical Infrastructures (CPIs, i.e. irrigation canals, field terracing, check dams, etc.) and Farmer Field Schools. c) A Project to assist ERRA and its partners in restoring livelihoods in the earthquake-affected areas of Pakistan was funded by the Swedish International Development Agency (SIDA) to support the implementation of ERRAs livelihoods rehabilitation strategy, facilitating the development of Community 6

Livelihoods Rehabilitation Plans (CLRPs) for GOP funding, and promoting participatory watershed management in pilot sites.

Relevance
The FAO response to the earthquake consistently used a livelihoods-based approach from damage and needs assessment to distribution of agricultural assets, to supporting coordination mechanisms, and to building the capacity of ERRA in the livelihoods sector. This approach significantly enhanced the relevance of the programme to the needs and opportunities of affected communities. Relief interventions were all responding to clear and pressing needs. The rehabilitation programme was also found very relevant, notably the idea of rebuilding damaged infrastructure through partnerships with Non-Governmental Organizations (NGOs) and Community Organizations (COs), and the fact that the programme decided not to restock animals but rather to provide shelter and feed for surviving livestock.

Efficiency
The programme is managed by the Emergency and Rehabilitation Coordination Unit (ERCU). Finances and administration are overseen by the Representation. The FAO representative has been very supportive of the emergency operations but his authority to approve procurements and payments is limited at US$50,000. The ERCU has been staffed through local recruitments of competent technical specialists and led for most of the programme implementation period by an experienced Emergency Coordinator, a key factor in the generally satisfactory results of the programme. However, some important managerial positions such as the heads of the two field offices in Muzzafarabad and Abbottabad were not filled at the time of the evaluation. The absence of a procurement officer in country contributed to delays and to some sub-standard goods being procured: 700 metric tonnes (MT) of maize seed procured in 2006 had serious germination problems, in spite of the now mandatory superintendence by an independent firm. The superintendence report clearing this seed was most probably fraudulent. The programme was implemented through partnerships with national and international NGOs and in collaboration with Government departments. The performance has been generally correct in spite of FAO not providing much capacity building and not formally screening Implementing Partners (IPs) before entering into agreements with them. Another issue was that the IPs performance was not monitored independently. While FAO was clearly committed to monitoring and lesson learning, the monitoring system depended largely on the honesty and capacities of the IPs staff.

Effectiveness
FAO and its partners delivered various inputs to a reported 260,000 vulnerable households in earthquake-affected areas from 2005 to 2008. These distributions started immediately after the earthquake and peaked in 2006. The total number of reported beneficiaries assumes that each kit was given to one and only one beneficiary. Taking into account possible double counting of beneficiaries across distributions, the evaluation mission estimates that FAO farm input distributions have 7

reached approximately 150,000 households, i.e. about 1 million people or a third of the 400,000 farming households affected by the earthquake. Farming Inputs The farming inputs given by FAO (wheat, maize, sorghum, oats and vegetable seeds, fertilizers, fruit saplings) were found very relevant, as subsistence farming and animal husbandry constitute the main sources of livelihoods in the earthquake zone. In most cases, the varieties distributed were appropriate for the farming systems, thanks to clear variety recommendations from the Department of Agriculture. However, the shortage of good quality maize seed in the country was a severe constraint which contributed to the procurement of seed with poor germination. Benefiting communities appreciated the distributed vegetable seeds for their nutritional advantage, but not the fodder seed as fodder is generally available from rangeland in the earthquake zone. Farming inputs were generally provided on time. However, villages situated near the main road and secondary roads received more inputs and received them earlier than villages situated along tertiary roads and at high altitudes. The distribution was well coordinated geographically by FAO, the Government and NGOs. The ADB-funded project helped cover the most remote settlement areas. Little targeting of vulnerable households seems to have happened in farming input distributions. Community dynamics tended to impose a blanket distribution (i.e. no targeting) in an attempt to avoid conflicts. There were limited cases where only few well-connected farmers in the village received the assistance. Despite a marked failure on maize, FAOs farming input distributions had a significant impact on food production, enhancing the earthquake-affected populations ability to meet its nutrition and livelihood needs. Yields of distributed wheat and vegetable varieties were higher than those of local varieties and the crops more resistant to diseases. More than 10,000 tonnes of chemical fertilizer were distributed alongside the cereal seed, and beneficiaries often identified fertilizer as the main reason for the success of their crop. Of all the distributed inputs, wheat seed and fertilizer appear to have been the most cost-effective. Livestock inputs FAO distributed material and training for the construction of 7,000 earthquakeresistant animal shelters through the Dosti Development Fundation (DDF) and provided 2,240 tones of feed and 700 tones of urea molasses block to the same households. Under the ADB-funded projects, the material for construction of 2,500 poultry sheds was also distributed to further improve food security. Poultry restocking was in progress at the time of the evaluation. The FAO focus on livestock shelters and feed rather than on restocking of ruminants was most appropriate. They helped reduce the sale of livestock and reportedly increased milk yields during winter. Buffaloes or goats donated by other organizations resulted in some cases in mass sales or slaughter of the donated animals for lack of feed and shelter to keep them. 8

Poor timeliness was identified as one of the main weaknesses in the delivery of livestock inputs. Distribution and construction activities occurred at the very tail end of year 2006 and then year 2007. Dosti and beneficiaries had to work during winter months to construct the shelters. The work was stopped for long periods due to heavy rainfall and snow. Some of the unprotected mud walls collapsed as a result. Delivery of building material during the winter also invited significant risks for the lives of the IPs staff and beneficiaries. Once constructed, the animal sheds were of good quality, earthquake-resistant and warm, to the extent that about 30% of recipients without proper housing temporarily lived in the sheds with their family. The majority of respondents suggested that larger sheds would have been more useful, as households typically keep more than the two large ruminants the shed is designed for. However, making the design bigger could have reduced earthquake resistance. Besides, the target group was small resource-poor farmers and those were expected to keep a small number of animals. The sheds were limited in number and very much sought-after. The intention was to give them in priority to widows and this was indeed implemented. In a few cases however, the shelters appear to have been given to the well-connected. Within each village, it was observed that households near the road tended to benefit more from the shelters than the most isolated households. The same focus on female-headed households was successfully pursued in the poultry restocking programme. Extension, Training and Farmers Field Schools FAO trained some 330 master trainers on improved wheat and maize production techniques, who in turn imparted training to some 2,300 male and 700 female beneficiaries in Muzaffarabad, Balakot, and Mansehra. FAO also laid out 143 demonstration plots throughout the earthquake zone. The ADB-funded project adopted the more sophisticated Farmer Field Schools (FFS) approach. The FFS visited by the evaluation mission were few and recently started and as yet, they did not seem to truly reflect the participatory concept of the approach. The short implementation time of rehabilitation projects may be a hindrance to FFS continuity. Rehabilitation of Community Physical Infrastructures (CPIs) Two rehabilitation projects, the CBLRP and the ADB-funded project, supported the rehabilitation of water control infrastructure as well as other small-scale infrastructure such as bridal paths and gabions for river bank protection. The CPIs were identified in consultation with the targeted communities and the civil work done by Community Organizations, supervised by the Department of Agriculture in the case of the CBLRP and by an NGO in the ADB-funded project. At the time of evaluation, both projects were estimated to be around the 40% achievement mark. The relevance of the irrigation rehabilitation work cannot be overstated. It is fair to say that FAO should have started rehabilitating irrigation channels sooner, during 2006 rather in 2007 as was the case in practice. The lack of attention to irrigation infrastructure during 2006 partly due to FAO and ADB taking a whole year to agree on audit and procurement procedures for the ADB-funded project impacted negatively on the effectiveness of the farming inputs distributed during the relief 9

phase. As late as May 2008, the evaluation mission found continuing demand for the restoration of irrigation infrastructure in nearly all visited villages. Irrigation rehabilitation work was ongoing at evaluation time. The evaluation found that the technical quality of the work suffered from high numeric targets, time constraints and initially insufficient supervision. A number of technical inadequacies are noted in the report, e.g. the construction of permanent structures across landslide areas rather than the recommended temporary crossing structures (pipes, roofing sheets or oil drums, easier to retrieve or to rebuild after a landslide) and excessive lining in places where no damage was done by the earthquake. Lining an entire earth channel goes well beyond rehabilitation and is not without environmental consequences. None of these issues are likely to render the infrastructure unusable, but they could result in decreased cost-effectiveness and durability. The mission witnessed clear changes in cropping patterns along sections of channels already rehabilitated. While direct beneficiaries are likely to be those farming the lower parts of the mountain slopes and valleys, indirect benefits will accrue to more vulnerable households farming the higher slopes and typically generating part of their income through working on irrigated farms further down. Watershed Management Watershed management activities were carried out under the SIDA-funded project and the CBLRP, using participatory approaches and in collaboration with the Forestry Departments of NWFP and AJK. At evaluation time the work was progressing faster for the CBLRP than for the SIDA-funded project, primarily because the former relies on simple approaches including approaches that have shown limits in the past, such as monospecific plantations and technical expertise available in Pakistan while the latter uses a more holistic approach to natural resource management and is dependent on technical assistance from ICIMOD in Nepal. The watershed component of the SIDA-funded project is trying to achieve too much in too short a time and is unlikely to meet its quantitative targets without compromising the quality of its work. Already, pressure to show results tends to skew the intervention towards classic landslip treatment at the expense of improving the management of key natural resources such as pasture, forest or cropland. The evaluation mission observed that the new earthquake-resistant design for homes in the earthquake zone appears to require two to three times more wood for heating than the traditional design. With all the hazards they posed, traditional houses did provide insulation in ways that cement blocks and light metallic roofs cannot match. The region is already deficient in forest resources and the increased demand for firewood will place an additional strain on those. The other environmental issue of importance is that mountain roads being constructed or widened in the region have caused widespread landslips. No attention seems to be paid to the treatment of the cuts and fills during road construction, resulting in accelerated erosion and loss of productive natural resources. Given such threats to natural resources in the area and considering that traditional natural resource management practices have reached their limits, small scale forestry and watershed rehabilitation operations such as the ones implemented by FAO are not likely to make a significant difference on current environmental degradation. Unless 10

large scale programmes are initiated, the natural resource capital will continue to erode, making communities more vulnerable to loss of life, livelihoods, and damage to physical infrastructure.

Community Participation
The evaluation presented an opportunity to document the progress made in insuring community participation in relief and rehabilitation programmes, the potential advantages and pitfalls encountered in doing so, lessons to be learnt and possible ways of making additional progress. The evaluation found that FAO and its partners have consistently attempted to secure some level of community participation in the planning and delivery of the Pakistan earthquake response. However, this interaction was in all cases hurried, characterized by rather brief dialogues, and relying on Community Organizations (COs) with insufficient attention to whether they constitute gate keepers or transparent access and delivery channels. There is no doubt that a certain degree of empowerment has been achieved in practice and that community participation brought value to the programme. Entrusting infrastructure rehabilitation to COs is cost-effective and logical, given that the concerned infrastructure was originally built by communities. On a politically plane, forging strong links with communities has allowed some NGOs to continue working in volatile and unstable areas in spite of a general backlash on NGOs. However, the risk of tokenism is real. The three rehabilitation projects are striving to meet tight deadlines while overcoming administrative and security constraints to delivery. This tension between the requirements of community participation and the realities of early recovery assistance leads to a substantial gulf between the rhetoric of participation and its actual practice.

Coordination and the Cluster Approach

This was one of the first times the Cluster Approach was implemented worldwide. FAO positioned itself at the forefront of a livelihoods coordination architecture, composed of three Livelihoods Working Groups (LWGs) established at the national, provincial (Peshawar) and state (Muzaffarabad) levels. The working groups were initially co-chaired by FAO and ILO, and later on by ERRA, PERRA and SERRA with FAO acting as secretary. In 2007, Livelihood Working Committees were also set up at the district level with support from the SIDA-funded project. Livelihood Working Groups and Committees worked well and were appreciated by central authorities and many participating organizations. By some accounts, these were the best clusters in terms of dynamism and participation, particularly when compared to the much more problematic, conceptually unclear and somewhat competing Early Recovery Cluster. Perhaps one reason why the Livelihoods Working Groups worked well is precisely that they were not formally called clusters, and hence could operate in more flexible ways than a normal cluster. The push by FAO to get livelihoods support and rehabilitation moving early and to set up livelihoods coordination forums at all levels of Government was a major factor in mobilizing funds for the rehabilitation phase and in avoiding a lull in activities after the relief phase.

11

Pakistan was also an early adopter of the Delivery as One initiative supposed to foster greater collaboration between the UN agencies at country level. The process of developing joint programmes has been a slow and painful one, not without similarities with the long design phase for the CBLRP. The Delivery as One initiative appears to focus too narrowly on administrative and programmatic issues at the expense of intangible but potentially more critical dimensions, such as developing team spirit among UN agencies.

Conclusion
Given the logistical, social and political difficulties involved, FAO and its partners in Government and in NGOs should be commended for having put together a rather good relief phase. Damage and needs assessments helped raise donors awareness about the need to reconstruct livelihoods. Strong links were established with the Government and NGOs. Coordination forums facilitated by FAO were widely perceived as open, transparent and reasonably effective in deciding who should do what where. The relevance, quality and impact of the farming and livestock inputs delivered were generally good (with the exception of maize seed distributed in 2006). Maintaining key staff such as the Emergency Coordinator for the entire relief phase and part of the rehabilitation period contributed significantly to the quality of the response. In the rehabilitation phase FAO tried to strike the right balance between quantity, quality and speed; between community mobilization and delivery; and between reconstruction goals and development imperatives. The work with community organizations was found useful though a bit superficial. The rehabilitation projects were racing to finish their work in time, sometimes at the expense of quality. In spite of these difficulties, there is no denying that the rehabilitation work initiated by FAO and partners was very relevant. Insufficient monitoring capacity within the ERCU constituted a significant weakness. Working through IPs being a standard modus operandi for FAO emergency programmes, an independent capacity to extract feedback about the quality of its response is vital to the successful future engagement of FAO in emergencies. Another key finding was that the SIDA-funded project suffers from too complex an institutional set up, which led to delays in implementation. The idea of building the capacity of ERRA, SERRA and PERRA by lending them personnel does not seem to work very well, resulting in unclear reporting lines and responsibilities. The watershed and forestry management activities implemented by the CBLRP and the SIDA-funded projects are commendable in that they address the environmental degradation which amplified the impact of the earthquake. Marked differences of approach were noted between the two projects, the CBLRP being perhaps a bit simplistic while the watershed component of the FAO/SIDA is a bit too innovative for a rehabilitation project. It is important to realize that introducing innovations in a rehabilitation framework implies significant risks, however well-meaning those innovations are. The new earthquake-resistant but energy-hungry houses built across the earthquake-affected zone provide an apt illustration that building back better can involve significant risks.

12

Recommendations
1. FAO should deploy a qualified procurement specialist early on in crowded emergency theatres where the flow of vast aid resources increases the risks of fraud and inflation. 2. In all emergency responses, the cadre of technical staff should include experienced national researchers or extension workers with good familiarity with the area, so as to strengthen varietal choices and other technical decisions. 3. FAO should consider raising the in-country signing authority to at least US$100,000 to improve the timeliness of operations. 4. FAO emergency personnel should be trained to physically check seed, identify storage pests and conduct a rough germination test if a seed quality problem is suspected. Such tests by FAO personnel should not replace the results of the superintendence as a basis for acceptance, penalty application or rejection of the seed lot, but would allow FAO to verify the integrity of the distributed planting material independently from superintendents. 5. FAO and its partners should continue to pursue cost-effective ways of ensuring community participation in early recovery and rehabilitation assistance, while holding realistic expectations as to what level of participation is possible in such contexts, i.e. modest yet significant involvement of communities in priority setting, implementation and evaluation. 6. Review operational arrangements for CPI rehabilitation to avoid delays in implementation. 7. Irrigation technical standards developed in Pakistan should be applied more systematically and consistently in the earthquake rehabilitation programme. 8. Simplify the institutional set up of SIDA-funded project and speed up implementation. 9. Retrain the people implementing the Farmer Field Schools to impart a greater sense of participation in experimentation and learning. 10. FAO and ADB should finalize a global umbrella agreement for all administrative aspects to avoid delays in the implementation of joint relief and rehabilitation programmes in the future. 11. FAO must build a capacity (either internal or contracted out) to monitor IP performance independently, e.g. on how well the IP performed in targeting vulnerable households. 12. The two watershed-related interventions implemented by FAO should coordinate their activities and bring technical standards closer to one another than is currently the case. 13. Technical assistance needs for the FAO/SIDA watershed component should be reassessed and flexibility introduced in terms of reference to address the emerging needs of the project and to build local capacities. 14. The SIDA-funded project should finalize a realistic work plan and clarify responsibilities. In view of the resources and time available, the watershed pilot activities of the project should be confined to eight sub-watersheds.

13

15. A project should be prepared and funds raised for extending the FAO-SIDA integrated watershed management approach. 16. In emergency operations of significant size, FAO should hire a co-ordination specialist to set up livelihoods clusters (or working groups) and carry out effective who does what where (3W) mapping as early as possible. 17. In the Pakistan earthquake response, FAO should update the 3W information and present it in a map form to identified underserved areas now that many NGOs have left. 18. In the absence of a capacity building cluster, FAO and other UN organizations should find alternative ways of coordinating who does what in capacity building and institutional development, so as to avoid overlaps. 19. Start the capacity building component of the SIDA-funded project as soon as possible, in order to improve IPs targeting, distribution processes, participatory planning skills, progress reporting and technical understanding of the issues at hand. 20. In countries with frequent disasters or compounded crises, FAO should build long-term partnerships with local NGOs as implementing partners along the lines of the global agreement between WFP and IFRC for the use of National Red Cross or Red Crescent Societies in food distributions. 21. All concerned actors including FAO should be careful with innovations during rehabilitation phases. For instance, irrigation channels should not be lined (cemented) in their entire length even where no damage occurred, on account of cost and environmental considerations. 22. In both NWFP and AJK, the need for a new environment-friendly livelihoods development policy in mountainous areas was voiced, and FAO might usefully help develop such policies under the SIDA-funded project. 23. Making the new houses of the earthquake zone more fuel-efficient for heating and cooking should be a foremost priority of the GOP and its development partners. 24. FAO should advocate for land stabilization and conservation practices along existing roads and for treatment of cuts and fills during any new road construction or widening project in the area.

14

PART I BACKGROUND
1. Introduction

On the morning of 8 October 2005 at 8:52 a.m. local time, a massive earthquake struck the northern parts of Pakistan, with an epicentre located northeast of Muzaffarabad, capital of Azad Jammu and Kashmir (AJK)1. Recorded at 7.6 on the Richter scale, the earthquake proved the deadliest natural disaster in the countrys history. It flattened entire towns and villages across the Northwest Frontier Province (NWFP) and Kashmir, killing more than 73,000 people and seriously injuring some 70,0002 in a wide swath of devastation. In a matter of seconds, houses, schools, hospitals and offices were reduced to heaps of rubble, burying thousands of inhabitants underneath. Huge landslides barred rivers, creating lakes which brought other hazards. A 10-storey building collapsed in the countrys capital city, Islamabad, and cities as far as Kabul in the west and New Delhi in the east were severely rocked. Overall, it is estimated that around 3.5 million people spread over an area of 20,000 square kilometres were rendered homeless by the earthquake and its subsequent aftershocks. The shallow depth of the October 2005 earthquakes epicentre (estimated at 10 km below ground by the Pakistan Meteorological Department) was a key factor explaining its force and destructive effect. Another factor that made the earthquake so deadly was the generally poorly constructed public sector buildings, such as schools, hospitals, government offices, etc. Eighty percent of educational facilities collapsed in the most affected districts, killing thousands of children and teachers. The earthquake was followed by a long series of aftershocks, each time forcing people to rush out of remaining homes and buildings. These aftershocks caused many damaged houses and buildings to collapse, triggered additional landslides that blocked roads repeatedly, and hampered rescue efforts. Their frequency decreased gradually over the month of October. The destruction was most severe in Balakot in the Mansehra district of North West Frontier Province, which was almost entirely destroyed and where barely 15% of the 40,000 inhabitants survived. A tremendous toll was taken on human lives, houses and buildings in the NWFP districts of Mansehra, Shangla, Battagram, Kohistan and Abbottabad and in the AJK districts of Muzaffarabad, Neelum, Bagh and Poonch. Beyond lives and buildings, the quake also extolled massive losses on assets supporting local livelihoods, notably in the agriculture sector. Harvesting of the maize and rice crops was underway when the earthquake struck, and remaining maize and rice crops either rotted or were eaten by unattended livestock. Food, tools and seed were buried under collapsed buildings. About half of the traditional irrigation and rainwater harvesting infrastructure (earth and stone canals, terraces, retaining walls)

1 Azad Jammu and Kashmir (AJK, literally free Jammu and Kashmir) is one of two regions in the Pakistanicontrolled part of the former princely state of Jammu and Kashmir. The other region, called Northern Areas, was less affected by the earthquake.

Federal Relief Commission 31 march 2006. An additional 59,000 were less seriously injured.

15

was deeply damaged, rendering it inoperative3. Grass cutting from farms and rangelands was underway, but had to be abandoned and part of the harvested grasses were lost for not being properly stacked. Estimated crop losses in the affected areas range from 30 to 75 percent. The earthquake also exerted a heavy toll on livestock, as animals died under collapsed buildings and in landslides. Livestock (buffaloes, cows, goats, sheep and chickens) represent a key asset in the area in terms of food production, earnings and savings. The earthquake magnified the impact of environmental degradation, causing significant land destabilization and damage to the natural capital assets of rural people. The seism resulted in several major landslides and thousands of minor landslips, affecting about 10 percent of hillside arable land, forests and rangelands. Flash floods and mudslides destroyed agricultural land and fruit tree plantations, and altered runoff routes on hillsides and in valley lowlands. Many natural springs feeding irrigation and rural water supply schemes dried up as a result. FAO estimated the total cost of damage and losses in the agricultural and livestock sectors alone at US$409 million (Table 1). The earthquake also had a major impact on the delivery of support services to rural populations, as most Government structures and facilities were totally destroyed.
Table 1: Damages and losses to the agricultural sector
Direct Damages AJK Livestock Crop Irrigation Total NWFP Livestock Crop Irrigation Total AJK & NWFP Livestock Crop Irrigation Total Indirect Total Losses Losses PKR billions 3.6 0.3 0.0 3.9 2.4 0.2 0.0 2.6 6.0 0.5 0.0 6.5 11.9 4.6 0.2 16.7 6.3 1.5 0.1 7.8 18.1 6.1 0.3 24.5 Direct Damages Indirect Total Losses Losses US$ millions 60.0 5.0 65.0 39.5 3.0 42.5 99.5 8.1 107.6 197.7 77.1 4.0 278.8 104.2 24.7 1.3 130.2 301.9 101.8 5.3 409.0

8.3 4.3 0.2 12.8 3.9 1.3 0.1 5.3 12.1 5.6 0.3 18.1

137.7 72.1 4.0 213.8 64.7 21.7 1.3 87.6 202.4 93.8 5.3 301.4

Source: Source: MINFAL/FAO (2005) post earthquake early recovery, rehabilitation and reconstruction programme for the agriculture and livestock sector.

It should be stressed that the earthquake-affected areas are some of the poorest regions of Pakistan. Prior to the earthquake, the income level per person varied from US$150 to US$200 per year, compared with US$480 for the rest of the country. Most farms are owner-operated and small, with an average cultivable land holding of 0.5 ha.

Pakistan Post-Earthquake Early Recovery, Rehabilitation and Reconstruction Programme For The Agriculture and Livestock Sectors - Ministry of Food, Agriculture and Livestock / FAO - 15 November 2005.

16

In the absence of shelter, people were forced to live under the open sky. Tents and blankets were extensively distributed during the months of October, November and December but the incoming winter was expected to aggravate the situation. In Islamabad, the Capital Development Authority (CDA) created a tent vi1lage, where over 10,000 persons affected by the earthquake were provided with accommodation. But for some, leaving was not an option. While the better-off elements of society quickly migrated to lower areas, most earthquake victims living in rural areas tried to remain as near as they could to their land, apparently for fear of losing it (most people with a land deed lost it in the earthquake, others did not have one) or to ensure that any asset buried in the debris of their houses was recovered. Moreover, moving everything that was left to an unknown destination was a daunting and costly task. Hiring a truck cost PKR 8,000, a big expense and only the first of a long series if people were to move to Rawalpindhi or Islamabad for the winter. Livelihoods would have been even more seriously disrupted by out-migration. The catastrophe brought to light the lack of effective disaster preparedness in the country. The absence of an efficient civilian disaster management apparatus was badly felt in the wake of the quake. This realisation led the Government to establish the Federal Relief Commission (FRC) within days of the disaster to coordinate relief efforts. Two other bodies were subsequently created: on 24 October 2005, the Earthquake Reconstruction and Rehabilitation Authority (ERRA) was set up with the mandate to plan, coordinate, monitor and regulate reconstruction and rehabilitation activities in earthquake-affected areas with a particular emphasis on promoting earthquake-resistant building norms; and on 21 December 2005, the National Disaster Management Authority (NDMA) was promulgated with a broader mandate covering all natural disasters in the country. A massive relief effort was mounted. Immediately after the earthquake, the Army provided the core of the relief mechanism, mobilising its entire helicopter fleet. Pakistan Army engineers worked day and night to reopen the blocked routes during the early days of the rescue efforts. Once the main roads were cleared, the Government, national relief agencies, NGOs, private and religious charities and trusts started to move food and non-food items (e.g. medicines, blankets, warm clothing and tents) into the quake-affected areas. People from every corner of Pakistan and overseas volunteered to help or donated money, clothes, food items, medicine, blankets, etc. The Government of Pakistan initiated the provision of rapid financial assistance to earthquake survivors, and later put in place a financial scheme to support the reconstruction of earthquake-resistant homes (US$2,917 per house). After a free for all approach in the beginning, relief agencies were assigned specific union councils (UCs) for implementation of livelihoods and social sector projects including health, education and water sanitation activities, while construction of private and public buildings and other infrastructure was taken up by the Government through the newly constituted Earthquake Relief and Rehabilitation Authority (ERRA). As a result of these massive relief and rehabilitation efforts, the living condition improved in these areas and people started returning to their villages. On the 1st of April 2006 the relief phase was officially declared over and the rehabilitation and reconstruction phase was initiated. By June 2006, 90% of the camp population in NWFP and 70% in AJK had returned to their villages (UNHCR 2006). 17

FAO has responded by providing various types of assistance, for the first time using a full-fledged livelihoods-based approach, from damage and needs assessment to distribution of agricultural assets, to supporting various coordination mechanisms in the livelihoods sector, to capacity building of ERRA. The FAO response is described in section 3.

2.

Purpose of the Evaluation and Methodology

The evaluation was designed to provide a review and validation of the analysis, strategies and interventions being undertaken by FAO, the Government of Pakistan, partners and donors in response to the earthquake emergency; identify corrective measures, lessons learned and opportunities for improvement of ongoing and future programme delivery; provide accountability to all stakeholders; and draw lessons with respect to the implementation of livelihood approaches in emergency contexts. The evaluation was scheduled at the end of the relief phase and at mid-term for rehabilitation projects, so as to derive lessons from the relief phase and contribute to the implementation of the rehabilitation phase. Following a document review conducted at Headquarters, a scoping mission was fielded in October 2007 to discuss the evaluation objectives and scope with the main programme stakeholders in country (FAO, GOP, UNRC and key non-governmental partners). This was immediately followed by a qualitative impact assessment (QIA) conducted by a team of national consultants from November 2007 to February 2008. This impact assessment was timed to prepare the main evaluation mission, conducted in May-June 2008. The qualitative impact assessment, led by Pr Bahadar from Abbottabad University, aimed to gather the perceptions of farmers regarding FAOs assistance during the relief phase4 and to qualitatively evaluate its usefulness and relevance. For this purpose field work was carried out in six selected union councils, where Focus Group Discussions were conducted with men and women beneficiaries and non beneficiaries. Individual discussions were also held with men and women, village elders, local politicians and religious leaders. The focus of the discussions was on the quality of inputs, the processes and timeliness of the interventions and the contribution of the assistance on recovery of faming systems and livelihoods. The team undertaking the main evaluation mission was composed of: John Watt, Engineer and Team leader; Olivier Cosse, Agronomist and FAO Evaluation Officer; Professor Paigham Shah, Agronomist, covering the distributions of farming inputs; Dr Javid Ahmed, Forester and Range Management Specialist, covering watershed management;

Distributions of seeds and fertilizers in spring 2006 (vegetable seed, maize seed, urea and DAP) and autumn 2006 (wheat seed and fertilizer), as well as animal sheds constructed in autumn 2006. The FAO rehabilitation phase was implemented for the most part after the QIA.

18

 Professor Naeem Khalid, Anthropologist, covering social mobilisation, participation and targeting; and Mohammed Riaz, Irrigation Engineer, covering infrastructure rehabilitation. Fielded between 19 May and 14 June 2008, the mission spent seventeen days in Islamabad and Peshawar and ten days in the field, visiting project areas all along the programme zone (Figure 1). Due to security constraints, the team had to split between a group of nationals, covering the area around Bisham (Shangla, Batagram and Kohistan districts in NWFP), and internationals, covering programme areas in AJK.5 The fielding of the main evaluation mission was synchronized with that of the midterm evaluation of OSRO/PAK/701/SWE - "Assistance to ERRA and its partners in restoring livelihoods in the earthquake-affected areas of Pakistan", in fact immediately preceding it, with sufficient time for the two teams to interact with one another in Rome.6 During all stages of the evaluation, consultations with stakeholders, including government authorities (at all levels), donors, programme beneficiaries, field partners (including national and international NGOs) and both international and national staff of FAO was given high importance. The evaluation was designed to focus at the programme level, with issues such as strategic planning and coherence between projects receiving a lot of attention. The fact that the FAO earthquake response is composed of a limited number of projects (nine) implied however that the evaluation teams had to analyse quite a number of issues at the project level, while maintaining a focus on response-wide themes. The evaluation covered classic evaluation criteria such as relevance, effectiveness, efficiency and impact. A substantial effort was made to collect evaluation issues and questions from all stakeholders so as to enrich the scope with questions that would be most relevant and useful to the evaluation audience. This led to the inclusion in the TORs (Annex 1) of a number of questions specifically related to the use of participatory approaches in needs and damage assessments, project design and implementation and also of a few issues dealing especially with the watershed management component. Given the importance of partnerships in the FAO modus operandi in emergencies and the FAO coordination role in such contexts, notably within the so-called cluster approach rolled out in Pakistan immediately after the earthquake, these were included as prominent issues for the evaluation. Finally, the mission also studied the Delivering as One initiative recently piloted in Pakistan, with a focus on FAOs own participation, contribution and stake in this interesting development.

Bisham, an important programme area for FAO, lays 5 hours drive north of Abbottabad. International staff and consultants cannot spend the night there as per UN rules and have thus to go and come back during the same day. A day trip to the area would have been rather unpractical; hence the choice was to send the three national consultants for a three-day trip. 6 OSRO/PAK/701/SWE - Mid-Term Review Mission Report - I. Cherrett, F. Egal, T. Hofer and P. Cecchi, FAO September 2008.

19

Figure 1: Evaluation Itinerary

PESHAWAR

Earthquake-affected area

Areas visited by eval. mission (meetings with beneficiaries)

PESHAWAR

Cities visited by the evaluation mission (meetings with partners)

Villages visited by the qualitative impact assessment team

20

In-country briefing and meetings were held with the FAO team. Others interviewed included key decision makers in the Government of Pakistan, State of AJK and provincial authorities, UN organizations, NGOs, Community Organizations and other Implementing Partners. Prior to departure, debriefing sessions were held with all available stakeholders in Muzzafarabad (5 June) and Peshawar (8 June). In Islamabad, a similarly wide debriefing workshop was planned but had to be cancelled due to a UNDSS security alert on that day. A smaller debriefing meeting was held with the FAOR and the Emergency Coordinator ai. Another debriefing meeting was held at FAO Headquarters on 20th June. The evaluation information collected on OSRO/PAK/701/SWE was also used to assist with the briefing of the mid-term evaluation team in Rome prior to their departure. The Evaluation is not without weaknesses. During field work, it appeared that community members could not always recall how the IPs approached them initially, the quantity, quality and timings of the inputs they received, or were mixing the inputs from FAO with those of other NGOs. Gender also constitutes a major blind spot of the evaluation. Cultural issues compounded access to female respondents and therefore the degree to which gender issues could be addressed. While the QIA team had female researchers, the main evaluation team could not secure the participation of a female community development specialist. Generally in rural Pakistan and most particularly in NWFP, women are not easily accessible. In some villages of Allai valley, even discussions between the female surveyors of the QIA and female members of the community could not be arranged. In the rest of the earthquake-affected area, village elders and activists helped the QIA surveyors organise discussions with females. However, interactions between the whole-male evaluation team and women in the communities were almost inexistent. This is all the more an issue because a sizeable number of men in AJK and mountainous area of NWFP are employed in the off-farm sector, often outside of AJK. Women therefore play an important role in farming.

3.

Overview of the FAO Programme

This section attempts to describe the FAO earthquake response in a factual manner, as it was designed, funded and implemented for the purpose of providing a rapid overview of the programme to be evaluated. Opinions, value judgements and recommendations are left for subsequent sections. 3.1. Damage and Needs Assessments The first requirement for FAO was to assess damages and needs in the agriculture sector. FAO approved a TCP project7 one week after the earthquake, allocating US$400,000 to the fielding of technical assistance for damage and needs assessment, strategy setting, and the provision of emergency assistance before the winter (200 tons of wheat seed distributed by the Department of Agriculture in December 2005 for some 4,000 households in Battagram, Mansehra and Bagh Districts8).
The Technical Cooperation Programme (TCP) draws from the Organizations regular budget to enable FAO to respond rapidly to urgent needs for technical and emergency assistance. 8 Additional seed for the same MINFAL distribution of November 05 was financed by the World Bank.
7

21

The project (TCP/PAK/3007) funded two damage and needs assessments lead by FAO and involving relevant counterparts within the Government of Pakistan (Ministry of Food, Agriculture and Livestock - MINFAL) and the Government of AJK (Department of Agriculture), in close collaboration with the Provincial Governments: 1. From 21 to 31 October, as part of a system-wide damage and needs assessment coordinated by the WB and ADB, two FAO/MINFAL teams assessed the extent of the damage on agricultural assets in affected areas. They estimated the financial and infrastructural damage and losses in the agriculture and livestock sector and the likely cost of the reconstruction effort (US$310 million or PKR 18.5 billion at the times exchange rate).9 2. The Post-Earthquake Rapid Livelihoods Assessment conducted by Peter Reid, FAO Livelihoods Adviser, visited the Muzaffarabad and Bagh districts in AJK from 27 to 31 October 2005. It aimed to provide qualitative information on how the earthquake had affected peoples lives and livelihoods. Building upon previous Participatory Rural Appraisals conducted in 1995, a description of how people made a living prior to the earthquake was provided, emphasising the centrality of livestock and describing three distinct cereal production systems (double cropping, intermediate, single cropping) corresponding roughly to three altitudinal zones (Figure 2 overleaf).10 Figure 2: Altitudinal Cropping Zones in the Northern Pakistan Rabi season (winter) Kharif season (summer)

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Single cropping zone

Fallow Maize * 1800 masl ** (5900 ft)

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Transition zone

Wheat or barley harvested green for fodder

Maize 1550 masl (5000 ft)

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Double cropping zone

Wheat Maize (or rice if good irrigation) 1000 masl (3500 ft)

* Maize takes longer to grow and mature at higher altitudes due to the colder climate, making doublecropping impossible there. ** masl = meter above sea level. Altitudes are approximate, as much depends on sun exposure. The real factor at work is mean temperature.

http://www.adb.org/Documents/Reports/pakistan-damage-needs-assessment.pdf The first livelihoods assessment was followed by another one conducted around June 2006: Pakistan Earthquake Second Rapid Livelihoods Assessment, FAO/Department of Agriculture AJK, undated.
10

22

The preliminary findings from these assessments, together with the Pakistan Post-Earthquake Early Recovery, Rehabilitation and Reconstruction Programme for the Agriculture and Livestock Sectors11, were presented at a workshop in Islamabad on 10 November 2005. The workshop was attended by representatives of the Federal, Provincial and District Governments, donors and NGOs. Later on during the winter 2005-06, the same TCP project funded the development of a Livelihood Rehabilitation Strategy for ERRAs livelihoods sector, a momentous piece of work that would have substantial influence on ERRAs and FAOs response to the earthquake. While the assessments were still being planned, FAO participated in the preparation of the UN Flash Appeal12, and appealed for US$25 million to fund emergency agricultural assistance and early recovery activities. Nine earthquake-related projects were ultimately funded, with a total budget of more than US$22 million, largely from bilateral and multilateral donors (details in Table 2 below). These projects focussed on the provision of seeds, fertilizer, animal sheds and feed, poultry units, training, rehabilitation of small-scale infrastructure (e.g. irrigation canals), and capacity building.

Table 2: FAO programme in response to the Pakistan earthquake

Project Symbol Donor Project Title RELIEF PROJECTS
Emergency assistance to support the rehabilitation of the agricultural sector and poor household livelihoods in earthquake-affected areas Saving livelihoods to save lives - Emergency assistance for the restoration of food security and protection of rural livelihoods amongst OSRO/PAK/605/UK UK earthquakes-affected farmers, women and other vulnerable groups in Pakistan Saving livelihoods to save lives - Emergency European assistance for the restoration of food security and protection of rural livelihoods amongst OSRO/PAK/602/EC Commission earthquakes-affected farmers, women and other vulnerable groups in Pakistan Pakistan: Saving Livelihoods to Save Lives OSRO/PAK/603/CAN Canada Saving livelihoods to save lives - Emergency assistance for the restoration of food security and OSRO/PAK/604/BEL Belgium protection of rural livelihoods amongst earthquakes-affected farmers, women and other vulnerable groups in Pakistan Emergency assistance to support the rehabilitation of the agricultural sector and poor household OSRO/PAK/606/USA USA livelihoods in earthquake-affected areas

Total Budget

Starting Ending Date Date

TCP/PAK/3007

FAO

$399,999

Oct-05

Feb-07

$1,131,492

Feb-06

Mar-06

$2,369,601 $2,179,163

Feb-06 Feb-06

Jan-07 Jul-07 Jun-07

$1,255,678 Mar-06

$500,000

Apr-06

Jun-07

REHABILITATION PROJECTS PAK/06/001/ /01/34 OSRO/PAK/701/SWE OSRO/PAK/702/ASB TOTAL UNDP Sweden ADB
Community based livelihoods recovery programme for earthquake-affected areas of AJK and NWFP Project to assist ERRA and its partners in restoring livelihoods in the earthquake-affected areas of Pakistan Immediate support to poor and vulnerable households in the 2005 earthquake

$3,312,258 $6,491,712

Sep-06 Jan-07

Sep-09 Dec-09 Dec-07

$4,904,999 Mar-07 $22,544,902

11 12

MINFAL / FAO - 15 November 2005. South Asia Earthquake Flash Appeal, UN-OCHA 26 October 2005.

23

3.2. The Relief Programme: Saving Livelihoods to Save Lives Five short-term relief projects were funded by DfID, the European Commission, Canada, Belgium and the USA (OFDA), under the common title of Saving livelihoods to save lives. These relief projects aimed at the rapid resumption of agriculture and livestock rearing in the affected areas of Northern Pakistan. They were implemented as a joint programme from February 2006 to July 2007. The objectives of the relief programme were as follows: Overall objective: Contribute to the protection and recovery of vulnerable livelihoods of the population of AJK and NWFP affected by the earthquake. Specific objective: Ensure the prompt resumption of agricultural and animal production activities for rural, vulnerable households affected by the earthquake thus reducing their dependency on food aid, through (i) the provision of selected agricultural inputs; (ii) reducing or preventing further loss of livestock while optimizing production from stock that has survived; and (iii) assisting ERRA to develop a medium-term livelihood rehabilitation strategy. Main expected results: 1. A Government strategy and action plan are prepared for the rehabilitation of the agricultural sector and poor rural household livelihoods in AJK and NWFP (main output of project TCP/PAK/3007); 2. Some 150,000 affected vulnerable farming households have the essential inputs (seed, fertilizer) to start the 2006 kharif and rabi agricultural seasons (0.2 ha per household); 3. Some 4,500 vulnerable households are maintaining their milk producing livestock thanks to feed and livestock shelters; 4. Coordination of interventions/operations of all actors working in the resumption/rehabilitation of agricultural and animal production is strengthened and technical assistance provided through weekly coordination meetings at provincial and national level; and 5. Technical assistance is provided for the strengthening and coordination of interventions and operations of all actors working in the recovery of livelihoods. It should be noted that some project documents included activities that were finally not implemented: 1. It was envisaged to support land preparation activities but this proved too difficult to organise in a socially equitable way. 2. The inputs were to be delivered partly through voucher schemes and input trade fairs, but large-scale distributions were opted for in all cases, apparently for expediencys sake. 3. The cash for work modality was also envisaged in the ECHO project document, although not budgeted for. 3.3. The Rehabilitation Programme The Pakistan earthquake response was noteworthy in that it managed to mobilize significant resources for a rehabilitation phase, following upon and in some cases overlapping with the immediate relief phase (cf. Figure 3 p.27). This well-funded rehabilitation phase was partly a result of good fortune and seized opportunities, but also a function of FAOs early

24

investments in damage and needs assessments and in strategy setting (development of the Livelihoods Rehabilitation Strategy), which gave the Organization significant profile with the Government and donors. The rehabilitation phase aimed to: (i) build the capacity of the stakeholders to implement ERRAs Livelihood Rehabilitation Strategy; (ii) assist the Government of Pakistan (GOP) to re-establish agricultural support services; and (iii) rehabilitate rural infrastructures to facilitate increased agricultural production and diversification and marketing. However, each project had its own specific objectives. a) Community Based Livelihoods Recovery Programme One of the grabbed opportunities was that the European Community wanted to pilot a programme implemented jointly by various UN agencies, in an early effort to promote a more coherent delivery of assistance by UN organizations in the country. The three-year EC- and UNDP-funded Community Based Livelihoods Recovery Programme (CBLRP) was designed in the beginning of 2006, as part of the ERRA-UN Early Recovery Plan13 to restore livelihoods in the most affected areas. The CBLRP involved UNDP as the organization in charge of overall programme management, financial administration, planning and community mobilisation; FAO as the organization in charge of delivering rehabilitation assistance in the agriculture and forestry sectors to communities identified by UNDP; and ILO and UNIDO in charge of supporting training and income-generating activities to the same communities. In summary, UNDP works on building social capital through community organizations (COs), FAO is restoring the natural capital by working on the revival of the on-farm economy, ILO focuses on human and physical capital by promoting off-farm livelihoods, and UNIDO attempts to develop financial capital by attracting more investment in the project areas. The project focuses on the two most-effected Tehsils14 of Balakot (NWFP) and Muzaffarabad (AJK). The FAO component included the reconstruction of damaged irrigation infrastructure, some gabion structures designed to protect agricultural land from bank erosion, the distribution of seeds (maize, wheat, vegetables, oats), fertilizer and fruit tree saplings to some 10,000 beneficiary households, the establishment of fish farms and fruit plant nurseries, and the reforestation of depleted communal and private forests.15 The ILO component focused on offfarm livelihoods and included so-called bridal paths or link roads (small, steep roads often hard-topped with cement and linking strings of villages to the main road network) as well as skill development (training in a variety of off-farm businesses). UNIDO was attempting to develop financial capital by attracting more investment in the project areas and entrepreneurship training. All these activities were to be implemented through a participatory, community-based approach in partnership with decentralized line departments and NGOs. In each affected community, Community Organizations (COs) are formed or revitalized by the UNDP teams of social mobilizers. The COs open bank accounts through which project disbursements are made. The CBLRP was clearly perceived by all stakeholders as a step between past and future UN collaboration efforts. The CCA and UNDAF are prominently mentioned in the project
ERRA UN Early Recovery Plan. May 2006. The Tehsil is the second-lowest tier of local government in Pakistan; each Tehsil is part of a larger District and is subdivided into a number of Union Councils. 15 Surprisingly, UNDP implemented a parallel forestry component under the same project.
14 13

25

document, and the project is viewed by the donor and concerned UN agencies as a sort of precursor to the Delivering as One initiative. The development of such a joint programme has not been without tensions and some jockeying for budget share (p.30). IFAD and WFP were originally approached and participated in the formulation mission but ultimately opted out due to disagreements over project design. Similar tensions may crop up again under the Delivering as One initiative. b) Immediate Support to Poor and Vulnerable Households in the 2005 Earthquake Funded by the ADB out of the Japan Fund for Poverty Reduction, this project stemmed in part from FAO collaboration with the ADB in assessing needs immediately after the earthquake.16 The project was originally formulated in January 2006, just three months after the earthquake, and was designed to be initiated on 1 March 2006. However, the implementation arrangements and notably the procurement procedures were only agreed upon in March 2007 after long, protracted negotiations between ADB and FAO headquarters. It was finally decided to implement the project through regular FAO procedures. The one-year delay was perceived as very frustrating by the ADB. However, it might have had a silver lining in that it prevented the project from undertaking risky activities such as restocking of goats after an FAO study identified significant feed deficits in five of the eight reviewed districts.17 Early restocking projects implemented by other organizations resulted in frequent sale or consumption of donated animals. The project inception mission in March 2007 opted to cancel goat restocking activities altogether, and focus on animal feed and shelter. More generally, it allowed for a number of activities to be deleted, scaled down or reformulated to take into account changes of the project context. The revised aid packages were reflected by FAO in a letter of interpretation approved by the ADB Inception Mission in March 2007. The ADB-funded project included the following five components: 1. Distribution of wheat, fodder, and vegetable seed and fertilizer to some 25,000 households; 2. Construction of 3,900 animal shelters (one per benefiting household) and provision of animal feed to the same 3,900 households; 3. Poultry restocking targeting 3,900 households (including provision of 20 hens and 5 cockerels to each household, shelter construction material and poultry feed); this component is targeted at women and Female-Headed Households; 4. Community Physical Infrastructures (CPIs): rehabilitation of irrigation watercourses, field terracing, protection walls, check dams, etc, to be identified in consultation with the affected communities; 5. Training farmers in crops, fodder, poultry production, animal health, soil and water conservation, and water management through a Farmer Field School approach.

Here again, a broader set of partners was envisaged originally: WHO and UNESCO were part of early discussions on the project but only the collaboration with FAO materialized. 17 NWFP: Mansehra, Battagram, Shangla, Abbotabad, Kohistan; AJK: Muzzaffarabad, Bagh, Rawlakot. Only Shangla, Kohistan and Muzzaffarabad had a reasonable balance between the demand from the districts herd and flock and the associated feed resource. OSRO/PAK/602/EC - Short-term Recovery, Rehabilitation and Reconstruction Programme; The Livestock Response, FAO / Roger Lough, June 2006.

16

26

The project focuses the above assistance on communities residing at altitudes higher than 1500 meters above seal level, in thirteen Union Councils considered particularly isolated, under the assumption that such areas have received less than their share of relief assistance. c) Project to assist ERRA and its partners in restoring livelihoods in the earthquakeaffected areas of Pakistan As explained above, FAO supported the development of a comprehensive livelihood rehabilitation programme for ERRAs livelihoods sector. This strategy was developed in consultation with ERRA and fine-tuned through an extensive bottom-up process involving national and local government officials, UN agencies, NGOs and other actors. The implementation of the Livelihood Rehabilitation Strategy was later supported through this project by the Swedish International Development Agency (SIDA). The project has a three-fold focus on 1) local livelihoods improvement and diversification through the facilitation of Community Livelihoods Rehabilitation Plans (CLRPs) developed by NGOs at the village level and to be funded by the GOP; 2) institutional capacity building for ERRA, line ministries, district administrations and NGOs; and 3) control of landslides through collaborative watershed management. The project, implemented by ERRA in collaboration with FAO since March 2007, is expected to continue until December 2009.

Figure 3: FAO Pakistan Earthquake Programme Chart

2005 2006
TCP/PAK/3007

2007

2008

2009

Initial planning, strategy development, wheat seed

604/BEL PAK/06/001 (CBLRP) 602/EC 701/SWE

Relief phase: Saving livelihoods to save lives

603/CAN 606/USA 702/ASB

Rehabilitation & reconstruction phase

605/UK

$100,000 $400,000

$2,000,000 Impact Assessment

Each shape represents a project. Shape size is proportional to financial size.

Evaluation Mission

27

PART 2 SYSTEMIC FUNCTIONS

1. Strategic and Programme Planning

Generally speaking, the programme was found well designed, with a few exceptions. The project documents for the five relief projects were written during the completion of the needs assessments and benefited significantly from those. It is important to stress that the same staff and consultants who had worked on the needs assessments were asked to develop the programme. This ensured a good utilisation of needs assessment results in programming. The relief programme was also realistic in its implementing arrangements as well as risks and assumptions. For instance it was envisaged that targeting should not be too narrow so as to avoid local conflicts, a pragmatic consideration in the view of the evaluation team. The risk of unavailability of inputs was also envisaged, but probably underestimated in the case of maize seed as the following sections of this report will show. The livelihoods needs assessment allowed for a rapid but fairly thorough understanding of needs within the general farming systems of the area, for instance in terms of describing the three altitudinal zones for cereal cropping, or in the animal husbandry area where the assessment pleaded for assistance in rebuilding animal shelters and providing feed to protect the surviving animals during the 2005-06 winter. As a consequence, the relief projects (and later the ADB-funded project) included animal feed and shelters rather than donations of animals. In the case of the ADB-funded project, an initial provision for goat restocking was scrapped in favour of feed and shelters during the initial review and reformulation of the project in March 2007, as explained above. Ultimately, the supply of material for animal shelters would only occur towards mid to end 2006, hence too late to have an impact on the survival and production levels of animals during the first winter after the catastrophe (2005-06). However, these goods were still relevant to help stockowners keep their stock during the second winter. One should also commend FAO for refraining to distribute animals and for having tried to explain to other agencies and NGOs in coordination meetings why it would be a mistake to distribute animals. Organizations which did not heed this advice would later come to regret it, as buffaloes and goats distributed in 2006 by ICRC and others were quickly slaughtered or sold by their recipients for lack of caring capacity. Over and above the immediate issue of whether or not earthquake-affected households could keep additional animals, FAO refrained to restock animals because it felt that the mountainous environment struck by the earthquake had long been overgrazed and that the grasslands needed a rest, a rationale supported by the present evaluation. In the area of cereal crops, one could possibly question the relevance of distributing wheat seed and fertilizer immediately after the earthquake in November-December 2005. The affected communities were still in shock, they were receiving food aid, and land preparation was made difficult by the lack of bullocks and tractors. However, this activity of modest if not symbolic size was still useful to ensure a harvest in June 2006 and may have served the purpose of focusing at least some attention from donors, government and communities on the need to get agricultural production back on its feet at the earliest possible time.

28

The interventions implemented later on during 2006 were highly need based. These included maize seed, the dominant crop in the area, vegetable seeds, livestock feed and shelters. It was also thought to distribute poultry units so as to benefit women and widows and improve nutrition. Unfortunately, the intervention had to be dropped momentarily due to a threat of bird flue in the region.18 It must be stressed that, unlike in other countries or emergencies where the need for seed aid distributed by FAO is not always evident, the communities affected by the earthquake in northern Pakistan did lose most if not all of their seed in the areas along the main fault lines, where the damages to houses was above 80%. Distributing seed was therefore most appropriate and relevant in those areas. What is even more commendable is that FAO stopped distributing seed and fertilizer after 2007, for fear of undermining the local markets.19 Perhaps the only design flaw of the relief phase was that it did not include the rehabilitation of damaged irrigation infrastructure, which could have started as early as 2006 and, if implemented, would have improved the impact of the seed and fertilizer assistance. The Emergency Coordinator explained that funding could not be identified for such work in 2006. In this perspective, the delays incurred in finalizing the ADB-funded project take on new significance: the ADB-funded project, if approved in early 2006 as originally envisaged, would have helped reconstruct irrigation channels a full year earlier than was the case. The focus of the ADB-funded project is on communities residing at altitudes higher than 1500 meters above sea level, which was found very useful. Such communities have tended to receive less relief assistance than those residing at lower altitudes and closer to the main road network, as evidenced by the Impact Assessment conducted as part of the present evaluation and corroborated by interviews with communities and Government officials.20 The rehabilitation programme was also found generally relevant, notably the general idea of rebuilding damaged infrastructure through partnerships with NGOs and Community Organizations, with the latter being directly involved in programme implementation and financial management. In effect, the programme used direct cash transfers to communities to fund their own reconstruction efforts. This was an innovative approach which, in the view of the evaluation team, goes in the right direction. The very existence of a rehabilitation phase means that resources were made available over a longer timeframe than just a year or two, for the reconstruction of complex and costly community infrastructure. This is in and by itself a significant development, as in the past the FAO emergency division (TCE) had found it difficult to mobilize resources over time frames longer than a year. The front loading of post-disaster humanitarian programmes, i.e. the fact that the immediate relief assistance tends to be much better funded than longer-term reconstruction efforts, has been found to be a significant problem for communities affected by past disasters. It is encouraging to note that this problem did not seem to arise in the case of
It was later on re-initiated thanks to the ADB-funded project. FAO emergency programmes sometimes rely on easily-implemented seed aid for too long after a disaster, which undermines the private sector and creates dependency. See for instance the Evaluation of the FAO Emergency and Rehabilitation Assistance in the Greater Horn of Africa (2004-2007) Nicholson et al. 2007. 20 See also: Road Less Travelled Analysing the Effects of Remoteness and Need on the Distribution of Earthquake Assistance, RISEPAK, October 2006.
19 18

29

the FAO Pakistan earthquake response, in that there was no reconstruction funding gap. The fact that Pakistan taxpayer money was made available to fund rehabilitation and reconstruction efforts implemented by NGOs and COs within the framework of the ERRA livelihoods rehabilitation strategy should also be seen as a significant breakthrough. However, the three rehabilitation projects are inherently more complex than the six relief projects, and their design could perhaps have benefited from greater simplicity and pragmatism: CBLRP The implementation arrangements used by FAO in the CBLRP imply channelling funds for the repair of irrigation channels via the Department of Agriculture, a provision which intends to promote capacity building of DOA officials and their longterm ownership of this rehabilitation work, but which in practice seems to add a layer of bureaucracy in an already complex system of fund transfers. As per the project document of the CBLRP, FAOs role is purely technical, i.e. solely concerned with the delivery of aid packages and Community Physical Infrastructures (CPIs) agreed upon between the concerned communities and UNDP social mobilisation teams. While this clear-cut definition of roles may seems rational, in practice the lack of involvement of FAO personnel in social mobilisation leads to what the evaluation team called relay problems during implementation: about half of the CPIs identified by the UNDP social workers are later on rejected by the FAO technicians on technical / feasibility grounds. These sorts of problems could perhaps have been avoided if the project design had planned for qualified FAO staff participating in the social mobilisation effort.21 The components and budgets of the various UN agencies were designed in a way that originally lacked transparency and involved some jockeying for budget shares. These issues were ultimately resolved with good will and a lot of to and fro negotiations, but the programme was delayed as a result and a few inconsistencies remained, such as the UNDP involvement in community forestry in parallel with a similar FAO component. SIDA-funded project The main design flaw is in the complex institutional set up: the staff hired by FAO to supplement the capacity of the central ERRA, the AJK SERRA and the NWFP PERRA have unclear roles and lines of authority. Such unclear roles lead to low motivation and team spirit within the project. In short, the idea of re-enforcing ERRAs capacity through staff does not seem to work well in practice. The number of CLRPs that the project intends to facilitate (1,600) is too optimistic. ERRA and FAO later adopted more optimistic numerical targets, such as the number of watersheds to treat (19). The processes for the screening of Community Livelihoods Rehabilitation Plans (CLRPs) and of the PC-1 forms22 for each proposed infrastructure is far too complex, and the proposed flow of funds still not approved by all; hence this component is very slow to start and some originally interested NGOs are currently dropping out.
21

As per the project design, initial social mobilization was the mandate of UNDP. FAO did facilitate some limited community mobilization later on, e.g. in the forestry component or to set up water user associations. 22 Planning Commission Form #1, a sort of simplified project document used for all projects involving treasury funds in Pakistan.

30

The project is not well integrated in the rest of the FAO programme: its management lines are parallel to those of other projects, in that the project CTA reports directly to Rome rather than to the Emergency Coordinator posted in country; the project capacity building component could benefit the entire FAO programme and implementation partners but is currently targeted at IPs of the SIDA project only.

ADB-funded project This was, especially after the March 2007 revision, probably the simplest and best formulated rehabilitation project in the portfolio, which helps explain why it was the most rapidly implemented.23 The only design issue identified by the evaluation was that, although the project focuses very appropriately on communities residing at altitudes higher 1500 meters above sea level, it included the distribution of wheat seed. This was probably not advisable given that these areas correspond to the single cropping and intermediate zones, where wheat is either harvested green as fodder (intermediate zone) or not cultivated at all (single cropping zone). The first draft of the Livelihoods Rehabilitation Strategy was found by some partners to place too much emphasis on agriculture-based livelihoods. This draft was improved upon following extensive discussions at all levels and now includes other sectors as well. Ways and means of integrating non-agricultural livelihoods in recovery efforts is an important issue for FAO, which tends to position itself as the lead UN agency in livelihoods restoration after disasters but lacks expertise beyond natural resource based livelihoods. Keeping a broad view of entire value chains and diversified livelihood means during damage and needs assessments appears critical in this regard, as well as a more systematic involvement of ILO alongside FAO in livelihoods coordination forums, strategy setting and programme implementation.

2.

Logistics, Administration and Human Resource

In the early days of the earthquake response, the emergency team worked from the FAO Representation, i.e. from the same offices as the regular country team. However, there was little space there and they were camped in the conference room. The emergency team moved to a separate office some kilometres away by mid-2006. The new office is less secure than the FAO Representation, which is located within the fenced area of the National Agricultural Research Centre headquarters, but much more spacious. Finances are still managed by the regular country team, but administration, logistics support and other systems are separate. This sometimes leads to inefficiencies. For instance, administrative actions are initiated in one office and approved in another, sometimes with considerable back-and-forth movements. There are two separate e-mail servers which communicate with one another through the central server in Rome, and therefore e-mails with large attachments sent from one office to another in Pakistan can take days to arrive. Local recruitments have helped fill the positions needed to run the emergency programme, but this has been done on the basis of restricted funding. Key positions such at as the heads of the two field offices in Muzzafarabad and Abbottabad were not filled at the time of the evaluation. The position of head of office in Muzzafarabad had in fact never been filled.

23

At least once the procurement procedures were agreed by FAO and ADB, which took a year.

31

Relationships and coordination with local authorities and other UN organizations have suffered as a result. FAO implements most of its programme through local organizations, but had no relationship with these partners prior to the earthquake. The partners do not seem to have been screened or evaluated before entering into agreements. FAO seems to have been quite fortunate with its Implementing Partners insofar as the performance has been generally correct, in spite of the relationship being left rather informal and FAO investing little in terms of capacity building.24 Consequently there is no longer-term relationship and if a similar need were to develop in the future, FAO would need to start the process of finding Implementing Partners again. Recommendations: FAO should build long-term partnerships with local NGOs as implementing partners in countries with frequent disasters. By finding and endorsing implementing partners outside of an emergency, a full review of the partner organization is possible and where necessary capacity building can be carried out to reinforce the ways of working that FAO wishes to use, along the lines of the global agreement between WFP and IFRC for the use of National Red Cross or Red Crescent Societies in food distributions. The capacity of Implementing Partners in the earthquake response should be more systematically supported, e.g. through the capacity building component of the FAOSIDA project, in order to improve IPs targeting, distribution processes, participatory planning skills, progress reporting and technical understanding of the issues at hand. The signing authority for LOAs and payments is still with the FAO representative, who has been very supportive of the emergency operations both in the earthquake zone and later in Balochistan following floods. The problem is that this authority is set at the very low level of US$50,000. Above US$50,000 the authority rests in FAO headquarters in Rome. It was reported that requisitions could take up to 2 months to be answered. Half of the 52 requisitions analysed by the evaluation team were over US$50,000 and had to be authorized in Rome. Procurement in a humanitarian emergency situation is competitive and other organizations which have pre-agreements with suppliers, a much higher signing authority in country and much smoother systems are able to arrange for the delivery of goods in less than half the time that FAO is taking. This allows these other organizations to work faster and better, overshadowing FAO in the fast implementation of programmes. Another factor which played against FAO was the absence of a procurement officer in country. This absence as well as the low level of authority of the field office contributed to some sub-standard goods being procured, as described below. It also contributed to the late delivery, well into the winter, of material for animal shelters and poultry restocking (see p. 46).

24

At evaluation time, the capacity building component of the SIDA-funded project had not started yet.

32

Recommendation: FAO should reinforce its capacity to implement emergency and rehabilitation programmes, though the following means: For emergency operations of significant size25, deploy to the field a qualified procurement specialist early on, in the first few weeks after a disaster; the procurement specialists presence in the field is particularly important in crowded emergency theatres where there are significant aid resources flowing, and hence significant risks of procurement fraud and high price inflation that needs to be monitored; Raise the in-country signing authority to at least US$100,000 to speed up procurement, logistics and financial management, and ultimately improve the timeliness of operations; Countries with frequent disasters or protracted emergencies require FAO/TCE to maintain a minimum presence of capable emergency staff and/or consultants, develop market intelligence and preferred supplier contracts for material used regularly in emergency responses. The failed procurement of maize seed in 2006 There were significant quality problems with 700 metric tonnes of maize seed (variety Sarhad Yellow) procured in 2006. The germination and varietal purity of most lots proved much below standard. The FAO office learnt a lesson and during future input distributions, established demonstration plots throughout the distribution area to facilitate the monitoring and analysis of crop germination and growth in situ. The mission studied the 2006 maize seed procurement and distribution in detail to find the source of the problem. FAO procured maize seed for the Rabi 2006 season through two purchase orders (PO): PO 183302 was handed out on 22 March 2006 to a company called Pakistan Agri Services. The order was for maize seed of the varieties Azam (500 tons) and Sarhad Yellow (500 tons) and was due by 24 April; PO 184447, a repeat order placed on 21 April with the same company at the same price (430 PKR/kg) once it became clear that the available funding would allow for greater quantities than originally thought, ordering 728 tons of Sarhad Yellow seed to be delivered by 20 May 2006. The total amounts ordered were thus of 1278 tons for Sarhad Yellow and 500 tons for Azam. However, the supplier found it difficult to source such quantities and ran into very serious delays, so much so that the contractor had not delivered the entire ordered amount when the planting season came to pass. As a result, FAO rightly cancelled the remaining amounts on 11 June. The amount actually delivered was 700 tons of Sarhad Yellow and 72 tons of Azam.

Suggested for country programmes with an annual budget greater than US$10 million. In 2008, this group included 16 countries with a total budget equivalent to 65% of all resources available to TCE.

25

33

By that time, many NGOs were refusing the goods anyway, because it was getting too late to plant and/or because it came to be known that the seed had failed to germinate in some locations. The germination problems concerned only the variety Sarhad Yellow, for which Pakistan Agri Services subcontracted an army farm and seed plant near Lahore, called Askari Seeds. Progress reports indicate that Askari Seeds scrambled to fulfil the order by buying all kinds of seed from farmers in surrounding areas. Azam was also procured from farmers, but by a different sub-contractor and in a cooler agro-ecological zone in NWFP. An independent examination of the seed, commissioned by the Mansera DOA, found germination rates of 13 to 30% in a majority of lots of Sarhad Yellow, and reasonable germination power (above 90%) in only a few. Furthermore, the description of grain shapes and sizes provided in the Mansera DOA test suggests a wide genetic variability (grains of very variable shapes). The germination power and varietal purity for seed of the Azam variety were fine. Our hypothesis is that the sub-standard Sarhad Yellow lots were either old, dating back from 2004, or poorly stored. Maize seed can easily die if stored for more than a few months in hot and humid conditions. When reports of poor germination started to arrive from NGOs in the field, FAO sent a mission to the earthquake area to investigate. This mission came back with various alternative explanations: low moisture content of seedbed or incorrect depth and method of sowing leading to poor emergence.26 These factors may indeed have taken a small toll on germination, but they characterize the planting conditions in the concerned area. One would expect that the seed distributed by FAO would be able to fit those conditions. It is worth stressing that FAO followed all necessary and regulatory checks, by way of an independent superintendence contracted to a global firm27. The superintendence report from their local subsidiary in Pakistan was reviewed by the evaluation mission and found to be most probably fraudulent. It reports 100% germination rates for Sarhad Yellow, a degree of perfection hardly credible which, in hindsight, should have alerted programme and HQ staff. The lesson is that, although superintendence by private firms is a good principle, it is not foolproof or in this case corruption-proof. Hence the importance for FAO of trying to verify the most basic characteristics (e.g. germination, which for seed is the key variable) by its own staff, in addition to superintendents, so as to be able to spot foul play. No system is fool-proof of course, and there will always be problems. Emergencies are a type of situation where sub-standard goods tend to be shovelled by all sorts of actors: pharmaceutical companies sending their expired medicine to disaster-stricken places to make a "nice" gesture; suppliers selling fake boats in Sri Lanka during the tsunami response, etc. The Pakistan earthquake emergency was no different: many unscrupulous suppliers tried to profit from peoples misery by selling sub-standard goods at a high price.28
Cf mission report from Dr Sartaj, NARC. Netherlands Superintending and Sampling Company Ltd. 28 Another example is provided by Dosti, which rejected sub-standard bamboo and corrugated iron sheets in spite of 1) the supplier offering a hefty bribe and 2) the dire consequences this rejection would have on the timely delivery of livestock shelters in 2006.
27 26

34

Recommendation: Superintendence, while a good principle, is not sufficient to rule out quality problems and the superintendent report often comes late, after the transportation of goods to distribution point. FAO emergency personnel should be trained to physically check seed, identify storage pests and conduct a rough germination test if a seed quality problem is suspected. Such tests by FAO personnel should not replace the results of the superintendence as a basis for acceptance, penalty application or rejection of the seed lot, but would allow FAO to verify the integrity of the distributed planting material independently from superintendents. FAO should also be prepared to prosecute service providers, including superintendents, who falsify documentation.

3.

Monitoring Performance

The role of implementing partners has been central to FAO achieving its relief inputs distribution and food security objectives. This would call for a strong monitoring capability covering the earthquake zone to be able to monitor the actions and performance of these partners. This is clearly not the case: the post-distribution and post-harvest surveys, the only tools available to verify targeting and assess impact, are conducted by the same IPs who implement the programme, and often by the very same staff who conducted the distribution, which places them in a conflict of interest. Some monitoring has taken place by members of the FAO emergency team during field visits but generally the results of these visits and discussions were not seen to be systematically logged and exploited. The restricted staff capacity of the FAO team in the area of monitoring and evaluation limited the influence of monitoring information in the decision making processes at the senior management level. Moreover, in 2006 FAO hired a specialized firm (Kimetrica) to set up a web-based monitoring system but this system failed upon testing and the contract had to be terminated. The FAO office was clearly committed to monitoring, impact assessment and lesson learning, to the best of its ability. FAO conducted workshops on reporting requirements and postdistribution surveys with relevant monitoring and evaluation personnel of implementing partners, and even conducted a survey of IP satisfaction with the services provided by FAO. This information and the lessons derived from it were transparently reflected in the final reports for the projects of the relief phase. However, the robustness of the system depended largely on the honesty and human resource capacities of implementing partners. For instance, the quality and depth of IPs activity reports was found highly variable, some being very good, transparent and informative and others rather shallow, depending on the capacity and interest of the respective NGO staff. Recommendations: Working through IPs being a standard modus operandi for FAO emergency programmes, FAO must build a capacity to monitor IP performance independently, e.g. on how well the IP performed in targeting vulnerable households. This capacity could be composed of 2 or 3 in-house monitors headed by a senior officer, or contracted to an independent organization with no other role in the FAO response.

35

PART 3 SECTORAL INTERVENTION AREAS

1. Farming Inputs
1.1 Description of deliverables to date

FAO delivered key agriculture inputs to a reported 260,000 vulnerable households for restarting agriculture activities in the earthquake-affected areas of NWFP and AJK from 2005 to 2008. Wheat, maize, sorghum, oats and vegetable seeds, fertilizers, fruit saplings, material for animal shelters, animal feed, poultry birds, material for poultry sheds and poultry feed were donated to farmers in the earthquake-affected areas of NWFP and AJK to restore agricultural activity (Table 3 overleaf). In the Autumn of 2005 i.e. immediately after the earthquake, FAO provided 200 metric tonnes (MT) of wheat seeds for distribution by the DOA to 4,000 households, and 140 MT of animal feed benefiting 1,500 households between November 2005 and February 2006 in the same area through various IPs. The largest farm inputs distributions were done in 2006: In kharif 2006, 770 tonnes of maize, 1,432 MT of urea and the same quantity of DiAmmonium Phosphate (DAP) were distributed to 61,600 households. Each beneficiary received 12.5 kg of maize seed and 25 kg each of DAP and urea, a kit designed for 0.2 ha (4 kanals); in addition, 738 MT of urea and the same quantity of DAP were given to 29,120 households without the corresponding maize seed, because the seed supplier did not provide the ordered quantity in time for the planting season (see p.33). In the 2006-07 rabi season, FAO distributed 2,050 tonnes of wheat seed coupled with the same quantities of urea and DAP to 82,000 households in the affected districts of NWFP and AJK. Each beneficiary received 25 kg of certified wheat seed and 25 kg each of DAP and urea, a kit designed for 0.2 ha (4 kanals). In November 2007, another 625 tonnes of wheat seed were distributed to households living in inaccessible areas devastated by the earthquake (ADB-funded project). To promote kitchen gardening and improve household nutrition, 31,000 households were provided with seven different kinds of vegetable seeds including tomato, okra, capsicum, bitter gourd, coriander, bottle gourd and squash for cultivation in the spring season of 2006 (0.05 ha = 500 m2 per household). In addition, 31,600 households received cool season vegetables (peas, turnip, onion and spinach seed for 250 m2) for the 2007-08 growing season. 106 MT of sorghum, oats and barley seed were also provided to 13,000 households in 2007, for an area of 0.1 ha per household. These emergency inputs were distributed free of charge in seven districts, 13 Tehsils (subdistricts) and 93 union councils.

36

Table 3. Agriculture inputs distributed by FAO and IPs in earthquake-affected area of NWFP and AJK
Rabi 2005-06 200 Kharif 2006 770 Rabi 2006-07 2 050 Kharif 2007 76 63 35 8 31 600 625 679 1 200 400 4 000 4 000 Rabi 2007-08 625

Agricultural Inputs Wheat Seed (MT) Maize Seed (MT) Sorghum Seed (MT) Oats Seed (MT) Barley Seed (MT) Vegetable Seed Kits (HH) DAP (MT) Urea (MT) Animal Compound Feed (MT) Urea/Molasses Blocks (MT) Animal Shelter (No) Poultry Restocking (HH) Tree Nurseries (No) Fruit trees Distributed (No) Households (No) Hectares Fertilizer only (HH)

Total 2 875 846 63 35 8 56 590 5 210 5 064 2 240 700 7 000 4 000 22 50 000 260 539 55 859 29 120

200 140

24 990 2 170 2 170

2 050 2 050 900 300 3 000

165 165

22 5 500 1 600 86 590 22 610 29 120 85 000 16 400 14 474 2 108 68 975 13 141

The total number of beneficiaries in the table above (260,539) assumes that each kit was given to one and only one beneficiary, and that no beneficiary received more than one kit. Taking into account the beneficiary selection hiccups described below and the possible double counting of beneficiaries across distributions, the mission estimates that FAO farm input distributions could have reached in the order of 150,000 households, i.e. about 1 million people. This represents slightly over a third of the approximately 400,000 farming households who were affected by the earthquake. Many other actors (e.g. NGOs, ICRC) were distributing seed of their own in 2006 and the total coverage is therefore significantly higher. 1.2 Relevance

The farm inputs given by FAO were found very relevant. The main source of livelihoods in the earthquake zone is a combination of subsistence farming and animal husbandry. Many farmers especially those located at the core of the earthquake zone had lost their seeds in the collapsed houses and had little spare money to purchase seed. Due to the huge demand for relief and rehabilitation of infrastructure (housing, basic health facilities and temporary shelters), the government could pay due attention to the revival of the damaged farming systems. ERRA mainly concentrated on financing construction of houses and repair of the roads, bridges and public buildings. FAOs relief package aimed at restoring the farming systems helped resume farming activities, and in doing so early, attracted attention to the needs of the sector. In most cases, the varieties distributed were appropriate for the farming systems. Clear variety recommendations by the Department of Agriculture exist and have been followed. The main

37

distributed wheat variety was Aqab-2000, suitable for a wide range of irrigated and rain-fed conditions in Pakistan. While wheat seed was found available in suitable quantity and quality, the shortage of good quality maize seed in the country was a severe constraint. Two varieties of maize, Azam (70 tonnes) and Sarhad Yellow (700 tonnes) were distributed in 2006. It is regrettable that not more than 70 MT of Azam could be procured in time for distribution, because Azam is better adapted to the earthquake zone than Sarhad. In 2007, the programme procured and distributed 76 tonnes of maize seed of the Pahari variety, probably the best variety for the area. A small quantity of hybrid maize seeds (Bemisal 202 from Engro Chemicals) was also distributed. Hybrid crop production is too sophisticated for most farmers of the area; it requires more inputs, water and fertilizers than they can afford to produce very high yields. In addition, farmers have to purchase seed from a reliable dealer each year, as the yield of second generation seed of hybrid crops is lower and unreliable. Recommendations: The cadre of technical staff / consultants hired by FAO in any emergency response should include experienced national researchers and/or extension workers with good familiarity with the area, so as to strengthen varietal choices and other technical decisions. Hybrid seed should not be distributed during times of emergency or reconstruction, unless they are already introduced in the concerned farming system and well known to farmers. Oats could be a good substitute for wheat at higher elevations, where some farmers cut wheat as forage. Sorghum produced very high forage yield compared to maize. However, in one union council of Kohistan which had received it, the farmers told evaluators that they prefer to grow maize instead of sorghum on the limited land they own, because there are plenty of grasses available in the mountains. This is perhaps still true in this particular valley but wild pastures are highly overgrazed in other parts of the earthquake zone, notably in AJK, and fodder crops may still be promoted there as a way to try and reduce over-grazing. The distribution of vegetable seeds was found very relevant. Farmers in most affected areas did not grow much vegetable before the earthquake. Interviewed beneficiaries appreciated this relatively low cost intervention for its nutritional benefits. 1.3 Efficiency and timeliness

In most of the areas inputs were provided in time. However, it was found that remoteness had a significant impact on input delivery. Villages situated near the main road and secondary roads received more inputs and comparatively on time compared to the villages situated along the tertiary roads and at high altitudes. The earthquake-affected area is highly mountainous, with small settlements scattered alongside valleys in a difficult terrain. Tertiary roads leading to the higher valleys are only drivable by light four-wheel drive vehicles. Road blocks are frequent both in autumn and spring. The distribution of inputs for both kharif and rabi was therefore difficult in the most remote villages. In some cases, wheat seed was given to farmers in late November and the first week of December, which is too late and leads to lower yields. Farmers plant wheat from the last week 38

of October until the end of November and in rare cases until the first week of December, e.g. when autumn rains are late.29 The first distribution in 2005 was understandably late as it was immediately after the earthquake. But in 2006 and 2007, all the seed should have been distributed by 20th of October at the latest. Data on sowing time from the post-harvest assessment showed that wheat crop sown in October produced on average 2872 kg/ha as compared to 2575 kg/ha for a crop sown in December, indicating a 10% reduction in yields30 in a year with a moderate winter. The loss could have been much worse in a cold year. Generally, maize seed and its accompanying fertilizer were distributed in time for sowing the crop. However, some of the seed lots were delivered and distributed late for kharif 2006 crop and were not planted. Recommendation: Greater efforts should be made to procure and deliver inputs on time, especially seed. In the earthquake zone, wheat seed should be distributed towards the end of September-beginning of October, with no seed distributed after 20th of October. Maize seed should be distributed in April-early May, and no later than 20th of May. 1.4 Effectiveness and quality

The quality of the wheat seed was good. It was all certified seed from the Punjab Seed Corporation, germinated well in farmers fields and produced greater yield than the varieties used before the earthquake. Similarly, the vegetable seeds were generally of good quality and the farmers were quite satisfied. However, there were severe quality issues with 700 MT of maize seed (variety Sarhad Yellow) distributed for kharif 2006, as reported above (p.33). The seed was apparently procured by a sub-contractor from a large number of farmers in Punjab. The mission concluded that they had sometimes bought grain stored for more than a year. In any case, the germination and varietal purity of most lots of Sarhad Yellow proved much below standard. The germination power and varietal purity of the Azam variety were fine, but only 70 tons of Azam could be provided as compared to 500 tons ordered. As a result of this quality issue, FAO and its partners ended up paying for the transport, storage and distribution of an estimated 500MT of dead seed, along with an estimated 270 MT of live seed. A comparatively minor issue is that in most of the area, white maize varieties are grown for human consumption. Sarhad Yellow, as its name indicates, has a yellow grain; Azam has a white grain. The taste and colour of the bread made of Sarhad Yellow were therefore found unpopular. Azam faired better than Sarhad Yellow against this criteria as well. 1.5 Targeting and equity

This issue is here subdivided into the selection and coverage of geographic areas in need of assistance (geographic targeting) on the one hand, and the selection of the most needy beneficiaries within communities (social targeting) on the other hand.
29 30

The largest proportion of wheat, 80 to 90%, is planted under rain-fed conditions. Summary of Kharif 2006 and Rabi 2006-7 post harvest surveys, Tim Vaessen & Mohsin Rose, 2007.

39

Geographic targeting The selection of union councils for inputs distribution was based on WFPs Rapid Food Security Assessment conducted in March 2006, and a subsequent gap analysis with activities of other implementing organizations. The distribution was well coordinated geographically by FAO, the Government and NGOs. As already explained above, some really needy persons have been missed in the most remote areas, i.e. in small villages and isolated homes scattered along the valleys at a range of altitudes. In the initial stages of relief, the inputs tended to be distributed to easily accessible locations along the main roads. Later on, some of the areas difficult to approach were included for input distribution under the ADB-funded project. This was a much welcomed development. As some of the evaluators can personally attest, SRSP and NRSP should be commended for delivering inputs to some of the very difficult areas involved. Social Targeting The specific objective of the FAO intervention was to contribute to the prompt resumption of agricultural and animal production activities with a priority for vulnerable households affected by the earthquake to reduce their dependency on food aid. The approach was to target the most vulnerable households in the earthquake-affected areas, including (a) female-headed households and households where one member has been killed or handicapped by the earthquake; and (b) farmers with small land-holdings. Through agreements a detailed criterion for selection of beneficiaries was given to the IPs (FAO, 2006). The foremost criteria were farmers vulnerability followed by the cultivation possibility (access to land and water) and suitability of seed varieties. Other criteria for beneficiaries included: farmers without seed or with difficulties to access seed and other inputs; farmers stricken by the earthquake and who have lost more than half of their production; widows, disabled, elderly persons or orphan headed families, who are capable to use the package and have no other source of income; farmers willingness to put their own labour and equipment as required for proper cultivation; farmers with land having good soil moisture; farmers with access to four kanals31 of land for wheat production; farmers who are farming in secure areas; From the post-harvest surveys, it appears that targeting was moderately more efficient during rabi 2006-07 as compared to kharif 2006: only a mere 5% of beneficiaries sampled represented vulnerable groups: women, widows, disabled and orphans (compared to 2% among non-beneficiaries) in both cases, but 70% of kharif beneficiaries declared less than 0.5 ha of land, compared to 90% of rabi wheat beneficiaries (compared to 50% of nonbeneficiaries).
31

Four kanals are equivalent to 0.2 ha.

40

The low representation of widows among sampled beneficiaries, in spite of the earthquake having left many women and orphans as household heads, may be due to the fact that in the earthquake-affected areas, women do not have land entitled in their names. Rather, male members of the family (even when very young) manage land on behalf of women, at least officially. It is probable that the number of widows having benefited from the distributions is underestimated. Furthermore, access to women by male enumerators is also challenging in culturally conservative areas. Likewise the WFP/FAO Holistic Food Security Assessment (HFSA) survey was not able to provide an accurate picture of the needs of widows in the area. Generally speaking, community dynamics tended to impose a blanket distribution (i.e. no targeting), in an attempt to avoid conflicts. The same dynamic was apparently at work in IP coordination meetings, since the list of agreed selection criteria was rather long and could supposedly fit a lot of people. In many cases, almost every household in the community received the input. The minimum land holding of four kanals was frequently relaxed or overlooked, with farmers having less land also receiving the inputs. Some local committees distributed it on the basis of land holdings. In most places however, equal kits were distributed to all, at the great displeasure of a few large landowners. Inputs given to landless persons may have ended on the market. IPs adopted various beneficiary selection methods. Perhaps not surprisingly given the magnitude of the task, the beneficiary lists were therefore sometimes made almost in passim, i.e. the IP spent very little time in each village/area and wrote lists of names without proper social consultation. Some among the most needy households were ignored as a result. The majority of IPs asked village leaders, or village committees formed earlier for implementation of various relief projects, to draw a first list of deserving households, and then verified this list independently (some visiting beneficiaries one by one, other calling for broad-based village meetings to review the list). Some IPs did not verify the beneficiary lists independently. There also were some limited cases where only a few well-connected farmers in the village received assistance, typically when the farming inputs were distributed in a point located far from the village, either because most farmers could not organize transport or because only a few were informed of the distribution. It is important to realize that there is no recent history of social mobilisation and community development in the earthquake-affected area of NWFP (more so in AJK). Due to political reasons, NGOs (particularly international ones) did not operate in these areas before the earthquake. Communal organizations do exist in the area but mainly for social and religious purposes. 1.6 Impact and sustainability

Impact was assessed based on three sources, none of which is without defects: post harvest assessment surveys conducted by the NGOs implementing the seed and fertilizer distributions on a random sample of 3% of all beneficiaries, these surveys are useful to quantify impacts, although the result is not always very precise due to the use

41

of the recall method and the fact that IPs are placed in a conflict of interest when evaluating their own work.32 a qualitative impact assessment study (QIA) conducted in the winter 2007-2008 as part of the build up to the evaluation the study used focus group interviews to determine what the main impacts of the emergency farm input distributions were. It provided generally reliable but essentially qualitative information. direct interviews with beneficiaries by evaluation mission members these interviews necessarily limited in numbers provided validation for insights and findings of the other two sources and occasionally yielded original information.

Despite a marked failure on maize, FAOs interventions had a two-fold impact; on the one hand the FAOs interventions provided immediate relief to the earthquake-affected farmers, and on the other hand it demonstrated that farm yields can be significantly enhanced by introducing improvements in the traditional farming systems. The 2005 distribution funded from FAOs own resources was largely symbolic, but important nevertheless. Its effects were limited by the small size of the intervention, the late distribution (November-December) and the lack of land preparation capacity of many farmers. However, the distribution showed that FAO and the Government were committed to restoring livelihoods and not just crying wolf in needs and damage assessments. Production As far as wheat production is concerned, FAOs assistance has enhanced the households ability to meet nutrition and livelihood requirements for the earthquake-affected population. It is estimated that 70 to 80% of the farmers planted the distributed seeds. The remaining did not plant their donated seed for many reasons, the main one being unavailability of bullocks for ploughing. Another reason was not receiving the seed in time for planting, for instance in Dalola. Yields of wheat and vegetable varieties were generally higher compared to the local varieties and the crops more resistant to diseases. The 2006-07 rabi post-harvest assessment indicated a 23% increase in wheat production from 2004 to 2006 for FAO beneficiaries as compared to 15% for non-beneficiaries. The distribution of 770 tonnes of maize seed in 2006 (700 tonnes of Sarhad Yellow and 70 tonnes of Azam) was far less effective than the wheat seed distributions because of poor quality of the seed of one of the maize varieties. The Azam seed germinated well but most of the farmers complained about the poor germination of Sarhad Yellow. The poor quality of the Sarhad Yellow seed damaged the reputation FAO and implementing partners and caused losses to farmers. Fields sown with the non-germinating lots of Sarhad Yellow produced very low yields. It is estimated that half of those who received a non-germinating kit of Sarhad Yellow in 2006 ploughed and replanted their fields using seed of local types from market or fellow farmers. The remaining fields were left because it was too late to plant. However, where the seed was of good quality and the variety appropriate, the intervention was effective. In areas where Azam was distributed, the farmers harvested good crops and kept some of the produce for next years planting. This helped the households in overcoming the food shortage to some extent. It also helped livestock farmers, as more stover was
Some IPs also scaled down their interventions in the EQ zone fairly quickly and could not complete the surveys in their area or collected only part of the information, which affected the accuracy of the comparison between beneficiaries and non-beneficiaries and between pre- and post-earthquake.
32

42

available for feeding their livestock. In 2007, 76 tonnes of maize seed were distributed in the affected area, mainly Pahari, which performed very well. The kharif 2006 post-harvest assessment indicates that there was a reduction in the average maize yield per unit area from 2004 to 2006 amongst both beneficiaries and non-beneficiaries. However, beneficiaries experienced a reduction in yield of only 8% whereas non-beneficiaries experienced a 21% reduction. This reduction in both groups is thought to be due to the lack of labour, machinery, bullocks, irrigation water and communities engagement in house reconstruction. In total, 5,210 tonnes of urea and 5,064 tonnes of DAP were distributed alongside the cereal seed. Beneficiaries often identified fertilizer as the main reason for the success of their crop. For most, it was the first time they had used DAP, and they were encouraged by the results. No post harvest data has been recorded to quantify the effect of fertilizer in the current case, but given that there was no particular quality issue in the case of fertilizer and that fertilizer application tends to increase crop yield, it is reasonable to assume that its distribution may have increased crop productivity, more food for the family and more feed for the livestock. Field interviews confirmed that fertilizer and notably a balanced fertilization of urea and DAP was a key factor in the good yields of beneficiaries fields in rabi 06-07 and 07-08.33 The official district-level crop statistics (MINFAL, 2005 and 2006; DOAE-NWFP, 2007) indicate that wheat and maize production have been restored up to pre-earthquake levels in the affected districts. This being said, an examination of the data sheets shows that the indicators collected (surfaces planted, total production and average yield) often have identical values from one year to the next, which raises questions about the reliability of the data. Impact on Food Security The increased production from the use of good quality seed of improved varieties and fertilizers helped the food security of the farming households. According to the post-harvest assessment, the wheat seed package beneficiaries were able to produce on average 328.6 kg per 0.2 ha/household. This production can support a household of 6 persons for 171 days34. The average maize production induced by a maize kit which germinated well was 259.44 kg per household. Maize is a major crop at higher altitudes. However, wheat forms the major staple food in consumption, providing 50% of the kcal value in the daily diet of the local people.35 Farmers complained about the colour of Sarhad bread and its taste. However, they were also aware that in the absence of an FAO intervention many would have not been able to cultivate their fields and produce any crops and hence would have sold the remaining assets to buy food.

33
34

Farmers in many areas were using only urea before earthquake.

Contrary to the high consumption in wheat, maize only forms 8 to 30% of the consumption pattern according to a study for NWFP by the Social Policy Development Centre in 2000.

35

Based on 3350 kcal/kg of wheat, 2150 kcal/adult person/day as Recommended Dietary Allowance and 50% reliance on wheat in the local diet (FAO/WFP Household Survey, March 2007, in the EQ affected regions).

43

Box 1. Impact of vegetable production food security.

In the village of Sarniyan, UC Machiara, Mohammad Zahir told the qualitative impact assessment team that vegetable seeds were distributed in their village by the NGO NRSP. Most of the people at that point were still busy constructing temporary huts and other relief activities. Farming, including vegetable production, was not given due consideration as people had others important tasks to accomplish for survival. The NRSP staff visited the village and discussed resuming farming activities including vegetable cultivation. Vegetable seeds were distributed in April 2006. Zahir received seeds for onion, spinach, peas and other varieties. Except spinach the rest of the seeds germinated well and the yields were excellent. Zahirs family used to grow onions before the earthquake, but only for domestic use. This time the harvest was higher - 200 kg out of which 80 kg was used at home and the rest 120 kg was sold in the market for PKR 800. This amount was highly useful to buy food items which were scarce at that time. Zahirs family also prepared onion seeds for cultivation in the next year.

Some farmers at altitudes higher than 1600 m above sea level harvested wheat for forage before maturity, a traditional practice in this altitudinal zone (see Figure 2 p. 22), not the direct planned-for impact of human food security, but an impact nevertheless given the importance of livestock products in the diet. The distribution of vegetable seeds was an important component of the FAO earthquake response from a nutritional point of view. In most of the areas seeds provided by FAO were cultivated and a good crop of vegetables was harvested. However, some villages were not interested in getting vegetable seeds even in cases where the IPs made seed available within the village. Due to the shortage of household labour, vegetable farming and other minor farming activities in some areas took secondary importance. Another reason might be that the value of a full package of vegetables is about PKR 40, not a relatively attractive donation. Cost effectiveness Of all the distributed inputs, wheat seed and fertilizer appear to have been the most costeffective. Their distribution helped farmers re-start crop production and supported a gradual return to (relative) self-sufficiency. The data available from 2006-07 rabi season post-harvest impact assessments was used to calculate the benefit accruing to wheat farmers in monetary terms (Annex 4), which is of US$283 per households. Most farmers kept the seed of improved varieties supplied by FAO for planting in the coming season; thus this intervention has and will have additional and more prolonged sustainable effect on food production and livelihood restoration. Evidently, the 2006 maize distribution was not as cost-effective due to poor germination of Sarhad Yellow. Many farmers ploughed Sarhad Yellow under and replanted seed from the market or fellow farmers. It is estimated that half of the fields planted in Sarhad Yellow were replanted; such replanting was late and incurred expenses, which makes the calculation of the operation cost-effectiveness difficult.

44

2.

Livestock Inputs
2.1 Description of deliverables to date

FAO distributed material (galvanized roofing sheets, bamboo, lime, iron wire, rice straw) for the construction of 7,000 earthquake-resistant animal shelters. The shelter was designed by a Swedish architect specialised in cob construction who was volunteering for the NGO Dosti Development Foundation (DDF). FAO, after examining alternative designs, opted for the Dosti model and entered in an agreement with the NGO to provide the material, while the NGO would deliver the material and train master trainers and villagers in the construction techniques through the construction of one model shelter per village. FAO also provided 2,240 tonnes of feed (wheat straw as roughage and cottonseed cake as concentrates) and 700 tonnes of urea molasses block to 7,000 households with livestock from 2005 to 2007. Under the ADB-funded project, the material for the construction of 2,500 poultry sheds was distributed, some in very remote and inaccessible earthquake-affected areas of NWFP. Beneficiaries were trained in poultry shed construction and each was provided with wire mesh, one drinker and one feeder. Poultry restocking was in progress during the main evaluation mission, with packages comprising of 20 ten-week-old pullets and five cockerels being distributed to 4,000 households. 2.2 Relevance

Animal husbandry is an important source of livelihood in the earthquake zone. Almost all households used to keep animals, including buffaloes, cows and goats. Most of the houses and animal sheds collapsed during the earthquake and stored fodder was lost. Rebuilding shelter for human beings took priority and animals were left out in open. Some animals were lost after the earthquake due to lack of shelter during the winter, either because of the cold or because their owner decided they could not keep and feed them and opted to sell or slaughter them. The FAO focus on livestock shelters and feed rather than on restocking of ruminants (buffaloes, cows, goats) was excellent. Other organizations donated buffaloes or goats, resulting in some cases in mass sales or slaughter of the donated animals. 2.3 Efficiency and timeliness

Lack of timeliness in delivery was identified as one of the main weaknesses of the FAO activities supporting livestock. In several successive cases, the goods were delivered to the IP very late, so late that distribution and construction activities had to occur at the very tail-end of 2006 and then 2007, i.e. during the Himalayan winter. This reduced the effectiveness of the intervention. Mud walls can collapse easily if exposed to rain during construction. A traditional house or animal shelter is built in stones and the construction can thus take place over a long period of time because the stones walls can withstand bad weather. In contrast, cob buildings have to be completed and the roof installed as fast as possible, especially during rainy periods. This weakness was revealed when Dosti, because of late provision of building material by FAO, had to work during winter to construct the shelters (winter 2006-07, and then again during the winter 2007-08 for some of the

45

shelters funded out of the ADB-funded project). The construction was stopped for long periods due to heavy rainfall and snow, and some of the unprotected walls collapsed as a result.36 Delivery of building material during the winter also invited significant risks for the lives of the IP staff and beneficiaries.

Figure 4: Makeshift protection of walls against rain during construction

The exact same timeliness problem initially marred the poultry restocking intervention: the material and poultry birds were provided by FAO late, in winter 2007-08. This has led to significant mortality among distributed young birds (from 30% at lower elevations up to 80% in the highest elevations, e.g. in Shangla district). Thankfully, the IP stopped the distribution when hearing of the abnormal mortality. The IP finalized the distribution in June 2008. Recommendation: Greater efforts should be made to procure and deliver inputs on time. In the earthquake-affected areas, for security and efficiencys sake, those activities requiring significant transport, construction or distribution should be completed prior to winter or postponed after it.

This problem was compounded by the process followed for construction. DDF first constructed one demonstration shed per village (usually handed out to a widow) in which all villagers selected as animal shed beneficiaries participated, as a way to teach them the new construction techniques involved. Selected beneficiaries were then asked to build the walls of their own animal shed themselves. Only after they had built their walls were they handed out the FAO-provided assistance (metal sheets, poles, lime, etc.) that make the shelter weather-proof.

36

46

2.4

Effectiveness and quality

The animal sheds were of good quality, earthquake-resistant and insulated. The design was well thought-through. A cob building is one monolithic unit reinforced by straw. It has no weak straight-line mortar joints, thus making it stronger than brick or block. In the Dosti shelter design, this basic characteristic of cob construction is reinforced by a semi-circular floor plan, the use of bamboos and steel wires inside the walls for added structural strength, and light roofing material (bamboos and galvanized iron sheets) to limit damages in case of collapse. Cob walls also provide adequate insulation against cold and heat. The painting of walls with lime keeps the shelters fairly dry and the design even includes metal protections placed above the stone foundations to prevent wall and roof infestation by termites. The majority of the respondents to the qualitative impact assessment suggested that larger animal sheds would have been more useful, as households typically keep more than the two cows (or buffaloes) or five small ruminants the shed is designed for. There was no flexibility to allow for modification in the design, mainly for two reasons: 1) the logistics would have been much more complicated to organize; it was already complicated enough with all these different types of materials involved; and 2) making the design bigger could have reduced earthquake resistance. Besides, the target group was small resource-poor farmers and those were expected to keep a limited number of animals. 2.5 Targeting and equity

Geographical targeting for the animal shelter focused on a few highly-affected and remote UCs. The sheds were limited in number and given to selected households in a large number of villages, some of them very isolated. Within each village however, it was observed that households near the road tended to benefit from the shelters, rather than more isolated households. Dosti reportedly used Participatory Rural Appraisal to determine the criteria for the selection of beneficiaries for their animal shelter. The criteria were as follows: Female-headed households, widows, disabled and orphans with animals but without their shelters. Have existing animals not exceeding one buffalo, two large or four small ruminants. Have lost an animal shelter in 2005 earthquake. Have a monthly income including remittances of not more than PKR 3,000 per month. (Changed to PKR 6,000 as per revised government criteria). Have five or more dependants less than 18 years of age. Have the ability to put in the necessary labour for construction of a livestock shelter. Have the ability to provide land, stones and clay for the construction.

The results of the selection process were reasonably adequate and the focus on widows was indeed implemented. In a few cases however, the shelters appear to have been given to the well-connected. It is important to remember that these shelters were very much appreciated and sought-after (Box 2).

47

Box 2: Beneficiary Selection for Animal Shelters A total of 45 animal sheds were built in Hungrai. The community elders helped DDF field staff finalize the list of beneficiaries. Some of the informants in the beneficiaries and the non-beneficiaries focus groups argued that at the time of construction of the sheds, very few households in Hungrai qualified with the criteria for getting shelter as either their livestock had died or was relocated to plain areas. Some of the respondents complained that sheds were given to the resourceful households. There could be some discrepancies in the distribution of sheds, but in general we found that the distribution was fair. It would have been difficult to develop criteria acceptable for all when only 45 sheds were available for 150 households. Out of 45 sheds, seven were allocated to widows. However, in the Hassa village (UC Garlat), discussions with non-beneficiaries highlighted a beneficiary selection process biased in favour of the well-connected: not a single one of the 10 widows in village got any shed. Source: Report from the qualitative impact assessment

The same focus on female-headed households was pursued in the ADB-funded project with a poultry restocking programme focused on female members of the communities. A majority of the poultry recipients are women, and some, though not all, are indeed widows. 2.6 Impact and sustainability

Although in limited numbers compared to the demand in the area, the FAO-financed shelters and other animal inputs have provided relief to the animals and helped their owners hold on to their remaining assets by preventing distress sales of livestock. Beneficiaries found that the Dosti design compared well to traditional sheds on account of earthquake resistance and insulation (warm in winter, fresh in summer). For these reasons some recipients who did not have proper housing temporarily used the animal sheds for living purposes. The qualitative impact assessment team estimated that 70% of the households use the sheds for animal shelter, the other 30% are used for living, storage and/or other purposes (see Box 3). As a result of the size issue (shelters found small in comparison to local flocks), some beneficiaries used the shed for the first year or two, and then constructed a larger shed as their flock was being rebuilt to pre-earthquake size, using their FAO-supported sheds as a store or a kitchen. Building of additional sheds has a cost and it also occupies additional land which is scarce in mountainous areas. While it is unlikely that farmers in the area will replicate exactly the same design (bamboos are not readily available in the area, for instance), they may emulate some of the techniques used in the sheds. Some interviewed beneficiaries said they were planning to modify the current design of the sheds to make them wider in order to accommodate more animals, for instance. In fact, most of the shelters visited by the evaluation team had been customized one way or another already, either structurally (e.g. addition of a manger37, widening of the roof span) or decoratively (paintings, wood carvings). Besides, a few NGOs not contracted by FAO replicated the shelter design for their own rehabilitation programmes (e.g. Relief International).

The design of the sheds did not include mangers. Some farmers have thus constructed a manger or dug a simple trench so that animal manure does not mix with fodder.

37

48

Box 3. Usefulness of animal sheds A widow named Razia Begum was interviewed in Hangrai to document her perception about FAOs interventions. She has 4 sons and 4 daughters. Her husband died during the earthquake. She also lost her house, three milking cows and one ox which were the critical source of income for the family. She is currently living with her children in a tent which is broken in many places. After two years of continuous use during summer and winter, the tent is torned in a number of places which are covered by cloth. Razia owns 50 kanals of land which before the earthquake produced enough to support the family. She received maize seeds from an NGO but could not sow them due to the unavailability of oxen which are used to prepare land. She still had the maize seed at home stored in a cooking pot and knew that the seeds were treated and poisonous to eat. Razia Bibi was supported for the construction of an animal shed by FAO through DDF. She contributed labour to the construction of the shed. After her husbands death, Razia is responsible for all the farm activities as well as household chores. Due to heavy labour and malnutrition she looked frail and showed cuts and wounds on her hands. However, she was very happy to have the shed for her pair of goats which was given by another NGO. Her goats have a safer and warm place but her family has to wait until she completes her house which is under construction. However, she can use the goat shed for her family in case of an emergency. Source: Report from the qualitative impact assessment

The majority of farmers were happy with the animal feed distributed by FAO. This intervention did help in reducing the sale of livestock and according to IP staff has increased milk yield in the winter months. There were however some complaints about molasses blocks being new to the area and difficult to feed to their animals. The evaluation team could observe some blocks still laying unused in beneficiaries homes. Finally, poultry restocking will most probably improve the nutritional status of the beneficiaries and will generate some income for the female members of the households from the sale of eggs.

3.

Extension, Training and Farmer Field Schools

The Pakistan earthquake response integrated a modest component devoted to training of earthquake-affected farmers, through three basic channels: Rapid extension messages were imparted on beneficiaries during distributions of emergency inputs; Demonstration plots were laid out in 2006-07 to promote modern agriculture technologies; Farmer Field Schools were used for practical training of farmers in vegetable production, crop production and integrated pest management activities, starting in 2008 under the ADB and CBLRP projects in partnership with the Kissan Welfare Association (KWA). Given the modest financial size of these activities, their usefulness is analyzed hereunder without going into a detailed review of the relevance, timeliness, quality, targeting and impact of these activities.

49

During 2006, FAO trained some 330 master trainers in improved wheat and maize production techniques, for them to pass on their skills to local communities through village meetings. Leaflets on improved maize and wheat cultivation techniques have also been produced to support trainers. On-farm training on various topics related to improved production technology for various crops was reportedly imparted to 2,266 male and 699 female beneficiaries in Muzaffarabad, Balakot, and Mansehra. The QIA indicates that in some areas, farmers have produced lady fingers and onion seeds for the next season for the first time thanks to the training on seed preparation received from FAO. FAO laid out 143 demonstration plots of wheat during the rabi 2006-07 season to propagate best integrated crop management techniques at village level and assess the crop performance under varying climatic conditions. The plots were managed by farmers under the supervision of FAO agronomists. Field days were arranged to show the crop of demonstration plots to farmers of the surrounding areas. Programme reports indicate that demonstration plots produced 60% more yield than conventional plots (FAO, 2007a). The data from these demonstration plots indicated an average yield of 2,636 kg/ha as compared 2,077 kg/ha in fields contiguous to the demonstration plots and to an average of 1,493kg/ha in conventional farmers plots. The demonstration plots were apparently located in easily accessible, flat and relatively well exposed land, which would explain why fields contiguous to the demonstration plots registered much higher yields than conventional plots. Even though demonstration plots may have been placed in favourable conditions, they still yielded 27% more than neighbouring fields, illustrating that significant gains in production could be made given sufficient training and extension work. Demonstration plots are of course a rather conventional extension technique, in which farmers are asked to come and witness the advantages of modern farming technologies without being able to see by themselves how these results were obtained. In the ADB-funded project, the approach taken is the more modern Farmer Field Schools (FFS). Conceptually, the FFS are based on the idea that farmers can learn faster and learn more useful, adapted techniques if they experiment themselves on what techniques work best, rather than just look at the results of a demonstration plot or variety trial managed by someone else. The FFS visited by the evaluation mission were few and had recently started. It may therefore be a bit unfair to comment upon them. However, the team would like to signal that farmers in those schools did not seem to be experimenting and learning that much. They appeared more interested in potential financial gains to be derived from the FFS in the form of investment opportunities and training packages than in what the FFS master trainers were telling them. These FFS left an impression of old wine in new bottles. The FFS, predicated on selfexperiment and a non-normative approach to learning, was being used to convey the same old tired extension messages e.g. about line planting of maize, a practice which keeps on being demonstrated but which only proves useful if farmers can mechanize. A technique is never good or bad per se. It has advantages and disadvantages which will play out differently in varied agro-systems. In the FFS philosophy, the role of an extension worker is to experiment with farmers to identify which technique leads to the best results in which environment.

50

Recommendation: Retrain the people implementing the Farmer Field Schools and secure increased technical backstopping by Headquarters to impart a greater sense of participation in experimentation and learning; develop a more practical and well-focussed curriculum and generally ensure that the FFS as implemented in Pakistan remain true to the core logic of the approach. The short remaining time of the projects supporting the FFS may be a hindrance to continuity and potential benefits. There seems to be no sustainability mechanism built into the concept of the FFS after the termination of the FAO assistance. One should note that the agriculture extension department also has a large FFS programme in the province. Recommendation: Closer coordination between the Department for Agriculture Extension and the FSS work by FAO / KWA appears in order to improve sustainability prospects.

4.

Rehabilitation of Community Physical Infrastructures

4.1 Description of deliverables to date

Among the three rehabilitation projects, two include the rehabilitation of CPIs as an objective: the ADB-funded project and the CBLRP.38 The types of infrastructures eligible as well as the modus operandi differ slightly. Link roads and bridal paths are to be delivered by ILO in the CBLRP, but are part of FAOs responsibilities in the ADB-funded project. Apart from that, the two projects have a lot in common: both deal with the rehabilitation of water control infrastructure (channels, check dams, river bank protection walls, etc.), identify the CPIs in consultation with the targeted communities, and implement the civil work through Community Organizations (supervised by the DOA in the CBLRP, and by an NGO in the ADB-funded project). At the time of evaluation, both projects were estimated to be around the 40% achievement mark. Data about the targets and achievements of both projects are provided in Table 4 overleaf. The two projects therefore represent a significant effort to rehabilitate key rural infrastructure damaged by the earthquake, focussing largely, but not only, on irrigation infrastructure. This section will review the relevance, efficiency, quality and impact of the irrigation rehabilitation work only, because the team did not include expertise in road building and river bank protection. Issues related to targeting and community participation in rehabilitation projects are discussed at length in part 4, section 1 on community mobilization and participation (p. 62).

The FAO-SIDA project aims at supporting a participatory planning process which will lead to the identification of CPIs, but these will then be funded out of the Government treasury rather than paid for by the project.

38

51

Table 4: Infrastructural Work and Progress Achieved by FAO in the CBLRP and ADBfunded projects Activity (unit) Targets CBLRP39 Irrigation channels (km) 50 Check dams (No) 150 Water storage tanks (No) 314 Animal drinking water ponds (No) 50 Field terracing (ha) 886 Stone protection walls (cu.ft) 50,000 Fish ponds (No) 32 40 ADB Irrigation channels (No) 41 Link roads and bridal paths (No) 21 Stone protection walls (No) 1 Fish ponds (No) 8 71 Total (No) Achieved/underway 14 10 80 6 43 43,500 6 28 11 0 7
46

4.1

Relevance

The relevance of the irrigation rehabilitation work cannot be overstated. One major effect of the earthquake was the damage caused to water conveyance and harvesting structures including water courses, channels and terraces, resulting in a prolonged incapacity to irrigate the concerned fields, lower yields, and a narrower range of crops cultivated. The cropping of high value crops such as vegetables or rice was greatly reduced. Wheat and maize yields were also negatively affected. The lack of the normal irrigation capacity was reported as one of the reasons for poor yields in 2006 in the qualitative impact assessment commissioned for this evaluation. Based on the results from the post-harvest survey, the final reports for relief projects highlighted a decrease in cereal production in 2006 related to damage caused by the earthquake to irrigation systems. The same reports recommended more investment in the repair and rehabilitation of agricultural infrastructure, a recommendation which was taken up during the rehabilitation phase. However, it is fair to say that FAO should have started rehabilitating irrigation channels sooner, during 2006 rather than towards the end of 2007 as was the case in practice. The evaluation found that this late start of channel repairs activities impacted negatively on the effectiveness of the farming inputs distributed during the relief phase. As late as May 2008, the present evaluation mission found high demand for the restoration of irrigation infrastructure in nearly all visited villages. Mindful of this situation, the District Administration in Abbottabad has recently placed a very high priority on the repair of damaged irrigation channels.

Data as of December 2007. Source: CBLRP Mid-Term Review - Final Report, February 2008. Data as of May 2008. Length and volume data not consistently available for ADB-funded project. Source: monitoring sheets for NRSP and SRSP.
40

39

52

According to FAO staff, the delay in starting irrigation repair activities was due to difficulties in identifying interested donors. The mission further notes that when donors were interested (ADB, EC), the start of their projects was delayed by bureaucratic wrangling on procurement rules between the ADB and FAO or prolonged design problems in the case of the CBLRP. It could also be that the full extent of needs in terms of irrigation repair was overlooked during the needs assessments. Irrigation in the area is usually through open community-built earth channels tapped from natural springs and perennial streams. As already reported, about half of the traditional irrigation and rainwater harvesting infrastructure (earth and stone canals, terraces, retaining walls) had been damaged by the earthquake in both AJK and NWFP. However, the figures from the various damage assessments do not always present a coherent picture. According to MINFAL data, only 6,200 ha were irrigated before the earthquake (8.5 percent of all cultivated land) in the three affected districts of AJK, whereas 71,600 ha (28.92 percent of all cultivated land) were irrigated in the five affected districts of NWFP. However, the value of irrigation mechanisms lost in the earthquake was estimated by the FAO/MINFAL damage assessments at US$ 4.0 million and US$ 1.3 million in AJK and NWFP respectively, i.e. much higher in AJK than in NWFP. MINFAL data is probably biased towards state sponsored, modern irrigation schemes. It seems little if any statistics exist about the traditional schemes built and maintained by communities themselves. Rapid qualitative tools such as focus groups and key informant interviews were the only possible information sources during the damage assessments. Judging from the fact that many irrigation channels were still in need of repair at the time of the evaluation mission, it appears that the areas reliance on irrigation was most probably under-estimated. Initial estimates would therefore need to be re-visited in more detail at the micro-level to come up with a more realistic picture. 4.2 Efficiency and timeliness

Irrigation rehabilitation work is currently in full swing, and the mission observed that it was beset by a number of administrative and logistical deficiencies (applying to a degree or another to all current rehabilitation work): Delays in approval of schemes: Approval of schemes by FAO has sometimes been delayed by a few months, which in the area is sufficient for cost estimates to become obsolete. FAO has only two civil engineers responsible for the approval of cost estimates and instalments for the CPIs of all the three ongoing rehabilitation projects, and they can not meet all requirements in time. Delays in transfer of instalments: communities rehabilitating irrigation channels are paid in three instalments which are frequently delayed, reportedly due to late approval for payment in Rome, implementing partners not submitting required progress reports in time, etc. These delays sometimes results in deteriorating relations between the NGOs and the Community Organizations who implement the work. The role of the DOA in approving progress reports and the release of instalments to COs in the CBLRP contributes to the delays and brings little added value. The rationale for DOA involvement was to build awareness of community-built irrigation channels within line agencies and strengthen links between them and the affected communities. In practice, the technical capacity of line agency personnel is limited and the FAO engineer does all the supervision and control work.

53

Insufficient coordination with local or provincial government structures: Executive District Officers (EDOs) are sometimes involved in projects without the relevant provincial authorities (DG Agriculture Extension & DG Agricultural Research) being informed. MOUs between FAO and the Forestry Department were also reportedly drafted with minimal involvement from the Department. Lack of standard material rates: there are wide differences regarding costs and qualities of material used by various IPs for similar work, reflecting wide market fluctuations in an area flooded by aid money. Non-suitability of vehicles: the procured vehicles are fairly heavy and long land cruisers which cannot access the most remote places due to sharp turns and steep slopes. Lighter vehicles would have been more terrain-friendly.

Recommendations: Operational arrangements for CPI rehabilitation should be reviewed to avoid delays in implementation, including through the hiring of one additional civil engineer and refocusing the role of the DOA on substantive monitoring and technical backstopping rather than approving expenditures. A District Rates Committee may be constituted which should be responsible for fixing and regularly updating standard costs for construction material. 4.3 Effectiveness and quality

The evaluation found that the technical quality of the rehabilitation work suffered from high numeric targets, time constraints and initially insufficient supervision. The mission noticed the zeal with which FAO and its implementing partners are working to achieve the maximum in the remaining three months of the ADB-funded project. To complete the target of 64 infrastructures in three months appeared a challenging job for the IPs. The current rate of implementation could lead to compromising quality for the sake of completing quantitative targets. While aware that rehabilitation work cannot always be of the highest possible technical standard and quality, the mission wishes to note a number of technical inadequacies it observed. None of these are likely to lead to unusable infrastructure, but they are likely to result in decreased cost-effectiveness and durability: Standards developed by the On-Farm Water Management (OFWM) projects not followed: insufficient backfilling; lack of drops on steep gradients (leading to high water velocity and potential scouring); use of permanent structures across landslides instead of the recommended temporary crossings (pipes, roofing sheets or oil drums) which can be easily retrieved by farmers in times of movement; lack of pre-cast turnouts (turnouts are usually cast locally in ad-hoc wood moulds); sharp turns (smooth curves are preferable); absence of topographic surveys sometimes lead to difficulties where there is a need to draw an entirely new route or for watercourses stemming from new water tanks. Lack of drinking water structures in water tanks: water storage tanks built by the programme cater only to irrigation needs; there are no arrangements for drinking water for animals, ablution, washing clothes, etc. Small adjoining drinking tanks for animals, taps for ablution and bathing and floors for washing clothes could be constructed at a minimal cost. The water used for these purposes could be recycled for irrigation.

54

Excessive lining: channels are often lined by IPs in places where no damage was done by the earthquake. Such work goes well beyond rehabilitation and is not without environmental consequences. Lining would normally require the uprooting of trees planted along the channels to avoid any future damages by their roots to the lines sections, and even when this is not done, the lining will result in decreased seepage and hence may cause the withering of those trees. Excessive lining also poses an equity issue in that its high cost will result in less resources being available to repair additional channels in other communities. Lack of training for Water User Associations: training is lacking in high efficiency irrigation techniques, operation and maintenance, book keeping and good warabandi practices (negotiation for equitable water distribution and turns among share holders).

Recommendations: Quality should not be compromised for quantity and speed. OFWM standards developed in Pakistan should be applied more systematically and consistently, notably in terms of drops, curves, backfilling, and use of temporary crossings of landslides. Unnecessary lining where there has been no earthquake damage should be avoided on account of cost and environmental considerations. Consider topographic surveys at critical points prior to implementation, e.g. for new courses, sites and header tanks. 4.4 Impact and sustainability

Since no irrigation channel was completely rehabilitated by the CBLRP and the ADB-funded projects at the time of the evaluation mission, it was a bit early to talk of impact yet. However, the mission witnessed clear changes in cropping patterns along those sections of channels already rehabilitated: terraces with no access to irrigation were either left fallow or planted with rainfed wheat or barley, while irrigated terraces were already being planted with rice or onion. It is therefore evident that this work will have, and is already having in places, a significant impact on restoring secure and profitable farming systems. In the programme area, the cropping of high value crops and fruit trees depends on irrigation. Erratic fluctuations in total rainfall and in its distribution make rainfed agriculture a risky endeavour. The damage and destruction to irrigation channels by cracks and landslides generated by the earthquake have therefore deprived farmers of a stream of food and income. Wheat and maize crops can be severely affected by prolonged dry spells at key times of their development (e.g. flowering). This explains why so many terraces were left fallow until they could be irrigated again: the risk of investing precious resources and work into a land that will bear no crop was deemed too high. Restored irrigation is thus expected to result in improved productivity in food and high value crops among the shareholders of the concerned channels: Increased and more predictable access to food and flow of income from farming due to increased cropping intensity, improved yields and crops of higher value. Appreciation of the value of land with access to water for irrigation back to preearthquake levels.

55

In addition, a range of indirect benefits can be expected41: Increased farm labour opportunities and improved wage rates; Lower food prices and better nutrition throughout the year for net food buyers; Growth in non-farm employment (e.g. in trade of high value crops); More water for non-agricultural uses, including domestic uses that improve health; Increased crop by-products (hay and maize stalks) that would reduce grazing pressure on the environment.

While direct beneficiaries are likely to be those farming the lower parts of the mountain slopes and valleys, where irrigation infrastructure is mostly located, indirect benefits are likely to accrue to more vulnerable households farming the higher slopes and typically generating part of their income through working on farms located further down.

5.

Watershed Management
5.1 Description of deliverables to date

In the FAOs earthquake response, watershed management activities are carried out under two projects: the SIDA funded OSRO/PAK/701/SWE, hereinafter referred to as FAO-SIDA project, is a three-year project (from January 2007 to December 2009), the third component of which aims to stabilize and restore the productivity of the landslips and eroded hillsides through a critical evaluation of all land uses and adoption of a holistic and participatory approach to natural resource management; and the EU-UNDP funded Community Based Livelihoods Recovery Programme for Earthquake-affected Areas of AJK and NWFP, hereinafter referred to as the CBLRP, a 36 months project which includes a small FAO-executed component (output 5 226,380) aiming to mitigate deforestation and ensure a supply of timber and fuel wood in the future; the project also includes another FAO-executed component on the rehabilitation of critical soil and water conservation infrastructure such as check dams, field terraces and protection walls to enhance agribusiness and links with markets (output 6 - 398,643); as well as a support to fruit trees nurseries. For convenience sake, these varied components are analyzed here as one watershed management intervention. Both projects are being implemented in collaboration with the Forestry Departments of NWFP and AJK with the technical assistance of FAO. In the SIDA-FAO project, FAO is providing technical assistance in social mobilization and entered into an agreement with ICIMOD to provide technical assistance for hazard mapping, soil and landslip stabilization through biological and engineering interventions, and intergrated, participatory watershed management planning. The CBLRP also follows participatory approaches and its focus is on social and agro-forestry, soil and water conservation, and Joint Forest Management. Technical assistance is provided by FAO staff, notably by a female forester to help liaise with women in AJK, and a consultant has been hired to develop agribusiness proposals.

41

See for instance: Poverty reduction and irrigated agriculture, IPTRID Issues Paper No.1, FAO January 1999.

56

The FAO-SIDA project proposes to establish two pilot watersheds for learning and training and then implementing the successfully piloted integrated watershed management techniques in 15 more watersheds (17 in total42). At evaluation time the project was in its early stages of implementation due to delays in recruitment.43 Planning and implementation was in various stages of implementation in eight watersheds four in NWFP, and four in AJK. The process of identification and prioritization with local communities and line departments has been slow and so far the plans of even the two pilot watersheds have not been finalized. Some demonstrations of biological and physical interventions for soil and water conservation and landslip stabilization are being implemented nevertheless, in the absence of a finalized plan. In contrast, the CBLRP relies on simple technical assistance available in Pakistan. The work is progressing well and is expected to be completed within the stipulated period. At the time of evaluation, the following activities had been implemented: 50,000 high-yielding certified fruit plants were provided to target beneficiaries in Balakot and Muzzafarabad Tehsils; 22 forest and fruit plant nurseries were established by private entrepreneurs supported by FAO; 200 acres of governmental and communal land were reforested by the Forest Department; 10 check dams and 43 ha of field terracing were rehabilitated. 5.2 Relevance

The mission found that the FAO-SIDA watershed management component was based on a sound concept, that of a holistic approach to natural resource management through active participation of the local communities. The distinction between large, tectonic landslides which cannot be controlled, and much smaller landslips which may be treated by vegetative or physical means to slow degradation, is a sensible and pragmatic one. However, the mission also found that the watershed component of the FAO-SIDA project is trying to achieve too much in too short a time. Testing, learning and implementing complex new approaches in a three year period is a rather ambitious aim. The project is unlikely to meet its quantitative targets without compromising the quality of its work. It basically has two options: reduce the number of watersheds or seek an extension of its timeline and budget. In short, a sound approach suitable for a long-term project is not always effective in a shortterm reconstruction and recovery situation. More efforts should have been spent to try and simplify the approach and tailor the qualitative and quantitative targets to fit the implementation period. Recommendation: In view of the resources and time available, watershed management activities of the FAO-SIDA project should be confined to the eight sub-watersheds where planning and implementation are presently underway. The scope of operations in these watersheds may be increased to compensate for reduction in the number of sites. The links with other components of the same project could also have been made clearer. Currently, the CLRPs developed under output 2 of the same project and the watershed management plans prepared under this component are not being coordinated or linked.
This number of sub-watersheds to be treated is not mentioned in the project document, but results from a decision jointly taken by FAO ERRA at a later stage. 43 A Chief Technical Advisor and two watershed facilitators were hired in 2007 but not retained after their trial period. A new CTA and other facilitators were hired in January 2008.
42

57

The CBLRP is designed in a more modest and straightforward fashion and hence easier to implement. However, one should note that the CBLRP output 5 (forestry) also includes a community forestry component worth 246,793 and implemented by UNDP, so in fact the same project has two distinct forestry components not much coordinated with one another. FAO focuses on guzara forests44 in NWFP and private forests in AJK, and works through memorandum of understandings (MOU) with the Forestry Departments of AJK and NWFP for implementation of project activities (participatory planning, reforestation, stabilization of landslips and soil conservation, etc.), while UNDP focuses on private forests and does not involve the DoF. This rather strange set-up leads to unnecessary duplications of activities. More generally and in spite of the project documents identifying relevant on-going natural resource management initiatives for collaboration, there are no linkages between the watershed management components of both projects, and between them and other similar initiatives in Northern Pakistan. As a result, the FAO-SIDA project does not take into consideration the 40 years of watershed management experiences in the country, as well as very similar work done in a recently completed EU-funded project in NWFP and Punjab. This fragmented approach is perhaps the result of having to implement complex watershed management operations within a narrow timeframe. Recommendation: The CBLRP forestry components should be streamlined to avoid duplication; FAO is better suited for forestry activities and should thus take over the component implemented directly by UNDP. The two watershed-related interventions implemented by FAO should coordinate interventions and exchange information to a greater extent than is the case at the moment. 5.3 Efficiency and timeliness

One reason for concern and delays in the FAO-SIDA project is the lack of clarity in reporting lines as well as the absence of a work plan accepted by all parties. Recommendation: The FAO-SIDA project should thrive to finalize a realistic work plan and clarify responsibilities. The Watershed Management Coordinator in ERRA should focus on coordination with little hands-on management, while the project CTA should manage the project, provided he can spend enough time in the field (remote management will not work). Another reason for implementation delays of the FAO-SIDA watershed component is the remote location of the technical assistance team (ICIMOD is based in Nepal). Assigning a capacity building and quality control role to a remote organization has slowed down implementation. These concerns were flagged to the ICIMOD technical assistance team through a telephone interview. They largely shared our concerns; however, their position was that they were simply following their TORs.

Subsistence forests set aside to meet the domestic needs of local communities, generally owned by them either individually or communally and managed in concert with the Forestry Department.

44

58

The issue of developing a national capacity to support the project from a closer location was also discussed with ICIMOD, e.g. whether or not it would be useful to build the capacity of such entities as the Integrated Land Management Project of AJK Forest Department, the Extension Services Management Academy of AJK, the NWFP Forest Management Centre, etc. in integrated watershed management approaches that ICIMOD is developing, so that these institutions not only become repositories of information but also can give further training if and when needed. ICIMOD fully supports the idea that the capacity of local institutions should be built and a training memory created. Recommendation: Technical assistance needs should be reassessed and flexibility introduced in terms of reference to address the emerging needs of the project and to build local capacities. FAO and ERRA should identify the best local institutions for ICIMOD to build a local training and backstopping capacity for use in the FAO-SIDA project or similar projects in the future. They also explained that the reason why normal sequential steps were not followed in watershed planning is the pressure to show results. The second ICIMOD mission (May 2008) was originally planned to be devoted to watershed planning, but was shifted to design a number of landslide treatments upon the request of ERRA. Such pressures to show results on the ground are understandable, but they tend to skew the intervention towards classic landslide/landslip treatment, hazard mapping and bio-engineering at the expense of improving the management of key natural resources such as pasture, forest or cropland. The activities and plans of the FAO-SIDA watershed component need to pay greater attention to the management of natural resources and the needs of agriculture, livestock rearing and other natural resource based businesses. Field teams should include agriculture, livestock and agri-business professionals with experience in participatory approaches. 5.4 Effectiveness and quality

The guidelines provided by ICIMOD for the FAO-SIDA project and covering interventions on agricultural lands, farm and community forestry, enrichment of forests, stabilization of landslips, soil erosion control, and development of income generating ventures, are very appropriate for regenerating the natural resource capital and developing alternative livelihoods. The design and demonstration of bio-engineering interventions appears technically sound. However, the watershed component is yet to fully take off, and therefore there was no opportunity for validation. Outsourcing technical assistance to ICIMOD was for the purpose of making their experience in watershed management across the Himalayan range available. However, except for the hazard mapping exercise, ICIMOD has so far not contributed anything for which there would be no equivalent experience in Pakistan, putting into question their value added. So far, it appears that the watershed plans developed through participatory approaches are a strong reflection of the community wish lists with little or no facilitation to enable communities to make informed decisions on solutions linked to situation analysis. The

59

watershed plans should also be written in the language that the communities can read rather than in English. The project also provides for the training of community watershed management facilitators, selected staff of line departments, NGOs, private sector and representatives of communities to make sure that watershed management issues are considered and understood in the framework of negotiations for village development plans. So far only one training course has been organized with no representation of teams working on CLRPs, and no attendance by NGOs. The forestry interventions of the CBLRP rely on tried and tested approaches implemented and backstopped by national stakeholders. The afforestation approach and its translation into action through detailed MoUs were reviewed in detail and found technically adequate, and likely to be effective in achieving the expected outputs. Implementation through the Department of Forestry is going well. The quality of plantations done by communities and private forest nurseries that the mission visited was adequate. The technical assistance fills in those gaps where the Department of Forestry lacks capacity e.g. female forester for training of women in nursery techniques, community mobilization and their trainings, and preparation of designs and cost estimates for gabions and other engineering structures for soil and water conservation. 5.5 Impact and sustainability

Some of the results of the CBLRP are already visible. The mission could visit a few afforestation sites. In one plantation area, the communities mentioned their wish to plant a wider array of species (e.g. walnuts) but such species were not available in the DOF nurseries. This illustrates the fact that, in contrast with the FAO-SIDA watershed component which is perhaps a bit too fancy, the work under the CBLRP is perhaps a bit too run-of-the-mill, relying on approaches (such as monospecific plantations) that have shown limits in the past. The project is trying to develop private nurseries and to widen the offer of species they hold. The woman entrepreneur whose nursery we visited had mastered the nursery technique very well and hoped to earn good money through the sale of various types of trees. Recommendation: In the CBLRP, the choice of species for afforestation should be diversified further. It is rather early to conjecture what the outcome of the FAO-SIDA project will be. Sustainability remains a question mark, as there is no clear mechanism for follow up after completion of the project and it may be nave to assume that the COs with whom the project works will have systems in place for management and wise use of natural resources after just three years of implementation. Recommendation: A project should be prepared and funds raised for extending the FAO-SIDA integrated watershed management approach to rehabilitation of environment and wise use of renewable resources in four to six large watersheds.

60

5.6

Environmental impacts of reconstruction

A few key environmental issues emerged during the evaluation and are reported here, in order to draw the attention of decision makers and donors to the need for remedial action. Even though these issues do not strictly fall within the mandate of the evaluation, they severely limit the overall effectiveness of the FAO earthquake response as far as the rehabilitation of a sustainable environment is concerned. Perhaps the most important issue is that the new earthquake-resistant design for homes in the earthquake zone appears to require two to three times more wood for heating than the traditional homes. With all the hazards they posed, the stone-walled and earth-roofed traditional houses did provide insulation in ways that cement blocks and light metallic roofs cannot match. Animals also used to live in the same houses as humans because warmth was at a premium. All this has changed now that the onus is placed earthquake resistance. The Government scheme to pay a given amount of money per house destroyed has also led to over-reporting and the construction of many more houses than were destroyed during the earthquake. Before the catastrophe, extended families would live under the same roof, and hence share heat and meals prepared in common. Now the tendency is to split extended families and build one house per nuclear family, resulting in more wood needed for heating and cooking. The region is already deficient in forest resources and the increased demand for firewood will place an additional strain on those. Recommendation: Making the new houses more fuel-efficient should be a foremost priority in the area, to cut down on the use of firewood for home heating and cooking. The other issue of importance is that mountain roads being constructed or widened in the region (notably in AJK) have caused widespread land slips. No attention seems to be paid to the treatment of the cuts and fills during road construction, resulting in accelerated erosion and loss of productive natural resources. Recommendation: FAO should advocate for land stabilization and conservation practices along existing roads and for treatment of cuts and fills during any new road construction or widening project in the area. Given such threats to natural resources in the area and considering that traditional natural resource management practices have reached their limits, small scale forestry and watershed rehabilitation operations such as the ones implemented by FAO are not likely to make a significant difference on current environmental degradation. Unless large scale programmes are initiated, the natural resource capital will continue to erode, making communities more vulnerable to loss of life, livelihoods, and damage to physical infrastructure. Recommendation: Both in NWFP and AJK, the need for a new environment-friendly livelihoods development policy in mountainous areas was voiced, and FAO might usefully help develop such policies, for instance under the FAO-SIDA project, to use as guidance by AJK and NWFP authorities. 61

PART 4 WORKING WITH PARTNERS

1. Community Mobilization and Participation - Between Rhetoric and Practice
1.1 Context

Genuine community participation is considered highly desirable in the context of development assistance to ensure relevance and sustainability, yet already difficult to attain in such context. Since a consistent commitment to the use of participation and sustainable livelihoods approaches was claimed in the FAO documentation on the response to the 2005 Pakistan earthquake, this evaluation presented an opportunity to document the progress made in insuring community participation in relief and rehabilitation programmes, the potential advantages and pitfalls encountered in doing so, lessons to be learnt and possible ways of making additional progress. The evaluation set out to study the attempts by FAO and its partners to ensure a fair degree of participation during project implementation, primarily by looking at the three rehabilitation projects under implementation during the evaluation mission: the EC/UNDP-funded Community Based Livelihoods Recovery Programme (CBLRP), the ADB-funded project (Immediate Support to Poor and Vulnerable Households in the 2005 Earthquake) and the SIDA/FAO project to assist ERRA and its partners in restoring livelihoods in the earthquake-affected areas of Pakistan.45 This analysis should take into consideration that the geographic and security contexts in which these rehabilitation projects are being implemented is quite varied: The CBLRP focused on the Balakot and Muzafarbad Tehsils, which as a general proposition are quite easy and safe to travel to; The ADB-funded project focused on communities located above 1,500 meters, i.e. difficult to access physically. In NWFP, this has meant working in isolated and socially conservative areas (notably in terms of gender roles) that were at the time of the evaluation used as a base by the Taliban (districts of Kohistan, Shangla, Batagram); The SIDA/FAO project works to support the drafting and implementation of participatory rehabilitation plans in all nine earthquake-affected districts. 1.2 Roles in Community Participation

The role of FAO in both the relief and rehabilitation phases has been chiefly of a technical and managerial nature. The function of ensuring community mobilization and/or participation has typically been devolved to international and national NGOs. The only case in the FAO earthquake response where a government agency was responsible for interactions with communities was during the first months after the earthquake when the DOA distributed some of the farming inputs of the emergency phase. Notwithstanding this particular instance where
Within emergency projects, community participation was largely limited to rapid interactions with communities in order to ensure that the farm inputs provided were targeted adequately. Please see the section on the targeting of farming inputs p.39.
45

62

speed in implementation was a key factor in the choice of the DOA as an implementing partner, in general FAO and its donors tended to consider that NGOs are better able to interact with communities than governmental authorities. This is appropriate and reflects the comparative advantage of NGOs vis--vis line agencies and UN organizations: NGOs are typically closer to the grassroots, they often pay more attention to communities priorities than line agencies which in Pakistan tend to be quite top-down, and tend to be more attuned to social issues than a technical organization like FAO.46 Besides, they are not subject to the rigid security regulations that severely constrain UN staff capacity to travel in the area, and therefore are more able to interact with communities in remote and insecure areas, such as those covered by the ADB-funded project in NWFP. In the ADB-funded project, FAO has contracted the Sarhad Rural Support Programme, the National Rural Support Programme, the Dosti Development Foundation and the Kissan Welfare Association47 to assess needs and provide services to grassroot communities. The same role of social intermediation is assigned to NGOs in the SIDA/FAO project, a project whose entire logic (support the capacity of NGOs to prepare Community Livelihood Rehabilitation Plans in consultation with communities and help ERRA review and fund these CLRPs) is based upon the consideration that NGOs have a distinct comparative advantage in community participation. In the CBLRP, UNDP has retained the community mobilization and social capital building role, but this is only possible because the two selected Tehsils of Balakot and Muzafarabad are easily accessible and secure. Whether it was an NGO, a UN agency or a line agency which was responsible for interactions with communities, the evaluation found that this interaction was in all cases hurried if not hasty, and in many cases rather shallow. 1.3 The CBLRP Experience

The CBLRP is due to complete its term in mid-2009. Partners are under heavy stress to meet their targets. The UNDP community mobilization teams are responsible for forming or revitalizing Community Organizations (COs)48 and for listing their priority needs in capacity building and small scale infrastructures. FAO is responsible for the supervision of physical infrastructure work as they relate to agricultural productivity. Each CO is responsible for physically constructing the infrastructure, hiring masons, procuring material, etc. and has to open a bank account to receive transfers from the project and pay for local expenses. COs have also been responsible for selecting beneficiaries of the FAO-procured maize and fertilizer distributions in spring 2007. During this evaluations field work, the work of the UNDP social mobilization teams was well under way, at least from a quantitative view point. In NWFP, the CBLRP target was to
46 While FAO has developed expertise in participatory approaches (e.g. in the sustainable livelihoods approach used in the Pakistan earthquake response), the point here is that NGOs have a comparative advantage over governments and UN agencies in the day-to-day implementation of participatory approaches in the field. 47 SRSP is responsible for farming inputs and CPI delivery in NWFP while NRSP plays a similar role in AJK; the Dosti Development Foundation provided the animal shelters; Kissan Welfare Association is helping set up Farmer Field Schools. 48 Many COs had already been formed prior to the earthquake in partnerships with national NGOs such as Sungi, NRSP or Islamic Relief. Pre-existing COs lost many of their members and leaderships in the disaster, and the CBLRP therefore attempts to revitalize these with nominal effort.

63

form or re-vitalize 522 COs (including opening a bank account) and establish their priorities through 750 dialogues49. The achievement by 15 April 2008 was 478 COs and 644 dialogues, i.e. about 90% of the numeric target. Approximately two-thirds of the COs have had one dialogue only with UNDP community mobilization staff, and about one-third of them have had two dialogues. Therefore, a substantial proportion of COs receives physical infrastructures and/or other inputs based upon just one dialogue or session. The depth and breadth of such fast-paced community participation appears minimal. Judging from discussions with project staff and the review of the project community mobilization guidelines50, these dialogues last a couple of hours. They are in practice little more than a registration process, whereby a community listens to a presentation of the project, enlists members to form a Community Organization, and nominates a few priority interventions. It could hardly be otherwise given the large quantitative targets the community workers need to achieve. Discussions with FAO field staff revealed their awareness and concern about such a shallow base for decision making. There are quite a few risks involved: the risk of benefiting only a small section of the local community (not all community members are automatically a member of the CO the project works with); of selecting deliverables based on what the most vocal members in the assistance want; of elite capture for some project benefits; etc. The CBLRP Mid-Term Review report indicates for instance that the most vulnerable were often excluded from the farm inputs distributions, and did not benefit from ILO off-farm training opportunities.51 Faced with a large number of COs to mobilize during a short period, the CBLRP has tended to rely heavily upon clusters of community organizations. These are federations of COs belonging to the same union council; a typical cluster may regroup from ten to twenty COs. Such reliance on CO clusters can alternatively been seen as a de facto cost transfer exercise where the project relies on non-paid structures to do its own work, i.e. reach out to a large number of COs, or as a convenient way to easily scale up community participation during pressing times. The issue is that relatively weaker and fragile COs located in more remote parts of the union councils may be under-represented in their cluster. Relying almost exclusively on clusters may be contributing towards re-entrenchment of pre-existing inequitable power structures, further marginalizing the most remote and less well connected communities, and favouring communities located on the valley floors and/or near the road. This being said, there should be ways of ensuring that remote villages are not ignored by clusters, e.g. through spot checking in some UCs. The federation of Community Organizations into clusters may be irreversible and even desirable for the empowerment of local communities, as such clusters wield a greater political influence on local administrators and service providers than mere COs.
Dialogue is a term in common use in Pakistan, originally coined by the Agha Khan Rural Support Programme (AKRSP) and meaning a participatory discussion and consultation which, depending upon the social mobilizers judgment, may range between being a brief introduction of the project to an elaborate profiling of community needs and priorities. Many personnel working in the CBLRP have at some stage of their careers worked with AKRSP or another Rural Support Programme, hence the term is a common commodity to all personnel in the project. 50 I.e. A short How to and community dialogue procedure document provided to the mission. No narrative report or progress analysis of community mobilization and development was present in the documentation. 51 See CBLRP Mid-Term Review Final Report, UNDP February 2008, and CBLRP Evaluation Report - Four Off-farm Training Programmes, UNDP/Alamgir Khan Gandapur, May 2008.
49

64

Another constraining factor for community participation in the CBLRP is the limited range of eligible infrastructures, a range constantly narrowing as the project progresses. The project targets for CPIs under the FAO component are as follows: 150 check dams for water harvesting; 50 km of water channel development; 314 water storage tanks; 50 animal drinking ponds; 866 ha of field terraces rehabilitated; and 50,000 cubic feet of stone protection walls.52 Such fixed numerical targets imply that the COs visited by community mobilizers early in the project life have a broader range of CPIs available to them than those visited late. For instance, all the 50 animal drinking ponds might have been allotted already and hence this type of infrastructure is no longer available. Finally, it was mentioned by UNDP and FAO teams that approximately 50 percent of the projects identified by UNDP social mobilizers and communities are subsequently rejected by FAO engineers on technical feasibility ground or because the proposed infrastructure is not agriculture-related. Given that the dialogue phase with the community is the sole responsibility of UNDP and does not involve FAO at all, such losses are understandable, yet frustrating for communities. 1.4 The Case of the ADB-funded Project

The ADB-funded project attempts to work in the most remote areas of NWFP and AJK, the main criteria being that the project areas must be located higher than 1500 meters above sea level. The two NGOs contracted for CPI planning and delivery (SRSP in NWFP and NRSP in AJK) conducted initial and (judging from the documentation) rather brief participatory rural appraisals of their respective project areas in order to document the baseline situation in terms of human and natural resources. Since then, community participation has been confined to short dialogues with communities and COs involvement in CPIs delivery. No social or natural resource mapping has been attempted with the COs as almost all of them are at an embryonic stage.
Figure 5: Remoteness of Community Organizations
formed by the SRSP in North-West Frontier Province (ADB-funded project)
11-15 km (21%) 16-20 km (31%)

6-10 km (4%)

21-25 km (8%) 26-30 km (4%)

0-5 km (21%)

31-35 km (8%) More than 35 km (2%)

Distance from Karakuram Highway (km) 0-5 6-10 11-15 16-20 21-25 26-30 31-35 35 plus Total

No. of Community Organizations 10 2 10 15 4 2 4 1 48

These targets, originally more ambitious, were revised and reduced through the Mid-Term Review of the project.

52

65

The evaluation mission travelled to communities benefiting from the project in NWFP and AJK, and can bear witness to the seriousness of the effort made to access remote communities. The chart and table above (Figure 5) illustrate that three quarters of the 48 COs formed by the SRSP in NWFP are more than 10 km away from the Karakorum highway (the main tarmac road in the area), and 22% are more than 20 km away. Such strong focus on remote areas triggered a number of difficulties, especially in NWFP: The physical terrains in both regions and the highly fluctuating nature of security in NWFP in particular cannot be underestimated. Operations have suffered in areas such as Shangla and Batera, where the Taliban and the Pakistani military are involved in violent conflict53, making implementation, supervision and monitoring of progress achieved very difficult; The highly challenging environment also results in a high turnover rate for already young and inexperienced personnel; The fact that these areas (especially Shangla and Batera in NWFP) had seen very little development work before the earthquake means that no community organizations were formed there until SRSP started community mobilization for the project a year ago; Isolated, culturally conservative and conflict-ridden communities have not permitted the relatively inexperienced and all male staff of SRSP and DDF to address in any meaningful ways the depth and breadth of community participation, nor allowed access to female members of their communities.

A positive aspect was that the absence of pre-existing CO clusters meant that SRSP and DDF have had to work with each and every CO even if they were remotely located. In spite of the difficulties, the evaluation team feels that the effort was worth it in that these communities were the least-served ones among earthquake-affected communities, and FAO was apparently the first UN organization to support them. 1.5 The Case of the SIDA-funded Project

As explained above, this project embodies and supports the implementation of the ERRA Livelihoods Rehabilitation Strategy elaborated by FAO and the Government of Pakistan, the only ERRA strategy which is based on a process of participatory community planning facilitated by NGOs. The projects livelihoods component aims to support this participatory planning process through capacity building for NGOs preparing Community Livelihood Rehabilitation Plans and helps ERRA review and fund the implementation of these CLRPs by NGOs and COs. The project therefore promotes the channelling through NGOs and COs of a portion of the Pakistan Treasury money for post-earthquake reconstruction, a very innovative idea in Pakistan for state-funded projects. The SIDA/FAO project is therefore a good test case for the role of participation in reconstruction. The CLRPs reviewed were generally of good quality and probably represent the most genuine example of community participation in the entire FAO portfolio. Recurrent community
53

Taliban have sized control of Shangla and Swat following the Red Mosque event in July 2007.

66

priorities expressed in village meetings attended by the evaluation team, such as the rehabilitation or development of new link roads and irrigation channels, feature prominently in CLRPs. So far, the project theory that NGOs are able to develop good quality participatory plans in the earthquake zone appears validated However, it remains to be seen whether FAO, ERRA and their partners will be in a position to answer to these priorities. The rate of progress of the livelihoods component has been rather slow so far due to many factors: difficulties in identifying a Chief Technical Advisor that would be acceptable to all parties; a complex institutional set up where FAO staff are posted within ERRA, SERRA and PERRA under unclear terms of reference; a poor working relationship between ERRA, SERRA and PERRA; and the sheer novelty of the project modus operandi. The processes for screening the Community Livelihoods Rehabilitation Plans and the subsequent PC-1 documents54 is quite complex and long, and the flow of funds from the treasury to the COs was not yet approved and/or understood by all stakeholders at the time of the evaluation mission. During the evaluation mission, accountants from SERRA and DCUs in AJK raised a long list of issues and questions about the procedure for flow of funds proposed by the ERRA Economic Sector Director.55 As a result, some of the NGOs which were very interested in this process and applied for a number of Union Councils are now dropping out of the process and have stopped preparing CLRPs. It is crucial to find a renewed momentum. The project also includes a watershed component in which communities are supposed to be consulted in the drafting of watershed management plans in 19 watersheds. While this work was just starting at the time of the evaluation, the mission observed that in one case, a basic, run of the mill afforestation approach had been followed whereas community members were asking for a broader range of species planted. The same community insisted that their main priority in terms of physical watershed management infrastructure (protecting a series of houses from a landslide) had been discarded by project technicians on the ground that it was not feasible or too costly. Without judging the merit of either side in this particular case, it is important to underline that community participation is a dialogue between community members and technicians in which everyone has his own perspective and distinct contribution to make. Not everything a community wants is technically and financially feasible. 1.6. Conclusions Overall, the three rehabilitation projects are striving to meet tight deadlines and often inflexible targets, while overcoming administrative and security constraints to delivery. In such a context, the low levels of participation from communities are hardly surprising. This tension between the requirements of community participation and the realities of early recovery assistance leads to a substantial gulf between the rhetoric of participation and its actual practice, often characterized by rather brief dialogues, identifying projects from a narrow range of options, and reliance on COs and clusters with insufficient attention to whether they constitute gate keepers or transparent access and delivery channels.
PC-1 = Planning Commission form 1, a standard format for planning micro-project such as a school or an irrigation channel in Pakistan. 55 Some of these questions were quite fundamental, such as who will certify the work done to release the next tranche of payments.
54

67

In spite of these limitations, community participation does bring value to the programme: the CLRPs do seem to portray the priorities of communities (e.g. access roads, irrigation); and CO role in infrastructure rehabilitation is cost-effective and logical given that the concerned infrastructure was originally built by communities. On a politically plane, working in tight coordination with community representatives and COs has allowed some NGOs (e.g. Sungi or the RSPs) to continue working in the earthquake-affected area in spite of the recent and sometimes violent backlash against NGOs. In volatile, unstable areas, forging strong links with communities becomes a security imperative. The fact that FAO and its donors and partners have formally and consistently attempted to secure some level of community participation in the planning and delivery of the Pakistan earthquake response should be commended, as well as the direct role given to COs in CPI delivery in all FAO rehabilitation projects. There is no doubt that a certain degree of empowerment has been achieved in practice. However, the risk of tokenism is real and the substantial gulf between rhetoric and reality needs to be addressed. Recommendations: Community participation is crucial to improve on relevance and cost-effectiveness, but also to strengthen the security of teams operating in politically volatile areas. FAO and partners should hold and communicate realistic expectations as to what level of community participation is possible in early recovery and rehabilitation assistance, i.e. modest yet significant involvement of communities in planning, implementation and, an often forgotten step, evaluation e.g. through regular accountability forums held at the UC level, as pioneered by Sungi in Northern Pakistan. Continue to work with NGOs to facilitate delivery and community participation, but keep in mind that line agencies and local authorities have their own comparative advantages: while their administrative capacity is often weak, they can usefully participate in the screening of plans and projects (as in the SIDA/FAO project); their technical expertise is often irreplaceable during project implementation; and their involvement in monitoring the work of NGOs and COs on a technical plane is desirable. Clusters of COs need to be monitored or spot-checked regularly to ensure broad-based participation and representation, including that of remote villages. While it is useful to follow a pre-set menu of eligible Community Physical Infrastructure during early recovery and rehabilitation phases, the precise numerical targets for each CPI type (numbers of irrigation channels, ponds, etc.) must allow for ample flexibility not to pre-empt communities choice. ERRA should clarify the procedure for flow of Government funds to CLRPs implementation, to allow for a smooth implementation of the livelihoods component of the SIDA/FAO project.

68

2.

Coordination and the Cluster Approach

Cluster Approach

FAOs early activities positioned it at the forefront in the development of a livelihoods coordination architecture. This was one of the first times the Cluster Approach was implemented worldwide. Experience was slim and guidelines were not yet completed. After some initial hesitancy about which format to adopt (a food security sub-cluster? a livelihoods cluster?), it was decided to focus coordination efforts on Livelihoods Working Groups (LWGs) established at the national, provincial (Peshawar) and state (Muzaffarabad) levels. Such working groups had a number of advantages, in particular the fact that it was not formally required to conform to global guidelines and to deliver all the outputs a normal cluster should deliver and could hence follow its own dynamic in a more flexible fashion than real clusters. The working groups were initially co-chaired with ILO, and later on by ERRA, PERRA and SERRA. The role of co-chairing or acting as a secretary for the LWGs together with the authorities helped to further raise the profile of FAO with the Government. Coordination worked well and was well appreciated by central authorities and many participating organizations involved with livelihoods. Livelihood Working Groups helped allocate geographical areas to specific NGOs for the distribution of farming inputs, served as forums to share information and lessons learned, and were used to review the ERRA Livelihoods Rehabilitation Strategy in draft form (June 2007) and then disseminate it once finalized. The meetings were in general perceived as useful, well focused and well chaired. By some accounts, these were the best clusters in terms of dynamism and participation of NGOs and Government56, particularly when compared to the much more problematic, conceptually unclear and somewhat competing Early Recovery Cluster. In 2007, information was also collected from agencies and NGOs about their programme locations, in order to produce an inventory of Who does What Where (3W). However, the 3W approach was never effectively completed. Information was collected from organizations about their livelihoods recovery programmes, but the information was never mapped and distributed. The process apparently became stalled on the issue of which software should be used to store and retrieve the information. This information is now out of date. Many organizations have finished their livelihoods work or are moving on at the end of 2008. Ideally a 3W mapping should be carried out again as early as possible, maps generated and and the ReliefWeb mapping section used to disseminate the results. The fact that FAO could afford to hire staff dedicated principally to setting up and facilitating the livelihoods working groups at national and province/state levels did help. Unfortunately, a key contributor to this process (Anila Aftab) resigned in 2007. It was found that the pace and dynamism of meetings had suffered from her absence. Her last contribution was to help identify and hire the team of livelihoods advisors which would facilitate the implementation

An evaluation of the cluster approach conducted in 2006 in Pakistan concluded that NGO representation changed frequently. Although many criticized the NGOs for lacking predictability in this regard, it was the nascent cluster system that failed to instil ownership and involvement among the NGOs. See: Real-Time Evaluation of the Cluster Approach - Application of the IASC Cluster Approach in the South Asia Earthquake, IASC February 2006.

56

69

of the FAO-SIDA project. This team, posted within District Rehabilitation Units (DRUs), has helped set up and service Livelihood Working Committees at the district level. Clearly the push by FAO to get livelihoods support and rehabilitation moving early and to set up livelihoods coordination forums at all levels of Government was a major factor in mobilizing funds for the rehabilitation phase and in avoiding a lull in activities after the relief phase. Over and beyond the cluster approach and in addition to the work undertaken by UNDP, ILO, UNIDO and FAO under the CBLRP, the earthquake response was noteworthy in terms of the efforts that were made by various UN organizations to work better together. WFP kindly offered field office space and other logistical services to FAO and both organizations discussed joint programming opportunities at some stage, even though this has not been implemented.57 ILO and FAO have had a good relationship, with their dual coordination of the livelihoods working groups. However, discrepancies and overlaps remain. For instance, WFP and UNDP have their own forestry projects/components, and UNICEF carry out supplementary feeding activities for the most vulnerable people without attempts to share beneficiary lists with FAO to sustainably strengthen the livelihoods of these people when they return home. Clearly more collaboration could prove useful, particularly in the areas where mandates of each organization can cross over each other. Delivery as One Evidently, the Cluster Approach was designed to improve coordination and accountability during emergency operations, and in Pakistan it has to some degree managed to foster improved delivery during the earthquake response. Hence the question arises of what to do with the clusters after the emergency is over. Aid workers now sense a need for additional coordination in areas which are not meant to be covered by the Cluster Approach, such as capacity building and institutional strengthening. This is beyond emergency relief, and would seem to be an ideal area for the application of the UN Delivery as One initiative also piloted in Pakistan. The initiative in Pakistan was officially launched on 1 March 2007, upon the request of the Government. Once again, Pakistan is an early adopter, as it adopted the approach before guidelines precisely defining it were even drafted. An Operations Management Team has been working on the realization of common services for procurement, human resources management and information and communication technologies. A Transformation Fund managed by the Resident Coordinator and covering a period of 24 months was established to finance the change process. As of May 2007, an estimated US$ 6.8 million was required to fund the transformation and US$1.2 million had been mobilized from Canada. Finally, a High Level Committee on UN reform supervises the work. Five Thematic Working Groups were set up and have started working, based on a review of the current UNDAF (2004-2010), to better define the scope of future joint UN interventions. The process of developing joint programmes has been a slow and painful one, not without similarities with the long design phase for the CBLRP.

One attempt at the Abbottabad level stopped when those involved were transferred to other areas. More opportunities for WFP-FAO joint work materialized during the flood response.

57

70

Five joint programmes are expected to be finalized by the end of 2008, on the following themes, each related to specific Millennium Development Goals (MDG): i. ii. iii. iv. v. agriculture, rural development and poverty reduction (MDG1); education (MDG2); health and population (MDG 4, 5, 6); environment (MDG 7); and disaster management.

As conceived at evaluation time, four issues cut across all joint programmes: gender equality (MDG3), human rights, civil society engagement, and refugees. FAO is co-leading the development of the joint programme for area i) above and participates in all other joint programmes, with in particular a significant presence in the disaster management programme. The lessons drawn from the design and implementation of the CBLRP bear relevance here. Given the current lack of collaboration culture between agencies, joint programmes often translate into long, protracted negotiations to allocate budget shares and servicing costs. In the case of the CBLRP, a lack of transparency during the design process led to sub-optimal division of labour, with UNDP retaining a forestry component it was poorly equipped to implement. During the programme implementation, roles were divided so sharply that there was little actual collaboration involving staff from different agencies on a common task. Such compartmentalisation led to a number of issues, for instance the fact that approximately 50% of the projects identified by UNDP social mobilizers are subsequently rejected by FAO engineers on technical feasibility grounds.

71

PART IV: CONCLUSIONS, LESSONS AND RECOMMENDATIONS

Lessons are drawn from both the relief and rehabilitation phases. However, since the relief phase was completed at the time of the evaluation, the recommendations pertain mainly to the management of the current rehabilitation phase, or more generally to the future emergency livelihood programmes implemented by FAO. A few recommendations at the end of this section are addressed to all stakeholders involved in the Pakistan earthquake response, including the Government of Pakistan.

1.

A Relief Phase of Good Quality

Given the logistical, social and political difficulties involved, FAO and its partners in Government and in NGOs should be commended for having put together a rather good relief phase. Damage and needs assessments helped raise donor awareness about the need to reconstruct livelihoods, and the response was designed and implemented using a consistent livelihoods lens. Strong links were established with the Government and with national and international NGOs. Coordination forums facilitated by FAO were widely perceived as open, transparent and reasonably effective in deciding who should do what where. Beneficiary targeting and distribution mechanisms were planned for collegially and implemented pragmatically. The logistics and field presence in the earthquake zone were admittedly flimsy and the level of signing authority for both the FAO country representative and the Emergency Co-ordinator was too low, factors which slowed down the response. Some of the local procurements and LOA approvals were late, but partners managed to muddle through the activities, even if at a cost in terms of effectiveness and impact. The relevance, quality and impact of the farming and livestock inputs delivered were generally good, with the exception of maize seed distributed in 2006. Placing the emphasis on livestock feed and shelter as opposed to restocking embodied the sort of well informed strategic decision and direction that one should expect of a UN specialized agency. While the irrigation infrastructure was still in great need of repair at the time of evaluation, basic wheat production activity appeared to have been restored to near pre-earthquake levels in the most affected areas, in part thanks to the good work of FAO and its partners. The fact that FAO actually stopped seed and fertilizer distributions in 2007 not to adversely impact on private stockists is an illustration of the general good quality and maturity of the response. Interviews with donors and Government officials indicate that the earthquake response was generally perceived as useful and even influential. The credibility garnered by FAO in the earthquake response seems to have helped the Organization in its fund-raising and advocacy efforts during the Balochistan flood response. Lessons: Assessing the damages and needs and designing and implementing the response through a consistent, pragmatic, simple and well-informed livelihoods lens can ensure that abstract technical considerations do not take precedence over human and social capacities, opportunities and goals. 72

Damage and needs assessments are best conducted by staff and consultants experienced in the area, and retaining the same team of people to design the ensuing projects is conducive to a relevant response. Maintaining key staff such as the Emergency Coordinator for the entire relief phase and part of the rehabilitation period contributed significantly to the quality of the response by raising the profile of the Organization and allowing a learning process to occur throughout the response. In spite of most procurements being undertaken locally, timeliness remained an area for improvement, in part because optimistic hypothesis were used to define the deadline for the last distribution. Late planting implies yield loses. Therefore, the fact that some farmers sometimes plant late does not mean that most farmers will wait for the FAO inputs until that last possible planting time. Farmers may also need time between the moment they receive farm inputs and the moment they plant them. In the earthquake zone, wheat seed should have been distributed between the end of September and the 20th of October, even if some farmers sometimes plant until early December. Maize seed should have been distributed in AprilMay, even if some limited planting may happen in early June. Climatic considerations should also be given more attention: those activities requiring significant transport, construction or distribution should have been completed prior to winter, or postponed until after it. Recommendations: 1. In any emergency response of a significant size, FAO should deploy a qualified procurement specialist early on, a few weeks after a disaster; the procurement specialists presence in the field is particularly important in crowded emergency theatres where there are significant aid resources flowing, and hence significant risks of procurement fraud and high price inflation that needs to be monitored. 2. In all emergency responses, the cadre of technical staff and consultants should include experienced national researchers and/or extension workers with good familiarity with the area, so as to strengthen varietal choices and other technical decisions. 3. FAO should consider raising the in-country signing authority to at least US$100,000 to speed up procurement, logistics and financial management, and ultimately improve the timeliness of operations.

2.

A Poor Maize Seed Distribution in 2006

The Sarhad Yellow seed distributed in 2006 proved an exception to the general good quality of the relief phase. The variety was adapted only to the lowest lying areas; the seed was not true to type and in most cases did not germinate; and when it did, the grain was found unpalatable by many. A number of reasons were identified in the report: the lack of good quality maize seed in the country, the pressure to deliver, and a dishonest local superintendent.

73

Lessons: Massive emergency operations are situations which present a high risk of fraud and where sub-standard goods are often delivered by suppliers. The Pakistan earthquake emergency was no exception: many unscrupulous suppliers tried to profit from peoples misery by selling sub-standards goods to agencies and NGOs at inflated prices. Superintendence, while a good principle, is not sufficient to rule out quality problems and the superintendent report often comes late, after the transportation of goods to distribution point. Sourcing and delivery of high-quality farming inputs being at the core of FAOs comparative advantage in emergencies, it is of the outmost importance that the Organization, while continuing to decentralize its procurement processes and capacities, strengthen its quality control mechanisms so that it can ensure a supply of good quality farming inputs. Recommendations: 4. FAO emergency personnel should be trained to physically check seed, identify storage pests and conduct a rough germination test if a seed quality problem is suspected. Such tests by FAO personnel should not replace the results of the superintendence as a basis for acceptance, penalty application or rejection of the seed lot, but would allow FAO to verify the integrity of the distributed planting material independently from superintendents. FAO should also be prepared to prosecute service providers, including superintendents, who falsify documentation.

3.

Rehabilitation as a Balancing Act

The report describes the efforts of the staff managing three rehabilitation projects to strike the right balance between quantity, quality and speed; between community mobilization and delivery; and between reconstruction goals and development imperatives. The work with community organizations was found useful though a bit superficial; the initiated Farmer Field Schools do not seem to truly reflect the concept of the approach; the ADB-funded project and the CBLRP were racing to finish their work in time, sometimes at the expense of quality. In spite of these difficulties, there is no denying that the rehabilitation work initiated by FAO and partners is very relevant, albeit a year late compared to what would have been the optimal time to start such work, i.e. 2006. The lack of attention to irrigation infrastructure during 2006 negatively affected the impact of the farming inputs distributed during the relief phase. As explained, this was partly due to long negotiations between FAO and ADB on procurement rules.58 It took 15 months from the initial approach by ADB until an agreement was signed. Some beneficial changes were able to be made to the original project which would otherwise not have been possible, but this was a fortunate side effect. No emergency rehabilitation project should be held up for this amount of time because of administrative reasons. The development banks have become important agents in rehabilitation, and FAO will probably have to work with them again. A particular emergency response situation is not the right time to discuss complex administrative issues.

58

And a similarly long process of finalizing the project document in the case of the CBLRP.

74

Rather, a general agreement between FAO and ADB on how to work together in the context of emergencies should be prepared and used consistently. Another key finding is that the FAO-SIDA project suffers from too complex an institutional set up, which led to delays in implementation. The idea of building the capacity of ERRA, SERRA and PERRA by lending them personnel does not seem to work. Reporting lines and ways of working should be kept as simple and clear as possible in such circumstances. Lessons: While NGOs offer irreplaceable capacities to facilitate delivery and community participation, line agencies and local authorities have their own comparative advantages: their administrative capacity is often weak, but they can usefully participate in the screening of plans and projects (as in the SIDA/FAO project); their technical expertise is often irreplaceable during project implementation; and their involvement in monitoring the work of NGOs and COs on a technical plane is desirable. Community participation is crucial to improve relevance and cost-effectiveness, but also to strengthen the security of teams operating in politically volatile areas. Recommendations: 5. Better define what can be expected from the rehabilitation phase in terms of community participation: FAO and its partners should continue to pursue the most cost-effective ways of ensuring community participation in early recovery and rehabilitation assistance, while holding realistic expectations as to what level of community participation is possible in such contexts, i.e. modest yet significant involvement of communities in priority setting, implementation and, an often forgotten step, evaluation e.g. through regular accountability forums held at the UC level, as pioneered by Sungi in Northern Pakistan. In the CBLRP, clusters of COs need to be monitored or spot-checked regularly to ensure broad-based participation and representation, including that of remote villages. While it is useful to follow a pre-set menu of eligible Community Physical Infrastructure during early recovery and rehabilitation phases, the precise numerical targets for each CPI type (numbers of irrigation channels, ponds, etc.) must allow for ample flexibly not to pre-empt communities choice. 6. Review operational arrangements for CPIs rehabilitation to avoid delays in implementation, including by: Hiring one additional civil engineer; Refocusing the role of DOA on substantive monitoring and technical backstopping rather than approving expenditures; Simplifying cost estimates preparation through the use of standard rates of construction material set up and updated regularly by a District Rates Committee.

75

7. Quality should not be compromised for quantity and speed: Technical standards developed by the On Farm Water Management (OFWM) projects in Pakistan should be applied more systematically and consistently, notably in terms of drops, curves, backfilling, and use of temporary crossings of landslides; Consider topographic surveys at critical points prior to implementation, e.g. for new courses, sites and header tanks. 8. Simplify the institutional set up of FAO-SIDA project and speed up implementation: Integrate FAO management: the CTA from the FAO-SIDA project should report to the same reporting line as any other emergency programme staff/consultant, i.e. through the Emergency Coordinator and the FAOR, and to an integrated management team in Rome; To allow for a smooth implementation of the livelihoods component, clarify as soon as possible the issue of how Government funding should be allocated and spent in support of the implementation of CLRPs. Withdraw some of the staff detached to ERRA, SERRA and PERRA if their roles and the flow of funds to CLRPs are not clarified. 9. Retrain the people implementing the Farmer Field Schools and secure increased technical backstopping by Headquarters to impart a greater sense of participation in experimentation and learning; develop a more practical and well-focussed curriculum and generally ensure that the FFS as implemented in Pakistan remain true to the core logic of the approach. Closer coordination between the Department for Agriculture Extension and the FSS work by FAO / KWA also appears useful to improve sustainability prospects. 10. FAO and ADB should urgently finalize a global umbrella agreement for all administrative aspects (audit, purchasing, finance, monitoring and reporting) to avoid delays in the implementation of joint relief and rehabilitation programmes in the future. This agreement should be prepared outside the context of a particular disaster response.

4.

Insufficient Monitoring Capacity within the ERCU

Insufficient monitoring capacity within the ERCU constituted another significant weakness of both the relief and the rehabilitation phases. Some of the reports from IPs were very informative, insightful and frank about the difficulties faced, but there was no attempt at raising the quality of reports across the board. The conduct of post-harvest surveys by the IPs themselves compounded the insufficient oversight capacity of FAO and led to a lack of monitoring of beneficiary selection processes. It is important to remember that FAO is totally reliant on its partners for all interactions with the recipient communities. FAO was quite fortunate with its implementing partners in the earthquake response, even though some of them were newly formed. While trust will always remain an important component in such partnerships, this good fortune cannot be guaranteed each time an emergency response is needed. The inability of some generalist NGOs to understand complex technical matters

76

raised by some farmers (e.g. about the adaptability of the distributed seed in a given context) also pleads for establishing a direct link between FAO and disaster-affected communities. Lessons: Working through IPs being a standard modus operandi for FAO emergency programmes, an independent capacity to extract feedback about the quality of its response is vital to the successful future engagement of FAO in emergencies. Recommendations: 11. FAO must build a capacity to monitor IP performance independently, e.g. on how well the IP performed in targeting vulnerable households. This capacity could be composed of 2 or 3 in-house monitors headed by a senior officer, or contracted to an independent organization with no other role in the FAO response.

5.

Forestry and Watershed Management

The watershed and forestry management activities implemented by the CBLRP and the SIDA-funded projects are commendable in that they address the environmental degradation which amplified the impact of the earthquake. Marked differences of approach were noted: the CBLRP relies on national expertise and tried and tested approaches (afforestation and soil and water conservation as far as the FAO component is concerned there is also another forestry component implemented directly by UNDP) with little value added to what has already been tried in the country. Contrarily, the watershed component of the SIDA-funded project relies on international expertise to promote a much more ambitious, participatory and integrated approach to protect and develop natural resources available to communities. The CBLRP has been progressing reasonably well, while the watershed component of the SIDA-funded project has so far achieved very little, probably because it is a bit too ambitious for a rehabilitation project and involves many stakeholders. Detailed inception plans are missing in the SIDA-funded project, resulting in lack of clarity on roles and responsibilities and modalities for various interventions. It is unlikely that it will achieve its stated objectives at project end (December 2009). Lessons: Experience demonstrates that projects designed and implemented within the rehabilitation timeframe can struggle to get started and implemented when they are too ambitious. Recommendations: 12. The CBLRP forestry components should be streamlined to avoid duplication; FAO is better suited for forestry activities and should thus take over the component implemented directly by UNDP. The two watershed-related interventions implemented by FAO should also coordinate their interventions and bring technical standards closer to one another than is the case at the moment. For instance, in the CBLRP, the choice of species for afforestation should be diversified further.

77

13. Technical assistance needs for the FAO/SIDA watershed component should be reassessed and flexibility introduced in terms of reference to address the emerging needs of the project and to build local capacities. FAO and ERRA should identify the best local institutions for ICIMOD to build a local training and backstopping capacity for use in the FAO-SIDA project or similar projects in the future. 14. The FAO-SIDA project should finalize a realistic work plan and clarify responsibilities. The Watershed Management Coordinator in ERRA should focus on coordination with little hands-on management, while the project CTA should manage the project, provided he can spend enough time in the field (remote management will not work). In view of the resources and time available, the watershed management activities of the project should be confined to the eight sub-watersheds where planning and implementation are presently underway. The scope of operations in these watersheds may be increased to compensate for reduction in the number of sites. 15. A project should be prepared and funds raised for extending the FAO-SIDA integrated watershed management approach to rehabilitation of environment and wise use of renewable resources in four to six large watersheds.

6.

Coordination: a Major Role

Coordination forums facilitated by FAO were widely perceived as open, transparent and reasonably effective in deciding who should do what where and addressing technical issues. Hiring a senior national consultant to support coordination was a key factor to explain this success. However, the collected 3W information was never analysed nor put on maps and now needs to be updated as some NGOs are winding down their operations. Recommendations: 16. A strong coordination function being critical to FAOs and other actors effectiveness in large size emergency responses, FAO should reinforce its capacity to co-ordinate all such large emergency and rehabilitation responses by securing the services of a national or international co-ordination specialist to set up the livelihoods cluster groups and carry out effective who does what where mapping as early as possible, and disseminate these 3W maps and other coordination information, inter alia in the mapping section of ReliefWeb; this position could also take responsibility for the development of a monitoring system. 17. In the Pakistan earthquake response, FAO should update the 3W information (who does what where) and present it in a map form to identified underserved areas now that many NGOs are leaving or have left. 18. In the absence of a capacity building cluster in Pakistan, FAO and other UN organizations should find alternative ways of coordinating who does what in capacity building and institutional development to avoid overlaps.

78

7.

Fostering Successful Partnerships

The quality of the NGO implementing partners has been rather good. FAO was quite fortunate with its partners, even though some of them were newly formed and in spite of FAO investing very little capacity building in those partnerships. The SIDA-funded project includes a capacity building component but it has not yet started to deliver. One cannot assume that FAO will always be that fortunate in every response, and therefore a more formalized approach to IP selection and capacity building appears in order. The role of line departments during the rehabilitation phase varied significantly but was found generally useful. However, the role given to the DOA in approving financial disbursements to NGOs/COs rehabilitating infrastructure in the CBLRP significantly slowed down implementation. The different approaches taken in the two watershed components of the CBLRP (afforestation sub-contracted to DoF with little attempt to improve on their work) and the FAO-SIDA project (capacities of DoF not sufficiently taken into account) also point at the need to define more precisely the necessary contribution of line departments to livelihoods rehabilitation programmes. Joint programming with other UN organizations was not fully effective under the EC-UNDP funded CBLRP, as roles were defined a bit too neatly and actual collaboration opportunities were missed as a result, notably in community mobilization. Competition for budget shares also led to a long, protracted formulation process. Lessons: The Delivery as One initiative is not just about administrative and programmatic issues (common premises, standard procedures, joint programmes, etc.). It should also focus some energy and resources to develop a more cooperative attitude and a culture of team spirit among UN agencies. Recommendations: 19. In Pakistan, the capacity of Implementing Partners should be more systematically supported by starting the capacity building component of the FAO-SIDA project as soon as possible, in order to improve IPs targeting, distribution processes, participatory planning skills, progress reporting and technical understanding of the issues at hand. 20. In countries with frequent disasters or compounded crises, FAO should build longterm partnerships with local NGOs as implementing partners. By finding and endorsing implementing partners outside of an emergency, a full review of the partner organization is possible and where necessary capacity building can be carried out to reinforce the ways of working that FAO wishes to use, along the lines of the global agreement between WFP and IFRC for the use of National Red Cross or Red Crescent Societies in food distributions.

79

8.

Building Back Better?

The participatory development of the Livelihoods Rehabilitation Strategy was most welcome and proved instrumental in getting ERRA and the government to take livelihoods rehabilitation seriously. However, the evaluators would like to caution against some elements of the strategy, notably the "Build Back Better" phrase59. This phrase begs the question: Who is to say what is better? Introducing any new, untested feature involves taking a risk. The bigger the innovation, the larger the risk. One can of course attempt to forecast the likely effects of a new technique or practice on the social or natural environment, for instance through a social or environmental impact assessment, but these forecasts take time and will never yield certainty. Examples of how well-meaning innovations can carry risks are provided by the small quantities of hybrid maize seed distributed by FAO in 2007, by the excessive lining of irrigation channels in places not damaged by the earthquake, or by the new earthquakeresistant but energy-inefficient houses built across the earthquake-affected zone. Sometimes it may be useless to rebuild things as they were in the past. A disaster may have brought about irreversible change or taught some lessons worth heeding. The Government or affected communities may decide to react to the disaster by establishing more secure, sustainable and diversified livelihoods (relocation programmes, ban on traditional resourcedepleting activities, diversification towards more secure activities, etc.). The goal should be to reconstruct sustainable livelihoods and not necessarily pre-existing ones. However, the Pakistan earthquake response demonstrates that the idea that one could use a period of reconstruction to aggressively promote new developmental ideas is rife with dangers. Lessons: Rehabilitation and reconstruction are rarely the right time for trying bold innovations, and in that they are very different from development, a fact often overlooked by development workers with little exposure to relief and reconstruction activities. The specificities of rehabilitation (or reconstruction) vis vis development need to be better recognized: o Time: Development takes time and cannot be hurried. In rehabilitation, time is less crucial than in relief, but still quite a significant scarce factor. Communities cannot wait forever to restore their livelihoods. Therefore rehabilitation needs to rely on simple, well tested and easily scalable solutions, as does relief, even if rehabilitation work is not as simple and expedient as relief tends to be. o Aim: Development is in most cases about creating or tapping into new opportunities, while rehabilitation or reconstruction may be less creative, more focused on building back the status quo ante. o Risks: Development involves more trial-and-errors than rehabilitation and reconstruction can afford. For this reason, rehabilitation / reconstruction programmes

59

A phrase introduced during the Indian Ocean tsunami response.

80

normally avoid major technological leaps, although they may try and introduce change in careful, selective ways. o Specialisation: Good development tends to be multi-sectoral; strong coordinating and multi-stakeholder planning are essential. In contrast, sectoral approaches have clear advantages during rehabilitation and reconstruction, in terms of speed, scalability, technical coherence and quality, particularly in the restoration of infrastructure (roads, power lines, irrigation channels, etc.), basic services (health, education, etc.) and livelihoods (agriculture, fisheries, etc.). o Participation: Similarly, development must be thoroughly participatory, while rehabilitation can afford less intense community participation in the interest of delivering faster services, because there are less uncertainties and risks involved than in development. o Technical Assistance: As illustrated in watershed management activities, in a rehabilitation context, simple and well-defined technical assistance available on site and preferably provided by national expertise is more efficient than remotely located sources of technical assistance. Recommendations: 21. The programme is about rehabilitation and reconstruction, not a trial for development. FAO and its partners should be careful with innovations during relief and rehabilitation phases. For instance, hybrid seed should not be distributed during times of emergency or reconstruction unless they are already well known to farmers; and irrigation channels should not be lined (cemented) in their entire length even where no damage occurred, on account of cost and environmental considerations. 22. In both NWFP and AJK, the need for a new environment-friendly livelihoods development policy in mountainous areas was voiced, and FAO might usefully help develop such policies, for instance under the FAO-SIDA project, to use as guidance by AJK and NWFP authorities. 23. Making the new houses of the earthquake zone more fuel-efficient for heating and cooking should be a foremost priority of the GOP and its development partners, to cut down on the use of firewood for home heating and cooking. 24. FAO should advocate for land stabilization and conservation practices along existing roads and for treatment of cuts and fills during any new road construction or widening project in the area.

81

ANNEXES

82

Annex 1: Terms of Reference

Evaluation of the FAO Response to the Pakistan Earthquake

1. The Disaster On 8 October 2005, an earthquake measuring 7.6 on the Richter scale, with its epicentre located 19 km northeast of Muzaffarabad, Pakistan, struck the northern areas of Pakistan. An estimated death toll of over 80,000 was reported and 3 million people were affected by the loss of their loved ones, homes and livelihoods assets. FAO estimated the total cost of damage and losses in the agricultural and livestock sectors at US$409 million. The earthquake also had major impacts on the delivery of support services to rural populations and triggered a significant out-migration during the 2005-06 winter. The earthquake magnified the impact of environmental degradation, causing significant land destabilization and damage to the natural capital assets of rural people. The seism resulted in several major landslides and thousands of minor landslips, affecting about 10 percent of hillside arable land, forests and rangelands. Flash floods and mudslides destroyed agricultural land and fruit tree plantations, and altered runoff routes on hillsides and in valley lowlands. Many natural springs feeding irrigation and rural water supply schemes dried up as a result. In downstream areas, major water channels, roads and paths were blocked by rocks and debris thrown down by the earthquake. A massive relief effort was mounted. The Government of Pakistan initiated action on many fronts, including through the establishment of the Earthquake Reconstruction and Rehabilitation Authority (ERRA), the provision of rapid financial assistance, and the clearing of roads blocked by landslides. Civil society also responded immediately with large relief efforts. 2. The FAO Response FAO has responded by providing various types of assistance, for the first time using a fullfledged livelihoods-based approach, from damage and needs assessment to distribution of agricultural assets, to supporting various coordination mechanisms in the livelihoods sector, to capacity building of the national Earthquake Reconstruction and Rehabilitation Agency (ERRA). Between 21 and 31 October 2005, three joint assessment teams from FAO and the Ministry of Food, Agriculture and Livestock (MINFAL) visited the affected Pakistan-administered Kashmir (PAK) and North-West Frontier Province (NWFP). As part of the UN Flash Appeal, FAO appealed for US$25 million to fund emergency agricultural assistance and early recovery activities in the quake-hit region: provision of seeds, fertilizer, animal feed and sheds, tools, and Farmer Field Schools. As of mid-2007, nine earthquake-related projects were funded, with a total budget of more than US$22 million, largely from bilateral and multilateral donors (details in Table 1 below). They focussed on the provision of seeds, fertilizer, animal sheds and feed, poultry units, training, rehabilitation of small-scale infrastructure (e.g. irrigation canals), and capacity building. FAO also invested significant efforts and resources in facilitating livelihoods coordination groups at the regional and district levels, helping map the livelihoods-related activities of all concerned departments, agencies 83

and NGOs (Who What Where or 3W) and co-chairing coordination meetings with MINFAL. A distinct feature of the programme is that it started with short-term relief projects aiming at the rapid resumption of agriculture and livestock rearing in the area, and continued with longer-term rehabilitation projects, implemented through a participatory, community-based approach in partnership with decentralized line departments and NGOs, such as the ECHOUNDP funded Community Based Livelihoods Recovery Programme (CBLRP). Table 1: FAO programme in response to the Pakistan earthquake
Project Symbol Donor Project Title RELIEF PROJECTS
Emergency assistance to support the rehabilitation of the agricultural sector and poor household livelihoods in earthquake-affected areas Saving livelihoods to save lives - Emergency assistance for the restoration of food security and protection of rural livelihoods amongst OSRO/PAK/605/UK UK Earthquakes-affected farmers, women and other vulnerable groups in Pakistan Saving livelihoods to save lives - Emergency European assistance for the restoration of food security and protection of rural livelihoods amongst OSRO/PAK/602/EC Commission Earthquakes-affected farmers, women and other vulnerable groups in Pakistan OSRO/PAK/603/CAN Canada Pakistan: Saving Livelihoods to Save Lives Saving livelihoods to save lives - Emergency assistance for the restoration of food security and OSRO/PAK/604/BEL Belgium protection of rural livelihoods amongst Earthquakes-affected farmers, women and other vulnerable groups in Pakistan Emergency assistance to support the rehabilitation of the agricultural sector and poor OSRO/PAK/606/USA USA household livelihoods in earthquake-affected areas

Total Budget

Starting Ending Date Date

TCP/PAK/3007

FAO

$399,999

Oct-05

Feb-07

$1,131,492

Feb-06

Mar-06

$2,369,601 $2,179,163 $1,255,678

Feb-06 Feb-06 Mar-06

Jan-07 Jul-07 Jun-07

$500,000

Apr-06

Jun-07

REHABILITATION PROJECTS PAK/06/001/ /01/34 OSRO/PAK/701/SWE OSRO/PAK/702/ASB TOTAL UNDP Sweden ADB
Community based livelihoods recovery programme for earthquake-affected areas of AJK $3,312,258 and NWFP Project to assist ERRA and its partners in restoring livelihoods in the earthquake-affected $6,491,712 areas of Pakistan Immediate Support to Poor and Vulnerable $4,904,999 Households in 2005 Earthquake

Sep-06 Jan-07 Mar-07

Sep-09 Dec-09 Dec-07

$22,544,902

Another originality of the response is that FAO supported the development of a comprehensive livelihood rehabilitation component for ERRAs post-emergency rehabilitation plan with Swedish funding. Developed and fine-tuned through an extensive bottom-up process, the Livelihood Rehabilitation Strategy was later supported by the Swedish International Development Agency (SIDA), which agreed to fund its implementation within the framework of project OSRO/PAK/701/SWE. This intersectoral project has a three-fold focus on 1) local livelihoods improvement and diversification through Community Livelihoods Rehabilitation Plans (CLRPs) developed at the village level and to be funded by the GOP; 2) institutional capacity building; and 3) control of landslides through collaborative watershed management. The project, implemented by ERRA in collaboration with FAO since March 2007, is expected to continue until December 2009. It is due to undergo a mid-term evaluation by mid-2008.

84

3. Purpose of the Evaluation The evaluation will provide a review and validation of the analysis, strategies and interventions being undertaken by FAO, the Government of Pakistan, partners and donors in response to the earthquake emergency, and identify corrective measures, lessons learned and opportunities for improvement of ongoing and future programme delivery. It will aim to provide accountability to all stakeholders including the government and donors on the effectiveness of the FAO response, and will attempt to draw lessons in particular with respect to the implementation of innovative livelihood approaches in emergency contexts. 4. Scope and Issues to be Addressed The evaluation will focus on the earthquake relief and rehabilitation programme in NWFP and PAK, and will analyse the portfolio and its implementation in the field toward the end of the relief phase and toward mid-term for rehabilitation projects. Lessons derived from the analysis of the relief phase will hopefully be useful to fine-tune implementation of the rehabilitation phase. The Evaluation will also take into consideration the cluster approach and the status of Pakistan as a pilot country for the UN Delivering as One initiative, in order to derive lessons for FAO and its partners vis--vis these important developments in the humanitarian sector. To the extent possible, the evaluation will be synchronized with the mid-term evaluation of OSRO/PAK/701/SWE - "Assistance to ERRA and its partners in restoring livelihoods in the earthquake-affected areas of Pakistan". This will depend on the degree of progress achieved by the project on the ground by Spring 2008. If such progress is found insufficient (which will likely be the case as implementation is just starting), the main project stakeholders (ERRA, SIDA and FAO) may decide to maintain a separate mid-term evaluation for 701/SWE at a later date. The following issues have been collated through consultations with FAO programme staff, MINFAL and ERRA representatives, as well as with a few programme donors and implementing partners. While they are based on standard evaluation issues, an effort was made to try and adapt the scope to issues and dimensions specific to the earthquake response and in some case, to the livelihoods rehabilitation project (OSRO/PAK/701/SWE). Strategic and programme planning relevance Assessment of the role and contribution of FAO in facilitating the formulation of the GOP Earthquake Livelihood Rehabilitation Strategy. How participatory was the process? Is the resulting strategy clear, well designed, fit for purpose and owned by GOP and main NGOs? Status of implementation of the strategy: has it informed / impacted upon various partners programmes? How is strategy implementation monitored? Should the strategy be revized or amended? If yes, how? Analysis of the programme response as intended; in particular, is the programme clear, relevant and well designed? Complementing other assistance to cover the main needs of the affected communities?

85

Did the response evolve with the recovery process to remain relevant at all stages? How long should the relief and the rehabilitation phases last? How much synergies exist between the projects composing the response (with particular reference to coherence / articulation between the input distributions and the longer-term rehabilitation projects). Are there remaining gaps in the response? Innovative approaches: participation and watershed management

Use of the SLA and participatory approaches by FAO and partners (in particular influence on the GOP approach) in needs and damage assessments, project design, implementation, strategy setting: value added and/or disadvantages. Are there clear processes for preparation and screening of CLRPs? Do COs have adequate capacity and incentive to facilitate their development? Are the COs preparing CLRPs representative and inclusive enough? Are we really listening to local priorities and needs or are we asking communities to rubber-stamp predefined solutions? Depth of relations with communities in the SIDA-funded, the UNDP-funded and the ADB-funded projects. Progress vs. the overall goal of 1600 CLRPs and realism of this goal in view of such progress. Is government funding being made available for implementation of the CLRPs? Were lessons from similar projects (IFAD local development fund, WFP watershed project and Aga Khan Rural Support Programme) taken into account? What are the priorities identified by communities in the CLRPs issued so far (access, productive, social assets)? Are these CLRPs realistic and doable? Is there a need for introduction of innovations from the top down? A need for more work at the level of several villages or communities (management / rehabilitation of natural resources such as forest or irrigation canals, transport infrastructure, markets, etc.)? How well are the CBLRP modalities for fund management and for community-led procurement implemented? Through which mechanism will the CLRPs be implemented at the community level? Use of local artisans? Watershed management: what progress and issues in identifying pilot watersheds and red zones? In implementing pilot actions? Are FAO operational systems generally supportive of a programme implemented in a participatory way at the grassroot level? Management, capacity and implementation issues efficiency

Adequacy, rapidity and flexibility of administrative and operational processes, procurement mechanisms (delegation of authority, procurement mission, etc.), preparation and approval of Letters of Agreement; flows of funds from Islamabad to provinces and districts; technical and administrative clearances; main bottlenecks and fast track mechanisms used; efficiency of fiduciary control mechanisms. How useful was the support (technical, administrative and financial) received from headquarters? Timeliness and usefulness of short-term consultancy inputs. Profiles and mix of skills in the ERCU and field offices; are technical and administrative capacities adequate and geographically or institutionally well placed? Are there important gaps? Balance between skill sets and between national and international staff. Appropriateness of accountability and reporting lines for staff paid by the programme and hosted within government structures at the provincial and district levels (e.g. livelihoods coordinators).

86

Are the response and its implementation arrangements sufficiently monitored by FAO? Is substantive and financial reporting to donors satisfactory? More generally, how well did FAO respond to donors' expectations in terms of programme delivery and reporting? Programme delivery and targeting

Description of the deliverable to dates and review of their relevance, quality, timeliness and geographic distribution. Did remoteness of some of the affected areas have an effect on delivery of FAO rehabilitation services (and that of other actors)? Did FAO adapt its programme locations to take into account other actors? Description of the targeting approaches taken by Implementing Partners and by the community (including possible cases of re-targeting / re-distribution by the community itself after the formal distribution). Was aid given to the neediest and most vulnerable? Was targeting relevant at all given the local situation? Effectiveness effects, impact and sustainability

Comparative assessment of the effectiveness and impact of inputs distributed during the relief phase. How far back to pre-earthquake levels are we in terms of agricultural production, livelihoods and marketing mechanisms? Are there attempts at building back better? Are there provisions in the programme to ensure the maintenance and/or sustainability of different programme results (e.g. maintenance of repaired canals, sustainable supply of agricultural inputs, etc.)? Are demonstration plots and Farmer Field Schools used to their potential? How effective are they in conveying technical messages to farmers? How are training and physical assistance combined in the FFS modality? Usefulness of the technical assistance provided by FAO to Government and to NGOs. Is the Earthquake Programme well communicated / publicized? Did the experience, capacity and visibility gained in the earthquake response help in the floods response? Partnerships and institutional framework:

Strength of the partnership with ERRA; FAO impact on ERRAs capacity to coordinate and monitor the overall response, including within the GOP. Roles of FAO, other UN agencies, NGOs and Government in programme delivery, relationships with line ministries (especially at the decentralized level), use and reinforcement of their capacity, etc. Clarity and adequacy of criteria for selection of local partners, notably for the distribution of relief inputs and for community-led planning and implementation in the rehabilitation projects. Partnership agreements, documentation review process for MOUs and LOAs, and pragmatism of the adopted partnership agreements on the ground (implementation modalities, cash flow, reporting, etc.). Coordination, UN as One and Cluster Approach (with a focus on FAO participation, contribution and assistance received from other UN agencies):

87

Synergies and collaborations at field level, notably with WFP in terms of logistics and office support, food for training, joint assessments, etc. Are roles and responsibilities well delineated among UN agencies in the livelihoods and natural resource management sectors, or is there some degree of overlaps and/or competition? Relations between the livelihoods and the early recovery clusters and use of clusters during the rehabilitation / early recovery phase. Degree of national participation in, and ownership and chairmanship of livelihoods coordination forums. Which agencies and NGOs come to livelihoods working groups meetings and for what incentives? What content and level of coordination and trust between actors are being achieved? What are the factors for success in coordination? Who-what-where mapping: systems and tools, comprehensiveness, quality of data, effectiveness for limiting gaps and overlaps, possible need and means to enrich and update the data with more detailed reporting of all partners involved in livelihoods recovery. Impact of coordination on the degree of coherence and geographical overlap of activities by various partners, on good understanding of needs, joint response analysis, best practises, accountability, reporting. Previous evaluations have identified a possible conflict of interest between cluster head and agency head: an issue for FAO? Funding:

Extent to which donors and funding mechanisms contributed to greater efficiency (flexibility, conditionality, time horizons). Were there important activities proposed by FAO to donors, which could not be implemented due to lack of funds? Use of the FAO Special Fund for Emergency and Rehabilitation Activities (SFERA) funds and the fast-track emergency TCP to kick-start operations, ensure an early presence and quick distribution of inputs, and establish credibility with donors.

5. Methodology The methodology will be consultative and participatory. Consultations will be carried out with the collaboration of main programme stakeholders and donors on the selection of topics for evaluation. The main donors for the programme will be invited to comment on the evaluation terms of reference and participate in the evaluation itself. However, in the interests of independence in the evaluation process, final responsibility for the selection of programmes for evaluation, terms of reference and the selection of the evaluation team rests with the Evaluation Service. The fact that the FAO earthquake response is composed of a limited number of projects (9) implies that the evaluation teams will be able and probably will have to analyse quite a number of issues at the project level, while maintaining a focus on response-wide themes. The evaluation will first comprise a Desk Review at Headquarters. The review will describe the earthquake response portfolio, collect documentation and highlight key issues and aspects which the main evaluation mission should pay particular attention. An initial Scoping Mission will be fielded in October 2007 to discuss the evaluation objectives and scope with the main programme stakeholders in country (FAO, GOP, UNRC and key non-governmental partners). The scoping mission will also review potential (national) candidates for an Impact Assessment, select a team of consultants for the same, and 88

train them on the methodology. The aim is to conduct the impact assessment towards the end of 2007. The impact assessment methodology will basically consist in focus group interviews conducted with a) male beneficiaries, b) female beneficiaries, and c) non-beneficiaries, among 12 benefiting communities in various locations of North-West Frontier Province (NWFP) and Pakistan-Administered Kashmir (PAK). The localities will be decided by the FAO Evaluation Service based on a thorough analysis of the FAO response. This exercise will prepare and inform an Evaluation Mission scheduled for April-May 2008. During all stages of the evaluation, consultations with stakeholders, including government authorities (at all levels), donors, programme beneficiaries (of both genders), field partners (both national and international NGOs) and both international and national staff of FAO will be given high importance. 6. Evaluation Time Frame Desk Review: Scoping Mission: Impact Assessment: Main Evaluation Mission: Draft Report: Final Report: 7. Team Composition The evaluation will be managed and supported by the PBEE of FAO in Rome. The Team Leader will be consulted as to the final terms of reference and the evaluation arrangements. The team will consist of national and international members: Team Leader (External) Senior Evaluation Expert The team leader should be external to FAO and have strong evaluation and communication skills, as well as extensive experience in post-emergency situations. Ideally, the team leader should be familiar with the context of the earthquake response. The team leader will particularly review the overall strategy adopted by FAO and the relevance of programme partnerships and interventions with respect to that strategy. He will also be responsible to coordinate the production of the evaluation report and to finalize it. Livelihoods / Community Development Expert A livelihoods expert external to FAO, with extensive field experience in northern Pakistan, familiar with the conditions in affected mountainous regions, agro-pastoral and pastoralist communities who can help assess the relevance, effectiveness and impact of the response from a socio-economic perspective. Technical Specialists Four technical specialists external to FAO, with field experience in northern Pakistan and knowledgeable of the key food security and livelihood issues in the region will be required, ensuring expertise in the areas of agronomy, livestock, forestry and smallscale irrigation schemes. The technical specialists are expected to be Pakistani nationals. FAO Evaluation Officer September 2007 October 2007 November 2007-February 2008 May-June 2008 October 2008 December 2008

89

An evaluation officer from the FAO Evaluation Service will be part of the team to bring institutional knowledge in the team and will second the Team Leader in the management of the evaluation process, notably its technical component. The evaluation team would need to include a suitable Government expert to reinforce the relevance of evaluation findings as well as ownership of recommendations within Government. Discussions are under way to identify the most appropriate level for this participation. 8. Evaluation Management The evaluation management should foster transparency and usability, as well as ensuring a well informed, quality product. The evaluation management will ensure that all stakeholders are consulted at different stages of the exercise and that the results of the evaluation are disseminated back to all those who have been respondents to the process. The evaluation process will be managed by the Evaluation Service. The FAO country office is expected to facilitate the conduct of the evaluation at country level. A Consultative Group will be set up at the country level, composed of the main programme stakeholders (ERRA, MINFAL, decentralized governments, FAO Representative and programme managers, interested donors). An initial meeting will be organized at the start of the evaluation mission to discuss the remit of the evaluation, help plan for the field visits and brief the team on the latest developments of the programme. A series of debriefing and ground-truthing meetings will be organized at the end of the evaluation mission with broad participation from programme stakeholders (including Government, interested donors, partnering NGOs, FAO and other UN agencies), possibly in Muzaffarabad, Abbottabad and Islamabad. It is important to organize some of these close to earthquake-affected areas to encourage participation by local actors. The last debriefing meeting, to be held in Islamabad, will include representation from interested programme donors. The objective of these meetings will be to present tentative findings, recommendations and lessons learned, and exchange views on these with stakeholders before writing a first draft of the report. 9. Evaluation Dissemination As explained above, the main evaluation mission will debrief with a cross-section of GOP, UN, donors and NGO representatives in country. Reports of the evaluation and the FAO management response will be available to all parties and will normally be public documents. The Evaluation Service will also maximize transparency by providing updated information on the web on the evaluation work programme, evaluation terms of reference, budgets and expenditures.

90

Annex 2: Persons Met

Government of Pakistan
Federal Government Aamir Ashraf Khawaja Mohammed Yousaf Bhattti Saleem Khan Jhagra Amjad Nazir Masood A. Rana Tajammul Rizwana Rafiq Ahmad Raza Sarwar Ahmed S. Shaikh Mohammad Fayyaz Alternate Permanent Representative Additional Chief Secretary Additional Secretary Joint Secretary Commissioner Special Crops Deputy Director Special Crops Sectoral Expert Livelihood Director Planning (Economic Sectors) Team Leader, Technical Advisor M&E Technical Adviser M&E Pakistan Representation to FAO MINFAL MINFAL MINFAL MINFAL MINFAL ERRA ERRA ERRA / DfID ERRA / DfID

Government of North West Frontier Province (NWFP) Faisal Rahman Fazrul Rahman Liaqat Ali Mohammad Atif Musaddiq A. Khan Nasir Azam Khan Niaz Mohammad Usman Ayub Zubair Asghar Qureshi Jehan Sher Khan Munawar Khan Rizwana Wasaich Musaddiq A.K. Tahirkheli Allah Had Khan Mohammad Diyar Mohammad Akhtar Baig Mohammad Rahim Saifullah Muqarrab Ali Khan S. Riaz Ahmad Shah Akhtar Zeb Fazal Rabi Naveed Iqbal Imtiaz Ahmed Awan Mohammad Younis Mohammad Rashid Mian Ghulam Hussain Acting DG Director (Monitoring & Evaluation) - acting DG Environmental Expert Media Coordinator Provincial Planning Expert Director General Senior M&E Officer M&E Director Director Planning and Technical Livelihoods Coordinator Agricultural Advisor SGC PERRA/UNDP Provincial Planning Expert PPD CMP II Deputy Director DFO Kaghan DG Agriculture Research System DG Agriculture Extension Director Executive District Officer EDO Agriculture EDO Soil Conservation Agriculture Officer Divisional Forest Officer DO Agriculture EDO Agriculture District Officer (Water Management) PERRA PERRA PERRA PERRA PERRA PERRA PERRA PERRA PERRA PERRA/FAO PERRA/FAO PERRA/UNDP PERRA / UNDP NWFP Agriculture Department NWFP Fisheries Department NWFP Forest Department NWFP University Agriculture Dept. Directorate of Livestock & Dairy Dvpt Agriculture Dept., Mansehra Agriculture Dept., Mansehra Agriculture Dept., Mansehra Agriculture Dept., Mansehra Dept. of Forestry, Abbotabad Agriculture Dept., Abbotabad Agriculture Dept., Abbotabad Dept. of OFWM, Abbotabad

Government of Azad Jammu and Kashmir (AJK) Syed Asif Hussain Farman Ali Muhammad Akbar Raza Raees Khan Shafqat Ullah Baloch Abdul Aziz Gorsi Haq Nawaz Khan Mohammad Bashir Maqsood A. Malik Mohammad Khalil Director General Livelihoods Management Coordinator Advisor, Agriculture and Livestock Livelihoods Management Coordinator Livestock Management Coordinator M&E Assistant Director General Chief, Agriculture Assistant Chief, Planning Chief, Foreign Aid Planning and Development SERRA SERRA / FAO SERRA / FAO SERRA / FAO SERRA / FAO SERRA / UNDP Dept. of Agriculture Planning and Development Dept. Planning and Development Dept. Planning and Development Dept.

91

Riaz Hussain Shah Director Kareem Qurashi Deputy Director Muhammad Siddique Sulehria Director General Basharat Hussain Project Director Shahid Mahmood Assistant Director Mumtaz Akhtar Qadir WFP ILM Project Director Kh. Ashfaq Ahmad Community Development Supervisor Sheikh Altaf Ahmed Fisheries Development Officer Mohammad Waheed Murad Assistant Warden Liaqat Hussain Programme Manager Gul Mohammad EDO Assistant Mohammad Bashir Abbasi Director

Animal Husbandry Dept. Animal Husbandry Dept. Animal Husbandry Dept. Irrigation Dept. On-Farm Water Management Dept. Dept. of Forestry Dept. of Forestry Fisheries Dept. Fisheries Dept. DRU, Muzaffarabad DRU, Muzaffarabad Hazara Agriculture Research Station

FAO
Headquarters Stephan Baas Thomas Hofer Regina Gambino Giancarlo Ferrari Nacif Rihani Simon Mack Florence Egal Jan Poulisse Juan Fajardo Daniel Ramirez Leon Ilja Betlem Daud Khan Purveen Kharas Elisabeth Beer Hilde Niggemann Marianne Ward Matthias Lichtenberger Neil Marsland Richard China Richard Trenchard Hans Page Maria Angela Bagnardi FAO in Pakistan Mohammed Farah Sayed Mohammad Ali Attila Bussay Ian Cherrett Tim Vaessen Faizul Bari Piet Vochten Rachel Percy Faman Ali Javaid Qaisar Habeeb Mustafa Edward Watt Mahjebeen Qazi Mohsin Rose Najma Akbar Raabya Amjad Sajid Mahmood Syed Sajidin Hussain FAO Representative FAO Representation Assistant Representative (Programme) FAO Representation Chief Technical Advisor SUPARCO Consultant SIDA Project MTR Senior Emergency Coordinator ERCU Emergency Coordinator ERCU Chief Technical Advisor ERCU / SIDA Project Consultant ERCU / SIDA Project Livelihood Management Coordinator ERCU / SIDA Project Project Manager ERCU / SIDA Project Operation Logistics Officer ERCU Operations Officer ERCU M&E Officer ERCU National Agronomist ERCU Programme Assistant ERCU Communication Officer ERCU Logistics Assistant ERCU Disaster Preparedness and Assessment Consultant ERCU Sustainable Rurual Development Officer Forestry Officer Procurement Strategy/Monitoring Officer Project Coordinator Animal Production Officer Senior Officer Nutrition Officer Senior Officer Agriculture Officer Internal Auditor Legal Officer Service Chief Senior Programme Officer Operations Officer AELA Desk Supervisor Senior Operations Officer Operations Officer Nutrition Specialist Service Chief Liaison Officer Service Chief Associate Professional Officer NRCB FOMC AFSP AGAH AGAP AGAP AGNP AGPC AGPS AUD LEGA OCD TCAP TCEO TCEO TCEO TCEO TCER TCER TCER TCOM TCOM

92

Pervez Akhtar Sadaqat Hanjra Chris Baker Khalid Rasul Mohammad Zia-ud-Din Mohammad Shoaib Mian Zia-ud-din Abida Sadia Mohammad Arif Khan Azhar Muhammad Rashid Chaihi Salah ud-Din Imtiaz Ahmad

Secretary Consultant Consultant Deputy Technical Coordinator Civil Engineer Sub Engineer Soil and Water Engineer Social Forestry Specialist Rural Sociologist Field Coordinator Soil and Water Engineer Sub Engineer Field Coordinator Assistant Field Coordinator

ERCU ERCU ERCU Garhi Habibullah Garhi Habibullah Garhi Habibullah Garhi Habibullah Garhi Habibullah Garhi Habibullah Muzaffarabad Muzzaffarabad Muzzaffarabad Muzzaffarabad Abbotabad

Donors
John Blunt Mian Shaukat Shafi Ayadurai Somesan Barbara Humick Mohammad Imran Ashraf Andrew McCoubrey John Hansell Marilee Kane Institutional & Governance Specialist ADB Team Leader, Earthquake Emergency Assistance Unit ADB Consultant ADB First Secretary, Development Canadian High Commission Agronomist / Development Advisor European Commission Delegation Infrastructure and Environmental adviser DfID Livelihood Adviser DfID Deputy Director, Earthquake Reconstruction Prog. USAID

Multilateral Agencies
Fikret Akcura Jan Vandemoortele Anwar ul Haq Fawad Hussain Alvaro Rodriguez Kris Hendrickx Riaz Ahmid Khan Muhammed Usman Qazi Fazal Rahman Allah Bakhsh Bashrat Khan Arthur M. Gaines Albert French Saad Gilani Ayesha Khan Peter French Mahe Ahmed Magdalena Moshi Iqbal Karim Isabelle Providoli UN Resident Coordinator UN Resident Coordinator Advisor UN Area Coordination National Humanitarian Affairs Officer Country Director Technical Advisor Programme Manager Programme Officer Provincial Coordinator Community Development Supervisor Community Development Specialist Deputy Security Advisor Head of Muzaffarabad Office Project Manager Chief Technical Advisor to CBLRP Country Director Deputy Country Director Head of Programme M&E Officer Soil and Water Conservation Specialist UNRC UNRC UNRC OCHA UNDP UNDP UNDP UNDP UNDP / CBLRP UNDP / CBLRP UNDP / CBLRP UNDSS UNICEF ILO UNIDO WFP WFP WFP WFP ICIMOD

Non-Government Organizations
Miraj Khan Raheela Khan Jasper van de Reep Raza-ul-Haq Irfan Rajput Faisal Ahmad Mohammad Sajid Fayyaz S.M. Abid Programme Director Acting Nutritional Programme Coordinator Head of Muzaffarabad Office Head of programming Executive Director Field Coordinator NWFP Field Coordinator Field Coordinator Social Mobilizer AKDN Caritas CRS CRS DDF DDF DDF DDF DDF

93

Chris Grose Munawar R. Kazmi Roshan Zada Khattak Mukhtar Ahmed Katar Tahir Waqar Shahid Iftikhar Shah Aziz Ahmed Imran Ali Shah Aftab Ali Ghazi Asif Jehanzeb Khan S. Aftab Ahmad Akhter Ali Arif Syed Mohammad Zahid Munir Ahmed Qamar Zaman Abbasi Saima Alvi Klaus Euler

Managing Director National IMP Facilitator Extension Specialist / FFS Technical Manager Programme Manager District Programme Officer District Programme Officer Programme Coordinator Programme Officer Team Leader Manager Social Organizer Engineer Engineer Team Leader Assistant Coordinator Senior Social Organiser Independent Consultant

IMA International Kissan Welfare Association / NARC Kissan Welfare Association / NARC Mercy Corps NRSP NRSP NRSP Relief International SRSP SRSP SRSP SRSP SRSP SRSP SRSP Sungi Development Foundation Sungi Development Foundation

94

Annex 3: Mission Itinerary

14 - 17 May 2008: Meetings between Team Leader and various FAO units involved in the response and with the Alternate Permanent Representative of Pakistan to FAO at FAO Headquarters in Rome. 18 - 19 May: 20 24 May: Travel to Islamabad Meetings with the FAO team, with MINFAL and ERRA, with donors of the earthquake response and its Implementing Partners Travel from Islamabad to Peshawar Meetings with DG PERRA and line departments; travel to Abbottabad Meeting with FAO, CBLRP, ADB, SIDA projects staff, line departments and NGOs in Abbottabad; field visit to CBLRP and SIDA project activities; Team 1 travels to Muzzaffarabad; Team 2 travels to Bisham Team 1 meets with Additional Chief Secretary, DG SERRA and line departments in Muzafarabad; field visits to CBLRP, SIDA and ADB projects activities; Team 2 meets with NGO partners in Bisham and visit the activities of the SIDA and ADB projects in the field; travels to Muzzaffarabad Wrap-up meeting with AJK Government and other partners in Muzzaffarabad Report writing in Islamabad Wrap-up meeting with PERRA in Peshawar Further meetings with programme partners in Islamabad, report writing Wrap up meeting with GOP, donors and partners in Islamabad cancelled due to security alert. Team Leader and Evaluation Manager travel back to Rome Debriefing meeting in Rome

25 May: 26 May: 27 - 29 May:

30 May - June 4:

5 June:

6 - 7 June : 8 June: 9 - 13 June: 12 June:

14 June: 19 June 2008:

95

Annex 4: Cost-effectiveness of the 2006 Wheat Seed and Fertilizer Distribution

A total of 2,050 tonnes seed of improved varieties of wheat was distributed in earthquake-affected area in 2006. The impact assessment indicates that beneficiaries obtained 1,622.5 kg per ha of wheat yield as compared to 1176.78 kg per obtained by non-beneficiaries. For assessing the benefit, the following assumptions were made: 1. 2. 3. 4. All the donated fertilizer was applied to field planted with the donated quality seed of Aqab2002 by beneficiaries. Non-beneficiaries applied half of the dose of urea and no DAP. Transportation cost from distribution point to field was not included in calculation, although it was substantial in some cases. Application cost of fertilizers and harvesting and threshing charges were not included, and Straw was not considered in benefit calculations.

The most important hypothesis is probably the first one, and it is most certainly false: benefiting farmers could easily have stored, donated or sold the fertilizer, rather than apply all of it. They could also have applied it on other fields, planted with some other wheat seed or another other crop The following calculation assumes there was not alternative use for fertiliser than on the donated wheat and therefore underestimates the overall cost-effectiveness of the donated fertilizer, because such alternative uses are potentially more cost-efficient than those of our hypothesis. For instance, using part of the fertilizer on onions or other cash crops would probably be smarter farming than using all of it on wheat. The kits were generously endowed with fertilizer, reflecting optimum fertilisation ratios established in research centres, but the conditions in Northern Pakistan are different from those of a research station in Peshawar or Pir Sabag. We can trust that if local farmers choose to use their resources in a different way than described in programme documents, its probably their choice which is the most cost-effective. Another factor which pleads for under-estimation of the benefits is the price used for wheat grain: The government support price is used in the calculation, i.e. its the minimum price rather than the real market price in the earthquake zone. Estimates of excess monetary benefits to the beneficiaries in comparison to non-beneficiaries from wheat seed and fertilizer distributed in 2006-07 Type of farmer Beneficiaries Non-beneficiaries Difference Yield kg per ha 1,622.50 1,176.80 445.70 Value $ Per ha 308.30 223.60 84.70 Variable cost $ per ha (seed & fertilizer) Seed DAP Urea Total 10.80 40.00 21.70 72.50 7.52 0.00 10.85 18.37 3.28 40.00 10.85 54.13 30.57 US$ 20,500 hectares 626,613 US $ 1,852 Rupees

Net benefit derived from the inputs, per hectare Total area planted by beneficiaries Total benefits to beneficiaries

Conversion rate: 60.6 rupees per dollar; FAO seed price: 273 $ per MT; Freight charges: 25 $ per MT; Total cost of FAO seed: 298 $ per MT; Grain support price (= farmer seed price) : 190 $ per MT; DAP price: 376 $ per MT ; Freight charges: 24 $ per MT ; Total cost of DAP: 400 $ per MT or 40$ per bag; Urea price : 193 $ per MT; Freight charges: 24 $ per MT ; Total cost of urea: 217 $ per MT or 10.85$ per bag.

96