ADBs Support to Gender and Development IED Evaluations of 2009 & 2010

Briefing for 11th Session of External Forum on Gender and Development

Monday, June 27 2011
Rajesh Vasudevan
The views expressed in this paper are the views of the author and do not necessarily reflect the views or policies of the Asian Development Bank (ADB), or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data included in this paper and accepts no responsibility for any consequence of their use. The countries listed in this paper do not imply any view on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology.

Monday, 27 June 2011

IED

IED Evaluation of ADBs Support to GAD in Two Phases

Phase I: Meta-Evaluation - Evaluation completed in December 2009 Phase II: Main Report and 6 Country Studies

- Evaluation completed in December 2010

Monday, 27 June 2011

IED

Phase I Meta-Evaluation

Monday, 27 June 2011

IED

Phase I Evaluation Criteria

(i) Relevance: Relevance at time of approval and in context of strategic priorities
(ii) Responsiveness: Covered institutional, operational, external responsiveness (iii) Results: Review of available published information on intended outputs and outcomes of loan and TA activities

Monday, 27 June 2011

IED

Phase I Methodology
(i) Desk review of about 500 related documents including 31 project completion reports, 17 higher evaluations, and 20 RGAs. (ii) Interviews with gender focal points in ADB headquarters.
Note: 212 Category I and Category II projects between 1998 and October 2009.

Monday, 27 June 2011

IED

Phase I Findings
(i) Determinants indentified that influenced gender achievements in departments: Realignments; uneven levels of influence of country gender assessments; lending sources and modalities; technical assistance distribution; and availability of gender specialists. (ii) Shift in ADBs portfolio from 2004, to sectors not amenable to gender mainstreaming. (iii) Gender action plans, a pragmatic and innovative approach.

Monday, 27 June 2011

IED

Phase I Recommendations
(i) Improve outcome orientation of GAD goals and their monitoring and evaluation; (ii) Provide clarity to operationalize GAD goals in ADB operations in the context of Strategy 2020; (iii) Provide adequate financial and human resources to support policy implementation.

Monday, 27 June 2011

IED

Phase II Country Studies

Monday, 27 June 2011

IED

Phase II Evaluation Criteria

(i) Study brings a country and project perspective to bear on the implementation and outcomes of GAD policy between 1998-2009. (ii) Study has assessed and rated gender-related components of 55 projects using the 4 standard IED criteria - relevance, effectiveness in achieving outputs and outcomes, efficiency in implementation, and sustainability.
(iii) PNG, Lao PDR, INDO, NEP, BAN, PAK

Monday, 27 June 2011

IED

Phase II Methodology
(i) Desk-review of available country and project-specific documentation;
(ii) Interviews with headquarters-based GAD specialists, and resident mission GAD staff; (iii) Purposive selection of 55 projects for in-depth assessment; (iv) In-country consultations; and (v) In-depth assessment of 55 sample projects

Monday, 27 June 2011

IED

10

Phase II Findings
Sample: 25 (Cat I), 20 (Cat II), 9 (Cat III), 1 (Cat IV)

(i) Overall: About 51% the projects were rated successful or higher in gender-related performance.
(ii) Relevance: 82% of the sample projects were rated GAD relevant or higher. (iii) Effectiveness: 44% effective or higher in achieving the intended GAD outcomes. (iv) Efficiency: 31% efficient or higher; and (v) Sustainability: 45% likely to be sustainable in terms of continuity of GAD outcomes.
Monday, 27 June 2011 IED 11

Phase II Findings
Missed Opportunities: Of the 10 category III and IV projects, 30% were rated successful or higher and the only highly successful project in the sample of 55 projects was in category III. Core and Non-Core Sectors: 20 (Core Sectors), 25 (Non-Core Sectors)

(i) 50% of the 20 projects in Core sectors rated successful or higher.

(ii) 60% of the projects in Non-Core sectors rated successful or higher.
Monday, 27 June 2011 IED 12

Phase II Findings
(i) Evidence weak on the influences of country gender assessments to country strategies;
(ii) Capacity to address gender issues, including in Category I and II projects, is often limited; (iii) ADB's business processes for project implementation and related information systems, are weak in capturing gender performance data; (iv) Review of gender categorization of projects during implementation is important.

Monday, 27 June 2011

IED

13

Phase II - Recommendations
(i) Make more effective use of country and sector level gender assessments in the preparation of country partnership strategies, sector road maps, results frameworks, operations business plans, and project preparation; (ii) Strengthen tracking of gender results in ADB project performance monitoring systems; and (iii) Increase collaboration with development partners, including non-government and community-based organizations in DMCs, to jointly deepen gender focus in country policy dialogue, technical assistance activities, and in strengthening capacity of executing agencies.

Monday, 27 June 2011

IED

14

GAD Assessment Criteria

Monday, 27 June 2011

IED

15

Relevance (20%)
Consistency of a projects impact and outcome with Governments development strategy ADBs lending strategy for the country ADBs strategic objectives at the time of approval and evaluation Adequacy of the design

Monday, 27 June 2011

IED

16

Relevance criteria and GAD questions used in Phase II study

Do the CGA , CPS (and its gender strategy) identify
the sector/project as priority for gender-related interventions? How important is gender to achievement of overall project objectives, as articulated in project documents? Is there evidence of gender-inclusive consultations and participation (C&P) on project design and intended outcomes? How well are the findings of the project gender analysis (or gender focus of social analysis) reflected in design of the project?
Monday, 27 June 2011 IED 17

Effectiveness (30%)
The extent to which the outcomes, as specified in the design and monitoring framework, either as agreed at approval or as subsequently modified, have been achieved.

Monday, 27 June 2011

IED

18

Effectiveness criteria and GAD questions used in Phase II study

Are gender-related outcomes and/or outputs included in the DMF, and are there associated targets and indicators? Are there missed opportunities for mainstreaming gender in project design? Are gender-related design features (activities) well conceived, and appropriately reflected in the Gender Action Plan (GAP)? Were gender-related outputs delivered as planned and on time, or, for active projects, are they considered on track?

Monday, 27 June 2011

IED

19

Efficiency (30%)
Describes (ex-post) how economically resources have been converted into results, using the EIRR (relative to 12% standard) cost-effectiveness of the investment (relative to industry practice) other indicators (implementation issues)

Describes the resilience to risk of net benefit flows over time

Monday, 27 June 2011

IED

20

Efficiency criteria and GAD questions used in Phase II study

Were sufficient human and financial resources
allocated to gender-related activities in the project? Are gender activities or indicators monitored during supervision? Has the project complied with all gender-related covenants and agreements, or is it on track to do so? Does the Gender Action Plan (GAP) provide a suitable foundation for monitoring implementation of gender-related activities?

Monday, 27 June 2011

IED

21

Sustainability (20%)
Considers the likelihood that human, institutional, financial, and other resources are sufficient to maintain the outcome over the economic life of the project

Monday, 27 June 2011

IED

22

Sustainability criteria and GAD questions used in Phase II study

Does the project build longer-term country
institutional capacity to address gender issues in the project or sector? Is there continued demand for project-provided gender-related products and services? Have project-supported policy, legal, and institutional reforms been implemented or are policy and institutional arrangements in place to sustain genderrelated activities? Are there gender-related risks to achievement of project outcomes? (Note: scale is inverted--high risk = 0; low risk = 3)?
Monday, 27 June 2011 IED 23

Assessment Rating - Categories and Ranges for four criteria

Highly successful (HS>= 2.7) Successful (1.6=<S<2.7) Partly successful (0.8=<PS<1.6) Unsuccessful (U<0.8)

Monday, 27 June 2011

IED

24