Monitoring and evaluation of community systems strengthening

Ms. Nathalie Zorzi Manager, Monitoring and Evaluation Support Team The Global Fund 18/19 March 2010

CSS indicator development process

August 2008: The Global Fund commissioned a review exercise in Pretoria to define indicators for community level service delivery and systems strengthening; January 2009: UNAIDS developed a guidance tool for including Community Systems Strengthening in Global Fund proposals which includes a number of CSS indicators; November-December 2009: A multi-partner Technical Working Groups oversaw the implementation of 9 field exercises which aimed at capturing best practices in community level service delivery and systems strengthening including the set of CSS indicators used at country level; February 2010: a harmonization workshop was organized in Geneva where M&E experts from various organizations reviewed existing CSS indicators and developed an updated list which was aligned with the proposed building blocks and SDAs of the new CSS framework; March 2010: An internal review and further definition of the indicators was done by the Global Fund. Comments from the Technical Working Group and other stakeholders were integrated into this process.

Challenges - Review of CSS case studies

Challenges at community level and CSOs - Community level workers have limited language skills, multiple reporting, and high turnover. - Electricity & internet have limited availability - Ability to analyze the collected data for decision making is limited resulting in a perception of purposeless data collection - When you cant make sense of your own data how can you explain it to others in the community for their meaningful involvement?

Challenges
Limited capacity at sub-national & national level for written feedback & supervision to show purpose in data Poor planning for trainings for M&E at all levels Poor correlation of program performance, logistics and finance (more purposelessness) Thus, loss of opportunity of learning at community level Poor data analysis and strategy at national level for systematic involvement of CSOs CSOs are registered with departments other than health causing counting difficulties

Approach
Include both M&E for service delivery and CSS Community level M&E means community level of both CSOs and public health system Large CSOs follow M&E for both community level and health facility level according to the level of application Community level M&E to be integrated with national level at the planning stage itself
Design jointly by various stakeholders Harmonized list of indicators Common reporting formats and data collection tools

Approach
MoUs between disease program and CSOs
Avoids double counting Gives baseline for calculation of % for indicators Better understanding of roles & responsibilities Facilitated by provision of guidelines from the national level for involvement of CSOs

Build capacity of community level in sharing information with the community Include meetings at community level and program review at various levels Include operational research at community level Include CSOs in evaluations and joint reviews and - learning opportunities for health facilities and CSOs

Approach
Assuring data quality
Publication of M&E guidelines giving standardized data collection tools and reporting formats Simple forms with minimal information at peripheral level Training & re-trainings of staff on M&E Guidelines on frequency and content of written feedback and supervision Ensuring dedicated trained M&E staff at all levels for supervision (and their travel); ensure data entry staff Standardization of data base

Current status of CSS indicators (March 2010)

Currently we have a list of 21 indicators with a description of the rationale, definition, numerator, denominator, proposed method of measurement, data sources and data collection frequency defined Page 21 of the CSS indicator document contains a number of additional indicators for consideration The list of CSS indicators should be updated and finalized to be ready for Global Fund Round 10 (May 2010)

Building Blocks, SDAs and indicators (1)

# and % of organizations working at the community level with a staff member responsible for monitoring and evaluation (CSS-ME1.1) # and % of organizations working at the community level using standard data collection tools and reporting formats that enable to report to the national reporting system (CSS-ME1.2) # and % of organizations working at the community level submitting timely, complete and accurate program reports according to nationally recommended guidelines (CSS-ME1.3) # and % of organizations working at the community level conducting reviews of their program performance in the past 3/6 months (CSSME1.4) # and % of organizations working at the community level with a costed annual work plan in place which includes monitoring and evaluation activities (CSS-SP2.1) SDA 2: Strategic planning # and % of organizations working at the community level with at least one staff member who received training or re-training in planning and M&E according to nationally recommended guidelines in the past 2 years (CSS-SP2.2)

SDA 1: Monitoring & evaluation BB1: Monitoring & Evaluation and Planning

Building Blocks, SDAs and indicators (2)

# and % of community health workers currently employed who received training or re-training in HIV, TB or malaria service delivery according to national guidelines in the past 2 years (CSSSB3.1) SDA 3: Community actors: skills building for service delivery # and % of currently employed staff members of organizations working at the community level that have been recruited more than 1 year ago (CSS-SB3.2) # and % of organizations working at the community level that received supervision in accordance with national guidelines in last 3/6 months (CSS-SB3.3) # and % of organizations working at the community level with a budget coming from more than one source (CSS-FR4.1) # and % of organizations working at the community level that submit timely, complete and accurate financial reports according to nationally recommended standards and guidelines (CSS-FR4.2) # and % of organizations working at the community level that are planning to implement resource mobilization activities (CSSFR4.3)

BB2: Resources and capacity building

SDA 4: Financial resources

Building Blocks, SDAs and indicators (3)

# and % of organizations working at the community level with the minimum infrastructure in place to deliver quality HIV, TB and malaria services (CSS-MR5.1) # and % of organizations working at the community level that have not experienced stock-outs of essential commodities in the last 3/6 months (CSS-MR5.2) % of organizations working at the community level that keep adequate logistics data for inventory management (CSS-MR5.3)

BB2: Resources and capacity building

SDA 5: Material resources infrastructure; information; essential medical and other products & technologies

# and % of organizations working at the community level with staff trained or re-trained in stock management in the past 2 years (CSS-MR5.4) # and % of organizations working at the community level that maintain acceptable storage conditions and handling procedures (CSSMR5.5)

Building Blocks, SDAs and indicators (4)

BB3: Organisational strengthening SDA 6: Management, accountability and leadership SDA 7: Building community linkages and collaboration for service delivery # and % of organizations working at the community level with staff who have received training or re-training in the past 2 years on management, leadership and accountability (CSS-MA6.1)

BB4: Community networks, linkages & partnerships

# and % of organizations working at the community level that are implementing activities in partnership with other organizations working at the community level or with the national health system (CSS-CL7.1)

BB5: Enabling environment

SDA 8: Communicatio n and advocacy for enabling environments and community engagement

# and % of organizations working at the community level that are planning to implement communication or advocacy activities to address barriers to equitable access for target populations (CSS-CA8.1) # of legal arrangements amended or put in place to build an enabling environment for community engagement (CSS-CA8.2)

Issues for discussion

Are the indicators useful for actors involved with CSS that aim at strengthening a limited number of organizations at the community level? Are there any considerations regarding the feasibility of data collection for the proposed indicators? If the list of building blocks and SDAs will be amended will this have an impact on the proposed indicators and how do we address this? How do we ensure that all actors involved with CSS have access to the indicators? Are there any suggestions on the next steps in the CSS indicator development process?

Asia feedback
1. Make the definitions more concise 1. Change from Community Systems Strengthening to Community SECTOR Strengthening (treat CSS as a strategy & not as a system) 2. Community Actors be changed to mean Key Affected Populations and grass-root community members. The framework should focus on GF programming A new set of 7 building blocks with 14 SDAs have been proposed The framework should be included in the expanded version of the M&E toolkit Technical support & assistance should be more local & peerbased

2. 3. 4. 5.

Proposed BB & SDAs

BB1: Sustainable funding BB2: Community sector workforce SDA1: Sustainable financial resources

SDA 2: Build community workforce and strengthen overall capacity of new and existing staff and volunteers SDA 3: Build technical skills for service delivery (of CBO staff and volunteers)

Proposed BB & SDAs

BB3: Organizational & management Strengthening SDA 4: Organizational/management support and training for small NGOs/CBOs

SDA 5: Transparent financial management structures BB 4: Planning and M&E SDA 6: Needs assessment, gap analysis, mapping SDA 7: Evidence-based strategic planning among community groups, linked to M&E systems. (including OR)

Proposed BB & SDAs

BB 5: Communication, Information & Networking SDA 8: Partnership building & collaboration, including networks of KAP

SDA 9: Information and knowledge sharing (including mechanism for information sharing, material development and resources) BB 6: Accountability and transparency SDA 10: Models & tools for community accountability & transparency to its constituency

Proposed BB & SDAs

BB 7: Social Mobilization & Advocacy SDA 11: Monitoring and documentation (of both community and government interventions) SDA 12: Advocacy, Campaigning & Community mobilization SDA 13: Enabling environment (policy, legal, governance) SDA 14: Reducing stigma & discrimination in all settings

Thank you

Covered in this presentation

1. Indicators for community systems strengthening 2. Challenges of community level monitoring and evaluation 3. Approaches to overcome these challenges 4. Feedback on the CSS framework from Asia