Health Facility Surveys and Quantified Supervisory Checklists

Health System Innovations Workshop Abuja, Jan. 25-29, 2010

Health Facility Surveys What are they?

Assessments of different types of health facilities using a standardized questionnaire Usually done through a simple random sample of all health facilities Surveyors are usually trained health workers (often doctors) Usually look at many different aspects of service delivery including technical quality of care
2

Health Facility Surveys

Advantages Can assess quality of care Can be independent of service providers Can be done more frequently than Disadvantages Complex to design Lots of data, can overwhelm Cannot provide information on coverage, equity
3

An example from Afghanistan:

600+ facilities surveyed every year 2004 to 2008 by a team led by JHU Contents developed through consultative process Very careful quality assurance Each facility rated on a score of 0-100, can be aggregated at county, state, national level Present results through balanced scorecard

What the BSC Looks At:

Presence of staff Knowledge of staff Quality of patient-provider interaction Availability of drugs and supplies (also quality on sample basis) Patient satisfaction (different from HH results) Waste management

Can Look at Provincial Progress Color Coded

80 75 70 65 60 55 50 45 40

32% Improvement in Total Scores in Contracted Facilities (from health facility survey)

M O P H A lo

P P A M e d ia

2004

2005

2006

2007

Looking at Provincial Progress on Total Score

Balanced Score Card Results from 2007 Compared to Previous Years

Province

2004 2005 2006 2007 Change from 2004 to 2007 48.7 59.3 49.8 80.2 55 53.4 71 60 59 71.6 78.6 62.8 75.8 65.4 70.2 31.5 23.6 22.4 17
8

Badghis Balkh PPA Median

National Median 53.2

Can Look At Areas Needing Attention

Index Patient Counseling Equipment Functionality Family Planning Availability Patient History & Exam Proper sharps disposal Obstetrical care BHC's with >750 patients HMIS Implementation Provider Knowledge Drug Availability 2004 2005 2006 2007 29.6 65.7 61.4 70.6 62.2 25.4 22.2 67.7 53.5 71.1 35.1 67 70 73.5 52 22.3 32.3 65.8 69 83.7 36.6 78.7 82.9 82.2 77.5 42.3 55 74.9 68.7 85.7 48.7 83.8 93.7 83.1 84.4 59.5 57.4 91.5 68.7 81 Change 2007-2004 19.1 18.1 32.3 12.5 22.2 34.1 35.2 23.8 15.2 9.9 9

Health Facility Assessment in Nigeria under Malaria + Program covering 327 facilities

Illnesses: Fever/Malaria; Pneumonia; Dysentery and Diarrhea

13

Poor awareness of PMVs regarding new Malaria treatment policy

What are the challenges with Health Facility Surveys? Deceptively difficult to do Requires talented technical staff experienced in survey design Need to do it every year or so to look at changes Costs about $300,000 per year (more during development)
17

What are the challenges with Health Facility Surveys?

Generates a lot of data (400+ questions on each facility) Tough to explain to managers need means, like BSC, to summarize data Quality assurance is a real challenge Easy to do badly consumers wont know

18

Quantitative Supervisory Checklist What is it?

A reduced version of a health facility assessment Objectively assesses a variety of indicators to come up with total score. Takes about 2-3 hours to complete A copy of results left in the health facility, easy to track progress QSC is both a management intervention and tool for M&E
19

Example of a Quantitative Supervisory Checklist

Date of Visit Availability of Drugs (010) Presence of staff (0-5) HMIS implementation (010) TB Case Detection Rate (0-5)
5/12 7/19 8/11 10/21

3 2 3 0

5 1 3 1 3 4

4 2 5 1 3 2

6 2 5 2 4 5
20

DPT3 coverage rate (0-10) 2 Consultations per capita 2

Development of QSC in the Philippines

New HMIS forms developed which were supposed to facilitate supervision Checklist Safari in 7 provinces found:
25 different checklists 95 items, average 4.5 pages long Rarely used, never found in health facilities Designed in such a way to make follow

21

Development of QSC in the Philippines

Discussions with key program managers led to definition of 20 indicators. Indicators scored from 0-3 with specific definitions and means of calculation Copy of QSC could be left in HF so future supervisors & staff could track progress Copy with supervisor so s/he could track which indicators were lagging Before & after assessments in 4
22

Example of a Quantitative Supervisory Checklist

Date of Visit Availability of Drugs (010) Presence of staff (0-5) HMIS implementation (010) TB Case Detection Rate (0-5)
5/12 7/19 8/11 10/21

3 2 3 0

5 1 3 1 3 4

4 2 5 1 3 2

6 2 5 2 4 5
23

DPT3 coverage rate (0-10) 2 Consultations per capita 2

% Change in Scores from Baseline

Evidence for the Effectiveness of QSC

24

Other Findings from QSC

Health workers liked it because it made it clear what was expected. Supervisors not angry Supervisors liked it because made interaction with HWs more focused on key results HWs tracked performance and became adept at tracking their own performance Was launched nation-wide but fell

25

Quantified Supervisory Checklists

Advantages Can assess QOC. Can be independent of service providers Can be done often Inexpensive Clarifies what is expected of HWs Can be adapted to conditions as Disadvantages Challenging to design Cannot provide information on coverage, equity Ensuring continued use is difficult

26