Wendy Walker and Claude Bodart EASS/EARD

Both nationwide programs designed and supported by ADB Both programs target 5% of the population
The food stamp program has designed and implemented a national Proxy Means Test (PMT) which is the basis for targeting in both programs.

Foodstamps

Response to impact of high inflation for the poor during food and oil crisis in 2008. The objectives of the Food Stamp program are to support the consumption of basic foods by extreme poor families. The Food Stamp program distributes benefits via foodstamps or electronic card to selected poor households so that they can purchase a specified number of high protein foods in the market place. Response to impacts of financial crisis (2010). The specific objectives of the Medicard program are to ensure access to health services of the poor by reducing the amount of out-of-pocket expenses paid by the poor, so that they seek medical care when needed and have access to discounted medicines.

Medicard

When fully rolled out nationally the programs are intending to reach about 18,000 households or over 100,000 people. The transfer in terms of food stamp is about USD 33 per household per month so that the program is a large percentage of food expenditure of eligible households. The Medicard program is open to the target population irrespective on their health insurance status.
For not insured poor the program will pay for outpatient, laboratory and diagnostic tests, impatient care at the secondary level of medical care and discounted medicines, and for insured poor it will pay for co-payments for most of medical care and for the cost of discounted medicines.

The purpose of the evaluation is to provide evidence to the government on the impact of the programs. It will provide the government with good indications of whether or not the program had achieved the expected outcomes or results In turn the evaluation results could also be used to decide whether to expand the coverage of the programs.

The selection of a control group is always a challenging task:

At the same time control and treatment groups must be comparable Control and treatment are chosen based on the targeting criteria: the proxy means test score The cut-off point for programme eligibility is exogenous and so households below and above the threshold should be ideal treatment and control groups

we need a control group because it is not enough to observe an improvement in the treatment group, but we also want to single out the specific improvement due to the programs

Non-equivalent control group design (Regression discontinuity design)

Baseline Treatment group 1 O1 (HHs in 5 % PMT score) Comparison group O3 (HHs in 5-10 % PMT score)

Period of one year

Endline FS and Medicard No treatment

O2 O4

Both treatment (beneficiary) and comparison groups received an initial observation (the pretests O1 and O3). treatment group was drawn from the eligible households in the bottom 5 % PMT score comparison group from the following 5 % PMT score (who is not eligible for food stamps or Medicard). The treatment group then receives the Medicard and FS programs, but the comparison group does not receive either of these programs. After the implementation period of 1 year is completed, a second set of measurement observations is made (O2 and O4).

We are assuming that the lowest 5% and the next 5% of households are similar in living standard. So we are expecting that O1 would be approximately equivalent to O3. This equivalence can be established by comparing O1 and O3. After both programs implementation, we are expecting O2 will be greater than O4 if the programs had an impact (looking at absolute and proportional change and also controlling for other factors). Because both groups were approximately equivalent at the beginning of both programs, any difference between the treatment and comparison groups will be attributed to the effect of implementation of both programs. This allows us to assess the impact of the two programs.

The Food Security analysis will use specially designed FANTA modules to measure food security. These modules should enable us to make cross country and even international comparisons of food security, ie. how the households in our sample compare to other households in food security issues. Validating the targeting. Under the food stamp program, we were able to get a Food Stamps module into the Household Socio-Economic Survey (a nationally representative survey, conducted independently by the National Statistical Office). Through these data we will be able to have national data on who is receiving or not receiving the food stamps and see to what extent they are poor based on the national distribution. This will enable us to determine whether the targeting by PMT and its implementation was effective or not. Not a lot of targeted programmes can assess the targeting method in this way using the national poverty survey. In the case of Mongolia, it was particularly good to achieve such cooperation between statistical office and Ministry of Social Welfare. The food stamps program has two rounds of qualitative impact monitoring built into the program. This will help to further contextualize the impact evaluation findings.

Status of impact evaluation work

Baseline: 1,000 hh surveyed. Completed in December 2011, exploiting the roll-out of the PMT data collection (which in October 2011 covered Ulaanbaatar, Orkhon and Selenge aimags)

Analysis: Ongoing Endline survey: Planned for OctoberNovember 2012. Analysis: Planned for January-February 2013. This is also when the HSES data will be available.

Not just a pilot and experimental real time and real programs Implementation of Food Stamps and Medicard in the sampled areas was delayed: households started to receive support only at the end of April/May and therefore they will be re-interviewed only after few months of treatment Threshold changes: The baseline sampled households on both sides of the PMT threshold. After the baseline data was collected, the Government decided to increase the threshold for foodstamps. This effectively meant that most of our control group was then targeted to receive the food stamps. These are the challenges of real-world quasiexperimental impact evaluations. New benefits with same target population : In June 2012 Government announced that part of a large cash distribution of mining revenue would be targeted to the same 5% households. Typical impact evaluations (that are done by JPAL and Innovations for Poverty Action) do not face such problems because in general, they control the intervention and the evaluation. At follow-up we will need to re-sample a control group (adding an extra 500 hh) because the initial control group was also partly covered by the programs. Treatment and control group will be compared using a regression discontinuity design.