Part IIB, Paper 1: Policy Evaluation

Lecture 1

Kai Liu

October 2017

Kai Liu Policy Evaluation 1 October 2017 1 / 21

Policy Evaluation

Key questions in economic policy:

I What is the impact of a particular policy?
I What is the mechanism through which the policy operates?

Evaluating the causal effect of a policy on individual behaviour

or outcomes.
Need both economics and econometrics
Key aim to provide framework for analysing policy. Focus on
examples based on microeconomic choices, especially education
and labour supply issues.

Kai Liu Policy Evaluation 1 October 2017 2 / 21

Policy Evaluation: Examples

Does participation in A level booster courses improve A levels?

I Students who take A level booster courses do worse on A levels
than those who do not. Causality

Does going to university cause an increase in wages?

I What would have graduates earned otherwise Counterfactual.

Does the introduction of a minimum wage lead to increased

unemployment?
Does the availability of health insurance make individuals more
healthy?

Kai Liu Policy Evaluation 1 October 2017 3 / 21

Example: Postgraduate Training

How many are thinking of doing MPhils or Masters’ degrees?

Want to know what impact this will have on your life!

Look at historical earnings of graduates

(Terminology: doing a Masters is receiving a treatment and puts

the individual in a treatment group)

Kai Liu Policy Evaluation 1 October 2017 4 / 21

Example: Postgraduate Training
Basic regression:
ln Yi = α + βDi + i

If an individual receives the treatment (does an MPhil):

Di = 1, otherwise Di = 0
Outcome for the individual (earnings in this case): Yi
Key questions
1 How do we interpret the estimate β?
Percentage: β = 0.05 means 5% higher
2 Whose β?

Kai Liu Policy Evaluation 1 October 2017 5 / 21

Example: Postgraduate Training

What comparison group?

I Idea of a control group and establishing a counter-factual for
the particular person
I Counter-factual: what would the outcome have been for that
individual if they had not taken the MPhil? Counterfactual is
unobserved.
I Correlation versus causation: high wages of those who have
MPhils does not mean that taking an MPhil increases wages
I Can we use earnings of those who leave straight after getting an
undergraduate degree as a counter-factual?

Kai Liu Policy Evaluation 1 October 2017 6 / 21

Example: Postgraduate Training
Issues
I Different occupation choices (banking vs academia)
I Different degree classification / university (need good degree fro
postgrad)
I Different financial situation (need finance - family, borrowing,
scholarship)
I Different preferences over studying
I Different unobserved ability to work hard

How to disentangle these selection effects (those who did MPhils

differ from those that did not in some additional way)
from the causal effect of the “treatment” on any particular
individual
Distinguish observable differences from unobservable differences
Kai Liu Policy Evaluation 1 October 2017 7 / 21
Example: Postgraduate Training

Need both economics and econometrics to answer fully

Economics - basic model:
I Individuals choose their education to maximising lifetime utility
I Individuals differ their preferences and in their opportunity sets
I Model links preferences and opportunities into choices - why
does selection occurs and what is mechanism for causal effect

Econometrics of policy evaluation

I Focus on how to separate out the causal effect empirically

Kai Liu Policy Evaluation 1 October 2017 8 / 21

Outline of Course
1 Framework for analysis:
I Selection Bias
I Whose treatment?
2 Randomised Control Trials : directly removing all selection
issues?
I Randomly allocate MPhil places
I Internal vs External validity

3 Basic regression (removes selection on observables)

ln Yi = α + βDi + γXi + i

I Xi can include degree class, occupation choice

I Assumed linearity: Xi does not affect β
→ Will occupation choice affect value of MPhil?
Kai Liu Policy Evaluation 1 October 2017 9 / 21
Outline of Course

4 Instrumental Variables (removes selection on unobservables)

I Use a predictor of taking an MPhil uncorrelated with
unobservables (eg. preferences)
I What is an appropriate instrument?

5 Difference-in-differences (removes selection on unobservables)

6 Fixed effects (removes all individual characteristics that are

constant over time )

7 Regression discontinuity (focuses on marginal people)

Kai Liu Policy Evaluation 1 October 2017 10 / 21

Outline of Course

Helpful text: Angrist, J. and J-S Pischke “Mostly Harmless

Econometrics: An Empiricist’s Companion” (Princeton
University Press, 2009)

Focus on the intuition - this is not an econometrics course.

Relevant articles showing applications for each method -

available online and on moodle

Kai Liu Policy Evaluation 1 October 2017 11 / 21

Framework for Analysis: Potential Outcomes

If an individual receives the treatment (does an MPhil):

Di = 1, otherwise Di = 0
For each individual, only observe one of these two conditions

Define Gi = 1 if a member of the group of people who were

treated; Gi = 0 if not in that group

Outcome for the individual (earnings in this case): Yi

For individual i:
outcome if treated Y1i
outcome if not treated Y0i

Kai Liu Policy Evaluation 1 October 2017 12 / 21

Potential Outcomes
The effect of the MPhil for individual i is given by:
∆i = Y1i − Y0i

Take 2 individuals:
I j does the course, k does not
I For j : Dj = 1, observe outcome Y1j .
I For k : Dk = 0, observe outcome Y0k .
We observe
∆jk = Y1j − Y0k

We do not observe the counterfactual for either person

∆j = Y1j − Y0j
∆k = Y1k − Y0k
Kai Liu Policy Evaluation 1 October 2017 13 / 21
Potential Outcomes
Two ways to see this problem:
1 Missing data problem:
F For each person, we are missing the observation of either the
treatment or the non-treatment (control) outcome.
F For the MPhil example: we do not observe the wages that those
who did the MPhil would have got if they had not done the
MPhil
F We do not observe the wages that those that got jobs right
away would have got if they had done an MPhil

2 Selection bias:
F The people who are treated are different from those who are
not.
F Those who do the MPhil are different from those that do not

Kai Liu Policy Evaluation 1 October 2017 14 / 21

Missing Data Problem
We can write the problem as
The causal impact of the MPhil
z }| {
E [∆i | Gi = 1] = E [Y1i − Y0i | Gi = 1]
= E [Y1i | Gi = 1] − E [Y0i | Gi = 1]
| {z } | {z }
Observed Counterfactual

I E [Y1i | Gi = 1]: the outcome when treated for the people who
were actually in the treatment group
I E [Y0i | Gi = 1]: the outcome when untreated for people who
were actually in the treatment group
Observe from data, averaging across those treated:
E [Y1i | Gi = 1]
Missing data: E [Y0i | Gi = 1]
Kai Liu Policy Evaluation 1 October 2017 15 / 21
Selection Bias
Again, the causal impact:

E [∆i | Gi = 1] = E [Y1i − Y0i | Gi = 1]

= E [Y1i | Gi = 1] − E [Y0i | Gi = 1]
| {z } | {z }
Observed Counterfactual

I Don’t observe E [Y0i | Gi = 1].

Possible solution: use the outcomes of non-treated individuals

I Can we use E [Y0k | Gk = 0] to measure E [Y0i | Gi = 1]

E [∆i | Gi = 1] = E [Y1i | Gi = 1] − E [Y0k | Gk = 0]

| {z } | {z }
Observed treated Observed not treated

I Observable!
Kai Liu Policy Evaluation 1 October 2017 16 / 21
Selection Bias (2)

Does
E [Y0k | Gk = 0] = E [Y0i | Gi = 1]?
| {z } | {z }
Observed Counterfactual

I Requires: expected outcome when not treated is the same for

those in the treatment group and those not.
I Those who are treated are not being selected for a reason that
affects their outcome.

Hence, the selection problem

Kai Liu Policy Evaluation 1 October 2017 17 / 21

Another way to see selection problem
Object of interest - the causal impact:

E [∆i | Gi = 1] = E [Y1i | Gi = 1] − E [Y0i | Gi = 1]

Adding and subtracting E [Y0k | Gk = 0] :

E [∆i | Gi = 1] = E [Y1i | Gi = 1] − E [Y0i | Gi = 1]

−E [Y0k | Gk = 0] + E [Y0k | Gk = 0]

Rearranging:

E [∆i | Gi = 1] = E [Y1i | Gi = 1] − E [Y0k | Gk = 0]

−(E [Y0i | Gi = 1] − E [Y0k | Gk = 0])

Kai Liu Policy Evaluation 1 October 2017 18 / 21

Another way to see selection problem (cont.)
Observed treatment effect
z }| {
E [∆i | Gi = 1] = E [Y1i | Gi = 1] − E [Y0k | Gk = 0]
| {z }
True treatment effect

− (E [Y0i | Gi = 1] − E [Y0k | Gk = 0])

| {z }
Selection bias

We can estimate causal treatment effect by looking at observed

treatment effect ONLY if
E [Y0i | Gi = 1] − E [Y0k | Gk = 0] = 0

This term captures the selection bias

Those who do the MPhil would earn more than those that do
not anyway: causal effect less than observed effect
Kai Liu Policy Evaluation 1 October 2017 19 / 21
Summary

Course uses both econometrics and economics: focus is on

actually doing economics!
Mechanisms for evaluating policy require counter-factuals
Notion of a treatment and a treatment group
Selection into the treatment group on observables and selection
on non-observables
Key issue in evaluation is minimising selection bias:
I Finding an appropriate control group

Kai Liu Policy Evaluation 1 October 2017 20 / 21

Recommended reading for Wednesday - spend 1
hour!

Cobb-Clark and Crossley, Economic Record, 2003

Angrist and Pischke, chapter 2.
Deaton, Journal of Economic Literature, 2010

Kai Liu Policy Evaluation 1 October 2017 21 / 21