Randomized Trials

IGS
Treatment status

Two states of the world

1 if unit is treated
=
0 if unit is untreated

Assume that is observed for every unit

2
Potential outcomes

Each unit has two potential outcomes

1 if = 1

0 if = 0

Potential outcomes are fixed attributes of each unit

Treatment effect for unit : 1 0

3
Average treatment effect (ATE)

Average treatment effect (ATE): How outcomes would change

on average if every unit were to go from untreated to treated.

We are after
[1 0 ]

. . . but we only observe

[1 = 1 [0 | = 0]

4
Outcomes and treatments: Example

Treatment: Course attendance

Outcome: Final grade

Student A Student B

Untreated
0 20 28
potential grade
Treated
1 27 30
potential grade
Treatment status 1 0

Actual grade 27 28

Treatment effect 1 0 7 2
5
Selection Bias
Unobserved
in RED
1 0
= 27 28 = 1
= 1 0 + 0 0
= (1 0 ) + (0 0 )

Treatment effect Selection

for student A bias
= =
7 -8
6
 Selection Bias (contd)
Absent random assignment Unobserved
in RED

[1 = 1 [0 | = 0]

= [1 = 1 [0 = 1 + [0 = 1 [0 | = 0]

ATE among treated Selection bias

7
 Selection Bias (contd)
Absent random assignment Unobserved
in RED

[1 = 1 [0 | = 0]

= [1 = 1 [0 = 1 + [0 = 1 [0 | = 0]

ATE among treated Selection bias

8
Random assignment

Random assignment creates two or more

groups of units that are probabilistically
similar to each other on the average
Under random assignment, treatment status
is independent of the units potential
outcomes and their background
characteristics
0 , 1 ,

9
Random assignment (contd)

Random assignment creates two groups of units that are, in

expectation, identical prior to treatment
The treatment group is a random sample of the population
The expected potential outcomes of the treatment group
are identical to the average potential outcomes in the
population
The control group is a random sample of the population
The expected potential outcomes of the control group are
identical to the average potential outcomes in the
population

10
Random assignment eliminates selection bias

When is randomly assigned, 0 = 1 = 0 = 0

Therefore
[1 = 1 [0 | = 0]

= [1 = 1 [0 = 1 + [0 = 1 [0 | = 0]

ATE among treated Selection bias = 0

11
Random assignment and internal validity

Selection bias is the only internal validity

threat that random assignment removes
As for the other internal validity threats,
random assignment simply reduces the
likelihood that these threats are confounded
with treatment

12
Randomized Trials for Social Policy

13
The basic design

At least two conditions

Random assignment of units to conditions
Posttest assessment of units

R X O
R O

14
The basic design Example 1

A clinical trial in which a random subsample of

participants receive a new drug and the rest gets a
placebo (a pill with no pharmaceutically active
ingredients)

R X O
R O

15
The basic design Example 2

A clinical trial in which a random subsample of

participants receive a new drug and the rest get the
acknowledged gold standard drug that is already in
the market

R XA O
R XB O

16
The basic design Example 3

A clinical trial in which a random subsample of

participants receive a new drug, a random
subsample get the gold standard, and the rest of
participants receive a placebo

R XA O
R XB O
R O

17
Basic design and attrition

The lack of pretest is risky if there is any

likelihood of attrition We dont know
Were those who dropped out of the study
systematically different from those who
remained?
Were those who dropped-out of one condition
systematically different from those who
dropped-out of the other conditions?

18
Basic design and attrition

Attrition is very common in field experiments

Useful to collect pretest measures of the
outcome measures unless
Pretest in not possible or impractical
Pretest is a constant
High risk of testing bias

19
The pretest-posttest control group design

Pretest is added to the basic design, either

After random assignment
R O X O
R O O

Before random assignment

O R X O
O R O
20
The pretest-posttest control group design (contd)

Two main advantages over the basic design

Able to deal with attrition as a threat to
internal validity
Allows within-subject analyses (e.g., repeated
measures ANOVA) that increase statistical
power

21
Alternative-treatments design with pretest

If posttests reveal no differences between conditions, this

design allows testing whether both treatments were
effective or neither was

R O XA O
R O XB O

Particularly useful when there is no control group due to

ethical concerns

22
Multiple treatments and controls with pretest

It can be extended to include more than two

treatments and more than one control group

R O XA O
R O XB O
R O O

23
Multiple treatments and controls with pretest (contd)

In dismantling studies, similar treatments that differ by one

part are compared to assess which parts contributed most to
outcome
In parametric or dose-response studies, different groups
receive increasing doses of the same treatment (e.g., drug,
monetary incentive, hours of training)
Increasing the levels allows a more precise estimation of the
functional form of dosage effect
Testing a wide range of doses helps avoid the risk of confounding
constructs with levels of constructs

24
Solomon four-group design

R O X O

R O O
R X O
R O
versus : Did the actual act of pretesting influence the results?

History or maturation threats?

Interaction between treatment and pretest?

Testing?

25
Factorial designs

Two or more factors (i.e. IVs)

Each factor has two or more levels

R XA1B1 O
R XA1B2 O
R XA2B1 O
R XA2B2 O

The levels of the factor can include control conditions

26
Factorial designs (contd)

Main effect: Difference in outcome means over levels of

one factor collapsed over levels of the other factor
Simple main effect: A main effect of one factor at a given
level of a second factor
Two-way interaction effect: change in the simple main
effect of one factor over levels of the second factor
(moderator)
Three-way interaction effect: change in the two-way
interaction effect over levels of a third factor (moderator)

27
Factorial designs (contd)

Interactions are more difficult to detect than main

effects
Power analysis is essential to determine adequate
sample sizes
Larger (i.e. probability of Type I error) may be
warranted if the interaction is predicted by theory
(e.g., Snow 1991)
Deliberate oversampling of observations that are
extreme on the interacting variables provides
more powerful and still unbiased test
28
Factorial designs (contd)

Factorial designs with empty cells may be used

when assigning units to all possible combination of
factors (i.e. cells) is
Too expensive
Unethical
Uninteresting (e.g., some combinations have no policy
implications)
Less power but practical advantages that may
outweigh statistical complicatedness

29
Factorial designs: Advantages

1. Tend to be more efficient and require fewer units

than non-factorial designs because the same unit
is exposed to all factors simultaneously
Not always the case
Larger sample sizes may be required to detect interactions
between/among factors
If the outcome is a low base rate event and both factors reduce the
likelihood of that event, more units may be needed with a factorial
design than with a non-factorial design
2. Allow assessing the joint effect of different
factors

30