Académique Documents
Professionnel Documents
Culture Documents
***************
FACULTY OF AGRONOMIC SCIENCES
***************
MSc. Thesis by
Eskender Girmay
2015-2016
Abstract
Regression discontinuity design (RDD) is originally a useful tool for evaluating a
program or policy when a single variable is used to determine program eligibility. The
RDD concept has been generalized to evaluate programs when eligibility is based on
multiple variables. Multivariate regression discontinuity design (MRDD) may provide
estimates of multiple parameters of interest (corresponding to the multiple possible
treatment contrasts and for different subpopulations) so that the analyst is faced with a
wider range of strategies for estimating treatment effects from a multiple rating score
regression discontinuity. The choice among these different strategies has important
implications for precision, bias, and generalizability. In this study, we compare the
relative performance of three MRDD estimation approaches (frontier, centering and
univariate) in Monte Carlo simulation study under different scenarios (different
dimensional space of the assignment variables, presence of white noise and presence
of correlation between score variables). Result attested that given correct model
specification with two assignment variables, all three approaches estimated treatment
effects without bias. Whatever the scenario, frontier approach provided unbiased
treatment effect and performed well even if the number of assignment variables
increases. However, when the dimension of the assignment variables exceeds two,
centering and univariate approaches yield biased treatment effect and reduced
efficiency.
iii
Table of Contents
Certification
Error! Bookmark not defined.
Abstract.iii
Acknowledgment
Error! Bookmark not defined.
Acronyms..E
rror! Bookmark not defined.
List of tables......vi
List of figuresvi
1 INTRODUCTION..1
1.1 Problem Statement .............................................................................................. 1
1.2 Objectives ............................................................................................................ 5
1.3 Research Questions ............................................................................................. 5
1.4 Research Hypotheses .......................................................................................... 5
2 PRINCIPLES OF REGRESSION DISCONTINUITY DESIGN...6
2.1 Regression Discontinuity Design with a Single Assignment Variable ........... 7
2.2 Regression Discontinuity Design with Multiple Assignment Variables ......... 9
2.3 Assumptions Required for MRDD .................................................................. 12
2.4 Estimation Strategies for Multivariate Regression Discontinuity Design ..... 15
2.4.1 Frontier Approach ........................................................................................ 17
2.4.2 Centering Approach ..................................................................................... 18
2.4.3 Univariate Approach .................................................................................... 19
2.5 Regression Discontinuity Designs in Practice ................................................ 21
3 MATERIAL AND METHODS26
3.1 Simulation Design for Monte Carlo Comparison of Approaches .................. 26
3.2 Model Specification and Data Generation with 2-4 Assignment Variables . 26
3.3 White Noise Imputation in the Data Generation Process ............................... 29
3.4 Correlation Inclusion between Assignment Variables ................................... 30
3.5 Comparison Criteria .......................................................................................... 30
4 RESULTS AND DISCUSSION...32
iv
4.1 Results................................................................................................................ 32
4.1.1 Effect estimates for MRDDs with different set of assignment variables . 32
4.1.2 Effect Estimates for MRDDs with different noise levels .......................... 37
4.1.3 Effect Estimates for MRDDs with different correlation among assignment
variables .................................................................................................................... 42
4.2 Discussion.......................................................................................................... 45
5 CONCLUSION.49
5.1 Conclusion ......................................................................................................... 49
5.2 Implications in practice .................................................................................... 50
6 REFERENCES..54
7 APPENDICES..57
Appendix I: R script for multivariate regression discontinuity design57
Appendix II: Graphical analysis of RDD using paramedic and non-parametric
method..62
v
List of tables
Table 1: Assumptions of regression discontinuity design ............................................ 15
Table 2: Summary of analytic approaches for analyzing sharp multivariate RDD .... 20
Table 3: MRDD estimates with different number of assignment variables for model
1(constant treatment effect) ............................................................................................. 35
Table 4: MRDD estimates with different number of assignment variables for model
2(with interaction treatment effect) ................................................................................. 36
Table 5: MRDD estimates at different noise level for model 1 with two assignment
variables (constant treatment effect) ............................................................................... 39
Table 6: MRDD estimates at different noise level for model 2 with two assignment
variables (with interaction treatment effect) ................................................................... 41
Table 7: MRDD estimates with different correlation among assignment variables
sample size for model 1(constant treatment effect) ....................................................... 43
Table 8: MRDD estimates with different correlation among assignment variables
sample size for model 1(with interaction treatment effect) ........................................... 44
List of figures
Figure 1: Discontinuity with 2 rating score variables ................................................... 14
Figure 2: Graphical visualization of treatment and control group ............................... 22
Figure 3: LATE estimation for indh dataset using parametric regression ................... 23
Figure 4: LATE estimation for indh dataset using non- parametric regression .......... 24
Figure 5: Sensitivity of the estimate to different bandwidths ...................................... 25
Figure 6: Treatment effect estimated using parametric regression .............................. 62
Figure 7: Treatment effect estimated using non parametric regression ....................... 62
vi
INTRODUCTION
1 INTRODUCTION
1.1 Problem Statement
Regression discontinuity design have become increasingly popular in
evaluating the impacts of social programs or in the economics literature. In a
traditional regression-discontinuity design (RDD), units are assigned to treatment and
comparison conditions solely on the basis of a single cutoff score on a continuous
assignment variable. Since the cutoff is arbitrary from the perspective of the unit of
intervention, units that are just eligible for the program or have values of the metric
close to the cutoff can be compared with units that are just not eligible, to measure
the local average treatment effect of the program.
However, units are frequently assigned to treatment and control group based
on more than one continuous assignment variables. More recent applications of RDD
in education have had multiple assignment variables and cutoff scores available for
treatment assignment. For example, Jacob and Lofgren (2004) examined the effects of
a remedial education intervention that was assigned to students based on missing a
reading cutoff, a math cutoff or both. (Gill, B., Lockwood, J. R., Martorell, F.,
Setodji, C. M., & Booker, K. 2007) examined the effects of schools failure to make
Adequate Yearly Progress (AYP) under a No Child Left Behind program by
missing one of 39 possible assignment criteria. Both are examples of the multivariate
regression discontinuity design (MRDD), where treatment effects may be estimated
across a multiple cutoff frontier, as opposed to a single point on the assignment
variable. Other MRDD examples in education research are by Kane (2003),
Matsudaira (2008), Papay, Murnane and Willett (in press), and van der Klaauw
(2002). With the exception of the latter, all of these studies analyze each of the
assignment rules separately. Multiple assignment variables in RDD are not unique in
education; they are also used frequently in other fields of research, such as evaluation
of labor market programs. Card, D., Chetty, R., & Weber, A (2007); Lalive and Lalive
(2008) in Marketing and other fields. In principle, they provide an opportunity to
obtain unbiased estimates of treatment effects, using the same logic as single rating
score RD designs.
Additional issue that may affect both centering and univariate approaches is
correlation between assignments variables. Treatment effect in a RDD is subjected to
scales of assignment variables (Wong et al. 2013) and thus to correlations between
assignment variables. In centering approach, the correlation structure of the data is not
taken into account in the dimension reduction process. In the univariate approach, the
presence of correlation affect the conditional distribution of individuals around cutoff
of each assignment variable. Consequently, the number of individuals at a particular
sub-frontier may decrease with the correlation and results in less accurate estimated
effects. On contrary, frontier approach takes into account the correlation structure of
the data and estimates should not be subjected to presence of correlation between
assignment variables. Among others, these three issues deserve further investigation
to guide researchers when dealing with MRDD.
1.2 Objectives
The main objectives of this study were to compare the relative performance of
three different MRDD estimation methods under different scenarios.
on a rating of their need for and for and/or ability to benefit from the intervention
(Bloom, 2009)
Since we never observe control and treatment cases at the cutoff, the causal
estimands is better defined in terms of the difference in limits of conditional
expectations as we approach the cutoff from below and above:
The second equality is with observed instead of potential outcomes. This holds
because we observe only the potential treatment outcomes below the cutoff, and only
the potential control outcomes above or at the cutoff. The difference in limits
California). In this case, students who score below the passing cutoff score on either
the test of academic English or the test of conversational English receive EL services;
students who score above the cutoff score on both receive standard instructional
services. In this case, although there are two rating scores used to determine treatment
assignment, there are only two distinct treatment conditions. In case of two
assignment variables, we partition the treatment space into three subspaces, we
assume that all cases receive exactly the same treatment (otherwise, more than one
potential treatment outcome needs to be considered). Although this is a fairly specific
implementation of a MRDD, the results presented here may also apply to MRDDs
where treatment and control conditions are swapped. The cutoff frontier is defined as
F {(r , m, p, q) : (r r , m m , p p , q q ) (r r , m m , p p , q q )
c c c c c c c c
(r r , m m , p p , q q ) (r r , m m , p p , q q )}
c c c c c c c c
At which the average treatment effect is estimated. Assuming complete treatment
compliance, the average treatment effect at the cutoff frontier is given by
Q frontiers. Let the difference in potential outcomes be G Y (1) Y (0) and the joint
i i i
density function for assignment variables at R, M , P and Q be f (r , m, p, q) then, we
can define the average treatment effect at the cutoff frontier F as the weighted average
of conditional expiations given the single frontiers F , F , F and F .
r m p q
E[G | ( R , M , P , Q ) F ]
RMPQD i| i i i i
where A denotes the k th assignment variable whose singular values are denoted
k
a ; and the cut point c ; A denotes the hyper-plan defined by all assignment
k k -k
variables restricted to values greater than their respective cut points; except a ; A
k -k
a -k denotes a singular point in A ; f (a |a c ) is the joint density function of all
-k j j j
assignment variables with the j th assignment variable fixed to c ; and
j
m m p p q q f r rc ,.m, p, q dmdpdq
c c c
W
R Dw
c c c
c c
D m m p p q q f r r ,.m, p, q dmdpdq r r p p q q
w c c
c c c c c
f r ,.m m , p, q drdpdq r r m m q q f R,.m, p p , q drdmdq
c c c
r r m m p p f R,.m, p, q qc drdmdp
c c c
m m p p q q g (r , m, p, q) f r rc , m, p, q dmdpdq
E[G | R F ]
R i i R
m m p p q q f r rc , m, p, q dmdpdq
c c c
2.3 Assumptions Required for MRDD
Given that the frontier average treatment effect may be decomposed into a
weighted average of frontier-specific effects, all required assumptions for the
traditional univariate RDD must be met for each discontinuity frontier (at all frontier)
for all design has two requirements: First, there must be a discontinuity in the
treatment probabilities at all frontier . Second, the expectations of potential outcomes
must be continuous at all frontier not necessarily along frontier F. With both single
and multiple rating score RD designs, it is assumed that the cutoff score(s)
determining treatment assignment is/are exogenously set; potential outcomes are
continuous functions of the rating score(s) at the cutoff score(s); and the functional
form of the model is correctly specified. Imbens and Lemieux (2012) provide a
detailed description of assumption checking in the single rating score RD context.
In order to assess the plausibility of the assumption that the cutoff scores are
exogenously determined, McCrary (2008) suggests (for the single rating score RD)
checking that the density of observations be similar on either side of the cutoff score.
In the MRDD design, it is useful to compare the density of observations near each of
the cutoff scores. For example, if there are two rating scores and four distinct
treatment conditions (as shown in Figure 1), this entails four Separate comparisons
A| B
(one comparing the density of observations on either side of frontier R , one
comparing the density on either side of frontier R B|C and so on). In general, if there
J 1
are J assignment and 2 J treatment condition, this will entails J * 2 separate
compassions. Evidence of discontinuity in the density of observations across any of
the frontiers suggests the possibility of manipulation of the rating scores and thus, a
failure of the erogeneity assumption.
be useful to include the other rating scores in the set of covariates X , as they should
not vary sharply at any frontier.
The third assumption is that the functional form of the model is correctly
specified. As Imbens and Lemieux (2008) point out, this is best done through visual
inspection of a plot of the Relationship between the outcome variable and the rating
score. For each of the methods that Reduce the multivariate rating score space to a
single rating score dimension (i.e., each of the methods except for the response
surface RD method), this can be done straightforwardly with the methods described
by Imbens and Lemieux (2008). When using the response surface RD approach,
however, identifying the correct functional form in a multivariate space through
graphical analysis can be difficult. In this case, we propose a visual inspection
technique for assessing the appropriateness of the functional form of the surface RD
model.
Note that if the functional form of the model is correct, the residuals should have an
expected value of 0 at each point in the multivariate rating score space. We check this
as follows. First, given a fitted surface RD model, predict the residuals. For each
A| B
frontier where there is a treatment condition discontinuity, (e.g., R in Figure 1),
consider the sample of observations that are in the adjacent regions of the rating score
space (e.g., regions A and B in Figure 1). Next, using this sample, construct a plot of
the residuals from the surface RD model against the single rating score that
determines treatment assignment at that frontier (e.g. R1 if we are examining the
A| B
frontier at R ) If the functional form of the model is correct, the residuals should
have a mean value of 0 at each value of the rating score. Most importantly, the
residuals should have a mean value of 0 as they approach the cutoff score from both
the left and the right. This can be checked by fitting separate nonparametric smoothed
curves through the data on either side of the cutoff score or by plotting the mean value
of the residual within small bins across the range of the rating score. We construct
these plots for each of the frontiers in order to assess the model fit over the entire
region of the sample. Deviation of the mean value of the residual from 0 particularly
at the cutoff score suggests the response surface RD model is not fit correctly in this
region of the data. Ideally, there should be no noticeable deviation from average
residuals of zero throughout the domain, though the most crucial region for no
deviation is around any cutoff score at which there is a treatment discontinuity. The
assumptions for traditional regression discontinuity design summarized in table 1.
RD factor Criteria
Cutoff Criterion must be followed without exception
Pre-post distribution must not be better explained with a curve
variances are the same across must have sufficient number of values
values of pretest
Continuous pretest distribution both groups (treatment and control) came
from same distribution
Equal treatment treatment administered in same way for all
participants
y f ( R1, R2 ,.....R J ) k T k X B
i i i i k i i i
Where {R1, R2 ,.....R J } D R Hence, R1, R2 ,.....R J are the J rating scores used to
i i i i i i
In case of MRDD with four assignment variables allows the estimation of five
different causal quantities: four frontier-specific effects ( , , and ) the
R M P Q
.
RMPQD
Step 1. Estimate the treatment function, which is the average size of the
discontinuity along the cutoff frontiers using parametric, semi-
parametric, or non-parametric approaches.
Step 2. Estimate the joint density function by using a kernel density estimator
or by estimating conditional density functions for each frontier
separately for observations that lie within a narrow bandwidth around
the frontier.
is contains all assignment variables and C is a vector contains cut points for each
assignment. For example, when we have 4 assignments variables RMPQ :
Z min( R r , M m , P p , Q q )
i c i c i c i c
Finally, apply standard RD analytic methods (e.g. local polynomial Regression) for
estimating treatment effects by using Z as the assignment variable and Zero as the
cutoff. The main advantage of the centering approach is that it allows the researcher
to Collapse scores from multiple assignment rules to a single assignment variable.
The approach also generalizes well to MRDDs with more than two assignment
variables, as Well as simplifies the analyses for estimating average treatment effects
across multiple discontinuity frontiers. However, it does not allow the estimation of
frontier-specific effects but that can be done by using whether frontier or univariate
approaches. The algorism of centering approach is presented in the following steps:
Step 1. For each unit i , center all assignment variables by their respective
cutoffs, that is ( X C ) where X matrix is contains all assignment variable
1i i
and C is a vector contains cut points for each assignment variables.
Step 1. Choose the first assignment variable and exclude all observations
values less than all cut point of assignment variables except the first
assignment variable so that the first assignment variable defined only
values greater than all cut point of assignment variables. Using the first
then analyze as a standard RD design with as the single assignment
variable. Example: if we have two assignment variable RM with rc and
mc cut point then exclude all observations with r values less than its
cutoff (m ).
c
Step 3. Estimates specific effect and the overall effect. To compute the overall effect,
take a weight of each assignment variables from frontier approach and
multiply each specific effect by its weight then add together to get the
overall effect.
ANALYTIC APPROACHES
Frontier Centering Univariate
First, model the response Center all assignment variables at Choose a single assignment
surface using a parametric, their respective cutoffs and assign variable and cutoff, excluding
semi-, or nonparametric each unit their minimum all observations that are
method, and then use kernel (or maximum) centered assigned to treatment via the
density estimates for assignment score. Use the other assignment variables and
integrating the treatment minimum assignment scores as the cutoffs. Analyze with
effects along the cutoff sole assignment variable and traditional univariate RDD methods.
Procedure
Frontier ATE ( overall effect) Only frontier ATE (overall effect ) Frontier ATE (overall effect) and
and frontier-specific ATEs frontier-specific ATEs
Estimands
content and
metric.
Correct modeling of the entire Correct modeling of the Correct modeling of the
response surface. bivariate outcome regression. bivariate outcome regression.
Nonparametric methods Nonparametric methods require an Nonparametric methods require an
require an optimal bandwidth optimal bandwidth choice and an optimal bandwidth choice and an
Requirements
Cutpoint: 30
Sample size:
-Full : 720
-Left : 362
-Right: 358
Covariates: no
The summary statistrcies shows all description of our dataset. The total number=720.
From the 720 individuals, 362 assigned to treatment group and 358 assigned to
Control group based on the cut point.
Plot the house_rdd dataset, whether to return a plot. Logical, default ot TRUE
>plot(rdd_dat_indh)
1.0
0.8
object$y
0.6
0.4
0.2
object$x
h=0.0485,\tn bins=39
>reg_para
### RDD regression: parametric ###
Polynomial order: 4
Slopes: separate
Number of obs: 720 (left: 362, right: 358)
Coefficient:
Estimate Std. Error t value Pr(>|t|)
D 0.22547 0.17696 1.2741 0.203
0.6
0.4
0.2
dat$x
h=0.05,\tn bins=41
Coefficient:
Estimate Std. Error z value Pr(>|z|)
D 0.144775 0.095606 1.5143 0.13
Using the function rdd_bw_ik(), we estimate the bandwidth for the non-parametric
model for our analysis so the Bandwidth is 0.79, for rdd analysis, we use individuals
only
Inside the bandwidth then we estimated the potential outcome of the two treatment
conditions.and visualizing the non-parametric estimation. The local average treatment
effect is the RDD parameter of interest. Therefore, estimated 0.144, which was a
positive impact but not significant at alpha level 0.05. So the giving grant to local
population under SNSF project no impact on addressing development needs to the
building a school or improving local water services.
>plot(x=reg_nonpara)
0.9
0.8
0.7
0.6
dat$y
0.5
0.4
0.3
dat$x
h=0.1581,\tn bins=18
Figure 4: LATE estimation for indh dataset using non- parametric regression
Regression Sensitivity tests: One can easily check the sensitivity of the estimate to
different bandwidths.
>plotSensi(reg_nonpara, from=0.05, to=1, by=0.1)
0.5
LATE
0.0
-0.5
Sensitivity of the estimate to different bandwidths. The graph shows that the
estimated local average treatment effect is less sensitive to bandwidth.
Model 1: Y T R M
i i 1 i 2 i i
Model 2: Y T R M R M
i i 1 i 2 i 3 i i i
The outcome for unit is a simulated based on two different specifications of the
true response surface using three Assignment variables.
Model 3: Y T R M P
i i 1 i 2 i 3 i i
Model 4: Y T R M P R M R P M P
i i 1 i 2 i 3 i 4 i i 5 i i 6 i i i
The outcome for unit is a simulated based on two different specifications of the
true response surface using four Assignment variables.
Model 5: Y T R M P Q
i i 1 i 2 i 3 i 4 i i
Model 6: Y T R M P Q R M R P
i i 1 i 2 i 3 i 4 i 5 i i 6 i i
R Q M P M Q PQ
7 i i 8 i i 9 i i 10 i i i
score less than p but R , M and Q assignment score greater than r , m and q
c c c c
(and 0 if otherwise). T equals 1 if unit i has an Q assignment score less than q but
c
an R , M and P assignment score greater than r , m and p (and 0 if otherwise).
c c c i
is a normally distributed error term with a mean zero and a standard deviation 2.
Model 1, 3 and 5 (model with constant treatment effect) shows a constant treatment
with no treatment by assignment variable interactions and level changes in the
treatment response surface and Model 2, 4 and 6 (heterogeneous treatment effects
model) defines heterogeneous effects based on continuous potential treatment
outcomes. In this model, effects along F , F , F and F are heterogeneous due to
R M P Q
interactions among assignment variables.
Given the above data generating equations and the distribution of assignment
variables, we first computed for each model the true treatment effects for frontiers.
These theoretical effects serve as our benchmarks for comparing the estimates
produced by the three approaches. All assignment variables are normally distributed
with the same standard deviation of 10, but have different means and cutoffs. For R
assignment variable, the mean is 45 and the cutoff is 40. For the M assignment
variable, the mean is 55 and the cutoff is 60. For P assignment variable, the mean is
50 and the cutoff is 45 and for the Q assignment variable, the mean is 50 and the
cutoff is 65.
The goal of the Monte Carlo study with 500 simulated samples of size 5,000 is
to evaluate the unbiasedness of the proposed approaches for estimating the true
treatment effects. Due to computational reasons, we do not directly investigate
variance estimators of the treatment effects (for each approach, standard errors may
be bootstrapped). However, for assessing the relative efficiency of the three
approaches, we report standard errors estimated from our simulation, which is the
standard deviation of estimated treatment effects across the 500 iterations,
2
i 1( i ) where is the average of the estimated effects and i is the
s 500
simulation standard deviation may be considered as an a average standard error
bootstrapped from the overall target population instead of the actual samples). To test
the unbiasedness of an estimated treatment effect with respect to its corresponding
Step 5. Generate the outcome variable, Yi using a defined model. Example: (model 1
the computed average of estimates. Hence, to assess the relative efficiency of the
three approaches, we report standard errors estimated from our simulation, which is
the standard deviation of estimated treatment effects across the 500 iterations,
2
i 1( i ) where is the average of the estimated effects and i is the
s 500
simulation standard deviation may be considered as an a average standard error
bootstrapped from the overall target population instead of the actual samples). To test
the unbiasedness of an estimated treatment effect with respect to its corresponding
true effect, we use simulation Student t-test with standard errors s / 500 . We use t-
test to determine the unbiasness of estimated effect because our data follows normal
distribution. Significant differences between estimated treatment effects and true
effects (at the .05 error level) are indicated by asterisks in the tables (simulation
standard errors are not presented in the tables).
Table 3). With four assignment variables. The frontier approach estimated the
treatment effects 8.00, 7.98, 7.99 and 8.00 at FR , FM , FP and FQ frontiers and 7.99 for
the overall effect, which is similar to the true effect estimated. Centering approach
estimated the overall effect 8.00. However, there is significant difference between the
estimated true effect and treatment effect estimated by the centering approach.
Moreover, standard errors produced by centering approach were very large compared
to that of the frontier approach. The univariate approach estimated the treatment
effects of 7.30, 7.79, 9.03 and 7.95 at FR , FM , FP and FQ frontiers and 7.95 of the
overall effect. Besides producing an estimated treatment effect with a very large
standard error (2.82) , there also was a significant different between the estimated true
effect and treatment effect estimated by the univariate approach. In General, the
frontier approach continues estimating unbiased treatment when the number of
assignment variables varies from two to four, while centering and univariate
approaches fails to detect the effect with four assignments variable. As can be seen
from Table 3, standard errors produced from all approaches increases proportionally
with increasing number of assignment variables. By examining these aforementioned
approaches in three differential dimension of assignment rules, we found that the
frontier approach yields the most precise estimates than the centering and univariate
approaches. While, the centering and univariate approaches fails to estimate the effect
with more than two assignment variables.
The first line of the column panels in Table 4 represents the theoretical
treatment according to the data-generating model (see Model 2). The second line
shows the true effects is 8.00. Table 4 shows result for MRDD estimate with different
number of assignment variables for model 2 (data generated by Model 2). First, we
considered MRDD with two assignment variables, Frontier estimated 8.00 and 8.04 at
FR and FM frontiers, and 8.03 for the overall effect with very a small standard error.
When MRDD consists of three assignment variables on the same scale and the same
distribution distributions, frontier approach produced estimations of 7.99, 7.98 and
8.00 for R , M and P frontier respectively with an overall effect estimated of 7.99
and a very small standard error. Centering and univariate approach estimated the
overall effects of 16.38 and 44.1 respectively, both of which are different from the
true values. Therefore with references to Models 1 and 2, centering and univariate
approaches fails to detect effects with three assignment variables whereas the frontier
approach qualified in estimating unbiased treatment effect with a very small standard
error.
Table 3: MRDD estimates with different number of assignment variables for model 1(constant treatment effect)
method of analysis
Number of Weights & effects wR wM wP wQ R M P Q RMD
assignment
variables.
True effect 8.00 8.00 8.00 8.00 8.00
Table 4: MRDD estimates with different number of assignment variables for model 2(with interaction treatment effect)
method of analysis
Number of Weights & effects wR wM wP wQ R M P Q RMD
assignment
variables.
True effect 8.00 8.00 8.00 8.00 8.00
Notes. Weights wR and wM are given in percent (WR WM 100) Standard errors (estimated from the simulation) are in parenthesis. * indicates statistical
difference (p < .05) between estimated and theoretical treatment effects (based on simulation standard errors).
It is true that extreme observations can increase variance and lead to bias
estimation. Table 3 and 4 shows the theoretical treatment effect according to the data-
generating for model 1 and 2 respectively (see Models 1 and 2 for two assignment
variables) at different noise level. The second line of each table shows that the true
effects are 8.00 .The first panel of the Table 3 shows the results for two assignment
variables on the same scale and same distribution at 0% noise level. That means the
data contains no extreme observation at all. All approaches provided treatment effects
similar to the true treatment effect. However with different standard errors. The
frontier approach estimates the effect with smaller standard error than centering and
univariate approaches. Hence all approaches estimated unbiased treatment effect
similar to the true effects, with frontier having higher precision at noise level zero.
When the noiseWe increase noise level to 5%, meaning the data contains 5% extreme
observations from the total sample size. First, we consider noise level at 5% and
frontier approach results in treatment effects between 8.04 and 8.01 at FR and FM
frontiers, and estimated 8.01 for the overall effect. Centering approach estimated 7.27
the oval effect with very a large standard error (33.7) and the overall effect estimated
by centering approach is quite different from the true effect so that means centering
approach fails to detect the effect at 5% noise level. The univariate approach results in
treatment effects between 10.55 and 10.80 at FR and FM frontiers, and estimated
10.71 for the overall effect with a large standard error but statistically similar
estimation with the true effect. Next, we consider noise level at 10% and frontier
approaches estimated the specific effect 7.94 and 7.87 at R and M frontier with very
small standard error. Frontier approach estimated the overall effect similar estimation
with noise level at 5% but the standard error is slightly larger than standard error at
5% so this is due to large number of extreme observations. Treatment effect estimated
from centering approaches is quite different from frontier and univariate centering
approach estimated. Centering approach provided the overall effect 4.51 with large
standard error (35.06). Therefore, the overall effect estimated from centering
approach is different from the true effect when we compare the true effect and the
estimated treatment effect from centering approach using t test. Univariate approach
estimated 8.17 and 8.59 for each frontier and 8.47 for the overall effect. When we
compare estimated effect from univariate and frontier approach, frontier approach
gives result more closely to true effect with small standard error. Table 4 shows
MRDD estimates at 25% noise level for model 2 with two assignment variables. The
Frontier approach results in treatment effects between 7.98 and 7.80 at R and M
frontiers, and 7.87 for the overall effect. There is no change in estimated the overall
effect at 5, 10 and 25 percent noise level but the standard errors are increasing slightly
when the noise level increase. The overall effect estimated from the Centering
approach is -0.64 which is quite different from the true effect and with the standard
error (35.53) however the negative sign doesnt mean that centering approach
estimated negative effect. The estimation value from all approaches depend on the
location of the true value so when the location of the true value is far from zero, the
sign of estimated effect from all approaches could also change. The univariate
approach estimated each specific effect 4.90 and 3.6.9 for R and M frontier
respectively and the overall effect is 4.13 with a large standard error (35.94) and
which implies that the overall effect and the specific effect estimated from univariate
approach are different from the true value. Hence, result from the simulation study
indicated that centering and univariate approach are sensitive to extreme observations.
Table 5: MRDD estimates at different noise level for model 1 with two assignment
M : 55, 10 cutoff 60
Noise Weights & effects wR wM R M RMD
level
True effect 8.00 8.00 8.00
Notes. Weights wR and wM are given in percent (WR WM 100) Standard errors (estimated
from the simulation) are in parenthesis. * indicates statistical difference (p < .05) between
estimated and theoretical treatment effects (based on simulation standard errors).
Table 6: MRDD estimates at different noise level for model 2 with two assignment
variables (with interaction treatment effect)
M : 55, 10 cutoff 60
Noise Weights & effects wR wM R M RMD
level
True effect 8.00 8.00 8.00
Notes. Weights wR and wM are given in percent (WR WM 100) Standard errors (estimated
from the simulation) are in parenthesis. * indicates statistical difference (p < .05) between
estimated and theoretical treatment effects (based on simulation standard errors
4.1.3 Effect Estimates for MRDDs with different correlation among assignment
variables
M : 55, 10 cutoff 60
Correlation Weights & effects wR wM R M RMD
Notes. Weights wR and wM are given in percent (WR WM 100) Standard errors (estimated
from the simulation) are in parenthesis. * indicates statistical difference (p < .05) between
estimated and theoretical treatment effects (based on simulation standard errors).
M : 55, 10 cutoff 60
Correlation Weights & effects wR wM R M RMD
Notes. Weights wR and wM are given in percent (WR WM 100) Standard errors (estimated
from the simulation) are in parenthesis. * indicates statistical difference (p < .05) between
estimated and theoretical treatment effects (based on simulation standard errors).
4.2 Discussion
Our analytic and simulation work highlights the complexities of choosing an
appropriate causal estimation method in multivariate regression discontinuity design.
In many cases, the frontier average treatment effect may not have a meaningful
interpretation because it does not make sense to pool effects across multiple frontiers.
If at one frontier, the estimate indicates no effect and at the other frontier, a significant
positive effect, then the average effect across the entire frontier rests dependent on
weights of each frontier. In these cases, we recommend that researchers estimate
frontier-specific effects because and can provide at least upper and lower bounds for
the overall treatment effect. In addition, without strong assumptions (e.g., constant
treatment effects), the frontier-specific effects and is less general than what would be
obtained from a traditional univariate RDD with a corresponding assignment variable
and cutoff. That is because the cutoff of a traditional RDD is not restricted by the
cutoffs of additional assignment variables (e.g., units with M m excluded for
i c
estimating treatment effects at F ). Still, the presence of multiple cutoff-frontiers has
R
the advantage of exploring the heterogeneity of treatment effects along different
dimensions.
test for both measurement periods has to be on the same metric. Two different reading
tests (e.g., for different grades), or a reading and a math test are not on the same
metric, even if their scores are calibrated to appear comparable. This is because the
test scores similarities are typically the result of deliberate standardization
procedures. Thus, the frontier average treatment effect and its interpretation would
depend on the calibration of test scores. researcher can then assess whether treatment
effects are constant across both cutoff frontiers, and if so, use the frontier or centering
approach to estimate an overall effect In general, the frontier approach performs well
in estimating treatment functions are correctly specified.
It also has the advantage of improved statistical efficiency for both the overall
and frontier-specific effects. However, because the true functional from of the
response is hardly ever known in practice, researchers may consider using the
centering approach for estimating complication of modeling a multi-dimensional
response surface. The issue here is that the centering approach may be prone to small
biases due to wider bandwidths and more complex functional forms caused by
pooling units from different cutoff frontiers. However, the bias might be mitigated by
using difference scores as the outcome whenever pretests are available to reduce the
complexity of the functional form and heterogeneity of the dependent variable.
scores as the outcome whenever pretests are available to reduce the complexity of the
functional form and heterogeneity of the dependent variable.
In this study, we have assumed that units are assigned to a single treatment
condition via two assignment variables and cutoffs, but this need not be the case.
Units may be assigned to one treatment condition for missing the R cutoff, another if
they miss the M cutoff, and both if they miss the R and M cutoffs together. This type
of MRD design raises several practical considerations for the researcher. First,
although the frontier and univariate assignment variable approaches can estimate
unbiased frontier-specific effects. In MRDD with multiple treatment conditions raises
questions about whether estimating a frontier average treatment effect is appropriate
and substantively interpretable given that it averages estimates across two unique
treatments. Third, treatment contrasts in an MRD design are limited to comparisons
along the cutoff frontiers. There may be cases, however, when the desired treatment
contrast is to compare outcomes from units that received the strongest treatment
dosage with those that received no treatment at all. In our example, units in quadrant
T2 might receive a stronger dosage of treatment because they missed both the R and
M cutoffs. However, the MRDD does not include these observations in the calculation
of treatment effects because they are not located at FM or FR. A researcher may
choose to redefine observations in Treatment units, and those in the remaining three
quadrants as the comparison cases. However, this treatment contrast would involve
comparing outcomes for units that missed both the R and M cutoffs (treatment cases)
with those that missed only one cutoff (comparison cases). Again, this may not be the
desired treatment contrast because treatment units are compared to those that have
received at least some treatment because they missed either the M or R cutoffs. This
predicament highlights one of the main design disadvantages of the MRDD, and one
that researchers should be aware of as they interpret their treatment effects.
Although this master thesis describes a set of strategies for analyzing data
from MRD designs, several issues in the analysis has limitations and some designs
deserve further attention. While, our study here has focused on the context of the
sharp MRD design, where we assumed no instances of treatment misallocation.
However, in many applications of MRDD, treatment crossover and no-show are likely
to occur. Traditionally, fuzziness around the RD cutoff is addressed by using the
assignment mechanism as an instrument for treatment receipt. Second, our study is
limited by the fact that frontier approach takes time to estimate treatment effect
because high dimensionality integral computation when computing average treatment
effects and weights for each assignment variables, besides, there is no statistical
package that handles frontier approach. So further work is need on development of
algorithms for frontier approach on the application of parametric and non-parametric
methods. Finally, our simulation did not examine the variation of distribution between
assignment variables, further work is needed for examining whether there are special
analytic issues and requirements for analyzing MRDDs with different distribution
between assignment variables.
5 CONCLUSION
5.1 Conclusion
Multivariate regression discontinuity designs are common in education and
many other fields. In principle, they provide an opportunity to obtain unbiased
estimates of treatment effects, using the same logic as single rating score RD designs
(Wong et al.2013). However, MRD designs contain some added complexity not
present in single RD designs. In particular, they present the researcher with multiple
possible estimands and multiple estimation strategies.
Regardless of the strategy used, researchers should assess the extent to which
the data appear to support the assumption of exogenous determination of the rating
scores and cutoff scores, the assumption that potential outcomes are continuous as the
cutoff scores, and the assumption that the functional form of the model is correct
(Wong et al.2013). Each of these assumptions are most easily assessed using frontier
RD methods, because sub setting the data by frontier reduces the assumption checking
Although this study described a set of strategies for analyzing data from MRD
designs, several issues in the analysis has limitations and some designs deserve
further attention. While, our study here has focused on the context of the sharp MRD
design, where we assumed no instances of treatment misallocation. However, in many
applications of MRDD, treatment crossover and no-show are likely to occur.
Traditionally, fuzziness around the RD cutoff is addressed by using the assignment
mechanism as an instrument for treatment receipt. Second, our study is limited by the
fact that frontier approach takes time to estimate treatment effect because high
dimensionality integral computation when computing average treatment effects and
weights for each assignment variables, besides, there is no statistical package that
handles frontier approach. So further work is need on development of algorithms for
frontier approach on the application parametric and non-parametric methods. Finally,
our simulation didnt examine the variation of distribution between assignment
variables, further work is needed for examining whether there are special analytic
issues and requirements for analyzing MRDDs with different distribution between
assignment variables
Section 2: Our next recommendation deals RD data with some extreme observations.
Data are likely to contain some extreme observation. Our simulation study shows that
at 0% noise level, frontier, centering and univariate approaches produced similar
treatment effect and relatively similar with the true effect estimated. In addition, at
5% noise levet al.l approaches estimated treatment effect similar with treatment effect
estimated at 0%. Centering and univariate approach fails to detect the effect when the
noise level exceeds 5 percent. However, frontier approaches estimated the effect
without bias and the estimated treatment effects were similar to the true effect
estimated. Thus, frontier approaches is robust in the presence extreme observation
while centering and univariate approach are very sensitive to extreme observations.
Computing treatment effect using frontier approach with more than two assignment
variables was challenging and computational intensive. Therefore, our simulation
study was limited to two assignment variables. Due to those reasons, our
recommendation restricted only RDD with two assignment variables. Therefore, we
recommend to use frontier approach to estimate frontier specific and overall effect.
6 REFERENCES
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J., & Hsuing, S. "Comparison of a
randomized and two quasi-experimental designs in a single outcome
evaluation: Efficacy of a university-level remedial writing program."
Evaluation Review 22 (1998): 207-244.
Angrist, J. D., & Lavy, V. "Using Maimonides' rule to estimate the effect of class size
on scholastic achievement." Quarterly Journal of Economics 144 (1999): 533-
576.
Baldauf, M. and J.M.C. Santos-Silva. "on the use of robust regression in
econometrics, ." , Economics Letters , 2012.
Berk, R. A., & Rauma, D. " Capitalizing on nonrandom assignment to treatments:A
regression-discontinuity evaluation of a crime-control program." Journal of
theAmerican Statistical Association 78 (1983): 21-27.
Berk, R., Barnes, G., Ahlman, L., & Kurtz. "When second best is good enough: A
comparison between a true experiment and a regression discontinuity
quasiexperiment." Journal of Experimental Criminology, n.d.
Bloom. "Modern regression discontinuity analysis." New York, NY: MDRC. , 2009.
Buddelmeyer, H., & Skoufias, E. "An evaluation of the performance of regression
discontinuity design on PROGRESA." World Bank Policy Research Working
Paper No. 3386, 2004.
Card, D., Chetty, R., & Weber, A. "Cash-on-hand and competing models of
intertemporal behavior: New evidence from the labor market." Quarterly
Journal of Economics 122(4) (2007): 1511-1560.
Cook, T. D., Wong, V. C., Steiner, P. M., Taylor, J., Gandhi, A., Kendziora, K., et al.
"Impacts of School Improvement Status on Students with Disabilities."
American Institutes for Research. , 2009.
Dimmery, D. "RDD: Regression Discontinuity Estimation R package ." , 2013.
Ellen Moscoe, Jacob Bor and Till Barnighausen. "Regression discontinuity designs
are underutilized in medicine, Epidemiology, and public health: a review of
current and best practice." Journal of Clinical Epidemiology 68 (2014): 132-
143.
Fan, J. "Design-Adaptive Non-Parametric Regression." Journal of the American
Statistical Association 87 (420) (1992): 998-1004.
Finkelstein, M., Levin, B., & Robbins, H. "Clinical and prophylactic trials with
assured new treatment for those at greater risk: I. A design proposal." Journal
of Public Health 5 (1996): 691-695.
Gelman, A. and Imbens, G. "Why High-order Polynomials should not be Used in
Regression Discontinuity Designs." National Bureau of Economic Research,
2014.
Gill, B., Lockwood, J. R., Martorell, F., Setodji, C. M., & Booker, K. " State and local
implementation of the No Child Left Behind Act. U.S. Department of
Education, Office of Planning, Evaluation and Policy Development, Policy
and Program Studies Service." 2007.
Hahn, J., Todd, P., & van der Klaauw, W. "Identification and estimation of treatment
effects with a regression-discontinuity design." Econometrica 69(1) (2001):
201-209.
Holland, P. W. "Statistics and causal inference." Journal of the American Statistical
Association 81 (1986): 945-968.
Imbens, G. and Kalyanaraman, K. "Optimal bandwidth choice for the regression
Discontinuity estimator." The Review of Economic Studies 79 (2012): 933-
959.
Imbens, G. W., & Lemieux, T. "Regression discontinuity designs." A guide to
practice. Journal of Econometrics 142(2) (2007): 615-635.
Jacob, B., & Lefgren, L. "Remedial education and student achievement: A regression-
discontinuity analysis." Review of Economics and Statistics LXXXVI (2004):
226-244 .
Jacob, Robin. "A Practical Guide to Regression Discontinuity." press from University
of Michigan , 2012.
Kane, T. J. "A quasi-experimental estimate of the impact of financial aid on college-
going." Working Paper 9703, National Bureau of Economic Research, 2003.
Lalive, R. "Unemployment benefits, unemployment duration, and postunemployment
jobs: A regression discontinuity approach." American Economic Review 97(2)
(2008): 108-112.
Lee DS, Lemieux T. "Regression discontinuity designs in economics." J Econ Lit
48(2) (2010;): 281-355.
Lee, D. S., & Lemieux, T. "Regression discontinuity designs in economics." NBER
Working Paper, 2009: 14723.
Matsudaira, J. " Mandatory summer school and student achievement." Journal of
Econometrics 142(2) (2008): 829-850.
McCrary, J. "Manipulation of the assignment variable in the regression discontinuity
design: A density test." Journal of Econometrics 142(2) ( 2008): 698-714.
Ou, D. " To leave or not to leave? A regression discontinuity analysis of the impact of
failing the high school exit exam." Economics of Education Review 29(2)
(2009): 171-186.
Papay, J. P., Murnane, R. J., & Willett, J. B. " The consequences of high school exit
examinations for low-performing urban students: Evidence from
Massachusetts." Educational Evaluation and Policy Analysis 32(1) (2011): 5-
23.
7 APPENDICES
Appendix I: R script for multivariate regression
discontinuity design
####### 1. Frontier approach #####
# Comments
## Input arguments
#y : response/outcome variable
# X : numeric n-by-d matrix, columns representing assignments variables
# If y is null, the first column of X is taken for y and removed from X.
# cutpoints: numeric scalar/vector of cutpoint(s) of assignment variables in X.
# If null or not provided, a vector of zeros is used.
# If cutpoints has length < d, an error is returned. If It is a scalar,
# it is replicated d times. If length(cutpoints) > d, only the d first
# elements are retained.
# bandwidth: numeric scalar/vector of bandwidth(s) to be used for local regression
# and density estimation. When omitted or null, the function
# uses (default) Imbens-Kalyanaraman optimal bandwidths.
# For d > 1, bandwidths are averaged.
# kernel : string, kernel to be used for local regression. The default is
# "triangular". See RDestimate in rdd package for other possible choices
# stand : logical or (0/1), Should variables be standardized?
# (default is FALSE)
#Treated : logical, Should a vector indicating treated individuals be returned ?
# (default is FALSE)
## Outputs
mardd <- function(y=null, X, cutpoints=NULL, bandwidth=NULL,
kernel = "triangular", stand=0, Treated=FALSE)
{mrequire(rdd, quietly=TRUE, warn.conflicts=FALSE)
## (1.0) Check data type
if (!is.numeric(X)) stop("Design matrix X must be of numeric class.")
X <- as.matrix(X);n <- size(X,1); d <- size(X,2); # d = Number of assignment variables; n = sample size
if (is.null(y)) { # The first column of X is taken as response if y is not provided
if ( d >= 3) {y <- X[,1]; X <- X[,-1];d=d-1;}else{ # and more than 2 assignments were given
stop("At least two assignment variable are needed with a response variable.")}}else {
if (d>2){
Q[i] <-cuhre(integrand=cdensity,ncomp=1,ndim=(d-1),lower=lowerLimit,upper=upperLimit, abs.tol=1e-03,
flags=list(verbose=0, final=0, pseudo.random=0, mersenne.seed=NULL) )$value
} else {
Q[i] <- integrate(f=cdensity, lower=lowerLimit, upper=upperLimit)$value
}
# Next, Integrate the product of conditional density and difference in potential outcome function
# Product of the two functions
gfi <- function(x){x <- as.vector(x);
gfunc(xgf(x,i))*cdensity(as.line(x))
}
if (d>2){
CEffectN[i] <- cuhre(integrand=gfi,ndim=d-1, ncomp=1, lower=lowerLimit, upper=upperLimit, abs.tol= 1e-03,
flags=list(verbose=0, final=0, pseudo.random=0, mersenne.seed=NULL))$value
}else{
CEffectN[i] <- integrate(f=gfi, lower=lowerLimit, upper=upperLimit)$value
}
}
# Get weights
Weights <- (Q*Margdc)/sum(Q*Margdc);
# Get conditional effects and overall effect
CEffects <- CEffectN/Q
OEffect <- sum( Weights*CEffects )
list(OEffect=OEffect, CEffects=CEffects, Weights=Weights, CDensity=Q,
MDenssity=Margdc, bandwidth=bd, Ntreated=sum(Itreat), Treated=Treated,gpar=par)}
Appendix II: Graphical analysis of RDD using paramedic and non-parametric method
160
160
140
140
120
120
dat$y
100
100
80
80
60
60
-40 -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20
dat$x
h=0.05,\tn bins=962