C U OC X O C IN V N OIE] N I 'I { d

S ST Ud AJISU E A IN N N OJEf , N I U d
qBnaqan1uuall)
r{freeseuIerfosroJ selnuue^es
Copr.right O 2008 by Princeton University Press
I,ullished by Princcton University Press,41 \Tilliam Streer,Princeton, New Jersel'08'540
ln the United l(ingdom: Princeton tlniversitv Press, 3 N{arket Place, Wbodstock,
Oxfordshire OX20 1SY

All Rights Reservecl

Librar,v of Congress Cataloging-in-PublicationData

Firebaugh ,G l e n n .
Sevenrulcs for social research/ Glenn Firebaugh.
p. c m .
Includes bibliographical referencesand inderes.
I S B N 97 8 - 0 6 9 1 - 1 2 5 4 6 - 6(h a r d co ve r : a lk. p a p e r ) - ISBN 9 7 8 - 0-691-13-567-0(pbk':
alk. paper) 1. Social sciences-Research. 2. Psychologv-Research' -3. Educatior
research. ,1. Public health-Research 5. Qualitative research. 6. Qu:rntitatire research.
L Tit le.
H62. I . 43 8 2 0 0 8
300.72-dc22 2007023942

British Library Cataloging-in-PublicationData is available

This book has beerrcomposed in Sabor.r

Printed on acid-free Paper.-

press.pilnceton.edLr

Printed in the United Statesof America

10 9 8 7 6 5 4 3 2 1
 To Sarah Elizabeth Gingrich Firebaugh-
louing daughter-in-law, engineer,athlete,
enuironmental actiuist, womdn of faith-
uhose life was cut short by an errdnt driuer.
We miss you.

. .' - '1r-ia rr0(p bk. :

.. :-j:-. .1.L.duc:rtion
,l::.irttit:rtileresearch

7007023942
Contents

Preface xt

Cnaprln 1,: The First Rule

There ShouldBe the Possibilityof Surprisein
SocialResearch 1
Selectingd Research Question 2
Resear chable Questions 2
Interesting Q,ttestions 4
Selectinga Sample 18
Samplesin Qualitatiue Studies 23
Is Meaningful Social Research Possible? 26
Summary 29
'l
Student Exerciseson Rule -)l

CHRprBn2: The SecondRule

Look for Differences That Make a Difference,
and Report Them JO

You Can't Explain a Variable with a Constant 37

Maximizing Variance to Find the Effect of a Cause 39
Size uersusStatistical Significance 41
Comparing Effects Where There ls a Common Metric 42
Calibration: Cr,tnuertingExplanatory Variables to a
Common Metric 44
Substantiue Profiling: The Use of Telling Comparisons 46
Visual Presentation of Results 51
Policy Importance 53
Importdnce for Theory 54
Conclusion 56
Student Exerciseson Rule 2 58

CHeptER 3: The Third Rule

BuildRealityChecksinto YourResearch 64
Internal Reality Checks 65
Realitl, Checks on Datd-Dlrbious Values and Incomplete Datcl 65
viii . Contents

Reality Checks on Measu'es-Aitn ior Consistencym

Con ceptualizatictt't ttntl iVleasurenrcnt 69
Reality Checks on Models-The Formal Equiualence Check 77
F.xternalRealitl Checks:\',tlitlatbrt with Other Data and Methods 76
Using Causal-ProccssObsert'aticlts to Test Plausibility of Results 77
Usittg Ethrtographic l)at,t to Help lnterpret Suruel, Results 79
Other Exatttples of I'tultiple-NlethodResearch 81
CctncIt td i ng Ret rt.tr k 82
Stttdettt Ere lcl,.c-.cttt Rttle.l 84

Cseprr,R 4: The Fotu'th Rttle

Replicate Where Possible 90
Sottrcesof Uncertain4, irt Socidl Rese,trclt 91
Ot,erL,iew:From Populatiott to Santpleartd Back to PopLrlittion 93
Measurentent Error as a Source of Uncertairttt, 100
Illustration: Tuo Methods fttr Estimating Global PoLtertJ' 101
Toward a Solution: Identical Analyses of Parallel Data Sets 105
Meta -analy sis: Syn76n trr,,'tf Result s Fornu lly acr oss St u d i es r06
Sttnunarl': Yottr Confidence Intert,als Are Too Nttrrou, r09
Student Exerciseson Rule 4 111

Cueprnn 5: The Fifth Rule

Compare Like with Like 120
Correlation and Causality r27
Types of Strategiesfor Comparing Like with Like 729
Matching uersusLooking for Differences 130
The Standdrd RegressictnMethod for Comparing Like aith Like 131
Critique of the Standard Linear RegressionStrategy, r32
Comparing Like with Like Through Fixed-Effects Methods 134
First-Difference Mctdels:Subtracting Out tbe Effects of
ConfctundingVariables 134
SpecialCase:Growth-RateModels 138
SiblingModels 140
ComparingLike with Like throughMatcbingon MedsuredVariables 146
ExactMatching 146
Propensity-Score
Metbod 147
Matchingas a Preprocessing Strategyfor Reducing
ModelDependence 151
ContparingLike with Like throughNaturally Occtrrring
Random Assignment 752
Instrumental Variables:Matching through Partial
Random Assignment 153
 ( ontrrrtr . ir

Matching Through Naturally Occttrring Rtrndom Assigrnnent to the

Treahnent Groult 1.58
Compdrison of Strategiesfor Cornpdring Like with Like r 59
Conclusion 162
Student Exerciseson Rule 5 r65

Cneprnn 6: The Sixth Rule

Use Panel Data to Study Individual Change and Repeated
Cross-section Data to Study Social Change 172
1a1
Analytic Differences hetween Panel and Repeated Cross-sectionData r/J
Three General Questions about Change 175
Changing-Effect Models, Part 7: Ttao Poittts in Time 176
Changing-Effect Models, Part 2: Multileuel Models tuith Time
ds the Context 182
What Vle'Wdnt to Knotu 183
The General Multileuel Model 183
ConuergenceModels 185
The Sign Test for Conuergence:Comparing Your rfs and 6s 186
ConuergenceModel uersus Changing-Effect Model 791
Bridging Indiuidual and Social Change: Estimating Cohort
Replacuncnt Lf[ects 195
An Accounting Schemefor Social Change 197
Linear D ecomp ositio n Met h o d 198
Summary 201
Student Exerciseson Rule 6 203

Ciraprnn 7: The SeuenthRule

Let Method Be the Servant, Not the Master 207
O bsessionwith Regressnn 209
Naturally O ccurring Randctm Assignment, Again 209
Decomposition'Vlork in the Social Sciences 218
Decomposition of Variance and Inequality 220
Decomposition of SegregationIndexes 222
The Effects of Social Context 226
Context Effects as Objects of Study 227
Conlexl E[[ectsas Nuisancc 230
Critical Testsin Social Research 231
Conclusion 235
Student Exerciseson Rule 7 ^!_JO

\eierences 241
. ,:,1ex 253
Preface

SeuenRules for Socidl Researcb is meant to serve as a second methods

tertbook for studentsin the social sciences.It can also serveas a refercrrcc
book for experiencedsocial researcherswho want to update their method-
ological perspectivesand skills. My aim in writing this book is to con-
tribute to the improvement of observational (that is, nonexperimental)
social research by setting forth seven key rules for doing that kind of
research.
This book is appropriate as a text in upper-level undergraduatemeth-
ods coursesand in graduatemethodscourses.Indeed,the rules can be use-
ful to anyone who is doing or wants to do observational research in the
socialsciences, from undergraduates writing seniorthesesto graduatestu-
d e n tsbegi nni nga di sserrati proi
on ecrto seasonedveteransstri vingr o im -
prove their research.In elaboratingthe rules, I assumeonly that the reader
has a working knowledge of standard regressionanalysis and of the ma-
jor modes of data collection and analysisin the sociai sciences.The book
is gearedtoward classroomuse for methodscoursesin economics,sociol-
ogy, political science,anthropology, psychology, geography and regional
science,education research, criminology and criminal justice, public
heaith and epidemiology,adminisrrativescience,policy analysis,and quan-
titative history. \Thatever your discipline, there should be little repetition
with your introductory methods tertbook. (In fact, becauseI wanted a
book that createsa new mold for methods textbooks, I studiously avoided
looking at other methodstextbooks while I was writing this one.) Nowhere
in SeuenRules, for example, do I present a step-by-stepdescription of the
researchprocess,as an introductory text would. InsteadI focus on the rules
for doing good social research.
The central messageof the book is "let's do better-not just statisti-
cally fancier-research." BecauseI want that messageto reach a broad
audience,I have tried to make the book as accessibleas possible.There
are sevenchapters, one for each rule. The chapters are self-contained, so
instructors can selectthe ones most relevant for their purposes. To facili-
tate classroom use, each chapter concludeswith a set of data analysrsex-
ercisesfor students.
In teaching this material, I have found that one chapter each week is a
good pace, except for chapters 5 and 6. Those chapters might require two
or three weeks each for instructors to cover the material adequately and
for students to complete the assignmentsat the end of the two chapters.
xii . Preface

Although chapters5 and 6 (coveringrules 5 and 6) are the most challeng- S eve r alot hcr indir r . i- : '
ing chapters in the book and could be omitted in some undergraduate tl-risbook in signific:rni ., .,
courses,the persistentreader wili be rewarded since rules 5 and 6 reflect ar A crdem ia Sinicl . r n. :: ' .
the current turn of social researchtoward issuesof causalinference(rule those lectures,As rhe 1.,
-5)and the analysisof longitudinal data (ruie 6). agementand suggesriir:1 l
The rules are liberallv illustrated with applicationsfrom economics, Mattl-rews,and Sal Or, ,:-.:
polirical science,sociologl',and relatedfields.I sometimesrely on exam- arrd pro vided euur r r r cr . , .
ples fror-r-rm\- own rvork in sociology, since I know that research best. S atvi kaChalasr ni,D. i. -
Equatior.rs are necessarv, but I keep them to a minimum, and in any caseI Frisco, David Johnso: . l.
presenr:lrgumentsand examplesthat rely more on intuition than on for- good, Alan Sica.. inJ l-- -
mal proofs. The book is much heavier on examplesand thought experi- w ere es peciallldet . r i . -
menrs rhan on formal statistics.Although the book is geared toward ron U niver sit yPr c. . ; - -
quantitative methods, the rules (and certainly the principles behind them) me to s peakm or e Jr : =- -
applv to qualitativeresearchas well. Fi na lly.I woLr l. l ir L=
It is fair to say that, for the most part, I learned the sevenrules either in bot-,k.but her supp, , r : . . ,
the processof my orvn researchor from my erperiencesas a journal edi- numero ust o m ent l( ] n.
tor. I did not learn them in graduate school-at least not in a codified
form. I expect to hear from other social scientistswho are in a hurry to
tell me about the rules that I missed.I concedethe point: It would be fool-
hardy to claim that the sevenrules here are exhaustive. (So send me your
additional rules,in casethere is a secondedition of this book.) What I do
claim is that the sevenrules are timely and important. Adherence to these
rules would materiaily improve current researchin the social sciences.
The idea for this book was hatched after a week-long seriesof lectures
I presented to faculty and others at Academia Sinica (Taipei, Taiwan) in
January 2005. The theme of the lectures was "Let method be the servant,
not the master." At the end of the week severalparticipants in the course
encouragedme to publish the lectures. My first thought was to carve out
severaljournal articles from the lectures. Upon further reflection, how-
ever, it became apparent that a seriesof articles would not do-I would
need a book. I resistedthe idea at first but later warmed to it, in part due
to the encouragementof Tim Sullivan at Princeton University Press.And
s o r his book w a s b o rn .
An early draft was written in sabbatical facilities kindly provided by
Harvard University. I thank Peter Marsden and Mary'Waters for inviting
me to spendthe 2004-05 academicyear at Harvard, and for providing me
with an office with such a grand view of Cambridge. I thank the other
'William
occupants of JamesHall, and Christopher \X/inship in particular,
for stimulating conversationsabout matters sometimesrelevant (and some-
times not) to the issuesof this book. I also thank Susan\felch, dean of
the College of the Liberal Arts at Penn State,for her efforts in providing a
timely and welcomesabbaticalto focus on a number of researchprojects.
 Preface . xi i i

Severalothcr individuals aiso deservenr1' tharlks for contributing to

rhis book in significantways. Tony Tam ilr'ited ure to give the lectttres
at Acaderr'riaSinic:rand then suggestedthat I should considerpublishing
those lecturcs.As the book beganto take shapc,I receivedinitial encor'lr-
agement i'rrrdsuggestionsfron Richard lrelsotr,(ieorgc Farkas,Stcphen
Matther'vs,,rnd Sal Oropesa. The followirtg colleaguesand friends reacl
and provided comments on one or ntore oi the chapters:Paul Amato.
Satvika(lhalasrrni,David Code, Daltort Cortler',GeorgeFarkas' Michellc
'WayneOs-
Frisco, Dirvid Johnson, Mich:rel Mass<lglirt,SteveNlorgan,
good, Alan Sica, and Eric Silver.The conrmetrtsof Christopher S7inship
rvereespeci:rllr,detailedand helpful. Thc anont'rnousrevierversat Prince-
ton Universitl' Pressprovided gentle bLrt probing comments that forced
me to spcak more directly and clearly on some points.
Finally, I would like to thank my' wife,.fudy Rae. Shehas not read the
book, but l.rersupport and encouragementhave come in other ways. too
numefous to mentlon.
C H AP TE R 1

The FirstRule
OF SURPRISE
THERE SHOULDBE THE POSSIBILITY
IN SOCIALRESEARCH

Socialresearchdiffers fundamentallyfrom advocacyresearch.Advocacy

researchrefers to researchthat sifts through evidenceto argue a predeter-
mined position. Lawyers engage in advocacy research in attempting to
prove the guilt or innocenceof defendants.Specialinterest groups engage
in advocacy researchwhen trying to influence the vote of lawmakcrs on a
particular bill. The aim of advocacyresearchis to convinceothers( juries,
lawmakers) to think or act in a given way. Hence advocacy researchpres-
ents evidenceselectively,focusing on supporting evidenceand suppressing
c o n trary or i nconveni entevi dence.
Social research,in contrast, does not suppresscontrary or inconventent
evidence.In the social scienceswe may begin with a point of view, bur (un-
like advocacyresearch)we do not want that point of view to determinethe
outcome of our research.Rule 1 is not intended to suggestthat you need to
check your idealism at the door when you embark on a social research
project. Far from that-if you don't want to changethe world for the bet-
ter, you might not have the doggednessneededto do good social research'
But rule 1 is intended to warn that you don't want to be blinded by pre-
conceived ideas so that you fail to look for contrary evidence,or you fail
to recognize contrary evidencer,l.henyou do encounter it, or you recog-
nize contrary evidencebut suppressit and refuse to accept your findings
for what they appear to say.
You must be ready to be surprisedby your researchfindings' It is the un-
certainty that makes social researchexciting and rewarding. If you always
knew the answer beforehand,what is the point of doing the research?
For first-time researchers,the surprise very often is in the finding of
small effectswhere large effectswere expected.Rarely do new researchers
obtain findings that were as strong as they anticipated. Sometimesresults
are even opposite to expectations. Perhaps you were absolutely certain,
for example, that in the United Stateswomen are more likely to approve
of abortion than men are (not true: see exercisesat the end of this chap-
ter), or that younger adults tend to be happier than older adults are
(which is also false:seeexercisesat the end of chapter2).
 1 . Llr:rpter I

Altlr'ugh v'u might irt first be disapl.r.intedin l'our results,it is rrnpor-

tant to lceepin mind that tt,ttottef'f'act is nor ir nonfinding. Any result is a
fi rrding.F indingno e ffc c tc o .l d b c rro re i n re re s t i ngthan fi ndi ng a bi g ef-
fcct, cspeciallyrvhencon'enri.rrrl ri'istl"r holds that there is a Lig effect.
Oftell the nrost interestingresLrltsirr sociirl researchare thclsethirt flr,i1
the frlcc<-rf conventionalu-iscfunr. so v.u should not be disa;lpclintecl u,hen
1 'or rdo not f ir r d t he h i g rl l e rt r o L rr.\| c e t(.d .r,r.u hen 1.orrr' rcsrrl.rrc
ts i rr-
c<l ns is t ent
wit h " u' h a t e v e r\' o n ek n o u .s ."

Srr.lc ; r r NGA RE S EA R CQu

H r.s .r.ro x

Thc first step in a researclrproject is to decideLlpon a researchqucstr()n.

A rc s ear c hques t io ni n th e s o c i a l s c i e n c e si s a q u esti on,rbol t t6e seci al
world that vou try to answer through the anali..sis of errpirical dara. The
errrpiricaldata may be dirta that you collecredr.ourself,or cl.rrecollected
l'tysorneoneelse.
Thcre are two fund:rrnentalcriteria for a researchquestion: interest
and researchability.In selectinga researchquesrion,r'our rirn is to 6ncl a
q ucs t iont hat is ( 1) re s e a rc l .ra bal e
n c l(2 ) i n te re s ti ngro \.ou anclro orhersi n
your field.

R Esp. qRc HReLE

Q uE S T T osN

Some researchquestionsi' the sclcialsciencesare not rescarchablebe-

ca us et hey ar e s im pl y u n a n s w c ra b l e(s e eL i e b e rs o n198-5,chap. t). In that
respecteconomics,political science,psvchologr,.and s,rci.iogl' are
the
sameas astronomy,biol'gy, chemistry,irnd ph'sics. some cluestions-such
:rs the existenceof God-are inherently u'knorv.ble wiih the scientific
method. Other questionsrniqht he rnans.r.erableas the questionsare cur_
rently conceived.otl-rerclrestionsmight be unanswerablewith the knowl-
edgeand methods we currentlr.hirve.
If you are a stlldent, some qllestionsthat are answerablein principle
might nonethelessbe be1,.nclvrur reach.The Nlinneap<>lis DornesticVi.-
lenceExperiment, for exrrnrple,sought to determinewhcther repeatinci-
dentsof domesticvi.lencc rv.uld declineif police irrrestedaccusedabusers
on the spot (Sher'.ran1L)92sherrnan:rnd Berk 19g4).Typically students
do not har-ethe time, rcs'urces,or credentialsrequired ,,, i".ru out an
ex-
periment of rhis r.nagnitlrcle. The possibiritiesfor student pr,rlectsare nor
as restrictedas one nrigl-rtthink. hou'ever,becausetherc are a surprising
number of datir sersrhirr rrrcar-ailableto studentsfor analysis(the student
exercisesat the end of this chapter use just such a data set, the General
 The Fi rstR ul e o .l

s.cial Survel'). l-r,en if r,ou plan to collect vorlr own srlrvL.ydata, yorr
should consult sr-rchprior survevsearlv in your project, if c,,riyfor guid
ance on hou. to u.ord vour questions.
other reseirrchis ruled out becauseit is r-urethical. TcrjLrr'p aheadto r.rr
e r amp l eth ilr is c enr r alin a s trb s e q u e ndti s c u s s i o no f c a usal i ry(chaprcr5),
rl'e did not kno\\', until a ferv decadesago, whether or rot sr-r-rokingcalrscs
lung carrcer-solle expertssaid it did, some said that tl.recvidencervasin
conclusive.Scienrificalh.,the best wav to settlethe disputewor-rldhave been
to .rssignindividualsr:rndomlvto one of two groupsra smoking group and
a nonsmokinggroup. In tenns of sciencc,then, the question.Lril.l b.
srveredb1'doing an experiment.hr practice,however,such an exDerlment ^n-
rvould haveraisedseriousethical quesrionsregarclingrln indiviclual.sright
to smoke (or not smoke),Thc important point hereis that ethicalconsicler_
atlons are irnportant-important cnough to rcndcr somc t)-pcsof rcsearch
un doabl e . r
\7hirt, then, d.es a researchable qrestion look likci Researchable ques-
ti<lnsin the s.cial sciencestend to be questionsthat are too \pe-
cific (for exarnple,about a specific individual or cvenr)'either n.r to<l grand
( "w hl 'i rre t her er v ir r s ?" ) T
. h e c l u e s ti .n" w h y i s my u rcl c an al cohol i c?"
is a gclodresearchquestion becausein the final anrrlysisit is imp.ssr-
'ot
ble to predict the behavior of a given individual; social lar.vsare nor dcrer-
n"rinistic.(Besides,the question isn't very interesting to anyone outsicle
vour farnilv.)It is possiblcto make that questionresearchableby gencral-
izing it to alc.holics in ge'eral. \wecould ask, for example,r,vhysomepeo-
ple tend to be more prone to alcoholisnrthan others are. To addrcssthat
questiorl, we firsr ask what characteristicsdistinguish alcoholics from
nonalcoholics.h.rother words, we ask what individual or communitr.trairs
correldtelvith alcohoLsm.
In the social scienceswbdt questionsgenerally are easier to answer
thitn why questions.As Lieberson(198.i, p.2I9, italics removed)purs ir,
"Empirical data can rell us what is happeningfar nrore readily than they
can tell us whv it is happcnrng."consider alcoh.lisrn again. with appro-
priate data it is a straightforward natter to find the correlate, .rf
h.lisrn, that is, to find the characteristicsthat distinguishalcoholics"1..,- from
others. \ffhether thesecorrelatesare cduses<lf alcohoiism is another mrrrrer
(w e take u p t he c aus alit y q u e s ti o n u n d e r ru l e 5 , " C ompare ri ke rvi th

lI n large p a r t b e c a u s e
o f f a mo tr s ( o r ir tfa n io u s)e xp e r im e n tso r ,e r the past century that
h arm ed ht t ma n s u b j c c t so r p l a c cdtlr e n ra t r isk o f h a r n r , u n ive r siticsn o rv
m,ri ntai ngui dcl i nes
regarding ar]y investrgation involving hurniln subjects. At the outsct of ttrrr
,..sJ,rrchvou
th , t l tlJ 6, v 111 . 1r ,;,1r r r r r n i r e r r r t r ' ' rir r r titL r tio lr llr e vit.g .h o lr J tlRRr fi,r
thc fr,,ce,l urr,, t,, i ,,1.
lo$' in nrur ou'n research.The central pnnciples are that participation b1, hun.,a,,
subjects
shoulclbe voluntarv. that anonvmitv or confidcnti;rlitv ot subjectsshor,rlcl be mailtainecl, arrcl
that the benefirsof fhe reseirrchshould out.,l,eighrrnl' foreseeablerisks to thc subjects.
! o Chapter.l

like"). But that does not make the correlates uninformative. Even if we
want to determine causes,generallythe first step is to find correlates,since
knowing the what gives us clues about rhe why.
Because"mere description" is sometimesdevaluedin the social sciences,
it is important to underscorethe point that "what" questionscome before
"why" questions.Factscome first, as SherlockHolmes statedfarnouslyin
his warning to Dr. Vatson that "It is a capital mistake to theorizein ad-
vanceof the facts" ("The Adventure of the SecondStain"). To punctuate
the detective'spoint, later I describean instancewhere social theorizing
went astravLrvattemptingto accountfor "rising global inequality" when
in fact global inequalitvwas not nsing. Obviously we needto get the facts
right about -,r,hatthere is to explain before we concoct an explanation. As
the first marim of Galileo's Discorsstates,"description first, explanation
s ec ond"( s eeP e a rl2 0 0 0 , p p . 3 3 4 -3 5 ).
Getting the facts right is critical, second, becauseaccurate description
sometimesis itself the end goal. Consider the problem of motion. Me-
dieval attempts to understand motion grappled with the nature and ori-
gins of motion. Current understandingtakes motion as given and attempts
to describeit preciselyand accurately.As Andrew Abbott (2003' p. 7)
polnts oul:

flsaac Newton] solved the problem of motion by simply assuming that

(a) motion existsand (b) it ter-rds
to persist.By meansof theseassump-
tions (really a matter of declaring victory, as we would now put it), he
was able to develop and systematizea general account of the regulari-
ties of motion in the physical world. That is, by giving up on the why
question, he almost completely answered the what question.

In this instanceprogress was stalled until the question was switched from
the unknowable (why things move) to the knowable (how they move). \7e
have returned, then, to the central point of this section: that a research
question must first of all be answerable.The secondrequirement is that it
is interesting.

INrnRn,srrNcQunsrroNs

Besidesfinding a question that is answerable,you want a question that is

interesting-to you and to others.Researchinvolves entering into a conver-
sation. The issueis how your researchwill contribute to that conversation.
A researchquestion might be interesting becauseof its scientificrelevance:
I n addr es s i n gth i s i s s u e .y o u j o i n a n o n g o i ng di scussi oni n the soci alsci -
ences.Or a question might be interesting becauseit is socially important;
 The First Rule r j

:'.ir.rrmative.tven if we perhaps it bears on some important krcal or national social problem. Or a

:' r find correlates,since question might be interesting becauseof its timclincss; a qucstion relating
to Americans' attitudesabout gun control might be particularly interesrrng,
r:. -, . ,.:.1i r - trhe s oc ials c ie n c e s , for example, when important legislationis pending on the subject.
- ,. ..l t' s r ionsc om ebe fo re "The heart of good work," Abbott (2003, p. xi) writes, "is a puzzleand
".
-:- -. j ::'r.' s s t at edf am ou s l y i n an idea." An interestingresearchquestion encompasses both. Thc puz;.le
- :: ": " r:-.-k et o t heor iz ein a d - is the issueabout the socialworld that bearsinvestigation.The idea is the
r - : . : - :-. i i .r in" ) . T o punc tu a te new twist that you bring to the investigation.
How do you find an interesting researchquestion? Often personal ex-
- : rrl inequality" when periences,or the experiencesof others we know, provoke our curiosity
.,.= need ro get the facts about some aspectof human behavior. You can also obtain ideas by read-
:: - - : .: rn erplanation.As ing the researchof others. New findings often lead to new questions, and
:: - . '- : -,. ::r() n f ir s t , ex plana ti o n researchreports often conclude by noting further researchthat is needed,
or by suggestingdirectionsthat subsequentresearchmight take.
- -. - , , ,.:: ,lcCUriltedescription Unfortunately, there is no foolproof recipe for cooking up an interest-
::' - . - : .--:m of rnotion. Me- ing research question. In that light, it is important to keep in mind that
:: : : , - , :l ' t he nat ur e and o ri - the ideas below are intended to stimulate your thinking, not to provide a
:. :- - -..,.sgilen and attempts roadmap to discovery.No one knows where lightning will strike, but it is
- .. - - :,', -\bbot t ( 2003, p .7 ) still a good idea to avoid sitting on a high rnetal roof in a thunderstorm.
'We
Inspiration is the sameway: cannot be certain where it will strike, but
we know that the probabilities are greater in some placesthan in others.
'
- .l,,--
lllnll
-
l-- ^^^., *:-^
as s um lngt-ln a t
If you have trouble thinking of a good question for a researchproject,
" .r " l:r r i t hes eas s u m p -
- most likely the problem is either that you have an idea that you like but
, : ',,.ould now put it), he
others don't, or you cannot come up with an idea in the first place.
. . -r, Llr . t o ^ tr .uhr-e -t -c-g, ,utl^afl-
Supposeyou have an idea for a project that enthusesyou but no one
'1 1 r ) g u p o n t h e wh ) ,
-r\ else. One reaction-perhaps most common among students doing their
: -. : -; 1 l l L l e S t i O n .
first researchproject involving actual data analysis-is to ignore what oth-
:.:, )l \ \ ' i 1 s s r v i t c h e d f r o m ers, such as your professors,think. That is generally not a good idea. The
.^ hou' t he, vm ov e).We interest criterion for a researchquestion dictatesthat you find an issuethat
- is interestingto you and to othersin your field. The "others in your field" re-
- ::Jirofli rhat a research
'': *c qLr ir em ent
is t ha t i t quirement is important becausethe "so what?" question is the first chal-
lenge you will face when writing up the results of your research.Early in
the report you must be able to explain, in a few sentences,why your re-
sults are of interest to your readers. Perhaps the question you address
relatesto an ongoing discussionin your field. Perhapsit extends prior re-
search in new directions, or replicates findings using a new popularion.
:. -- ',\.11rta question that iS Or perhaps it sheds light on an empirical puzzle. The point is that you
' . :' \ :nf irins need some rationale for doing your research.The fact that you cannot get
'". '_' _' b 'info
' "- .a
. COn v er -

. -:: .'-.:i to that conversatlon. anyone else interested in your research is a telltale sign that you need to
, : -.,: ,i irs scientificrelevance: rethink the project. Rethinking the project does not necessarilymean that
- - :i.iision in the social sci- you need to abandon your idea. But it does mean that you need to rework
:, : . , .:: i i i s s oc iallyim por t a n t; your question to make it interesting to others.
 ( o Chapter1

The issue,then, is how to reviseyour researchquestion to make it more surpri set hat dist ingur ihi: . -
interesting to others. (I assumehere that you have done the necessarypre- vour soc ialr esear chpr - ( ,: , . .
liminary review of other researchon the topic, so you have a basic idea of of surprise,you need ro I' r: - '
what othershave found before you.2)First you should discussthe project forehand. Sometin-res rc\-: r\ -
with others to find out why ir is uninteresrillg to them. Most likely they searsh uninteresting) l.r,; ..- -
will say that your question is uninrerestingeither becausewe already -
are measured,So if orhs:r ..,
kr-rowthe answer, or becausewe would not care about the answer even if to make sure that \-olrr -:-:. -
we knew it. That is: resul t.In exam iningf hr ; : : . , -
o The no-surprises objection: We won't learn anything from the re- i l cti on. f or exam ple.ir - ' -
searchbecausethe answer either is already well docume'ted by prior tw o con cept s,so t hat r h; : : .. ' .
research or is preordained by the research method. In other words, ticlnship between the frr', .
the researchis gratuitous, sincewe know the answer before we do the A l ternat ively.per h. r p.: : -- .
research. prior researchlyou'r'el.t.:' --
. The "so what" objection: The answer to the researchquestion either ment, however, it is rnr: , '
has no relevancefor social sciencetheory or for everydaylife, or mat- f ront well-estqblishetl t';.;; ; . - -
ters to such a small handful of people that the question is trivial. that resear cherhcve s r r - r . : '. .--
' el f can st im ulat er e\ eJr - : - .
"\flhy is my uncle an alcoholic?" is a good example of a trivial research
the opp osit e.Bccauser - i: , - -
question. No matter how important the answer might be to you, the ques-
rnixed, inconclusir.e.0r ., :l
tion is trivial to social scientistsbecausethe answer marrersto such a .-
u,hereresearchis u-ell
small handful of people. To make the question inreresting we musr gener- -s:,,-^
firlls into that categorr'.f i:: r ,
alize it. Instead of asking why your uncle is an alcoholic, we ask what jection, sinceprior findin-. : -
general traits distinguish alcoholics from others. So the triviality problem
piestof circumstances. \'(,-. .,
generally is solved by casting a wider net in the researchquestion.
l oncl usion and shou'. u : - . : -
Irrelevanceis also solved by casting a wider net. In truth, of course,few
i s great pot ent inl f , 'r . r . - ' - -
questions are completely irrelevant; relevanceis a matter of degree.As ed-
much researchbut littlr :: : .
itor of the American Sociological Reuiew I found thar one of rhe most fre-
That s it uat ionis qr r ir c- ". ',
quent complaints of reviewers was "\fhy should I care about the results
aonsens us on r esear eh: : * '- ,
of this paper?" As social scientistswe care about researchwhen the re-
rvel l -es t ablished f indinc. : - --
sults speak to ongoing conversarionsand debaresin our field. The more
pri se.S o if you helicr t '.: - ^
such connections we can make, the more interesting our results tend to
stratedthat married f-(,: :
be. so it is important to show how our researchquesrion links to theories
rrnmarr iedpeople.r hcr . . . .
and other empirical work.
marital statusand happ,, ,.-
The other objection (no-surprises)is consequentialbecause,as rule 1 -..
some new direction 1L-\,-.: -: :, -
states,surprise is a hallmark of social research.It is the possibility of
t'ou belieueprior rese.;' .
2 In early stagesof your literature
search you should look particularly for revier.varticles. terestingand significaflrr:-: -- -
A good review article will provide ( l ) a theoretical grid or template for thinking about the prior research.There li :- :- --
important issuesin the field, (2) an overview of key findings and key unresolved issues,and
ti ndi ngsin t he soci. r \l ! 1. - : . -
(3) a description of the most i'rfluential and important studies in the area. The
Journal of those findings.
Economic Perspectiuesand the Joumal of Economic Litercture are yaluable sourcesof sum-
marv articles of the latest findings in particular areas of economics. In the other social sci- In sho r t , if you ul: : : :
ences review articles are available in the Annual Reuiew serjes (e,g., Annual Reuieu, of establishedfindings. \ o,r i :. : -
sociology). \(/hen beginning a new researchproject it is a good idea to scan the table ofcon- r[1'/extendthe findittg:. t - '.
t e n t s o f t h e A n n tu lRe u ie w to se e if te ce n tsu m m a r ya rti cl esar:eavai l abl e,andgofrornthere.
.rnd i nduct ivewor k: Fr : . : '
 The FirstRule o J

surprisethat distinguishessocial researchfrclm advocacy research.so if

vour sclcialrese:'rrch project is uninterestingbecauseit lacks the element
clf surprise,},ou need t<l make the casethat the answer is nol know' bc-
forehand. Sometimesresultscan be known aheadof tinre (mal<ingtlrc rc-
searchuninteresting)becausethe resultsare built in bv rhc wily L-orceprs
irre measured.So if otherssav vour project is unintercsring,you firsr neccj
to make sLlrethat vour mcthod is not ro s<lmeclcgrcepreordaiuing y,our
result. In cramining the effect of mariial sirtisfacrionon overall lifc saus-
factior"r,f.r example,it is importanr to fincl independentnreasuresof the
rwo concepts,so thirt the measuresuseddO not guaranreerr posiriverela-
tionshrp betweenthe two.
Alternatively,perhapsthe possibilit.vof surprisehirsbeeneliminatedby
prior research;you've been "scooped." Lesryou be too hasty in that judg-
rnent, horvever, it is import:rnt to distinguish well-established researcb
from uell-establishedfindings.Lots of researchdoesnot necessarilymean
r h et rese ar ehc rhav
s t ' r eac h e dc c o n s c n s u sl.n c l e e tib. c c a usc!o' rro\ €rs\ i t-
self can stimulateresearch,an extensiveresearchliteratr-rrcmight signal
the opposite. Bccauseresearchin the social sciencessometimes
mixed, inconclusive,or controversialfi.dings, there are issuesof imporr 'ielcls
where researchis well esrablishedbut findings are not. If vour rescarch
falls into thar categor)',then it d.es not run afoul of the n.-s,rorises ob-
jection, sir.rceprior findingsdo not rell vou what yor wiltfind. In rhe hap-
piestof circurnstirnces, you lvill devisea critical testthat reachesa ciefinitive
conclusior-r and shorvswhy prior researchhas becn inconclusive.So there
is great potential for a significant rescarchcor-rtributionr,vherethere is
much researchbut little in the way of establishedfindings.
That situation is quite different from the situation rvhcie there is general
consensuson researchfindings. Unlessprior researchis seriouslyfl,rwed,
well-established findingsfrom prior researchrule our rhe possil.,iliiyof sur_
prise.So if vou believe,for example,that prior researchhas anrply demon_
strated that married people in the United Staresrend to he happier rh;rn
unmarrieclpeople,therc is little point in an<ltherstudy of the associationof
marital sratus and lrappinessunlessthe nerv study extends that finding in
some new direction (examples below). Nore rhe critical qualifier hcre, i/
you belieueprior researchhas antpll, demrmstratcd.
Sonreof the most iu-
teresting and significant researchin scicncehas ilverturned the finclinss of
pri.r research.There is nothing wrong *'ith challengingwell-establlhed
findings in the s.cial sciencesif t,ou can sbow that thie is redson to doubt
those lindings.
In short, if ,vou rvant ro investigatea topic with (apparently) well_
establishedfindings,you have rwo .pti.ns: cballengethe
findings or cldr-
ify'/extendthe findings. challenging pri.r researchinvolves both*deductive
and inductive work: First you demonsrratethe logical or methodol.gical
 $ o Chapter1

flaw in prior research (deduction), then you show, empirically. that the others rate you on lelel , : :,
flaw matters (induction). vou may 6nd t hat nr , r r r : . : - .
The other option is to clarify or extend establishedfindings. The clari- you measurehappines.. ) -,
fication of findings is deductive(or largely so), involving further concep- that marriage is rel:rred :. .
tualization and theorizing. The extensionof fir-rdingsis inductive, involving different ways. In orhcr ',',
further data analysis.I considereach of theseoptions in turn. lenge ar' old findir-rg.\r-,.- .:

Ch allenging Prior Researcb

h"rchallenging the findings of prior research,the burden of proof rests Box 1.1
with you to identify some shortcoming or flaw that is serious enough to CaseStudy:Rethinking:
raise questionsabout the reliability of earlier resulrs.personalanecdotes Incomc[n.- _
are not enough in this regard. For exampie, with regard to the estab-
lished finding that married people tend to be happier, it won't do lo say Contrary to some ea:-.:
"I'm not married and I am happier than most married peopleI know" or inequality find that. u:
"I know lots of unmarried people who are huppy and lots of married global income ln.qu.'r,,,'
people who aren't." If you believeprior results are off the mark, you 2002; Fir ebaugirr i, l''- . ,
need to focus on the reievantresearchitself and identify preciselywhere S al a-i- M ar t in2006: >.
it has gone astray. Perhapskey concepts are not measured adequately,or gl obal incom eineqr r ' :
the statistical rnethods are rnisapplied, or there are problems with the rapidly.as once ic.rr.u.
sampleor the data. income inequalin pr'
Box 1.1 gives an example of a presun.rably well-establishedsocial sci- know ing whar it i. r ^.-.- ..
ence finding that r,vaseffectivel_vchallenged by new research. Earl1. ac- expl ain it . I n a n r ir : : : : -
counts suggestedthat global incomeinequalityshot up dramaticallyin the has beenlinked r o . ,
last decadesof the twentieth century. Using more appropriate data and i n social science.r , , - , -
methods to analyzethe trend over that period, howe'er, recent research and th e "explosion"
now finds that global income inequality did not grow at all over that pe- that globalizar ioni. : :
riod, and most likelv declinedsomewhat. w rong headed- ir r r : . . :
The example in box 1.1 is not typical. Not all researchis easily char- It i s inst r ucr ir et , '. . .
lenged. In the first place, you rnust find something clearly dubious about comei nequalit rr eisr - :.
earlier research, and explain how your suspicions could account for the across individuali . - -
observed findings, You might, for example, question the finding about w ays t o m easur eJr . : :
marriage and happinesson the grounds that the finding is based on self- applying theserneasri.:.
reported happiness,and you have good reason to believethat self-reported w hetherincom einc; - . . . .
happinessis an unreliablemeasure.Your argumentmight he that, because the whole world.
American culture places a premium on being married, the married are Global income ine: - :
more likely to say they are hnppy whether they are or not, and the un- tion of income acros. ,-. :
married are more likely to say they are unhappy whether they are or nor, gi ven equal weighr . \ \ : -
and it is that bias in the measurementof happinessthat accounts for the tant point t o not e is ir . . , t- -
associationbetween marriage and happiness. i nequa lit y in ear lr . r : - . - . :
The next step is to devisea test to show that your cririque matters em- restedon a ser ierr , t - -
pirically. In the example of marriage and happiness,you would hypothe- Theseincluded:
size that the associationbetween marriage and happinessdisappearswhen
happinessis measured in (presumably) more reliable ways, such as how

' '.'..empirically, that the
. -, -i r r r dings. T he c l a ri -
'.r nq t ur r herc onc e p -
'-- . ' r J r r c t iv e,
inv olv i n g
- ..: J c r i uf pr oof r e s ts
' '.. .: s er iollsenoug hto
- -..::, Ir-rsot-ta1 anecdotes
: :r i- r f d t o t he es ta b -
. : ::. ri ri'on't do to say
. - :; : eoplc I k now" o r
- . . ,'-:-r.llots of rnarried
- ,.-- , r it r he m ar k , y o u
r ::'.:ri\' preciselywhere
- : ,.:-:red adequately,or
: .:: lr t ) blem s wit h th e
: -:.:,rl.Lrshed social sci-
.' - r r c , I r c h. E ar ly a c -
- -' : ,.: J r am at ic allyin th e
--. .',;'''ar. recent reSearCh
' : '.'. .rt .rll over that pe-
-:
'- -r. r r . h is eas ilych a l -
: ' : - :.i r r lv dubiousab o u t
r - -- -,,uld ac c ountf or th e
- . :'. ,,n tl-refinding about
- ' , -:-.rng is bas edon s e l f-
':..c\ e that self-reported
- : -.-:rLqirtbe that, because
...::ic.'I.the married are
-
: .::. or not, and the un-
, : : ''.:.ther t hey ar e or n o t,
.: . - r :: rh.rt accountsfor the
j-ir cnttque mattersem-
::i. \'ou rvould hypothe-
:tiness disappearswhen
:.'-i.leu'a1's,such as how
The Fi rsrRule . e
others rate you on level of happiness.Very possibly you will be wrong-
you may find that marriage is associatedwith happinessno matter hou'
you measure happiness. Your study nonethelesscontributes by showrng
that marriage is related to happinessevenwhen we measure happinessin
different ways. In other words, when you use a new approach to chal-
lenge an old finding, you add to knowledge whether you are right or
B ox 1.1
Case Study: Rethinking the Findings of Rcsearch on Global
Income Inequality
Contrary to some earlier research,recent str-rdiesof global income
inequality find that, when China and India are given their due weight,
global income inequality is not rising (Bourguignon and Morrisson
2002; Firebaugh 2003; Goesling 2001; Melchior and Telle 2001;
Sala-i-Martin 2006; Schulz 1998). (To be clear, there is massive
global income inequality in the world today-but it is not growing
rapidly, as once feared.)The "yes it is/no it isn't" story of rising global
income inequality provides a good example of the importance of
knowing what it is that you are trying to explain before you try to
explain it. In a number of popular articles and books, inequality
has been linked to economic globalization, one of the hortest topics
in social science,to fashion a growing literature on globalization
and the "explosion" of global income inequality. But the argumenr
that globalization is the cause of rising global income inequality is
wrongheaded-it tries to erplain a "fact" that isn't.
It is instructive to seewhere the earlier researchwent wrong. In-
come inequality refers to the disproportionate distribution of income
across individuals or households. Social scientists have devised
ways to measure disproportionality in income (Allison 1978). By
applying thesemeasuresto income data over time we can determine
whether income inequality is rising or falling for a country-or even
the whole world.
Global income inequality refers to the disproportionate distribu-
tion of income across all the world's citizens, where each citizen is
given equal weight. \flith regard to the discussion here, the impor-
tant point to note is that the claim of rapidly growing global income
inequality in earlier research was subject to challenge because it
rested on a seriesof dubious assumptions and incomolete analvses.
These included:
(continued on next page)
10 . Chapter1

wrong. If you are right, your results pose a challengeto conventional

findings. If you are wrong, your results extend the conver-rtionalfindings,
albeit that was not your intenticln.
Probably the most comnlon outcome is that challengesserve to refine
rather than dislodgeestablishedresearchfindings. Of course,the refinement

rcuntinttcdfrom PreuiuuspoKc)

o Extrapolation
from resultsfor the tails of the income distribu-
tion. Some earlier studies concluded that global income ine-
quality is rising rapidly on the basis of a comparison of income
trends for the very richest and very poorest countries. But the
vast majority of the world's people live somewhere else. \fe
cannot reach defensibleconclusions about the inequality rrend
for all the world's population by examining income trends only
for those who live in the very richest and very pooresr regrons.
. Unequal weighting of indiuiduals. Early studies often weighted
all countriesequally.Ho-,vever, becauseglobal income inequal-
ity refersto the unequaldistnbution of income acrossindiuid-
uals-and manv more individuals live in poor countries (e.g.,
China) l'here incomesare grorving rapidly than in poor coun-
tries u'here incomes are gror,r.ingslowly (Firebaugh2003)-
the failure ro \\reighrcounrriesby their population produced
misleadingresultsin those studies.
o Extrapolation from historical trends. Global income inequality
rose rapidly over the nineteenthcentury and the first half of the
twentieth (Bourguignon and Morrisson 2002), as the \il/esrtook
off economically and Asia and Africa lagged behind. Global in-
come inequality rose rapidly becausethe richest regions of the
world were the world's growth leadersand the poorest regions
tended to grow slowly, if at all. In recent decades,however,the
world has changed.Now many of the fastest-growingeconomies
are in poorer regions of the world (Garrett 2004)-a poinr not
fully taken into account in some early studies.Becauseincome
inequality is now declining across counrries, rhe longstanding
growth in global income inequality has halted. (At the same
time, income inequality has increasedwithin many countries.
Declining inequality across counrries and rising inequality
within them implies a "new geography of global income ine-
quality" in which countries are receding in rmportance as eco-
nomic units: seeFirebaught20031.)
 T h c Fi rstR ul e . l

of pri<trresearchcould serveas \rour startingpoint. In th:lt

caseyoll do ll()r
set.outto challe'ge prior research,but t. clarift or c,xre'd
pri<x research.
Indeed,unlessthe flarvsin prior reseirrch,rre .bviour, i.,-rport"nr,
.rncieasil,
correctable,clirrificati.n ancl exte.si.rl rrrc thc ,-,-rnrtp--iring
straregres
for avoiding the no-surprisesobjection,especiirllyfor new
rcsJarchers.hr
the rwo secions I describethe pri'cipal \\,aysthat r.ou might
'exr crarify .r
ext cn.lpri or r c s t ' lr c h.

Claifying Concepts in prior Research

vru might clisc.verthat the p.blern with prror research
is not that the
finding.sare wrong, but that ihe impricationsrlre
unclearbecauseof c.n_
ceptual fuzzinessin the mocrer.r the variabres.Erinrinating
rhe frrrin.r,
might not lead to nerv results, but it coulcl lead
to ,r.r" i,n.r..rtandine
:rb outwhat th e r es ult sm ean .In th i s c a s eth e e re m e n t
o f ,u . p;;;; our re_
searchinvol'es the implicatio's of .ur findings,no,
n...rr"rilv the find-
i ng s t l re rn se lv es .
It is especiallf i'rp.rta't t. be clear ab.r-rtthc r'earrirs
of our clcpe.<i
ent varirrble,the virriablewei:lre trying ro explai'. Richarcll'elsc,n
(2002.
pp. 209-10) givesexamplesfr''.r thcory arrclrcscarch.n
crime. As Fclson
notes,the first step in explaining crime is to determine$,hat
sorts ,f be_
havior we are trving to crplain. s'me tvpes cri'rc i'vorve
r.Jcvi;r'tacts
(for.exarnple,illegal drug r-rse), rvhererrsorh..'f t},p.s i.v'lve ;rggressior.r as
u'ell as deviance(for cxample, assrr.rt).App.rpriare exl,la,ratlJrrs
of iilc-
gal drug usenrav lre ver1,different fro- ."pl"nations
of ,rggressive behav_
ior. To test varioustheoriesof crime, the., Felson.ugg.r,r'ii.,"
,.,.",,f *h^,
he calls "discriminant prediction" (f'r exarnpl.,
ti.,lori.. of ,ggr.rrion
should betterpredicr assaulrtha'iliegal drug use,
whereastheories.f de-
vianceshould predict both assaulto,ld ilt.g") clrug
use).

Extending Prior Research

suppose you have no g'od reasor to clotrbtthe accurac,v

or conceptnal
framew'rk of pri.r researcrr_bearing on vorlr rcscarchqrresti'n. [n tnat
caseyou should use prior findingsas the point of departure
ro,. ,r*,,
researchproject. To make the researchinterestrng-to "ou,.
pernrt surp'ses-
one possibilityis tr extend that researchin new directions.
He.. n're,,uo.
generalideasalong that line.
o Foc,ustn a keysubpopuration.The
idea here is ro seewhether o'or
eirrlier researchfindings are reproducible fo*o,rr"
,r."i.gi. ,r1.,.., nt
the population. Suppose prior research shows
,t r, ..rfi"g.-*n,,r.,
tend to be more liberar politicalry than freshmen y,,i nrt.na
"r.. "
12 . Chapter1

parochial school, and in your review of the literature you find that no
one has examined that issue separatelvfor parochial schools, which
(you suspect)tend to be more conservativethan other colleges.In that
case you might design a survev for freshmen and seniors in your
school to seeif you find the same result for your college (better yet,
administer your survey in a broad range of parochial collegesin addi-
tion to your own). Becauseno one has studiedthis issuefor parochial
schools,you can be surprised b,vthe results.The results of your study
likely would be of interest to a number of people,including adminis-
trators and others in parochial schools, educational sociologists,and
those who study hor,v political attitudes are formed. (A word of
warning here: subpopulationssuch as "my friends" or "my sororiry
sisters" are too narrow. You might begin by pretestingyour survey on
your friends, but you do not want to end there. Not many people are
interested in the political attitudes of your friends or those in your
sorority. But a number of people are interested in the political atti-
tudes of studentsin parochial colleges,a much more substantial sub-
set of the generalcollege population.)
. IJsea new population.In this casethe issueis whether or not earlier
research findings apply to some new population. You might want to
know u'hether there is an associationbetween marriage and happi-
nessin the caseof arrangedmarriages.To find out, you could inves-
tigate the effect of marriage on happinessin societieswhere arranged
marriagesare more common than they are in the United States.
o Use a different time period. Here the issueis whether or not prior re-
searchfindings apply to a different era. In some studiesthat new time
period might be historical: '$ilasthere a positive associationbetween
marriage and happinessin the nineteenth-centuryUnited States,or has
that associationcome about more recently?Historical data to address
such questionsoften are lacking, however,so rnost replication studies
of this type are updating studies-studies that seek to update earlier
researchwith more recentdata. For example,studiesof racial attitudes
in the United Stateshave consistently found that white southernersare
more likely than whites outside the South to disapproveof marriage
between blacks and whites, to disapprove of racial integration of
schools,and so on. Many of those studieswere done in the 1970s and
1980s (for example, Middleton 1976;Tuch 1987), however,so peri-
odically new studiesappear (for example,J. S. Carter 2005) to deter-
mine how quickly regional differencesare disappearing(if at all).
c fJsea new population or subpopulation to test a specific hypotbesis.
In this caseyou choose a population or subpopulation that provides
a test of some hypothesis. For example, you might hypothesize that
the positive relationship between marriage and happinessis basedon
 The FirstRule . I.l

: - : -: lcrilture You find that no companionship,so you expect iittle or no effect of marriagc on hap
' :--rrochialschools,which pinesswhere long periods of separation are comrnon. You rr-righttest
--- .
',:-- ri ,. :l'.,.r1 orhercolleges.In that this hypothesis by estimating the effect of marriagc on happiness
- .j -.,.:-.: rrtr ] d s enior sin y o u r among those in the navy (where long periods of spousal separarion
r - , .' : ',(lrlr college(betteryet, are commonplace) and comparing that effect with the marriage effect
' :.;:,, ; hr alc ollegesin a d d i - in branches of the military where separation is not the norm. Note
- - -
: - .. - . . - :..:i t r s is s uef or P ar o c h i a l that in this instance a noneffect (no association between rnarrrage
: - :- l '.:-resultsof Your studY
-: and happinessamong those in the navy) provides support for your
, - -: - - .: r. e. inc ludingad m i n i s - companionship explanation of the general marriage-happinessrela-
a- -. :: ....::. , - r n,s1l oc iologis tsa, n d tionship.
1--- r: -.:: iornled, (A word of
- ::::; r J s " or " m y s o ro ri tY In short, one possibility for well-establishedresearch rcsults is to see
-- '.
-: - , - - , -l :.:r s t illg v our s urv e yo n whether or not old results apply in new contexts (strategicsubpopulations,
: . : - ':::. \ ot m anYP eo p l ea re new populations, or different tirne periods).
Another possibility is to extend or refine the cawsalexplanations of the
:: - - -.: ::.c i- r ds or t hos e in y o u r
"i :::.i :i in t he P olit ic a la tti - observedempiricai patterns found in prior research.The social sciences
: - -:
. . --' .-:: : lor e s ubs t anti asl u b - are pregnant with claims about what causeswhat. The evidencewe have
:- :
for these causal claims, however, is often very weak. A critical task in the
:
or not e a rl i e r social sciencesis more and deeperresearchon causal relations. There are
rr ,: r- : -- ., ..
" r ilet her severalgeneralstrategiesfor doing so. The major possibilitiesare discussed
: : - .. ,:.:. ,r'1. \bu might want to
' - .'.:::1 m.lf riage and haPPi- below.
:: ::
-: :: : ' - -:-.; O Llt v
. O U CO ul di n Ve S -
CAUSAL C H AIN S: IN T R OD U C E M ED IAT IN G VAR IABLES T O AC C OU N T
::: : -- -l rvhere arranged
- 'alaties FOR R ESU LT S OF PR IOR ST U D IES
- - ' . - . .:: .:-.ihe Unit ed S t at e s .
: - - . . . . - .'.r lc t lr c ror not pr i o r re - The idea here is to developtheories that identify the mechanismsor parh-
-' :t'. .'- - ::le studiesthat new time ways by which some independent (explanatory,exogenous)variable X af-
:-r :: - I :l i l v c t ls s O c iat iObe n tw e e n fects some dependent (outcome, endogenous) variable Y. One way to
r,:'.:: -. --r:-.:!1rv United States,or has account for the effect of X on Y is to find some set of variables Q that me-
' : :: , : ' . i Hi sr or r c aldat a t o a d d re s s diate the effect of X on Y, that is, X -+ Q + Y. This strategy is often em-
: ::. i\) Illost replicationstudies ployed in social scienceto account for the effects of ascribed traits such as
--- . .. - .:: :h.rt seekto update earlier gender or race on outcome variables such as earnings. It is well knorvn,
t : , .'.:-:... studiesof racial attitudes for example, that women tend to earn lessthan men in the United States.
'
:-t r ,.--.ilhet n'hite southerners are Explanations of this gender gap in earnings typically focus on mediating
i-: ) .::: :o disaPProveof marriage or intervening variables Q in this chain: gender -+ Q -+ earnings. In this
: -,',::: ic of racial integration of chain Q might include things like interrupted careers (women are more
*: i. -.-.:: \\ er€clonein the 1970sand likely to have interrupted work careers,and that's one reason for gender
differences in earnings) and hours worked (men are more likely than
:' ,, : ': ': :. J. S. Car t er2005) t o d e te r- women to work full-time). One way to make a researchcontribution, then,
' : - : - . ::: .| sa ppear ing( if at al l ). is to find important mediating variables that have been missed or under-
| : : . - . 1 .' t :' , tes ta s pec if ichy po th e s i s . valued in prior research.
1: .. ; .ll.population that provides The mediating variables strategy works best when reverse causation
:...j:-: .'ou might hypothesizethat can be ruled out. That is why the strategy is often popular for explaining
- .-. , : : .,i-. the effects of ascribed traits such as qender and race. In the illustration
:e a nd happines sis ba s e do n
14 . Chapter1

above, for example, it is clear that gender affects hours worked, not vrce
versa.
Sometimescausal direction is not so obvious. In the caseof marriage
and happiness,it is not clear which con-resfirst, marriage or happiness
(seereversecausationbelow). In other rvords,there could be a kind of se-
lection effect in which those who are selectedir-rtornarriagetend to be
happier than those who are not selectedinto n-rarriage.Even if marriage is
not selectivewith respectto happinessper se, marriage could be selective
with respectto possiblecorrelatesof happiness,such as physicai attractive-
ness,or cluses of happiness,such as physical health. If in fact the beautiful
and the healthy are more likely to marry and more likely to be happy, then
at least some of the associationof marriage u'ith happinessis due to the as-
sociation of marriage with attractivenessand health. Thr-rsrnarriage could
be associatedwith happinessevenif marriagehas no cdttsqleffect on hap-
prness,
This observation-that nonmediating variables could account for the
empirical associationbetweenmarriage and happiness-leadsto another
possibility for extendingprior research.

IN T R O D U C E N ONM EDIATING VARIAB i- F- STO ACC] OU N T

F O R R E S U L TS OF PRIOR STUDIf,S

The idea here is the same as in the mediating variables case:'We want to
develop theories that explain rvir-vr'veobserve an association between X
and Y. In the caseof nonmediating variables,though, the mechanismsare
not causal. Sometimesthe term sptn'iousdssociationis used to describeas-
sociationsbetweenX and Ythat can be accountedfor by their associations
with other variables,where those other variablesare not causalmechanrsms
linking X and Y. (For example, X and Y might be associatedbecause
some third variable Z causesboth of them. Or X and Y might be associ-
ated becauseX is associatedwith other variablesthat are causesof Y.)
The term spurious here means that there is no causal link from X to Y. It
does not mean that the associationof X and Y itself is a mirage. Ifhen we
say, for example, that the associationof marriage and happinessis spuri-
ous, we do not mean that, on secondthought, married people aren't hap-
'What
pier after all. we mean is that getting married typically won't make
you happier. (If the distinction is unclear, it should become clearer with
the discussionof causalityin rule 5, "Compare like with like.")
Often an associationbetween X and Y is partly causal and partly not,
so some of the so-calledcontrol uariablesthat are added to a model to ac-
count for an associationof interest are mediating and some are not. Ifhat-
everthe situation-whether the associationof X and Y is due to mediating
variables or nonmediating variables or a combination of the two-it is
nearly impossible to overstatethe importance of careful conceptualization
 The FirstRule . 1-i

and theorizing in determining what controi variablesto add. To assistin

this theorizing, you should draw a path diagram that depicts your model
(chapter5 providesexamples).In a path diagram,variablesare nodes,and
arrows connect the variables.A straight arrow from X to Y indicatesthat
X causesY, and a curved double-headedarrow between X and Y indicates
that X and Y are associated, with no causaldirection assumed-If there arc
no reciprocal effects among the variahles,path analysis methods (Ahvin
and Hauser 7975) can be used to estimatethe rnodel. The irnportant poirtr
here, however,is to draw the model so that you can seeat a glancehow all
the variablesare interconnected.s
Finally, it is important to stressthat you should not add control vari-
ables willy-nilly to a model to find out if that affects your r:esulrs.Vru
need to justify the variablestheoretically before you add them. More con-
trol variablesare not always better than fewer conrrol variables,as Lieber-
son (1985, chap.2) emphasizesin his discussionof the dangersof what
he calls "counterproductive controls" and "pseudocontrols." Careful the-
orizing is the best protection againstsuch dangers.

IN TRO D U C E M OD E R A T 'IN C V A R IABLES TO ACCOUNT FOR,

AN D Q U AL I F Y , R E S U LT S O F P RIOR STUD1ES

A moderating uariable is a special kind of nonmediating variable that

conditions the effect of one variable on another. For erample, if the effect
of education on income is greater for whites than it is for nonwhites, then
we say that race moderdtes or conditions the effect of education on in-
come. The effect itself is called an interaction effect (Jaccardand Turrisi
20 0 3) .
By introducing moderating variables, Scott Myers turns a common
observation-that religious parents tend to have religious children-into
an interesting researchproject. After noting that "Religiosity, like class,is
inherited," Myers (1996, p. 858) theorizesthat there are facrors that would
"condition the ability of parents to transmit their religiosity." In orher
words, he theorizes that there are interaction effects. He finds that the
transmission of religious practices (such as praying, reading the Bible, and
attending church) is stronger when parents are happily married and do
not divorce, when parents agree on religious beliefs, when parents show
affection toward their children and are moderately strict in rearing their
children, and when the husband works in the paid labor force and the
wife is a housewife. In short, Myers finds a number of interesting interac-
tion effects.
:rPath diagrams also help you to see rvhich variables are exogenous (determined outside
the model) and which are endogenous(determined b.vother variablesrn the model).It rs irn-
portant to find exopienousvariables, since you cannot estimate a model when all variables
are endogenous.
16 . Chapter1

\7e encounteredthe concept of interaction effects earlier, most no-

tably in our hypothesisthat the effect of marriage on happinessrvill be
reducedin navy marriages(asopposedto marriagesamong thosein other
branchesof the military), In fact, if \\'e think again about comparing the
findings of new populationswith findings for other populations,we see
that such comparisonsare best carried out in a moderator-variableframe-
work, that is, by combining dara for the old and new populations and
usinE;interaction terms. In the caseof comparing studentsin parochial
coliegeswith studentsin other colleges,for example, we could append
the data collected on parochial schools to existing data on other colleges
and employ "type of school" (parochial/other)as the moderating variable.
(Merging the parochial school data with the earlier data should not be a
problem if the replication was done properly.) Or, to determinewhether the
effect of marriage on happinessis weaker in the case of navy marriages
than for marriagesin general,we could combine data from two samples-
a sample of those in the navy, and a sample of those not in the navy-and
use "in navy" (yes/no)as the moderating variable.

C H E C K F OR R EVERSE CAUSATION

The extension of prior researchmight also turn on the issue of which

uaridbles are dependent (called the endogeneity problem). You might
concede,for erample, that the married in fact tend to be happier (no mat-
ter how happinessis measured),but not becausemarriageleadsto happi-
ness.As noted eariier, the associationcould reflect the fact that happier
peopleare more likely to attract a mate, so happinessincreasesthe likeli-
hood of marriage instead of the other lvay around. If prior researchrnter-
prets the marriage-happinessassociarion as reflecting only the effect of
marriage on happiness,you can make a contribution by designing re-
searchto separatethe effect of marriage on happiness(marriage-+ happi-
ness)from the effect of happinesson marriage (happiness-+marriage).If
the conventional interpretation is that marriage causeshappinessand you
show that happinesscausesmarriageinstead,then you have found an ex-
arnpleof reversecausation:There is a causaleffect,but prior researchhas
the arrow going in the wrong direction.

B R O A D E N T H E RESEARCH QUESTION BY EXPANDIN G IT AC R OSS SU BF IELD S

The idea here is to reframe old issuesby synthesizingperspectives,con-

cepts,or methods from more than one subfield.Synthesizingstudiesare
hard to do, but successfulstudiescan have high payoff in terms of new
insights. Consider again the issue of the transmission of religiosity
across generations.Social class is also transmitted across generations,
and there is a huge sociological literature on class inheritance. It is

: t:Ji S er r r lier ,m os t n o -
- ,:: rn happines swil l b e
. -:: -:.;-llong thosein other
- ,i. or r rc om par in grh e
.. =: i. , r P nlat ionswe , see
'"' -:'r.rtor variableframc-
. .. .ri\\ popul;rtionsand
' tempt to span the two fields could have high payoff, however, in terms
- ::. r ! lent s in P ar o c h i a l
: : r'. u'e could append
' " i -:-trJon clther colleges
. r - . - : . . ^ .vdrr<
. ^-:rb l e .
- _ililufrdtrrr5
- :: J.rt.1should not be a
-. -. ::rermine whetherthe
.-, ,-:.: i of nay y m ar ri a g e s
j i ::-'.:l-o111 two Samples-
Lrr in the navy-and
- tl,.' issue of which
' , hlat n\ . Y ou m i g h t
're
happier (no mat-
.:: r r ge leadst o ha p p i -
::r.' i.rct that happier
:: irlCredSeS the likeli-
: : r ior r es ear c hil nter-
:-: .rt onh- the effect of
- -..:., r i. br i i des igni n gre -
- - -- r t t . t r r iage-+ happi-
-.,::r nc s s- + m ar r iag e ).If
: - :..::s happines sand y o u
- ::-. '. ()u l-ravefound an ex-
-:,:. i '.r r pr ior r es earc hh a s
- - : . Lri i S U B F IE LD S
- '. :: Zi 1 Qper s pec t iv e sc,o n -
- -- -- >.n r i. r es iz ing
s t udie sa re
: - - :.r\ off in terms of new
- - '-, '..rr. r is s ion
of r elig i o s i ty
- '. - r- :rtl .lCrOSS generations,
:' .iess inheritance. It is
The First Rule o | -
useful to think about how the study of religious inheritance could be
linked to the study of classinheritance,which has been studied in much
greater depth. Are the processesof classinheritanceand religious inher-
itance similar? Are parents in the United Statesmore effective in trans-
mitting their classstanding or their religious beliefsl To my knowledge
no one has tackled that question (I supposebecauseir is difficult to find
a common metric for comparing the two types of inheritance).The at-
'We
of both substanceand method. could generalizethe issuemore-and
make it even more interesting-by asking what overarching principles
govern the transmissionof beliefs,attitudes,values,and statusesacross
generations.For example,are parentsgenerallymore successfulin trans-
mitting their beliefs and values (political, religious, and so on) or their
material statuses(such as their social class)to the next generation?In
synthesizingstudiesof this sort, prior findings for particular fields of in-
quiry become the touchstones for more general theories and research
about human behavior.
In short, the answer to the question of concern here-'$7hat if your re-
searchidea is not interestingbecauseit has been thoroughly researched
before?-is: Dorl2t give up on your idea too easily. You might be able to
update previous findings with more recent data, or extend findings back
in time with historical data. Or you might try to replicateprior research
by focusing on a strategic subpopulation, or by using a new population
altogether.
Alternatively, you can use the samepopulation as in prior research,and
look for ways to extend that researchby accounting for well-documented
associations in terms of either mediating variables, nonmediating vari-
ables,moderating variables,or reversecausation (or some combination of
the above). Your project can be especiallyinteresting if it bears on empir-
ical regularilies in societies-fundamental associationsthat are well doc-
umented and appear to be generally true of most societies (for example,
the inheritance of social class across generations). In general, the more
tightly you can link your researchto key empirical regularities, the more
interesting it is for social scientists.
In offering these suggestionsabout how to build upon the foundation
of prior research,I am assuming that there is prior research on the issue
you want to study. Of course you do not need to worry about prior re-
search if there is no prior research. So it is tempting to try to think of a
novel research idea, something that no one else has investigated before.
That often posesa dilemma. Recall that interest and researchabilityare
the key requirements for a research topic. Yet if we find a topic that is
 l8 . Cl.raprter
I

r nt er es t ing: r n d re s e a rc h rrb re
r' e, r' ri k e l ' th e i ssuehas bee' studi ed
ber of rimcs, s. tl.rereis cxrensir.erescarch . num_
,,n i, ni..".ii."iri.'*.n.rn,r,
difficult, ther, ro fircl a feirsirrlercsearc-rr
c.lucstlon
r !'v s r r v r r rhat
( r r or ris
J rir,porrant
r r r Pu vet
ha s ro t be c, stLr cr ied
That dilenrmapr'babl' erprai's ri.h' researchers-in-trairring
stru,qglcto find a rcsL.arcitro1.i.-.11.,.,. sornerirncs
tr.r-in vai. to fi.d o,r'ar.rar,,ng
rluesri.n rhat n. .ne has inresrigated
tefirre. Trre exarr.rple, ,ru. are i._
to make,thepoi'r rhat rl.t.b.rt soci;rlresearch "t
:;rd;d m'sr oftcn is research
rrcshperspectives
a'ctnewi'sightsto old and_niinu,,rgu..o.
:ji.::#:r,li

S r r . r c r r nc A S AMt,rr.

After detcrnining u,bat \.ou w.lnt

to stud\,, r,ou need t. creterminctt,hont
to study' First y.u need ro deterrni*e
rvhat rarget p.pulati.' your results
are nle:lnt to describe.(scechapter 4).
Are r,gu intcrestedin clrawingcon-
clusi.ns ab.ut :rll c.lrcge st'clents,r,'orldll,ide
? (Tlrat c.r-rld bc .verlr. anr_
as_'btaining a sa'rple of college srucle'rs
li:i:,tt r,vorldr,r.ide ,";jj";;
difficult') or American-c.llege ,n,"d.ntr?c)r o'lv
'nlv American corege
srude.ts i,r p,rr,rihial schools?f,itnorr r"hom r. sarnple,
idertif,r.'vour rarget popul:rti.n carefr-rlly r --' l.ou
.' -" l
need t<r
ard precisely.
After
.cleterminingvour target l.ropulation,vou t' determrneir
strateg)'f<rrselectinga srbset of i'diljicl.als-cailed 'ecd
,, ,n,r1rrr_frrm rhirr
populatior.r'Bccause\"ou attempt
to draw c'ncrusi<lnsabout the entire
populatior.r.' the basisof ,.r,,r,.
from a subser;ith,;
critical that y.ur samplerrereprese'tatiue o,ro,i*-", ,, ,,
ot 1,'r:r pclpurati.n(clrthe s'b_
populationstbat you wanr to compare:
seethe thiri principle,,f ,a,npling,
be'rrv). The most str:rightf'rw"ri;;)';;;r",rnin
o represenrative sarnple,
c.nceprualll', is to rhink ab'r-rtserecting
individuar, .,;il[;;i'ni ronaun.,
frorn.the populati.n, so each individu"al
in rhe target popurati.n hrrs an
equal probabilitv being selectedf'r the ,,r,'r-rple.-A ,",rrpt.-r.i.*.a in
this waf is callecla'f sirltpri t'ancr.ctm
tn,"jir. rn p.u.ti.., rhough, a simpre
randorn sirmpie usually,is hard r,
olrtain, ,rth., ;";;;;'f;,r.,etirnd,
hirve devisedfor obtai.ning,"r,pf.l ilr*"ni
u.. ,.pr..e.tirtive, or vrrru_
'een
f<x target popurationsi.ruti a.r..ftions
"lly.,r:: of rhese merhods irrc
av ailablcels ew h e re(Ka l t< > n1 9 g 3 ; K i s h
1 9 6 sr,andI seeno needro cover
the sameground here.what I u'ant
,o a,, ;,r*."a is enrphirsizesome
arcl"ringprinciples. over-

F,irst.princiTtle
of sampring: you cl.n't need ro eat
rhe whore ox ro know
that it is tough.

.;- 1135[e en stu die d a n um -
: -rlready. It is generallY
,: ,:r thirt is imPortant Yet
, .;:. irt-rt'.rinin gsoln c t im c s
' r"' rind a n itttere s t ing
- ': r'\.lnlPles above are ln-
: -.r iltost often is research
. ,.J .rnc1continuing areas
. .. ..::J ro cleterminewhom
' '' ' t:- lr rpl1l31iot-tyour results
'-.'-rr . d in dr awing c o n -
. ::.'.tcould be overlYam-
. - - -'-i : r ior ldr v ide wor-rl db e
, ' -. ' '- ,n lv A r ner ic anc o l l e g e
-' : i.lrrple, you need to
, :-:-:Jl \.
. r r r ' c dt o det er m i n ea
:: r s'l|l1?le-from that
.,
- -.> lr r r lsabout t he e n ti re
- --:r : ih. lt P oP r iiat io ni,t i s
. I iopulation (or the sub-
' George C. \7allace, four-term governor of Alabama and unsuccessful
- : : :: nrirtcipleof samPling,
- I r' -':r.presentative sample'
- -:. .-.-..clmPletelyat random
'. : ': :.ti!et PoPulationhas an
- - - : :... ^\ san-rPle selectedin
.' ::,t;tic e, t hough, a s i mP l e
" -, ,,rher sampling methods
- - :-. r(rr es ellt at iv e.or v i rtu -
.- '.-:. :ri oi rhese methods are
'- ' . --.:r.1 I s eeno needt o c o v e r
. :..i rs emphasizesome over-
'--:
- r ' ..1tthe whole ox to know
The FirstRule . 19
SamuelJohnson is often credited (probably incorrectly) with this color-
ful way to express the idea-later applied to the sampling of human
populations-that characteristicsof the whole can be inferred from char-
acteristicsof the part. All sampling is based on this principle'
The same laws of probability that underlie the first principle of sarn-
pling also lead directly to the first principle of sample sile.
First principle of sample size: For practical purposes,very large popula-
tions do not require larger samplesthan smaller populations do.
Suppose we want to know, with a certain degree of precision, which of
two candidateswill carry the statesof California and\Tyoming in a pres-
idential election. If the race is equally competitive in both states, we do
not need a larger sample for California than we do for the state of
'lfyoming,
even though California has seventy times more people than
'Sfyoming
Wyoming does.aIn fact, if the race is closer in than it is in Cal-
ifornia, then we would need a larger sample to predict the winner in
Wyoming.
The first principle of sample size implies an important corollary.
Sample size corollary:'We can make confident generalizationsabout a
large population from a sample containing only a tiny fraction of that
population.
Although students generally have little trouble grasping the first principle
of sampling, they often are surprised by the first principle of sample size
and its corollary. It seemsto defy common sensethat anyone could deter-
mine what millions of Americans are thinking by talking to merely 1,500
of them. Yet that is exactly what pollsters try to do.
third-party candidate for president in 1968, provides the most memo-
rable attacks on the reliability of political polls and the logic of polling.
aNote that the principle states
for practical purposes. Strictly speaking, all elseequal, we
to the finite population correction
ryglq1neg4 ejlightly larger san.rplein California, due
rt(N-ny(N-t), rvhere n is sample size and N is size of the population. (This correction is
neededbecausethe standard error approacheszero as z approachesN.) In practice z is usu-
ally so small relative to Nthat the correction is trivial. According to U.S. Census Bureau es-
timates for 2003, California had 35.48 million people and Wyoming had 0.50 million
people. Thus for a sample size of 1,000, the finite population correction factor is 0.999986
for California and 0.998999 for Wyoming. In other words, becausea sample of 1,000 con-
stitutes a larger fraction of the population in Wyoming than in California, the standard er-
ror shrinks more in'Wyoming (by a factor <tf 0.998999) than in California, where the
standard error is reduced only by a factor of 0.999986. The practical implication is that, all
elseequal, we would need a sample size of 1,001 for California to match the precision of the
results for a sample of 1.000 in Wvomins.
20 . Chapter1

rWallace'spopulist "Stand up for America" campaign resonated

with
large segmentsof the U.S. public,'tand polls showed surprising supporr,
particularly in the South, for his third-party candidacy. However, when
polls showed his support slipping in the final weeks of the 1968 elecrion,
\Tallaceattackedthe polls themselves.In an article "\X/allaceAssails'Lv-
ing' Election Polls," the Neu, York Times (October 27, 1968) reports:

George C. Wallace delivered a blistering attack on national political

polls todaf in responsero one that indicates support for his Presiden-
tial can-rpaignhas slipped substantiallyin the last week. Informed of
the results of rhe latest Gallup Poll, which shows that the Amencan
Independent party candidate dropped 5 percentage points, he said:
"Thev lie rvhen they poll. They are trying to rig an election. Eastern
monev runs everyrhing.They are going to be pointed out the liars that
t hey ar e. "

Two days later Wallace dismissedpolitical polls as "comic srrips" and

charged that the polls represent"a deliberateand desperateattempr on
the part of the other two parties to deceivethe American people" ("Sfal-
'Vlashington
lace, Irritated b_vPolls, Insists He Is Doing V/ell," Pos/, Octo-
ber 29, 1968, p. A7). In a campaign rally in Hannibal, Missouri, on
October 28, 1968, he posed this famous quesrion,as describedthe next
day in the Washington Post: "'The Gallup poll and the Harris poll-they
talked to 1600 people in the country and say rhey can decide how the
election's coming out,'the former Alabama Governor sniffed. '\Well,how
many of you have beentalked to by the Harris poll and the Gallup poll?'
he demanded.'Not a one of you, not a one of you.'"
Governor 'il/allace'sviews notwithstanding, reputable preelection polls
of likely voters have been fairly accurate in forecasting the results of re-
cent elections, even though the polls typically are based on a sample of
only a dozen or so respondents per milliotl voters. On the eve of a presi-
dential election, for example, most national polls rely on a sample of
1,500 or so likely voters. C)ver 1,22million Americans voted in the 2004
election-so a sample size of 1,500 works out to a ratio of one person
sampledfor every 81,000 voters,or 0.0012 percentof the voters.
How then are we to account for the fact that we can use 0.0012 percent
of the population to predicr so accurately the behavior of the remaining

5\x/allace'scampaign brochure describeshim

as "the undisputedleaderin the fight for per-
sonal and property rights, ar-rdagainst excessivetaxation and the takeover of personal rights
by the 'great societ1,."' Regarding the Vietnam War, rWallace is quoted as follows in the
brochure: "These few people today who are out advocating sedition and raising money and
clothesand suppliesfor the Viet cong-these college professorswho are making speechesad-
vocating victor,v for the viet cong communists-I would deal with these people as they ought
to be dealt with, as trairors." (http://www.4president.orglbrochures/wallacelg68brochure)
 The FirstRule . 2l

99.9988 percent? This principle is demystified somewhat by the second

principle of sampling.

Second principle of sampling: How casesare seiectedfor a sample is

more important than how many casesare selected,

In other words, the representatiueness of the sample is more impclrtant

than the size of the sample. Of course I do not mean to suggest that a
sample of two or three is all right, so long as those people are selectEd
properly. But I do mean to suggestthat a sample of only a few hundred
can give a surprisingly accuratepicture of a population of many millions,
so long as the individuals in the sample are representativeof the individu-
als in the target population. Obviously a larger sample is better than a
smaller sample, other things equal-but the superiority of larger samples
over smaller ones is itself due to the fact that (for a given sampling
method) larger samplesare more likely to be representative.Consider the
extreme caseof a single individual. A sample of one cannot be very repre-
sentativeof the entire population, unlessindividuals in the population are
all alike. At the other extreme, a census is clearly representativeof the
population, since it contains everyonein the population.
It is important to understand two points here: first, that a representa-
tive sample of a few hundred people can be remarkably accurate; second,
that a nonrepresentative ("biased") sampleof a few million peoplecan be
remarkably inaccurate.Failure to appreciatefully the secondpoint has led
to some infamous blunders in the polling business.In the United Stares,
"the poll that changedpolling" occurredin the 1936 presidentialelection
between Democrat Franklin D. Rooseveltand Republican Alfred M. Lan-
don. In its October 31,1936, issue,the venerablemagazineLiterary Digest
predicted-on the basis of a sample of ouer two million-that Landon
would win the election in a landslide. Three days later, Rooseveltwon in
a landslide. The error by the Literary Digest was causedby their sampling
method, which oversampled Republicans and those favoring change in
the government (Rooseveltwas the incumbent). On the basis of a more
representativesample of just 5,000 people, a little-known pollster named
George Gallup correctly predicted Roosevelr'slandslide victory. Not long
after the 1936 election, the Literlry Digest went out of business.The
polling methods perfected by Gallup are srill in use today.6
Becausescientificpolling has come a long way since the 1936 presi-
dential election, it is hard for individual researchersto match the general

6Controversies over the results of reputable polls (such

as Roper, Harrris, or Gallup
polls) rarely center on sampling issues any more. Most of the focus today is on validity
issues-on what respondents really mean by their responses(see Sudman, Bradburn, and
Schwarz 1995; Tourangeau, Rips, and Rasinski 2000).
22 . Chapter1

quality of datacollected by professionalfirms. Most social scientisrs,then,

rely on survey firms for the collection of the survey data they use. That is
one reason for the popularity of secondary analysis in the social sciences,
that is, the analysisof data collected by someoneelse.Of course,with sec-
ondary analysisof national data you cannot tailor the data to very specific
or local populations-you would not, for example, be able to study the
political attitudes of students in your own college with national survey
data. National survey data on the attitudes of college students do, how-
ever, provide a benchmark for the attirudes of students in your own col-
lege. In short, secondary analysis of national survey or census data very
often is called for evenin studieswhere researcherscollect their own data,
since the national data provide a point of comparison for the more local
or specializeddata.
Large survey data sets have become increasingly accessibleand user-
friendly over recent decades.Many of these data sets have been under-
written by government money, and researcherswho use public funds tr,r
collect data generally are required to place their data in the public domain
for others to use. Major social sciencedata setsare located in severaldata
archives in the United States and other countries. Probably the best-
known U.S. social science data archive is housed at the University of
Michigan, at the Inter-University Consortium for Political and Social Re-
search (ICPSR), web site http://www.icpsr.umich.edu.Undergraduare
and graduate students typically have accessto the national data setsmost
commonly used in social research.
Third principle of sampling; Collect a sample that permits powerful
contrasts for the effects of interest.
If you are interested in the effects of cohabitation, for example, then
make sure you have enough cohabiters in your sample to permit the com-
parison of cohabiters with others. If you are collecting your own dara,
you will want to ouersamplecohabiters. You can oversamplecohabiters
by using a method called stratified random sdmplin& First stratify on rhe
basis of union type (cohabiting versus married); then randomly sample
within each group. To oversamplecohabiters, simply selecta larger frac-
tion of cohabiters than marrieds.
As this example suggests,simple random sampling of the overall popu-
lation is not always the best method. Sometimesit is better to divide the
population into homogeneousgroups and oversamplethe minority groups.
If there are G groups, we can think of G samples,one for each group. So
long as you randomly sample within each group, each sample is represen-
tative of that group, and there is no problem in making comparisons
across groups. But the G samples merged are not representativeof the
overall population. Suppose,for example, that a preelection poll heavily
 The FirstRule . rj

oversamplesAfrican Americans. To predict results for each candidate,

you would need to downweight the African American sample to reflect
the proportion of voters who are African American, or your predictions
will be entirely too rosy for Democratic candidates(sinceAfrican Ame ri-
cans tend to vote overwhelmingly for Democrats, and whites do not)' I
discussthese issuesin more detail in the next section.
Finally, note that the probability of selection into a sample should not
depend on potential outcomes. Typically you stratify on the basis of ex-
planatory variables.By doing so, and oversamplingcasesfrom the minorrty
groups, you increasethe variance on your explanatory variables (demon-
strated in the next section). Becauseyou can't explain a variable with a
constant (chapter 2), it's a good idea to boost the variance on your ex-
planatory variables.Increasing the variance of your explanatory variables
is especiallyimportant when you have a small sample' as we now see.

Senprps rN Quarrrerrvn' Sluprr,s

The third principle of sampling provides a nice bridge to the discussionof

sampling in qualitative studies,sincesamplesin qualitative studiesgenerally
are much smaller than samples in quantitative studies. All the sampling
principles above apply to qualitative studies: You don't need to eat the
whole ox; large populations do not require larger samplesthan smallerpop-
ulations do; how casesare selectedis more important than how many cases
are selected;and you might need to stratify your sample.
lWhat rs different, usually, about qualitative studiesis the limited number
of casesstudied. To understand what difference that might make regard-
ing sampling strategy,we need first to identify more precisely the advan-
tagesof large samplesover small samples.By identifying those advantages,
we can think about sampling strategiesthat qualitative researchersmight
use to compensatefor small sample size.
Unless the sample size is very small (say 50 casesor fewer), the chief
limitation of a small sampleis not that it fails to provide a modestly reliable
description of key featuresof a population. A small representativesample
of 100 generally yields fairly accurate point estimates (to use statistical
parlance) of a population, where that term refers to a sample statistic that
summarizessome important population characteristic,such as averagein-
come, the ratio of men to women, the percentagemarried, or (as in the
'Wallace
George example) the percentageof the electoratewho will vote
for a particular candidate.
Instead, the chief limitation of a small sample is its lack of analytic
power-its lack of leveragefor separating the effects of multiple causes.
The problem is especiallyacute when causeshave little variance and (as is
24 . Chapter1

most often the case)covary wirh other causesor correlatesof the depend- and a minority population .::
ent variable. In general, the more bighly correlated the cduses,the more estimates of differences be:'.'.
caseswe need to sepdrate tbeir effects. sample of casesfrom each :
To illustrate, suppose you want to tesr a theory about only-children. representativefor the PoP;.::
Your theory argues that children reared in a home without siblings grow encesabout that overarch::-: :
up in a very different environment from that experiencedby children with sampl esto m ake t hem r eP: . . . :
siblings, and that the difference in childhood environment implies differ- lustrate, imagine we find. ::.
ent adult outcomes-perhaps only-children tend to be more self-confident ers, that adults who \\-ere , :
as adults, and more likely to assumeleadershippositions. You want ro ex- averagewhile other adults -:..
amine this theory by doing a qualitative study of 100 American adulrs, ucational difference is ts'i' .
some of whom have siblings and some of whom do not. .Weassumethat ti on for a ll adult s is 11, . ej -
N= 100 is a fixed upper limit-time and budget consrraints will not allow weights of the two group: .: :
you to collect data on more than 100 individualstotal. In short , t he use oi 5f r . r :- :
One issue you must worry about is obtaining enough variance on your populationsis often strilt-i,- :
explanatory variable. The problem arisesbecauseonly about 5 percent of groups, particularlv in th; -:
American adults are only-children. So in a random sample of 100 adults, sample size may he nto.:..:
you expect about 5 only-children-hardly enough to make confident in- oversamplingof this [\ Fs ,..
ferencesabout the differences berween only-children and others. The so- small samples:the lack (rt :
lution, as before, is to use stratified sampling: Divide the population into C onsi dcra lr er nat ivcc\ f t - "
those who are only-children and those who are not, and randomly select no-sibs (no siblings) i-Ind: -'
50 adults from each group. In other words, you deliberately oversample ri st, for ex am ple,m ight 'l: : - . .
only-children. In that way you increasethe variance on your independent fi rst-bornef f ecrin dt r {t t : - . .
variable(let'scall it no-sibs)more than five-fold, from0.0475 to 0.25 (the senceof siblings,but bcr : :
variancefor a dichotomy is n(1-n), where n is proportion in one of the lated with the variable first-'r
categories). children. The correlation :-. ',
By increasing the number of only-children from 5 to 50, you increase children are only-childre: --.:,
the reliability of your estimatesof Y (your dependentvariable) for only- the bi rth-or der ef f ect f r , : : : -
children. Of course, you also reduce the reliability of your estimatesof y principle. But to do that ',',. -
for others, since you have reduced the sample of others from 95 down to groups, not two-onlr.-;:-: :
50. But the gain in reliability in moving from 5 to 50 casesmore rhan off- not first born-and comi.,-:: :
sets the loss in reliability in moving from 95 to 50. You can have more l f i t i s no-sibst hat m . t r r r : - :
confidence in your estimatesof growp differenceswhere the groups con- group should differ from l :
sist of 50 each than in estimateswhere N= 5 for one group and N= 95 for i f i r i s fi rst - bor nt hat m lr t ; : - -
the other. should be between the :.:.:
But, you ask, doesnor oversamplingfly in the face of the notion of rep- for fi rst bor n wit h or r r i: - - '
resentativesampling? It was the oversampling of Republicans that led to first born.
the infamous Literary Digest blunder on rhe 1936 presidential election.
How are we to reconcile the advice here to oversamplewith the earlier -Some
m erhodol ogi .t: .rr- :-
discussionabout the importance of using representativesamples? mi ni sti c rath er than probebrL.:::. .
The key is to note that the aim of the Literary Digest poll was to draw tion. If so, then small sample. ; - : -,
to eliminate a different erpl.::.-:: -
inferencesabout a single population (voters in the 1936 election), whereas
r [q9l ) poi nts out, s ma]l \ - -- .
the aim of the only-child researchis to draw inferences about differences here becauseI believethere rs ..-:.: :,
between two populations, a majority population (adults who had siblings) social research (seerule 5 t.
 The First Rule . 25

and a minority population (adults who were only-children). For the best
estimates of differences between populations, we want a representative
sample of casesfrom each population. The sample does not need to be
representativefor the populations merged unless we want to draw infer-
encesabout that overarching population-then we need to reweight our
samplesto make them representativeof the overarching population. To il-
lustrate, imagine we find, in our 50/50 sample of only-children and oth-
ers, that adults who were only-children attain 14 years of education on
averagewhile other adults average12 years. Then our estimate of the ed-
ucational difference is two years, but our estimate of the averageeduca
t ion for a ll adult s is 12( .9 5 )+ 1 4 (.0 5 )= 1 2 .1 y e a rs ,re flecti ngthe rel ati ve
weights of the two groups in the overarching adult population.
In short, the use of stratified random sampling to oversample smaller
popul a ti o nsis of t en s t r ate g i cfo r c o mp a ri n gm i n o ri ry groups ro maj ori ty
groups, particularly in the case of qualitative research, where the total
sample size may be modest. In the final analysis, though, purposive
oversampling of this type cannot alleviate a fundamental problem with
small samples:the lack of power to adjudicatecompeting explanations.T
Consider alternative explanations for an association between the variable
no-sibs (no siblings) and some outcome variable Y. A birth-order theo-
rist, for example, might argue that any only-child effect that you find is a
first-born effect in disguise. In other words, what matters is not the ab-
sence of siblings, but being born first. And no-sibs is inherently corre-
lated with the variable first-born, since only-children are also first-born
children. The correlation nonethelessis not perfect, since some first-born
children are only-children and some are not-so it ls possible to separate
the birth-order effect from the only-child effect empirically, at least in
principle. But to do that we need to divide our sample further into three
groups, not two-only-children, first born among those with siblings,
not first born-and compare the outcome variable Y for the three groups.
If it is no-sibs that matters for the outcome, then Y for the only-children
group should differ from Y for the other two groups. On the other hand,
if it is first-born that matters for the outcome variable, then the difference
should be between the first born and others: Y should be the same
for first born rvith or without siblings, and differ for those who are not
first born.

TSome methodologists and philosophers of social scienceconceptualize cause as deter-

ministic rather than probabilistic, so a purported cause can be eliminated by a single excep-
tion. If so, then small samplescould be analytically powerful, sinceeach casehas the potential
to eliminate a different explanation for some phenomenon. In that tradition, as Lieberson
(1991) points out, small N's can lead to "big conclusions." l do not pursue that approach
here becauseI believethere is an alternative view of causality that holds more oromise for
social research (seerule 5).
26 . Chapter1

The important point here is the demand placed on data by the need to
examine alternative explanations, even when we stratify on explanarory
variables. The more explanations there are, the more such comparisons
we need to make; the more comparisonswe make, the more finely we must
stratify; the more finely we strarifv, the smaller the subsamples upon
which our comparisons are made. The solution to this dilemma is a large
overall sample. That is what I mean rvhen I say that a larger sample has
greater "analytic power"-a larger sample permits more comparisons.
and with more comparisons you can investigatemore explanations.
Without the luxury of a large sample to provide the basis for one com-
parison after another, qualitative researchers very often must decide
which comparisons are strategic,and selecta sample accordingly. It helps
to stratify on explanatory variables and oversample the smaller popula-
tions within those strata, but a stratified sample helps only up to a poinr:
In the final analysis there are limits to the number of strata you can use
when the overall sample is small. As a result, qualitative studies generally
are not well-suited for omnibus tests of many.'different explanations for
some observed association. Qualitative methods are well-suited for pro-
viding thick description that can help place quantitative results in proper
context. Qualitative methods sometimescan also provide precise critical
tests-tests that often are difficult with conventional quantitative ap-
proaches. Indeed, through strategic data selection, qualitative approaches
often extend or correct quantitative studies, as rve discover subsequently
with examplesof rule 3, "Build reality checksinro your research."

I s M T I NT Nc F U LS o c rA LR rs s e n c n P o s s r err?

October 7,1903, is famous in aviation history for the tesr thar failed. As
news reporters watched expectantl,v,test pilot Charles Manly assumedhis
position aboard a curious-looking airship perched on a houseboat in the
Potomac River. To avoid fatal landings, the plan was to crash into the wa-
ter after demonstrating the possibilitv of human flight in a heavier-than-
air machine. As propeller wheels whirred a foot from Manly's head, a
mechanic cut the cable holding rhe carapult, and the airship tumbled ig-
nominiously into the warers of the Potomac.
That spectacular failure, along with a second well-publicized plunge
into the icy Potomac two months later, resulred in public ridicule of the
airship's designer,the distinguishedphysicist and astronomer Samuelpier-
pont Langley. Armed with a $50,000 grant from the ITar Department
and another $20,000 from the Smithsonian Institution (which combined
would be worth over $1.5 million in today's money), Langley had set out
to build a heavier-than-air aircraft piloted by humans. His highly visible

-:: : : ,:-:i o n dat a by t he ne e dto
: .i .: '.'.: stratifl.'on explanatory
-: : - : . :-.: ::u r e : uCh c om pa fi s o n s
! .r : ::--.... rhe more finelywe must
:- . ; : - .,,..c: th e s ubs am plesu p o n
: i," : r :,r this dilemma is a large
.,r:.- - :.:'. :har a larger samplehas
.:. -- : . : ; J: : : l!5 mo re COmp ar iSO nS.
' -: - (rre exnla na tion s.
'. . :....
: : :. ..;. rhebasisfor onecom-
'"; ; . - -: . : '..rr often must decide
!;
-i : :,.::rrle accordingly.It helps
::i-,-:.rie the smaller popula-
: -': - ^ :.:: onlv up t o a p o i n t:
: r -..r-.r:lof strata you can use
:. - --'--ri.rtilestudiesgenerally
::crent explanationsfor
r : . - i: .;'e $-ell-suitedfor pro-
r : -: : .:..-:::ilatile resultsin proper
:,: .;- .r.:r provide precisecritical
: . , :. ":::uonal quantitative ap-
:-z '.: :-: -:1.qr-ralitative approaches
-* : : . .r:'.'.edis c ov ers ubs eq u e n tl y
- l.:- : : : ! (l V O U f f e s e a f c h . "
. :,,r rhe test that failed. As
i::rles Manly assumedhis
,:.J on a houseboatin the
\\'asto crash into the wa-
, - - '_r. "'b'_
rl i 'eht -in
__
*a heav
' ier -th a n -
:::: , -': '!rt irom Manly's head, a
:- : : - : . ,: J the air s hip t um ble d i g -
t- .
:-- i . . , :J ri ell- public iz edpl u n g e
-_ i ;- ^PuuuL
, , k t ; ^rI. idic ule o f th e
'rr
-i
:: :t -. lstronomer SamuelPier-
'. -
!-,: ::,tm the 'war Department
-:
-:..ritution (which combined
': - - I ^-^l^-, L^l
:r\)ne\'),
-----,\ Langleyhad set out
^^- ^,.-
:'. r'limans,His highly visible
The First Rule . 27
failures added credibility to Lord Kelvin's famous claim, eight years ear-
lier, that "Heavier-than-air flying machines are impossible."
After Langley's firsr failed attempt a New York Times editorial had
this to say about the prospects of human flight ("Flying Machines Which
Do Not Fly," New York Times, October 9,1903)t
[k] might be assumedthat the flying machine which will really fly might
be evolved by the combined and contir-ruousefforts of mathernati
cians and mechaniciansin from one million to ten million years. . . .
No doubt the problem has attractions for those it inreresrs, but to
the ordinary man it would seem as if effort might be employed more
profitably.
In one of the ironies of history, as thesewords were being penned \Wilbur
and Orville'Wright were preparing their own launch in a remote area on
the coast of North carolina. The first manned flight did nor take place a
million years later, but just a little more rhan two months after the Times
editorial. On December1,7,1903, Orville and'sfilbur took turns piloting
a flying machine (that cost about $1,000 to build-$22,000 in today's
money) over the sand dunes of Kitty Hawk. With no governmenrfinansial
backing and little fanfare, two determined bicycle builders from Ohio
had proved Lord Kelvin and the New York Tinres wrong,
In academiathere are Lord Kelvins among us today who make similar
assertionsabout the impossibility of meaningful social research. Let me
be very clear: I am not talking about those who say that good social re-
search is difficult, or rhat social researchis often badly done, or thar some
questions about human behavior are inherently unanswerable. (I happen
to agree with all those assertions.)Rather, I am talking about those who
are skepticalabout the possibility of discoveringgeneralprinciplesof hu-
man behavior. In the view of such skeptics, either such principles don't
exist, or the principles are so "contextualized" that we cannot generalize
from situation to situation. Empirical nihilism of this sort tends to be most
prominent in schools of thought containing a "post" prefix: postmod-
ernism, postpositivism,poststructuralism,post-Kuhnian philosophy of sci-
ence,and so on.
In its most extreme version, empirical nihilism in the social sciences
denies the possibility of discovering even regularities in human behavior.
That position is obviously silly. consider the life insurance industry. Ac-
ru ari al tabl espresumeknow l edgeabout regul ari ri esi n death r "ur . r .Al-
though we cannor predict when specific individuais will die, we do know
that women tend to outlive men (in most Western countries anyway), that
nonsmokers tend to outlive smokers, and that it's good to exerciseand eat
your vegetables.or consider government efforts to monitor economic
growth and control inflation. If the discoveryof generalprinciples beanng
28 . Chapter1

on economic behavior were beyond the ken of humans, the Federal Re-
serve Board would have no reason to meet to discuss alterations in the
prime rate.
Empirical nihilism is nor new, nor is it unique to the social sciences.
Before attempting to discover principles about the world of narure,
physicistsand other natural scientistspresupposethat observablenatu-
ral phenomena,such as motion, are real, and that they are arnenableto
investigation. The ancient Greek philosopher Zeno of Elea believedother-
wise. Zeno, like his teacher Parmenides,vierved the world of senseas an
illusion. To defend that view, Zeno used paradoxes to demonstrate the
impossibility of motion. One paradox can bv illustrated by imagining an
arrow in flight. At each mornent in rime, rhe arrow is located at a specific
position. In subsequentmoments, the arrorv is at rest for the same reason.
So the arrow is at rest at all moments. Thus if time is a continuous series
qf moments there is no time for the arrow ro move, and motion is impos-
sible. The paradores are challenging-thev have engaged philosophers
and otherssincethey were introduced by Zeno almost 2,500 yearsago-
and important, since they undermine the con\ienrional view that the sen-
sory world is real. (The arrow paradox and other paradoxes of motion
credited to Zeno are generallythought to have been resolvedby calculus
and Newtonian physics, although quantum mechanics has complicated
that resolution in some instances.)
Is the social world real? Social sciencer,r'ouldbe a strangechoice of vo-
cation for someonewho answered that question negatively.Yet if we press
the social contextualization argument far enough rve arrive at an appar-
ent denial of the social world. Substitute the term "context" for the rerm
"moment in time" and you see that the contextualization argument is
similar to Zeno's argument about the arro\\'. In effect Zeno viewed each
moment as a separatecontext and concluded that the physical world is an
illusion. If social acts are so radically contextualized that there are no
general principles to discover, then one u'onders what is the glue that
holds human societiestogether. The existenceof routine, predictable hu-
man interaction suggeststhat there is something "there" in the social
world, even as the existence of motion suggeststhat there is something
"there" in the physicalworld.
The best responseto empirical nihilism is to ignore it and do the re-
search. The aim of this book is to help both novice and experiencedre-
searchers"do the research." I have tried to make the book eminently
practical.I provide examplesof good social research,as well as examples
of the other kind. obviously I believe that meaningful social research is
possible-imperfect research, but research that can be valuable nonethe-
Iess.(I say "can be valuable" becausethe findings of social researchare
subyectto misuse.In the worst-casescenario,social sciencecould be used

-:::n.ut q,t he F eder a lR e -
- .;.rss :rlterationsin the
: -- --.: i o t he s oc ial s c ien c e s .
- , - - ..: :n e u' or ld of nat u re ,
. . . : i : :h. r r obs er v ablen a tu -
: -- .--r: thev ar e ar nenab l eto
- : * :.. ,.'iE leabeliev edoth e r-
: : - ::- .. or ld of S ens ea Sa n
. . 1---\r'i ro demonstrate the
. .::.1. r eJ
bv im aginin ga n
- : .'- .,,-- locatedat a specific
, - -: :::: f or t he s am er ea s o n .
. .r uor.rtinuousseries
. - ;. .:rd motion is imPos-
': - ',,: :lrg.lged philosoPhers
, -. - : :r ,,s r3. 500 y ear sag o -
-- :..... ::l vieu, that the sen-
-
1 - i :.:: :J r adox es of m o ti o n
- r -:::-. r es oh' edbY c alc u l u s
- :, -.,..ai.sharscomPlicated
.- - i: -1itrange choiceof vo-
: - - .::.'.ti\.el\'. Yet if we press
:- .-.-.-...a .lrrive at an appar-
Jrlnfext" for the term
: : 1.:,.,1l1zi1tion
argument ls
:-- -' .:::;t Z eno v iewed e a c h
. ,.: :re
-:: - . P hv s ic alwor ld i s a n
- - '-, :-.'. :ze d t hat t ner e are n o
-,: 'i::: \\'ht1tis the glue that
-i'. - -: : :, ,tltine,predictablehu-
: "there" in the social
::r.rt there is something
:i n or e it and do t h e re -
: - :-. .,:Je and experienced re-
: ' ::,:.{- the book eminently
: ::-:--.:.h. as rvell as examples
.: r':.r:lrrgful social researchls
- : .:: nonethe-
-.rn be valuable
- :.:.:s of social researchare
. . . ';ial sciencecould be used
The FirstRule . 29
by authoritarian governments for the purpose of large-scalesocial engi-
neering, which generally has been a miserable failure.) The pitfalls of so-
cial researchmay be many, but so too are the potential benefits.The seven
rules are intended to move us closer to that Dotential in our social re-
searchprojects.
Sul.uenv
There should be the possibility of surprise in social research.Social re-
search requires that researchersremain open to new evidence,whether or
not that evidencesupports their cherished theories or beliefs. This open-
ness leads to the possibility of surprise. Openness also makes research
more exciting. If we know the results beforehand, what is the impetus for
the research?
An important first step for effective research is choosing an appropri-
ate researchquestion-one where surpriseis possible.Researchquestions
should be both researchable and interesting.Researchquestionsare unin-
teresting when there is no possibility of learning anything new, lvhen
there is no perceivedrelevanceto theory or everydaylife, or when the an-
swer is significant only to a very small number of people.
Becauseimportant social issuesof the day typicaliy have been investi-
gated in one form or another by social scientists,most often researchevi-
dence already exists on the question you want to investigate.In that case
you need to design your research to ensure that surprise is possible. The
first step is to distinguish well-establishedfindings from the other kind. By
"the other kind" I mean findings that are mixed or inconclusive, or find-
ings that are new and have not yet been replicated in subsequentstudies.
Surprise ls possible in those instances,becausein those instancesthere are
lingering questions about earlier results. Hence your research,if propcrly
executed, contributes to knowledge.
By contrast, if prior findings are widely viewed as settled, you con-
tribute little or nothing by merely repeating the analysesof others. Instead
you must either challenge the earlier findings or extend/clarify them. If
there is a good reason to doubt the results of prior studies, then of course
it is worthwhile to challenge them with new research. In that case,
though, the burden of proof rests on you to identify the flaws in pnor re-
search that are serious enough to cast doubt on their findings (flaws that
you will correct in your research).The other strategy-to extend or clar-
ify earlier findings-can be implemented by examining results for a sub-
population or for a different population or time period. New understanding
of old findings can also be gained by introducing mediating variables or
moderating variablesor nonmediating variables (or some combination of
30 . Chapter1

all three), or by testing for reversecausation. In some instancesa synthe- \P P E N DI X

sizing study may be possibleby bridging prior researchin different sub-
fields.
After deciding on a researchquestion, you must decidewhom to study. StudentExerciseson
First you decide what population vou want to examine. Then you selecta
sample of that population since generallr.,vouwill lack resourcesto collecr
data on the entire population. The important feature of a sample is not its
size (within reason)but its represenrariteness. til/hensamplesare properly
selected,it is possibleto draw defensibleconclusionsabout large popula- The exercisesfor chapt.'l1
tions (for example,all U.S. citizens)from analysesof only a tiny fraction Survcy(GSS)-The GSSr. ,
of the population. These underlying sampling principles are rhe same for English-speaking indiviJ::,:-
qualitative and quantitative research.Qualirative research,however, usu- From 1972 to 1.991the G>
ally relies on smaller (sometimesmuch smaller) samples.Stratifiedran- missed);after1994 it hi. :.
dom sampling that purposely oversamplessmaller populations might samplesizesof eachu',rr'; ', '-
compensatefor the limited samplesize,particularlf if the aim is to tesr an to about3,000 f<-rrtlre L.i.::
uncomplicated model. A small sample does, hoq'ever,limit your ability ro To do the exercises b.-. ''
make multiple comparisonswith different variables.For this reason, qual- elickon "SDA ar.hir(. .r..
itative studies generally are ill-suited for pror-iding omnibus rests of mul- tions shouldgenerateall lr- .'
tiple explanations for some observedassociarion. you should not need to .j,' ...
about the statisticai:ig,nri.,.:
.01 significancelevelas thc .
the p-valueis .01 or less.i', .
nificant. To assistyor.rin ;, :
boldface in the comput€r -r::
problems,go to http://pre...
N ote well: ( 1) The hgu: .
most recent CSS dat . r . r r :
working with more recen: i-
low you will need to use : :
move years after 200+. -
format used by SDA ar th.
formar might changesligi: :
directions below might b'c
low the logic of the instrr;:
the questions.

GsNoEn DrpppRpNcrsir .j.

The GSS asks the folior..-

"Pleasetell me whether c,: ,-
nant woman to obtain a ..
'
son?" [possibleans'ur,ers:
.i: - .ome instancesa synthe- AP PE N D IX
: - . . :i:earch in different sub-

StudentExerciseson Rule 1
;- - . . ,:-::rine. Then you Selecta
. ,', --rckresourcesto collect
. .i..;,r:c of a san-rple is not its
-i- -.r.,-.:ns am plesar e pr op e rl y
; . . . r-
The exercisesfor chapter 1 use 1972-2004 data from the General Social
: - - ..i:,rnsabout large PoPula-
,i- ,- -. '. ... 1-fonl,v a tiny fraction Survey (GSS).The GSS is a probability sample of noninstitutionalized
- : ::.:;i pl e s ar e t he s am efo r English-speakingindividuals, age eighteenand older, in the United States'
-: From 1972 to 1.994 the GSS was an annual survey (with a few years
\-: : : : : ::sear c h,howev er ,u s u -
- :.-. :r ..:mples.Stratifiedran- missed); after 1994 it has been conducted in even-numberedyears' The
r :. ,:'.:- populations might sample sizes of each wave vary from about 1,500 for the annual surveys
-r to about 3,000 for the biennial surveys.
r - - - .. .-. . :l r he aim is t o t es t a n
,..''.::. limit your ability to To do the exercisesbelow, go to the web site http://sda'berkeley.edu,
--1r.
click on "SDA archive," and follow the instructions below. The instruc-
:-: : : :: :r. Fc lrt his r eas on,q u a l -
t es t sof m u l - tions should generateall the statisticsyou need to answer the questions,so
: :: :.:--: '-r m nibus
t-
you should not need to do any calculationsby hand. Severalquestionsask
:: l-
about the statisticalsignificanceof associations.For theseexercisesuse the
.01 significancelevelas the criterion for statisticalsignificance,that is, when
the p-value is .01 or less,we consider the associationto be statisticallysig-
nificant. To assistyou in completing the assignments,variable namesare rn
boldface in the computer assignmentsin this book. For answersto selected
problems, go to http://press.princeton.edu/titles/85 93.html.
Note well: (1) The figures given below are for the 1'972-2004 GSS,the
most recent GSS data available when this book was written. If you are
working with more recent data, to replicate exactly the figures given be-
low you will need to use the "selection filter" in the SDA program to re-
move years aker 2004. (2) The instructions below are based on the
format used by SDA at the time this chapter was written, in 2006. That
format might change slightly in subsequentyears, so some of the specific
directions below might be out of date, but you still should be able to fol-
low the logic of the instructions to obtain the tables you need to answer
the questions.

GpNonRDrrrsnENcns rN AeoRtroN ATTITUDES

The GSS asks the following question, denoted abany, about abortion:
"Pleasetell me whether or not you think it should be possible for a preg-
nant woman to obtain a legal abortion if the woman wants it for anv rea-
son?" [possibleanswers:yes, no]. This question is askedafter a seriesot
 32 . Chapter1

questionsabout particular reasonsfor abortion (examples:should abortion

be permitted if a woman is married and does not wanr any more children?
if there is a strong chanceof seriousdefect in the baby?), so in context the
question askswhether a woman should have the legal right to an abortion
for any reason at all.
Use the Berkeley SDA rveb site to cross-tabulateabany and sex (male,
f em ale) .F < ll l o wrh e s ei n s rru c ri o n s :
. From the SDA homepage, click on "SDA archive," then go to
the
" GSScumulativedatafile."
. A new screen should appear. On the new screen,type in abany
for
the rorv variable and t1'pein sex for the column variable.
. Ignore rhe boxes for "control." Use the drop-down box for ,,\ileight"
to select"Oversamp-Weight for black oversamples."
r Under "Table options," you wanr (for now) both column percent-
ages and row percentages,so check those boxes if they are not al-
ready checked on your screen.
o Check the box beside "statistics," and use the drop-down menu to
indicate that you want 3 decimal points for statistics.
. Check the box beside"Question text." This gives you the wording
of questions.
. Removethe check beside"color codine."
o Under "Chart options," use the d.op:dor.'., menu to indicate that
you want "(no chart)."
r Click on "run the table."

"Run the table" should give you a 2x2 table with abany as the row
variable and sex as the column variable. Nore the legend for the table
under "Frequency Distribution." The numbers in your table should in-
dicate that there are 11,389 male respondentsand 14,7rs female re-
spondents, for a total N of 26,104 cases.Your table should say that
41. 5 per c ent o f m a l e re s p o n d e n rsa n d i 9 . 4 percenr of femarerespon-
dents responded ves, that a woman should have the legal right to an
abortion for any reason.You should also find that 55.1 percent of those
who agree (that abortion should be a iegal right for any reason) are
women, and 44.9 percent of those who agree are men (secondrow of
t able) .

Question 1. (a) which of the two comparisons_47.s percenrversus39.4

percent, or 55.1 percent versus 44.9 percent-is the relevant comparison
here?Explain. [Hint: Percentagesshould sum to 100 percent within cate-
gories of the explanatory variable. Does gender affect abortion attitudes,
or is it the other way around?]
(b) Summarize your finding in one or two sentences.

:' :. ' :\amples: shouldabortion
. J :: - : \\ .lnt any more children?
, , ,s o in c ont ex tth e
:: - . . . . : .-.;1 '31.?)
i: : :'- l.'galright to an abortion
- r.-t :- -: .rreabany and sex (male,
- -r- -....rch ir . e, t"hen go t o th e
:--.: -:.' :-iecn. tvPe in abanY fot
: : : - .-.:In lar iable,
;: rr: rr :-iou'n bor for "'Weight"
'::>-rm ples . "
l. i- ,
'.'. both column percent-
: : :
:-. : - .: ^ ,-'res if t hey ar e no t a l -
- tl- ..-; :re d r oP - downm en u to
\--1L l)Lll) .
. gives 1'ou the wording
::
:-: .,.:-nenu to indicatethat
: :. :re legend for the table
"
- -' r : :: : n vour table should in-
: : -: : : -.:: ,rn d 14, 715 f em al e re -
- l: : ': ).:l r ta ble s hould s ay th a t
- - - r r-r.ent of f em ale r es p o n -
-:,- ;r...-:r.rvethe legal right to an
.. . i: . : rh ar 55. 1 per c entof th o s e
. :i. . . - rrght for any r eas on ) a re
.*, -:-:::. .rre men (secondrow of
tr: .: : ' .:--1 1.5 per c entv er s us3 9 .4
. : - --:'.:, ' 100 percentwithin cate-
,:: i::.'.:: rffect abortion attitudes,
.\ .:
-iitlq).
The FirstRule . iJ
Now look at the figures under the heading "summary statistics." The
statisticslabeled ett) R, Somers' d, gamma, taw-b, and tau-a rcfer to mea-
sures of association,that is, they refer to how strongly thE tw<.rvariables
are related. In this casethe figures are close to zero) since the association
between sex and abany is small, as you can tell from the percentap5es.
Question 2. (a) Is the genderdifferencein abany statisticallysignificant?
To determine this, look at the p-valuesfor Chi-square (labeled Chisq(P) in
the printout).p=.00 in the printout means that the p-value is less than
.0O5-which is highly significant. The smaller the p-value, the more con-
fident we can be that an associationwe observe in our data is not due to
sampling error.
(b) Is the genderdifferencevery large?What doesthat suggestabout statrs-
tical significanceand substantiuesignificance(how large an effect is)?
(c) Are you surprised by the direction of the gender difference here? Ex-
plain.
Question 3. Are women more likely (than men) to approve of abortion
for any of the specific reasons investigated in the GSS, such as when a
woman is pregnant as the result of rape? To find out, return to the cross-
tabulation program and replace abany with abrape as the row variable
(dependentvariable),then "run tables" again. To avoid confusion, this
time suppressthe row percentages(that is, ask only for column percent-
ages).Repeat using abdefect, abnomore, abhlth, abpoor, and absingle as
the row variable. \X/hat do you find?
Question 4. One thing we learn from these results is that, for Americans,
the reason for an abortion matters. \il/hile most Americans believe that
abortion should be legal for some reasons, such as when a woman's
health is endangered,more than half believethat there are conditions un-
der which abortion should not be legal.
(a) Use the appropriate percentagesin your seven tables to rank-
order the sevenconditions here (pregnancy is due to rape, etc.) from
"strongest approval for legal abortion" to "least approval for legal
abortion."
(b) Is the ranking exactly as you would have predicted, or are you sur-
prised by some of the results?
Question 5. Now cross-tabulateabany (row) and race (column). The total
N should be 26,104 cases.
(a) Are race differencesstatisticallysignificant?(Again, use p<.01 as the
cutoff point.)
(b) Are the differencesin the direction you expected?Explain.
34 . Chapter1

MRnntece, HuNtINc, eNo FnrqurNcy or. Srx

The GSS variable sexfreq asks, "About how often did you have sex dur-
ing the last 12 months? [not at all, once or rwice, once a month, 2-3
times a month, weekly, 2-3 times per rveek,4+ times per week]."
In this exercisewe use the cross-tabularion program to investigatethe
associationbetween marriage and frequencv of sex, then to investigatethe
association between hunting and sexual frequency. The first thing you
want to do is convert the GSS variable marital rvith five categories(mar-
ried, divorced, widowed, separated,never married) into a new variable
with just two categories,married and not married (which includes sepa-
rated couples).Let's call that new variable married. Return to the original
screenwhere you selected"frequenciesor cross-tabulation," but this time
select "recode" to create the variable married, as follows:
r Under "createvariables,"click on "recode variables."
r Type in married for "name for the nerv variable to be created."
. For "Replacethat variable,if it alreadvexists?"click on "Yes."
. Type in marital for "Var 1" under "Name of existing variables to
use for the recode."
r For row 1 under "Output variable,"ti'pe "1" for "valuer" "married"
for "label," and "1" for "Var 1." For the secondrow, type "0" for
"valuer" "not married" for "label," and "l--i" for "Var 1."
o Click on "start recoding." If you have done this correctly, you should
have 20,674 not married casesand 25,828 married cases.

Question 6. Return to the "frequenciesor cross-tabulation"screen,and

this time cross-tabulatesexfreq with married. Enter sexfreq as the row
variable and married as the column variable. To make sure you focus on
the relevant percentages,select only column percentages.Otherwise, se-
lect the same options as in the abortion exercise.Your total N should be
18,594 when you run the table.
(a) Is the association statistically significant? (If you don't see p-values,
you must have forgotten to check "statistics.")
(b) What is the association between marriage and sexual frequency-do
married peoplerend to have more sex rhan others?ls there u"riun..
in sexfreq for the married or unmarried? -or.
(c) Summarize your substantivefindings in two or rhree sentences.Is this
what you expected?

we can also use the GSS to investigate the association between hunting
and frequency of sex. The hunting variable in the GSS is hunt: ,,Do you
(or does your [husband/wife]) go hunting? [respondent hunts, spouse
hunts, both, neither]." Repeat the procedures you used to recode marital

.:
i : lrd r-ou have sex dur-
L!: : :'.;,-:-. once a month, 2-3
. -l- :::i per lveek]."
- :: ::Jm ro investigatethe
l-r , :;r.. :hen to investigatethe
;. : . : : . .r-.;.. The f ir s t t hing y o u
r*:::.:, .' .:-r fir'e categories(mar-
-. : : = : :i i n t o a new v ar i a b l e
,i '. . :.. j .r hic h inc ludess e p a -
- . : ':::.J. Rerurn to the original
.:: -ilation," but this time
:,'l l o u' s :
- :: - : : . -'-:t:b les . "
- : .: = : ,.:'-eto be c r eat ed. "
" (b) Summarize your findings in two or three sentences.Is this whar you
:: : : , .:.l c lic k on " Y es . "
-
:: ). ,,-' : ,: eristing variablesto
- : :'- - ' :,rr "r'aluer" "married"
. - : . - . r:-,rod r ow, t y pe " 0" fo r
:- : " --i " fo r " V ar 1. "
. : : ": :.^.:S CoflectlY,You ShOuld
: - t -' -' ::--..i r ied
c as es .
, : -: -,':-.-:ulat ion"s c r een ,a n d
:,i:::.: t-::.r sexfreq as the row
- -:: : I :-,-ii<esure you focus on
-- : ::-::i ,1 qe5. O t her wis e ,s e -
: : r : : : r.. \,l u r t ot al N S hO u l db e
-., - : i ,: . c,u don' t s eep- v al u e s ,
r:: :i: .::i .erual frequencY-do
':- : - '-: I' r her em or e v ar i a n c e
.:
,: rhree sentences.
Is this
:-:: '.rJtion between hunting
: ::.e GSS is hunt: "Do you
::.pondent hunts, Spouse
. ...,u used to recodemarital
The First Rule . 35
into married to recode the 4-categoryvariable hunt into a 3-category vari-
able, called whohunts, coded 0-2 for number of people who hunt: 0 if
neither hunts, 1 if respondent or spousehunts, 2 if both hunt. (Hint: The
first row under "Output variable" should be "0, neitherhunt. 4"-that is.
type in "0" under "Valuer" "neither hunt" under "labelr" and "4" under
"Var 1." The second row should be "1, one hunts, l-2" and the third
row should be "2, both hunt,3.") ff you have recoded correctly, you
should have 18,874 casesof "neither hunts," 4,650 "one hunts," and
576 "both hunt."
Question 7. Cross-tabulatewhohunts with sexfreq, with sexfreq again as
the row variable and whohunts as the column variable. Again, ask for
column percentagesand statistics.The total N should be 10,894; if your
N is somewhat different, make sure you used the proper "Oversamp"
weighting.
(a) Is the associationstatisticallysignificantat p <.01?
expected?
Question 8. (a) Give two explanations for the association between who-
hunts and sexfreq.
(b) Think about the nature of the two explanations you have given. Do
your explanations imply that the association is causal-that is, do your
explanations imply that sexfreq will increase for people who take up
hunting?
(c) If you were to test your first explanation, would the test include non-
mediating variables?Mediating variablesin a causal chain? Moderaring
variables that condition the effect of whohunts on sexfreq? Reversecau-
sation?Explain.
(d) \fhat about a test of your second explanation-would that test in-
clude nonmediating variables? Mediating variables in a causal chain?
Moderating variables that condition the effect of whohunts on sexfreq?
Reversecausation?Explain.
 CHA P T E R 2

The SecondRule
LOOK FOR DIFFERENCESTHAT MAKE A DIFFERENCE.
AND REPORTTHEM

"How big is big?" is a question that students in introductory statistics

courses in the social sciencesinvariably raise. Students want to know:
How should one describea correlation of r=.10 betweentwo variables?
Is it large or small? \7hat about a correlation of r=.20? Of r=.30? At
what point does an associationgo from small to large?In short, what rs a
"large effect"?
The major point of this chapter is that it is your responsibility, as a re-
searcher,to translate computer printouts into empirical substanceusing
words and visual displays that enable you and your readers to decide
whether effectsare large or small, important or unimportant. If this is done
properl1. it should be an easy marter to find, and emphasize, differences
that make a difference. In the r,vorsrcase-which occurs more often than
we like to admit (Mccloskev and Ziliak 1996; ziliak and Mccloskey
2004)-researchers do not evenatrempr to discriminareresults on the basis
of the size of coefficienrsbur merelv provide a printout dump for readers,
with annotation limited to which coefficients are statisticaly significant
and which are not. In the best case,researchersrecognizethat size matters
(and report results accordrngly), bur other things matter as well. Empiri-
cal bignessis not necessarilythe same as theoretical importance or impor-
tance for policy. Effective social researchfocuseson differencesthat make
a difference, and it tells readerswhat those differencesare and why they
are important.
There are actually rwo issuesinvolved in the searchfor differencesthat
make a difference-the issue of size and the issue of importance-and I
discuss both of them in this chapter. The issuesare related, but they are
not the same. Supposeyou measuredthe length of right regsand lefi legs
for a sample of individuals. You would no doubt find a sizJle correlation
between the two lengths-r= .99 or greater.So you would find a very big
effect-but one that isn't very interesting or important. Big effects are not
necessarilyimportant effects.
Nor are small effects necessarily unimportant. There are instances
where small effects matter for theoretical or policy reasons. yet in those

:.FILRENCE,
rrrtroductory statistics
. : : -: . >:-i d enf swant t o k n o w :
: -= . ieru'een two variables?
-: " - - ' r=.20? O f r = . 30? At
1-. . ..,:q c i I n s hor t ,what i s a
:. - - ..: i i 5pons ibilit y ,as a re -
.: - - ::..;.rr, , r l s ubs t anc eus i n g
,- . ,-j '.(nr r eader st o de c i d e
:t-: : ..:-..::tlorrant. If this is done
t - - . , j .mp has iz e,dif f er e n c e s
- - . ,,..urs more often than
: r - - --: Zrl ra k and M c Clos k e y
: - - - - :- -.,1re resultson the basis
- - . -.:.:',r- rt
dum p f or r ead e rs .
j, -
: - : -, .:: )i .tt is t ic allys ignif i c a n t
: : - - - -. -: - '-rriz et hat s iz em atte rs
': ' : - -i : ::l -1t f Cf aSwell. E m p i ri -
-..*..,eor rmpor-
_ rntn nr lana
- : ".-::: n Jifferences that make
- j '::::t'tces afe and why they
'.:
- r-, i:.rrChfor differencesthat
, im nt t r r anc e- a n d I
'i
; - - ..: - ..:e related,but they are
r -- - : ng, htlegsand lef t l e g s
: .^: :nJ a sizablecorrelation
::: \ . ,l u'ould find a very big
- - ^ ::.i nr . B is ef f ec r sar e n o r
-: -:i r:r. Ther e ar e ins t a n c e s
: : .:.r' reasons.Yet in those
The SecondRule . .i:
instances it does not suffice to say simply "There is an effect." As a re-
seafcher you need to provide criteria for gauging the size of effects, and,
if the effects afe small, you need to explain why they are nonetheless
worth noting.
You CeN'r ExplerN A VARIABLEwITH A CoNsrRNr
In chapter 1 we looked at the issue of selecting a reseafch project. There
I stressedthat results from prior studies can be extended and clarified by
adding variables-variables that mediate an effect, that moderate (condi-
tion) an effect, or that act as a common cause of some observed associa-
tion. Here I add the point that differencescan only be explained by other
differences. So if you want to know why an outcome variable Y varies,
you need to find explanatory variables (Xs) that also vary.
Social scientistsunderstandthat you can't explain a Yariablewith a con
stant, so I don't know of any researchwhere sorneoneliterally trics to do
that. But there are instanceswhere theorists and researcherspay too little
'We
attention to the amount of variance in their explanatory variables.
know, for example, that demographic changestend to be slow and contin-
uous, so precipitous change is unlikely to be due to population change.
Likewise iulture-the beliefs, values, traditions, behaviors, and material
objects that collectively form a people's way of life-does not change
overnight. The samecan be said of a nation's geographicfeatures(distance
from the equator, whether it is landlocked, system of navigable rivefs, to-
pography, rainfall, etc.), a factor that has received renewed attention in
some recent theorizing about why some countries are so rich and others so
poor. Of course population, geography,and culture are important. But in
generalthey are unpromising explanations for rapid social change.
Consider the trend in fertility rates in Romania over the secondhalf of
the twentieth century (fig.2.1)- The generalpicture is that of a secular de-
cline in fertility that was interrupted by a stunning increasein fertility in
1967. From 1.965 to 1'967 fertlltty in Romania nearly doubled, from an
averageof about 1.9 children per woman to an averageof about 3.7 chil-
dren per woman (Bradatan and Firebaugh 2007).
How are we to accountfor the 7967 baby boom in Romania?Cultural
explanations aren't promising, since people'sviews of ideal family size dg
not change overnight. Nor is it likely that the jump was due to changes
in Romania's population composition (for example, an influx of high-
fertility immigrants) since a nation's population composition usually does
not change suddenly either. In this instance' the most promising route is
to examine the historical context.'What eventswere occurring in Romania
at this point in time?
 38. Chapter2

4.0

3.5

3.0

2.0

t.)

t.0

0.5

0
1956 1960 1965 1970 1975 1980 1985 1990 1995 2ooo
Figure 2.1. Total Fertility Rate in Romania, 1956-2000. source: Romanian
DemographicYearbook,2001.

From figure 2.1 we nore that fertility had declined sharply in the de-
cadebefore 1967. rn 1956 Romanian women were giving birth to almost
three babies,on average,over their childbearingyears.By the early 1960s
fertility had dropped below replacemenr level, meaning that Romanians
were not having enough children to replace themselves.Hence, as Ro-
manian government officials well understood, if that fertility rate contin-
ued the population of Romania would be destined to level off and then
decline.
It is not surprising, then, that the communist government took stepsro
try to boost the fertility rate. First a media campaign was launched in
scanteia, the official newspaperof the Romanian communist party. Tyo-
ically the first two pages of scanteia were devored to political atrd eco-
nomic news: foreign dignitaries visiting Romania, .r.*..o.ro-ic successes
of the Romanian communist Party, and so on. That focus chanqed some-
what in the summer of 1966, however, as articles abo.rt m"teinitv as a
woman's duty to the nation also began to occupy centerutag. in th.
newspaper.On August 4, 1966, Ursula Schiopu,a psychologist,argued
that "feminine charm is regeneratedby maternity" (s,canteii, Attgust 4,
1966, p.2). Two weekslater there appeareda first-pagearticle on schools
and children, and on August 21 another article about "children's love."
In a September19 article titled "Maternity's Advantages," a physician ar-
gued that pregnancyheals acne, eczemas,depression,and melancholy. on
september 23, women who ask for abortion were called "superficial" and
legislation was blamed for letting them obtain abortions so easilv.and the
 The SecondRule . 39

next day Zaharia Stancu, an important Romanian writer and member of

the Romanian National Academy, describedpast times when families had
many children and all of them were happy (p. 1). The next day an article
titled "Maternity" presented various stories about abortions, concluding
.,very few women ask for abortion because of objective reasons"
that
(September9,1.966, pp. 7-Z).
On October 2 the other shoe dropped. Decree 770, called "Measures
to regulate pregnancy interruption," was published on the first page of
Scanteia. The new decree outlawed virtually all abortions (which, until
the decree,had been very common in Romania). In addititln, the Roman-
ian government ceasedto produce or import modern contraceptive meth-
ods. Although not prohibited by the new law, modern contraceptive
methods becameinaccessible-availableonly on the black market. The
effect of thesepolicies was sudden and dramatic, as figure 2.1 shows.l
I use this example to show how one might "look for differences that
make a difference." Typically the explanation is not as straightforward,
of course. But the principles are the same. You need to know something
about your subject matter. Learn to think like a detective,using appropri-
ate tools at your disposal, whether qualitative or quantitative or both.
And remember that you must have variance to explain variance.

MexrmrzrNc VenreNcp,To FIND lun ErrBct oF A CAUSE

In the example we have just considered, our aim was to account for the
1967 baby boom in Romania. In other words, we were trying to find
the cause or causesof an effect. Increasingly in social research,however,
we want to isolate the effect of a particwlar cause,We start witl-r the sause
and try to design an experiment or some other method to determine the
magnitude of that effect on some outcome variable.
This strategyis commonplacein medical research.Before a new drug
for the treatment of asthma is approved for the general public, cxperi-
ments are performed to determine how much effect, if any, that drug has
on asthma, and what side effects the treatment might have. Or-to use
an example that is discussedin detail in chapter 5-we might want to es-
timate the causal effect of smoking on lung cancer. Smoking is not the
only cause of lung cancer, of course. But there is not much we can do to
control many of the other causes, and smoking is something we have
l Over time Romanians figured out \"'aysto circumvent the restrictions-for example, bv
bribing doctors to say the mother's life u'as endangered, so a legal abortion could be
performed-so fertility began to track back down, as we see in figure 2.1. The abortion las-
was rescinded in 1990, and since then fertility in Romania has fallen to one of the los'est
levelsin the world.
 40 . Chapter2

sorne control over, so we are especially interested in the causal effect of

smoking.
The same logic applies to much social research.Although it wourd be
nice to understandall the causesof residential segregation,or domesticvi,
olence, or the rise in income inequality in the United States,the most in-
teresting causes very often are rhose that are amenable to public policy
(more on this later in the chapter). Thus, instead of trying to understand
some phenomenon (such as the rise in income inequality) in its entrrety-
which might result only in the dim understanding of many causes,or in
the understanding of many causesthat we can't do much about-effective
social researchtoday often focuseson understanding one causewell.
To understand one cause well, w.e want to maximize the variance of
that cause.As discussedin chapter 1, very often we can increasevarrance
through our sample design.lwe do this by stratifying the target population
and ov er s am p l i n gm i n o ri ry g ro u p s .
F{ere we want to maximize the variance of the causar variable, so we
oversamplethe minoritv groups of the causal variable. To illustrate, think
again about cohabitationeffects.Supposewe want to determineif there is
a link between cohabitarion and domesticviolence.Although cohabita-
tion is rising in the Unrted states,at an,ypoint in time heterosexualmar-
ried couplesstill outr.rumbercohabiting couplesby more than ten to one
(U.S. CensusBureau 2000). Supposecohabiting couplesmake up 9 per-
cent of heterosexualunions in the united states.Then in a simpleianJom
sample the expected variance for the r.ariablecohabit (coded 1 for cohab-
it er s and 0 f o r m a rri e d c o u p l e s )i s (0 .0 9 )(0.91)= 0.082.B y sampl i ngan
equal number of cohabiting and married couples, we rriple the variance
on cohabit from 0.082 to 0.25 [(0.5)(0.5)]. By tripling its variance,we
make cohabit a more powerful tool for investigating the consequencesof
cohabitation.
In short, to "look for differencesthat make a difference" you need to
c ons iders am p l ed e s i g na s w e l l a s th e o ri a nd pri or res." rch. B ecausei r
takes varianceto explain variance,you can't evaluatethe effect of a partrc-
ular causeunlessthe causevariesenoughin your sample.To test properly
for "differences that make a difference," then, you mighr needlo over-
sample some categoriesof the explanatory variables,especiallywhen your
sampleis small.
Finally, it is importanr to srressthat the purpose of oversampling is to
create more variance on X in the sample, without losing sight of the ac-
tual distribution of X in the population. obviously, if there were only one
hundred cohabiting couples total in the United Stares,cohabitation could
not be a leading cause of domestic violence in the united states, since
domestic violence is much more widespread than that. (In fact, though,
cohabiting couples number roughly five million-so cohabitation very

.i.:-:-,:- in the causaleffectof
I r::: r', ^. \lrhoughit would be
-- - .: ':r--:l;rton, or dorneStiC vi-
: -: - : :iJ States,the most in-
':: :. : .,:.-.;n.rlrle to public policy
.. -'i : r-: ,f rrling to understand
-. - .:t i r v r in ir s ent ir e ty -
-: - - ' ' - '' . , 'f mA nVCA US eS O f, i n
, : -,-- I : :nlt.h about-effective
.-i : '. il: 'll a q o ne Caus ewell.
. : - ' . .:,-:rrm iz et he v ar ianc eo f
:' --: ' .',i a .t ninc r eas ev ar ia n c e
-: .,: - . ::1: rhe targetpopulation
- cance,resulting in deemphasison the size of effects. Generally researchers
;-.'j : -.: ',lusal Yariable,so we
i -i j ,.: .;.r1e. To illustrate,think
',: , : , -,:: ro determine if there is
, :: .r. -\lthough cohabita-
-
: . - :ime heterosexualmar-
- , - : - :' . more than ten tO One
,r.nles nreL e rrn 9 ner -
: \: . : : : . l rcn i n a s im pler and o m
-: - :
- 'r-rb i t 'c oded 1 f or c oh a b -
- - . = '.r,)S 2. B y s am plinga n
:- - I :.. \\'c r r iple t he v ar ia n c e
i fects. This blurring occurs when researchersfail to distinguish statistical
-: :::lling its variance,we
- . :: ..-.:.i.:tsthe consequences of
, --':- : -. .lrffcrence"you need to
-. i : ::.,rr re s ear c h.B ec aus ei t
i . : ': -:,t. rhe effect of a partic-
- . r: i-1mple.To test properly
. : ::. '.,ru might need to over-
- r - r -:. i . es pec iallywhen y o u r
: : .--:: !e of oversamplingis to
:, .,.:.. -:t losing sight of the ac-
- :. ::: >:,ries.cohabitationcould
: : . , : " ::rc L' nit ed S t at es ,s i n c e
r : r - - - .',r rhrt. t ln f ac t , t hou g h .
: - . :r-so cohabitation very
Thc Second R ule . 4l
well could be a significant factor in domestic violence.) The point of over-
sampling is not to magnify a minute effect in the population, but to en'
sure that there is enough variance in the sample to provide a fair test of
the effect of a cause.
SrzEvpnsus Starrsrtcer SrcNrprceNcr,
Before we can talk about the size of an effect, we want to have some sense
of how confident we can be that the apparent effect observedin our sam-
ple is not due to sampling error. That of course is the purpose of conven-
tional significancetests in the social sciences.ln practice, however, social
scientiststoo often conflate statistical significanceand substantivesignifi-
test the null hypothesis that the effect is zero. When the null is rejected,
researchersconclude that there is an effect in the population. When thc
null is not reiected, researchersconclude that there is no effect.
Researchersoften appear to forget, however,that statistical significance
is a function of sample size as well as size of the actual effect in a popula-
tion. Thus a sample coefficient can be insignificant for two very different
reasons: becausethere is no effect in the population, or becausewe have
estimated it so imprecisely, due to a small sample' (This problem leads
some statisticiansto recommend the use of larger p-values with smaller
samples.)
Even worse, when findings are statistically significant, significancetests
sometimes are misused to blur important differences in the sizes of ef-
and substantivesignificance,resulting in a flattened world where there are
only two outcomes,effects and noneffects.The world is flattened because
statistically significant effects are all treated the same-stupendous ef-
fects, minute effects, and all effects in between are dumped into the same
basket. The reader wants commentary on the size of the effect. Vhat she
often is given instead is an asterisk indicating how strongly the sample
data contradict the null hypothesisthat the effect is zero in the population.
That practice reduces the effectivenessof social research, as McCloskey
and Zlliak 11996) and Zlliak and McCloskey (2004) have forcefully
noted.
So the first point I want to make regarding the size of effects is "Talk
about the size of effects in your report." In most contexts such advice
would be gratuitous, but it seems necessary here, given the apparent
trained incapacityfor describingeffect sizesin some fields of social science.
'We return, then, to the question "How big is big?" It is useful to dis-
tinguish relative size and absolute size (although the two are related of
42 . Chapter2

course).In the caseof relative effect size the other causesprovide a ready-
made referencepoint. How large is the effect of X on Y relative to the ef-
fects of other causesof Y? In the caseof absolute effect size, researchers
must sometimes search outside their model for an appropriate yardstick
or referencepoint-some sort of benchmark to assistintuition. This is not
always easy to do, and perhaps for that reason it is not always done in
social research.

CompanrNc Er-rBcrs'Wur,nr, Tnsnt Is e Colm.loN Mnrntc

In the case of relative size we are interesredin which predictors have the
strongest effects. What are the most important determinants of one's so-
cioeconomic status (SES)in the United States(Jenckset al. 1972)? Educa-
tional attainment? Intelligence?Your parents' SES? Racial or ethnic
identity?Ambition? Hard work? Luck?
Determining the relative sizes of effects is particularly important in
testing theories sincein social sciencetheory very often it is effect size that
distinguishesone theory from another. There are, for example, numerous
hypothesesand theoriesin social sciencethat attempt to explain why some
countries are so rich and others so poor. Sometheories emphasizenatural
resourcesor geography, some emphasizeculture, some emphasizepoliti-
cal systemsand the importance of appropriate policies, some empha-
size the pattern of technology diffusion, and still others emphasrze
international relations or colonialism or exploitation. These causes are
not mutually exclusive, and cultural theories of developmentdo not say,
for example, that political systemsdon't mamer at all-just that culture rs
primary. Thus tests of theories very often are tesrs of the relative sizesof
effects.
It is not surprising, then, that much of the empirical work in the social
sciencesturns on the relative importance of causes.\X/hat ls surprising is
how frequently one finds a disjuncture between theory and findings in re-
searcharticles, with the findings secrion giving equal billing to all statisti-
cally significant effects, as if all statistically significant effects are the
same.But size matters, and effective researcharms readerswith the infor-
mation neededto distinguish bigger effects from smaller ones.
There are two situations to consider: the case where the explanatory
variableshave a common metric, and the casewhere they do not. It is eas-
ier to compare effectswhen the explanatory variableshave the samemetric
(dollars,years,etc.). Supposewe want to know, among married couples,
whether family size (the number of children a couple has) is linked more
closely to the number of siblings the mother has or the number of siblings
'We
the father has. might hypothesize,for example, that individuals who

a :. : :.-:: -al rs espr ov ide ar e a d Y-
:- : r -: -\ o n l' r elat iv e t o t h e e f-
. : :,- a -,.:; eif ect size,researchers
' ,i:' r : .rr appropriate yardstick
- j-.
-r.i ri t int uit ion.T his is n o t
fr ' : . : - :: i 5 not alway s don e i n
' .i '.i t)\ -\lE r nr C
,..rich predictorshave the
'-: i;rerminants of one's so-
i-:-..kset aL l'972)? Educa-
- :.' SFS ? Rac ial or et h n i c
rr:-i: - :.liticularlY imPortant in
'. :
:':'. rrftenit is effect sizethat
l: : -: .,1 :.i o r ex am P le,num e ro u s
. , , L . , SOme
^ - . * l ^ : - why
: :: -:,:-,::::lpt tO explain
-': ::i()ries emPhasizenatural
'r :
:: , - :,.:J. some emPhasizePoliti-
:r: : : ,.-:. --- ^,-l;-i-" ies ,s
polic "^-o
om e o-^ho-
-. ,.-.i sti l l ot her s em
-:
: : , : . ::.]ti o n.T hes e c aus e sa re
-: : :: : jeveloPmentdo not sa5
'i --rir:i ,..1ell-just that culture is
:: ::: i::!i of the relativesizesof
: :-: :::rl i ri ca l lv or k in t he s o c i a l
-.-. ' - .:...1\. V ' har ls s ur pr is i n gi s
:,:: i ::r iheorv and findingsin re-
- : : rquai billing to all statisti-
-
. :. . -; . . . r:g ni fic antef f ec t sare th e
'!(r:-:-. ...:::rs readerswith the infor-
:: , . . : ,:-.sma llerones .
:: : : :,.-.e il'here the explanatory
,-: -,:.: .'.irerethev do not. It is eas-
:: : . .- ...:'i chav s er he s am em e tri c
r i r.r ',.,.among married couples,
: ::- .: ;Ouple has) is linked more
:::: :-.-1! or the number of siblings
: : :\..:.:tple.that individuals who
The SecondRule . 43
grew up in large families tend to have more children than individuals who
grew up in small families, but that the family size experienced by a
woman in her youth is more consequential for her fertility than the fam
ily size experiencedby her husband.
'With
appropriate data on completed fertility for couples,we could test
the hypothesis by regressingnumber of children for a couple on number
of siblings in the wife's family and number of siblings in the husband's
family. Because the explanatory variables have the same metric, it is
straightforward to compare their effects. If size of the wife's family tends
to be more consequential for the number of children a couple has, then
the slope for the variable wifesibs should be steeperthan the slope for the
variable husbandsibs.
It is useful to think more concretely about how we would graph the re-
sults in two-dimensional space. Although generally we need a separate
graph for each explanatory variable, the graphs can be superimposedwhen
explanatory variables have a common metric. Here, for example, the
graphs for B*r, dfld Ft,u,bo,dcan be placed on the sameX-Y plane because
the X-axis iJ the same for both variables. By superimposing graphs and
comparing the steepness of the slopes,we can seeimmediately which vari-
able has the greater effect.
"^-.
The issue of which graphs can be superimposed and which cannot is
central, then, to the comparison of effect sizes. Consider a different sort
em p h a -
of comparison, one where we compare the effect of the same X across
ph a s i z e different groups. \Wemight want to know, for example, whether the effect
of religiosity on altruistic behavior differs across religious groups (for ex-
ample, Christians, Jews, Muslims, and Hindus), where religiosity is mea-
sured by an index based on frequency of attendanceat religious services'
participation in other forms of religious activity such as prayer, the inten-
sity of beliefs, and so on. The key here is that X is the same' so the X-axis
is the same, so the graphs can be superimposed. It is straightforward to
merge the separateX-Y planes (one for each of the religious groups) and
compare the size of the slopes,even if the units of measurementhave no
clear intuitive meaning.
As a third example, suppose we want to know which has the greater
effect on a woman's fertility-her religiosity, or the number of siblings
she has. In that casewe could regressfertility on religiosity and number
of siblings, but the slopes would be more difficult to compare. The two
X-Y planes could not be merged becausereligiosity and number of sib-
lings are not measuredin the same units.
As these three examples demonstrate, one way to think about the size
comparison issue is to think about how to combine separateX-Y planes
(one for each of the explanatory variables)into a common plane. Because
the outcome variable Y is the same for all Xs, the Y-axis is the same for
44 . Chapter2

each of the separateX-Y planes, so the metric for Y doesn't affect com-
binability. But the metric for X does. S7herethe metrics differ, we must
calibrate the different merrics to compare the effects. That is the subject
of the next section.

Car,rennrr o\r : C oNvEnrrNc ExpreuATo Ry VeRreerE s

ro e ConvoN MprRrc

Sometimes you find yourself in the happy circumstance that your ex-
planatory variables have the same merric, so you can compare the coeffi-
cients directly. In the caseof longitudinal srudies,for example,the variables
might all be expressedas growth rates. or perhaps all the explanatory
variables are expressedin years, or in dollars, or as a simple count, such
as number of siblings.
Very often, however, explanatory variablesdo not come ready-madein
the same metric. In that caseyou must calibrate the various metrics of the
explanatory variables to gauge the relative sizesof the effects. In the case
of policy analysis,for example, ir mav be possible ro convert variables to
a common standard by asking how much it costs to change each of the
variables by one unit.
Perhapsthe most common practice, though, is to calibrate metrics by
converting them either to standarddeviation units or to percentagechange.
The first strategy results in coefficientsrhar are called standardized coeffi-
cients, and the second results in coefficienrs rhar reflect elasticities. The
approachesare somewhat similar. In rhe caseof standardizedvariableswe
ask how many standard deviations we expect an outcome variable y to
change for a one-standard-deviationchange in an explanatory variable X.
In the case of elasticitieswe ask what percentagechange we expecr in y
for a 1,percent change in X.
For the sake of illustration, I will focus on standardization. Because
standardizedvariableshave a standard deviation of 1.0, a one-unit change
in a standardized variable refers ro a one-standard-deviationchange in
that variable. Thus if all our explanatory variables are standardized, the
slope for any given explanatory variable reflects the expected change in
the dependentvariable given a one-standard-deviationchange in that ex-
planatory variable, controlling for the other explanatory variables in the
model. In this way we can compare the size of effects directly by compar-
ing the sizesof the slopes-assuming it makes senseto think of standard-
deviation changesas having equivalent theoretical meaning for the Xs.
Standardization is common in studies where variables lack a natural
metric. such variables are commonplace in anthropology, education and
testing, evaluation studies, psychology, political science,and sociology.

. tor Y doesn't affect corrr-
::rL'mctrics differ, we must
Jflects.That is the subject
' 1-1 1a DLtl
i: : : ' jt:aunlstan ce th at you r ex -
:r i. . . ),r can compare the coeffi-
i i. - :.::. frtr example, the variables
r::. : :::haps all the explanatory
' : .::.. ,i as i1 simple count, such
r .:. j , iror come ready-made in
. .r -:.:..:' :he r-arious metrics of the
::
:: : -: - a to convert variablesto
-- - - : , i i s to c hangeeac h o f th e
:. :- -, -... :s ro c alibr at em et r i c sb y
:: - - :'.:: or to percentageChange.
: : - ,, : -: ::,lIed standardized coeffi-
-- :-i: ::-..lireflect elasticities.The
.-.: - .,.: : .tarrdardizedvariableswe
: :', ::-: -rr1outcome variable Y to
-:- ' rr erp lanat or yv ar ia b l eX.
- -.
: ': - - - .:.:-s eh. r ngewe ex pec ti n y
: ,,. . :: sr.lndardization.Because
-: ..:. :r oi 1.0, a one-unit change
, -.-.:-'-:-,.l.rrd-deviation change in
: :-. , .:.---i.les are standardized,the
- : . : : . :,:. rh e ex pec t edc han g ei n
:-i . : - -'-.': i r.rrr onc hangein t ha t e x -
:-.:: :rplanatory variablesin the
: :.:: : cffectsdirectly by compar-
i r-=.lr:: ienseto think of standard-
I - - : - . - .'tl me : lninqf or t he X s .
:.;: ',.,:::revariableslack a natural
-1.,^^ ^r ^- , . - , luc ar io na n d
r. : ,.::ical science,and sociology.
The SecondRule . 45
Measures of individual traits such as altruism, intelligence, self-esteem,
authoritarianism, prejudice) tolerance' risk-taking, and religiosity' as well
as national traits such as degree of political freedom, very often are in
dexes that are constructed from the results of test items or from the rat'
ings of experts. In such instances the common unit of measurement
acrossthe indexes is the standard deviation of the tests or the ratings'
By using standard deviations as the unit, with standardized variables
we in essencecompare effects on the basis of rankings on diffErent vari-
ables.Where distributions are the samefor explanatory variables-normal
or approximately normal, for example-a one-standard-deviationchange
in one X is, in terms of change in rank, comparable to a one-standard-
deviation change in another X. On the basis of standardizedscores,then,
we can compare the relative ranking of individuals across variables. We
can ask, for example, whether people who score high on a measureof re-
ligiosity also tend to score high on a measureof tolerance-which is what
we mean when we say that there is an associationbetween religiosity and
tolerance.
Introductory statistics textbooks often warn students not to confuse
the association of X and Y with the effect of X on Y. That warning bears
on the discussionhere becausethe regressionof one standardizedvari-
able on another yields the measure of association r. That's not what we
want when we are comparing effects. As a general rule, then, standardi-
zatron is not a reliable way to solve the different metrics problem when
comparing the size of effects. Other methods should be used where pos-
sible.
Consider a simple example. Supposewe collect a large random sample
that includes data on individuals' earnings (in thousandsof dollars), edu-
cation (years completed, 0 to 20), and background socioeconomic status
(an index of the socioeconomic status of the respondent's family of ori-
gin, with values ranging from 0 to 100). Supposefurther that the regres-
sion of earnings on education and socioeconomicbackground (SES)yields
this equation. where standard errors are in parentheses:
= 2.0 Education+0.4 SES+ a.
Earnings (2.1)
(0 . 1 ) (0 . 0 2 )
From thesefindings it isn't obvious whether education or SESbackground
has the greater direct effect on earnings. The issue turns on the calibra-
tion of SESunits with education units. If we say one SESunit equals one
education unit (one year of education), then we conclude that educarion
is five times more important than socioeconomic background since the
education coefficient is five times larger. The claim of equivalent units is
problematic, though, becauseSESranges from 0 to 100 whereas educa-
tion ranges from 0 to 20. Some might sa5 then, that we should compare
46 . Chapter2

individuals at the highest and lowesr levels on the two variables. Using
that method, we find that education and socioeconomicsrarusare equally
important, since the predicted difference in earnings is $40,000 more for
those with 20 years of education as opposed to those with no education
(controlling for SES),and $40,000 more for those in the highest SES
category as opposed to those in the lowest SEScategory (controlling for
education).
The problem with using the range for comparison is that we don't
know how realistic that comparison is. Perhapsmost individuals cluster
tightly in the middle of the SES distribution, with no one in the highest
and lowest categories of SES, whereas individuals are more evenly dis-
persed across years of education. Standardization takes into account this
difference in dispersion. In other words, by standardizing variables we
obtain summary coefficients that take into account both the steepnessof
t he or iginals l o p ea n d th e d i s p e rs i o no f e x p l anaroryvari abl esi n the parti c-
ular sample that we have. The dependenceof standardizedcoefficients on
sample variances is important to nore, since that dependencecan affect
the generalizability or external ualidity (Campbell and Stanley 1966) of
your results using standardized coefficients. The issue arises when vari-
ancesin the sample deviatefrom variancesin the population to which you
want to generalize.
Of the three bad options above-compare the unstandardized coeffi-
cients for education and income, compare predicted Y-valuesusing the
high and low valuesof education and income, and compare coefficients
for education and income as srandardized variables-the comparison of
standardized coefficients is the least likely to mislead. Generally, though,
there are better options, and you should work hard to find them. Where
a common metric is not readily available, you should attempt to supple-
ment your discussionof coefficientsize with a more targeted discussron
that employs concretecomparisons.That is the topic of the next section.

SussreNrrvp Pnor,rrrNc:Tsr Usp oF TELLTNG

CoupenrsoNs

In some instancesthe calibration of variables' merrics is informal or im-

plicit. For example, a researchermight dichotomize a conrinuous variable
to enablecomparisons with a true dichotomy, such as marriage. \Wemight
want to know, for example, whether money or marriage has a greater ef-
fect on one's happiness.As a first stab at the answer, we might first create
a dummy variable for income coded " 1" for those with above-averagein-
come and "0" for those with below-averageincome, and regresshappiness
on the income dummy and a dummy for marital status. (A dummy vari-
able is a variable with two values, 0 and 1.) In comparing the coefficienrs

:a: t\\'o variables.Using
. :-()micstatusare equallY
(n,
:.r:. ...:s\ c r n n n n morefor
. - 540,000
- -. rs ^n,o
- . ' , -')i e in t he highest S ES
',' .-: r : ) ( c ont r ollin gfo r
--'.I.g or \ '
: - , .:-r-rrisonis that we don't
, : . - -. . .! :ros t indiv idualsc lu s te r
. .,..:::'r no one in the highest
:*,rls are more evenly dis-
: :' rekes into account this
. . .- r.- r : - : - ^ ,v .dL
^-rdu
: ^Ll e sw e
-.:..U.1LUt41116
.-- -.:rrboth the steepness of
- -', . ar iahlesin t he pa rti c -
: -:,.:-.Jrrdized coefficientson
: .'.,..rdependencecan affect
- -:'.j l a nd S t anley1' 96 6 )o I
. .: t::u e ar is eswhen v a ri -
: - . :opulation to which You
:.r; unstandardized coeffi-
,i :cte d Y - v aluesus in g th e
.:rd compare coefficients
: ,-.i',es-thecomparison of
::--.'e.rd. Generally,though,
. :,rrd to find them. Nfhere
>:rruld attempt to supple-
. :tore targeteddiscussron
-,,pi . of t he nex t s ec ti o n .
]. L O \ I P A R IS ON S
-
:. :r3trics is informal or im-
: llrze a continuousvariable
'We
: :-'. . . -:chas marriage. might
- r r.ri: s e has a s r eate re f-
: .:-r:\\'eJ. rve might first create
: : - -e ri ith above-average in-
-- ,:e. and regresshappiness
: : :-,::::ri status.(A dummy vari-
-: - .:. ;omparing the coefficients
The SecondRule . 47
for the income and marital status dummy variables, we compare the dif-
ferencein the expectedvalues (means)of Y for the statuses"narried" and
"not married" with the difference in the expectedvalues of Y for the sta-
tuses "above-averageincome" and "below-average income."
Dichotomizing a continuous variable to compare its effect with rhar of
a true dichotomy is a simple example of a more generalstrategythat could
be called substantiueprofiling: the translation of model coefficients into
substantivelymeaningful results by repordng predicted Y-values across a
range of strategicX-values. Effects of two variablesare compared by not-
ing the size of the differencesin the predicted valuesfor Y at the values se-
lected for each of the Xs. Profiling in this way gives substantiveflesh to the
coefficientswhether or not the variableshave a common metric.
The use of substantiveprofiling for determining the relative sizesof ef-
fects is most effective when you can defend your choice of values for the Xs
as representing comparable differences on the Xs. Most often the com-
parison is somewhat loose, since it is difficult to determine precisely what
differences are comparable when variables are measured in dif{icult-to-
calibrate metrics (for example, years and dollars). An inexact comparison
generally is better than no comparison at all, however' sincethe aim of the
exerciseis to assistthe reader'sintuition in gauging which effects are the
larger ones.
Probably the most common example of substantiveprofiling is the sim-
ple comparison of differences in means (for categorical Xs) or expected
values of Y for strategicvalues of X (for continuous Xs)' Imagine a regres-
sion analysis of annual salariesfor university faculty where we find that
the coefficientfor gender(a dummy variablecoded female=1) is b=-1.0
with a standarderror a=0.1. If salary is coded in thousandsof dollars,
and we have controlled for other determinants of faculty salary such as
academicfield, researchproductivity, successin obtaining researchfund-
ing, teaching evaluations, experience, and so on, then we can say that
"Other things equal, faculty salariesare on average$1,000 lessfor women
than for men."2
It is hard to evaluatewhether a gender gap of $1,000 is big or small in
rhe absenceof context: $i.000 compared to what? If the issue is the size
2ln many instancesa great deal of information can be imparted very simply by translat-
ing results from statistical language into substantive language, as in the previous sentence.
This translation is easyrvhen the dependentvariable is already measuredin some readily un-
derstandablemetric, such as dollars or percentages.Someresearchersmay hesitateto include
such translations in their report for fear of insulting the intelligence of their audience.That
point may be valid when the audience is statistically sophisticatedand gender is not a focus
of the analysis.If gender is a focus of the study, however,it is important to state the effect in
plain English, if only as a starting point for a discussionof whether the gender effect is large
or small.
48 . Chapter2

of the gender effect compared to the effect of the other determinanrs of

faculty salary,the referencepoints are obvious: other effectsin the study. Is
the $1,000 genderdifferenceabout the sameas, say,the effect of publishing
one more article in a peer-reviewed journal? Is it larger or smaller than
the effect of obtaining one more researchgrant? How large is it compared
to the effect of publishing a book? How large is it compared to the effect
of being in an engineeringdepartment as opposed to a humanities depart-
ment? And so on.
To assistreaders' intuition about the magnitude of the $1,000 gender
gap in faculty salaries,we might compare that gap to gaps for other di-
chotomies of interest, such as the difference between associateDrofessors
and full professors,or berweenfacultv in humanities departmenisand fac-
ulty in engineeringdepartments.Let's sa'full professorson averagemake
$40,000 more than associateprofesroir, but this difference shrinks to
$2'000 when we control for cumulati'e research productivity, teaching
evaluations,and so on. lWecan sa).,then, that the unaccounted-forgendei
gap in faculty salariesis one-half as large as rhe unaccounted-for gap in
salary between associateand full professors, and let the reader decide
whether that is large or small. Or we might find rhat, evenwith all our con-
trol variables,market forces not captured in our model dictate that average
salaries for engineering faculty exceed those for humanities faculty by
$25,000, so the unaccounted-for gender gap is 4 percent as large as the
unaccounted-for gap between salaries for engineeringversus humanities
faculty.
In short, to gain a better senseof the size of the gender gap, we com-
pare differencesin the differencesin predicted values of yfor selectedval-
ues of X. The income difference (aspredicted by the regressionmodel) for
men versus women is compared with the income difference for engineer-
ing professors versus humanities professors, with the income difference
for an additional journal publication, and so on. 'we can benchmark usrns
results from other studies as well. \(le might ask, for example, how the
gender gap in faculty salariescomparesro the gender gap in related fields,
such as the gender gap in salariesfor public high school teachers.Or we
might ask how the gender gap in facult,v salariestoday compares to the
gendergap t e n y e a rsa g o o r fo rrr' \' e a rsa g o.
\flell-chosen comparisons, then, provide context for readers to deter-
mine the magnitude of the effectsof interest (seeexamplesin bor 2.1).
Although that seems to be common sense,many researchersappear to
give the marter little thought. It is surprising how often meaning?.tl.oto-
parisons are missing in research articles. The issue is not just that re-
searchersmiss an opportunity for a clear story that will leave a lasting
impression. Equally serious is the danger that-absent the discipline im-
posed by well-chosen comparisons-articles become fixated on small

::::- : ::t- other determinantsof
i; : .... :,,,..the effectof publishing
',..:-: : . l: l.lrger or smaller than
- - -,.'.: Hou large is it compared
''r' r:i-l: : :: .ompared to the effect
:- - : -r; ro I humanitiesdePart-
': - i*:. :..J.' of t he $1, 000 g e n d e r
.: : : - ,-: r,rP to gaps for other di-
::::- -: -'-::.1.
een associateprofessors
' -.\,l- ner r m ent s an d fa C -
-,1 . .- :: riessorson averagemake
. 'r. - : ::rs differenceshrinks to
: : :.:..:,ir productivity, teaching
.:-. i- -,-::": unaccounted-fOr gender
i'f :: - :lr unaccounted-forgaP in
::.. .. -r:re1 let the reader decide
i:: -:'-r :1rr. evenwith all our con-
': j " ,.: ::rcldeldictate that average
:- i- .: : ,r humanities faculty by
::: :,: .. -1 percentas large as the
' : : .-'..::rering versushumanities
-: : rhe gendergap, we com-
: : - . ...lLlesof Y for selectedval-
:i 'r rhe regressionmodel) for
- - -'c di i ier enc ef or eng i n e e r-
-. . ..,,i rh th e inc om e dif f e re n c e
:-..\\ e can benchmark using
::: ,:sk. for exarnple,how the
:-: i:nLler gap in relatedfields,
, --:h school teachers.Or we
:: -:::.s todal' comparesto the
. . ::
- rre\t for readersto deter-
r . i: i: - r >!-eex am plesin box 2 .1 ).
:,:l -:. :r,ir]\' researchersappear to
:: - . :l,.ri' often meaningful com-
- cent down to 17 percent for white applicants and from 1,4 per-
- :. - .:- rssueis not just that re-
i: :-' rhat will leave a lasting
- :::
-.:: :- :-:-.rbsent the disciplineim-
-r:: - :: hecome firated on small
The SecondRule . 49
effects while missing the five-hundred-pound gorilla in their midst. Even
seasonedresearcherssometimesmake that mistake. In fact, "missing the
five-hundred-pound gorilla" was a complaint I received fairly often from
re v i e w ersduri ng my sti nt as a i ournaledi tor.
A final point: It is possible to miss the interesting story by focusing lust
on means or on predicted Ys more generally. Sometimesthe "differcncc
that makes a difference" lies in dispersionrather than in averages.Consider
the question of whether working women are happier than housewivestn
B ox 2.L
Examples of Telling Verbal Comparisons in Gauging Effect Sizes
The following claims-accurate or not-provide good exarnplesof
comparisons that are memorable becausethey help readers gauge
the size of observedeffects.
o C)n the effect of religious attendance c-tnlife expectancy. On
the basis of longitudinal data from a national sample of the
U.S. population,Hummer et al. (1.999,p.277) find substantial
differencesin life expectancy by religious attendance:"For the
overall population, the life expectancygap [at age 201between
those who attend more than once a week (62.9) and those who
never attend (55.3) is over sevenyears, similar to the female-
male and white-black gaps in U.S. life expectancy."
. On gender differences in the effect of marriage on life ex-
pectancy. "Other factors constant, nine out of ten married
women alive today at age 48 would still be alive at age 65; by
contrast, eight out of 10 never-marriedwomen would survive
to age 65. The corresponding comparison for men reveals a
more pronounced difference: nine out of ten for the married
group versussix out of ten for those who were never married"
(\Waiteand Lehrer 2003, p.256).
. On the effect of race on employment Using an employment
audit where job applicants were similar on other background
characteristics,Devah Pager (2002) found that callbacks were
much higher for applicants without felony convictions. Among
both white and black applicants, a felony conviction reduced
one'schancesof a callback by one-half or more (from 34 per-
cent down to 5 percent for black applicants). As expected,
(continued on next page)
50 . Chapter2

the United States.The General Social Survey (GSS),a biennial survey of

U.S. adults, includes this question: "Taken all together, how would you
say things are these days-would you say that you are very happy, pretty
h^ppy, or not too happy?" Coding "very happy" as 3, "pretty happy" as 2,
and "not too happy" as 1, the averagehappiness of American adults is
2.203-between pretty h^ppy and very h"ppy, but closer to the former.
Making this comparison for housewivesversus women who work full-
time or part-tirne, we find no meaningful difference in averagehappiness:
The mean is 2.222 for working women and 2.219 for housewives(seethe
exercisesat the end of this chapter).

(continuedfrom preuiouspage)

then, a felony conviction hurts one's employment prospects.

The telling comparison, though, is the 17 percent callback rate
of white applicants with a conviction versus the 14 percenr
callback rate of black appiicants withowt a conviction. In this
study, being black hurts one's prospectsof a callback more
than being an ex-felon.
o On the political impact of felon disenfrancbisementin the
United States:The issueof whether felons and ex-felons should
have the right to vote is usually framed in terms of civil rights
for individuals. In the United States,however,the issuehas pro-
found political implications for the nation as a whole. On the
basis of their demographic characteristics, disenfranchised
felons and ex-felons would tend to vote heavily for Democratic
candidates if they were allorved to vote (Uggen and Manza
2002). Becausedisenfranchisedfelons and ex-felons currenrly
make up about 2.3 percent of the voting age population, and
the margin of victory in three of the last ten presidential elec-
tions has been 1.1 percent or lessof the voting-age population,
the rising population of felons likely has been a decisivefactor
in recent U.S. political history: As shown by the projections of
Uggen andManza, in 2000 Republican George\W. Bush would
certainly have lost the Florida vote, and thus the election, if dis-
enfranchisedfelons had been allowed ro vote in Florida; and
the results of the 1960 and 1976 elecions (won by Democrats
John F. Kennedy and Jimmy Carter, respectively)might have
been reversed had current (higher) levels of felon disenfran-
chisementprevailed in those elections.

CiS . a biennial survey of
. .,-,qcther, how would you
', rrll i:rfevery happy, pretty
' "pretty happy" as 2,
-.5-i.
i:ii of American adults rs
. rui closer to the former.
i*: \\'orlen who work full-
in averagehapprness:
-:.ice
L v for housewives(seethe
:,:tre,lfrom preuious page)
.:rplovment prospects
. - pcrc entc allbac kr a te
. i 'rqust hc l4 per c e n t
: :'' :!: J s onv ic t ion.I n t h i s
: :-;i s of a c allbac k m o re
.-:;.-,it'r,tncbisement in the
:.., n. and ex-felonsshould
:'.:J ir.rterms of civil rights
.- ,.\'.\'er!the issuehas pro-
.-r::on as a whole. On the
-:-:.:istics, disenfranchised
:- rc.rvilr for Democratic
'. )ie (Uggen and Manza
:'.. -rnd er-felons currently
:-: :::rg age population, and
: : .-:iit ten presidential elec-
- not until 1750-1800 that statistical graphics-length and area to show
: ::; i'oting-age population,
'. :.-..ibeen a decisivefactor
:^ ,',\'flbr-the projectionsof
.:.,-::George\7. Bush would
.::; rhus the election,if dis-
,,:: io I'ote in Florida; and
: . : -:r(rns(w on by Dem oc ra ts
::::. :ispectively)might have
:: .="els of felon disenfran-
The SecondRule . 51
Yet that comparison masks a telling difference in happiness between
housewives and working women. Compared to women who are in the
paid labor force, housewivesare significantly more likely to report being
either very huppy or not too happy: 3.1 percent more housewivesreport
being very hnppy (35.3% versus 32.2%) and 3.4 percent more house
wives report being not too happy (13/% versus10.0%). As a result, the
cross-tabulation of happiness by housewife status yields a Chi-square
value (a test of statistical independence)that is highly significant, indicat-
ing that happinessfor American women is nol independent of their loca-
tion in the workplace versusthe home. Here, however, the difference that
makes a difference is not the averagelevel of happiness,but the uariance
on happinessin the two groups.
The moral is that good social research looks at r,ariancesas weli as at
means.And in presentingfindings-whether basedon variancesor means-
good research most often relies on pictures as well as on words. That is
th e topi c of the nexr secti on.
Vlsuer PnnsENterroNop REsurts
A good graph is often worth a thousand words of text in a research arti-
cle. Graphic displays are pictures of numbers. The aim is to make large
data sets coherent by revealing the data without distorting what the data
have to say. Edward Tufte's The Visual Display of Quantitatiue Informa-
tion (19831 is a landmark work for presenting data graphically, and it is
worth quoting from the book's introduction (p. 8):
Data graphics visually display measured quantities by means of the
combined use of points, lines, a coordinate system, numbers, symbols,
words, shading, and color. The use of abstract, non-representational
pictures to show numbers is a surprisingly recent invention. . . . It was
quantity, time-series,scatterplots,and multivariate displays-were in-
vented, long after such triumphs of mathematical ingenuity as loga-
rithms, Cartesiancoordinates,the calculus,and the basicsof probability
theory.
"Excellence in statistical graphics," Tufte writes (p. 13), "consists of
complex ideas communicated with clarity, precision, and efficiency." The
best way to make the point is through examples. Box 2.2 provides two
examples focusing on displays depicting trends in American society (see
Tufte for other examples).
52 . Chapr er2

Box2.2
of Trends
Visual Presentations

o l5O
a'-

o 1nn
E
c I
l
f
F
50

I L
nr r '
" 1930 1940 1950 1960 1970 1980 1990

1930-199-5.
Example 1. Prison Adrnissionsby Race in the United S rares,
8'
Sour ce: http://purl.access.gpo.gov/GPO/LPS2026

1970 1990 1998 2050

( p r o j ecti on) (P roj ecti on)

f - . . l Wh ite ,n o n - Hisp a n ic f- - l Bla ck,n o n-H i spani c E H i spani c

t - - - l Asia n I Am e r ica nln d ia n

E x a m p l e 2 . Ra cia l/Eth n ic Co m p o sitio n o f th e U .S . Popul ati on,

8.
1970-205 0. Source: http://purl.access.gpo'gory'GPO/LPS2026
 The SecondRule . 53

Porrcy IupoRreucE

size matters, but it.is not rhe oniy rhing

that matters.soneti'res a s'ra,
effectcanhavebigimpricarions
r"' r"ig. p"p;i;;;
in a simple numericar sense:smaI "
cl.-rr',ir,
is *ue
effects ln terms of rates or percentages
imply big numbersfor big popurations.
Fo. the u.S. ccnsus B.-
reau estimatesthat the u.S. population ""r-pr.,
surpassed threc h'ndred million
sometimein 2006.If the percentage
or Americanswho are <lbeseincreases
by just one percenragepoint, thai's
three mition n.,";-;;;;;"p"nptr_n
population about the size of the
city of Chrcago.I' chin., rhe effect
one percentagepoint increasewould ,f a
be even greater-thirteen million
thenumber
orieople
inxe* v.,,kcity
;Til,tiTrnJl:#ffithan
!7e might call this the market sbare
pr|nc-iple_in a large market,
small changein rnarket share.makes a
t i! airr...,r..;"-;;;;ir"i... r_o__
paniesunderstand theprincipre "
welr,and"th*': ;d;;'ffir";;;
share is so inrense.If juit one Am..i.".r market
i" itty buys a Ford this year, that,s
about six million Fordssold. If pepsi
.rn .onuin.. just 1 percentof the pe.-
ple in china to buy o1...."r of pepsi
each week, that's about 675 rnillion
cans of Pepsisold in China p.. u.rr.
The marker share principi. u.ry ofren
drives public policy in the United
Statesand elsewhere.smai effects-"r-uti"
in the senset'hof tr,.y urr..t
a minusculeportion of the popularion_can
caffy enormouspolicy impli_
cations' particularrv in rhe,realm.of
publicheart'h
sequencesfor the affected population
il;;;#;r,'J." ..,n_
can. be ."or_o;, i,f;r' .*"_pt.,
death).As I write, the U.S. io.rgr"r,
l, .orrria..ing new regularionsman_
dating srrongerroofs for sp.ort_itirity
(sUVs) and light rrucks ro
reduce the number of deaihs rr.- ";;i;l.r
t.r-tili. rollover. The new srandards
are estimatedto cost,$gg_95 million
per year, and to prevent 73 to
deaths each year in the_UnitedSt"t"r'1R.'uters,..U.S. 44
proposesplan
StrengthenAutomobile to
Argrr; i;, 200s). If *. ;;-;e
estimateof 44 lives saved, \oojs,,, higher
that *ou'id h"ue..du.ed the number
highwav deathsin 2004 from 42,63i of U.s.
il; to 42,s92.In tcr-s of risk,
for the average American the annual
p-rrruiriry, of d.ath ,."- r,rgr.'
wa y acci d entw ill be r educ edfro m
a b o u t 1 1 6 ,9 1 9o r.0 0 0 1445 to" about
1/6,926 or .0001444. terls of
money, J" .,._ standardswill crst
I
million per life saved..calcurations $2
.r.h'nr'thi, inclicaterhar Amerrcans
indeedplace a high value on preventing
J*1fx on_U.S.highways,though
the value is not infinite, sincewe
.oulJp.evenr vehicularhighway
altogether by banning vehicular deaths
does not appear to be
consideringthat option at the moment,""rn..-c""gress
t o*"u.r.,

rTo be
sure, a ban on vehrcular travel
*ourd nor save 42,000 rives, since
would find alrernative means of t.rnrpn.tutinn Ame.cans
;;;l* carry the risk of cleath. The ba'
 54 . Chapter2

There is another reason why small effects can be important for policy:
Sometimes major determinants cannot be manipulated, or they are un-
known, so we focus policies on the small effects that are within the reach
of policy and that we know something about. Examples are commonplace.
'We
know, for example, that many diseaseshave a genericcomponent that
we cannot do much about. Hence policies focus on what we can more
readily control, such as the promotion of health pracricesthat are known
to foster good health and reduce the risk of disease(exercise,eat your
vegetables,don't smoke). We promote rhese practices even though in
some instancestheir effects are relatively small compared to one's genetic
endowment. Consider physical exercise.There is some debate over jusr
how much one's life expectancy is increasedby regular exercise-the an-
swer might be "not much." Yet the effect, even if small, is important,
since people desire long and healthy lives, and amount of physical exer-
c is eis s om eth i n gw e c a n c o n tro l .

IupoRteNcn FoR THEoRy

Relatively small effects sometimescan have big implications for theory as

well. The effect of income on happiness,for example, is fairly modest in
rich countries. The issue of income's effect on happiness is nonetheless
theoretically loaded. Economic theory has long assumedthat income has
positive utility-the more money, the more satisfaction or happinessone
has. Studies show that richer people are somewhat happier than poorer
people, at least on average.However, in a classicarticle three decadesago,
economist Richard Easterlin asks "Does economic growth improve the
human lot?" In that article, and in a body of work since,Easterlin(1974,
1996,2001) answers,"No, it doesnot-at leastnot in rich countries." In
his view the rising material well-being in the west is not being translated
into increased subjective well-being or happiness, as demonstrated, for
example, by the lack of growth in the general level of happinessin Japan
and the United States,rich countries where incomes have grown substan-
t ially ov er t he p a s th a l f c e n tu ry .
Robert Frank reachesthe same conclusion in his book Luxury Feuer:
Money and Happiness in an Era of Excess(1.999).There ls an income ef-
fect on happiness,he says, but it is a relative income effect: Happiness is

would no doubt dramatically reduce the number of miles Americans travel, however, and
that alone shor-rldreduce the number of deaths.In any case,the point is that there are limits
to the price Americans will pay ro reduce deaths.ril/e are not u'illing to ban cigarette smok-
ing, for example, eventhough we knou'that a smoking ban would improve the overall health
of Americans and savethousandsof lives each year.

: ::: - .: , ,rn be important for policy:
: r: -- r.:tipulated,or theY are un-
: . :-. : -:s that ar e wit hin t he r e a c h
::" -:. i remples are commonPlace.
t::: 1.,:- .: I geneticcomponentthat
' ;consumption arms race" akin to a military arms race-That is, if your con-
- :i .-ilSOn What we can more
'':-'. 11 frilctises that are known
: - - . -rrsJs e ( ex er c is e.eat y o u r
:: :-: : : rra cti c es ev en t houg h i n
;- . ..:'-,,. Jt)mpxredto one'sgenetic
.: l:.: . ls some debate over just
...' r'. reqular erercise-the an-
: , i. : ' ,:r1 i f s m all, is im porta n t,
.. .. ,- -).lrolrntof physical exer-
..: : :-.: lrrtplicationsfor theorY as
-:. r : :\.lrlple, is fairlY modest in
::: - r :'. happinessis nonetheless
-:. ::: :ssumed that income has
-- -. - .::-i .tetion or haP P ine sos n e
:: : :---.:"\'hat haPPierthan Poorer
. : - -r.i:J articlethree decadesago,
,:: :. :tomic growth imProve the
.: '. : '...,,rksi nc e,E as t er lin( 1 9 7 4 ,
--:- .r-,,.\t not in rich countries'" In
- -^,: \\est is not being translated
:r:riss. as demonstrated,for
:: : ... -.'r'elof haPPinessin JaPan
-.::: r-.a(lmes have grown substan-
--.-:. :r in hts book LuxurY Feuer:
:.- r-.-i -'r99t. There ls an income ef-
. : . : : . , : i nco m eef f ec t :HaP P i n e s si s
.. \mericans travel, however, and
.:. rhe point is that there are limits
:..,,rii rlling to ban cigarette smok-
: ri ould improve the overall health
The SecondRule . 55
affected by how much money you make relatiue to your peers. In othet
words, income gains produce no gains in overall happinessin a society be-
causeour standardskeep rising as we try to "keep up with the Joneses"'
This argument has profound implications. In a rich country that is
growing richer, individuals become engagedin what Frank (1"997) calls a
sumption lowers my happiness,and incomes are rising, then I must con-
sume more just to keep up, resulting in a never-ending"hedonic treadrnill"
for individuals (Layard 2005, chap. 4). This treadmill could haveperYerse
effects for society in generalas well as for individuals, since we could end
up consuming more than is socially optimal.
To test the hedonic treadmill theory with data at the individual level,
Laura Tach and I analyzed survey data for over sixteen thousand respon-
dents in the 1'972-2002 General Social Surveys (Firebaugh and Tach
'We
2005). examined responsesto the generalhappinessquestion in the
GSS ("Taken all together, how would you say things are thesedays. . . .").
Note that the preliminary statement "Taken all together" invites respon-
dents to think of happinessin terms of a durable trait rather than a tem-
porary emotion or mood-which is what we wanted, since our focus was
on happinessas an enduring individual trait rather than a temporafy state
o, of course,it might be argued that individuals don't really know
whether they are huppy or not-is self-reported happinessa valid indica-
-ood.
tor of one's true feelings?A number of studies have addressedthe issue,
with the weight of the evidenceindicating that a respondent is very often
in the best position to judge his or her own happiness(Konow and Earley
2003 ).
Becausethe hedonic treadmill requires a specific type of relative in-
come effect-one where keeping up with the Jonesesmeans continually
increasing one's own income, becausewe can be sure that the Jonesesare
increasing theirs-we used age as the basis for our peer groups. Age is
strategic for testing the hedonic treadmill hypothesis, since average in-
comes in the United Statesrise over most of the working lifespan. So as
Americans we must continually increaseour income to keep up with the
Joneseswho are our age peers.
Do, then, Americans compare themselvesto other Americans the same
age, resulting in a hedonic treadmill as individuals try to keep up eco-
nomically with their same-agepeers?In line with the treadmill hypothe-
sis, we found that the higher the income of others in one's age group' the
lower one's happiness,other things equal (we controlled for the individ-
ual's own income, age, physical health, education, marital status, and so
on). The relative income effect we detectedwas modest, however. Health
and marital status have much stronger effects on happinessthan relative
income does (at least as we were able to measurerelative income).
 56 . Chapter2

The important finding here is that age-basedrelative income effecrsap-

parently do exist. Regardlessof their size,they are important theoretically
for two reasons. First, age-basedrelative income ef?ects(or relative in-
come effects in general,whether or nor age-based)suggestthat individu-
als' utilities are interdependent.My happinessis affectel by the income of
my peers.I tend to be lesshuppy, for example,when my peersreceiveraises
and I do not. Second,age-basedrelative income effectsraise difficult ques-
tions about the value of continued income growth in rich counrries.
Rather than promoting overall happiness, continued income growth in
rich countriescould promore an ongoing consumption race where indi-
viduals consumemore and more just to maintain a constant levelof hap-
piness (for further discussion,see Frank 1997; Layard 2005). At
the
extreme) if peer incomes matter as much to us, dollar for dollar, as our
own income does, then relative income is the entire income,happrness
story (nor rhe enrire happiness5rory. since income is not the onlv det..-
minant of happiness). And if relative income is rhe entir. in.o-"-
happinessstory-that is, if there is no absolure income effect at all-then
overall happinessin a society does not increaseas individuals in the socr-
ety become richer.
In short: Even modest effects can have immodest implications for the-
ory. very often modest effectsare just that-modest, with limited implica-
tions. Readersneed to know which is which. It is up to you, the reseaicher,
to make that distinction.

CoNcrusroN

This chapter can be summarized in a single sentence:Look for differences

that make a difference-and tell us about them. Be sure to reporr the dif-
ferencesin such a way that your readerscan gauge the sizesoi the effects.
Reporting differencesin such a way that readerscan gauge the sizesof
the effects implies well-chosen comparisons. An apt .o-pa.iro., can be
worth a thousand asteriskedcoefficients. consider Devah pager,s (2002)
dissertation researchon the effect of imprisonment on job prospects (box
2.1, above)' To isolate the effect of a criminal record on employment
chances,Pagerconducted an audit study of employers, sending iwo pai.s
of well-groomed, well-spoken college men (one white team anJ one brack
team) to apply for 350 advertisedentry-leveljobs in Milwaukee. Therr re-
sum6s were identical, except that one member of each team said he had
served an eighteen-month prison sentencefor cocaine possession.within
each team, one member was randomly assignedthe '.ciiminal record" for
the first week; then they took turns playing the ex-offenderrole each week.
Pager found, unsurprisingly, that job prospects are poorer for those with
 The SecondRule . 57

a prison record. Surprisingly, though, the imprisonment effest was

trumped by a race effect, resulting in the comparison widely reported in
the media: "it is easierfor a \fhite with a felony conviction to get a iob
than it is for a Black with a clean record" (New York Times, Arts & Ideas,
March 20,2004). The point to emphasizeis that it is the comparison that
makes Pager'sresults compelling. Pager did not merely report that race
has an effect on employment prospects-she gave us a well-chosen corn-
parison to help us gauge the magnitude of that effect.
Instead of comparisons such as this, what we find too often in the so-
cial sciencesis the findings dump-authors present a laundry list of effects
("XpX2, and X. affectY") with little or no commentary on the relative
or absolute sizesof the effects.There are probably two reasonswhy find-
ings dumps are so common. The first is that variables in social research
very often are measuredin different units (dollars) years, points on a test)
that are hard to calibrate. Second,emphasison significancetests has had
the unintended consequence,perhaps, of devaluing calibration work. To
put it more bluntly, the inability or unwillingness to distinguish big ef-
fects from small effects may be a learned incapacity stemming from a
preoccupation with statistical significance.Although we all know better,
too often social scientistsconfuse statistical significancewith substantive
significance-a problem that is exacerbated by the convention in some
journals of replacing standard errors with asterisksto indicate statistical
significance at some arbitrary cutoff point (one asterisk means p<.05,
two asterisksmeans p<.01, etc.). The net result too often is a two-step
degradation of information, first compressing information on sampling
errors into two or three categories,then confusing those categorieswith
substantivesignificance.
This rampant imprecision in reporting the sizes of effects stands in
stark contrast to the careful attention usually paid to using estimation
methods that will yield the most precise estimatespossible. Given the ef-
fort required to collect and analyzedata in social research,it is amazing
how often researchersquit iust before the finish line.
 A P P E NDI X

Student Exerciseson Rule 2

The exercisesbelow use 7972-2004 data from the General Social Survey.
The GSS is a probability sample of noninstirutionalized English-speaking
individuals, age eighteen and older, in the United states. Fro^ i972 to
1994 the GSSwas an annual survey (with a few years missed); after 1994
it has been conducted in even-numberedyears. The sample sizes of
each
wave vary from about 1,500 for the annual surveysto about 3,000 for the
biennial surveys.
To do the exercisesbelow, go to the web site http://sda.berkeley.edu.
The instructions below should generateall the statisiics you need ro an-
swer the questions,so you should not need to do any calcuiations by hand.
Use the .01 significancelevel for questions about statistical signihcance.
Again, variable names are in boldface in the computer assignmentsin this
book, and answersto selectedproblems are provided at http://press.prince
ton.edu/titles/8593.html.
Note: The student exerciseson rule 1 (at the end of chapter 1) use the
samedata and web site. The site is user-friendly,so if you did the chapter 1
exercisesyou should have no trouble doing the exerciseshere. To avoid re-
dundancy,the instructions below include only the key commands;I do not
give step-by-stepinstructions as I did in rhe exercisesfor rule 1. Recall that
the figures given below are for the 1972)004 GSS, the most recent GSS
data available when this book was writren. so if you are uorkins with
more recent data, to replicate the figures exactly you will need to ise the
"selection fiber" in the sDA program to remoue years
after 2004. Also re-
call that the format in the SDA web site might have changed somewhat
sincethesedirections were wrirren, so some of the specificdiiections below
might be our of dare, but you still should be able to follow the logic below
to obtain the tablesyou need ro answer the questions.

AssrcNupNr 1. ITHer DetpntrrNrs HepprNrss?

The GSSasks the following question, denoted happy: "Taken all together,
how you say things are these days-*o.trd-you say th"t y"ou
_would
"r.
very happy, pretty happy, or not too happy?" Note that the question asks
about happinessin general,not about happinessin a specificiomain, such
as happinesswith one's job or marriage or financial situation.
 The SecondRule . 59

You have probably heard someone such as your grandmother say

that if you have your health, you have it all-you should be happy. Let's
see if there is empirical evidence for the proposition that health is
linked to happiness.From the Berkeley SDA homepage,click on "SDA
archive," then go to the GSS cumulative data file and follow these in-
structions:
o Recode h"ppy so high values denote greater happiness,as follows,
' : . GeneralSocialSurvey' From the recode screen!type in happy-re for "name for the new vari-
,:; .
- *i: : . : r:l i ze d E nglis h- s pe a k i n g able to be created." Type in h"ppy for "Var 1" under "Name of ex-
isting variables to use for the recode." For row 1 under "Output
. t om L97 2 to
. : . - . . :-: ..:,: Stat es F
variabler" type "1" for "valuer" "not too huppy" for "labelr" and "3"
.,. ., : .' '.irrs rnissed);aftet 1994
for "Var 1." For the secondrow, type "2" for "value," "pretty happy"
:J : : i: . Tre s am P les iz esof e a c h
for "label," and "2" for "Var 1." For the third row, type "3" for
- - r , --r',:" ) to about 3,000 for the
"value," "very happy" for "labelr" and "1" for "Var 1." Then click on
"start recoding" to convert happy to happy-re, a variable that ranges
:.' http ://sda.berkeley.edu.
from 1 for "not too happy" to 3 for "very happy." You should have
::.llisticsyou need to an-
43,317 valid cases.
calculationsby hand.
-rr.r,' . Recodehealth the sameway, by reversingthe coding so that it ranges
-:r statisticalsignificance.
from 1=poor health to 4=excellent health. Call your new variable
:'.:ure r assignmentsin this
health-re (for "health recoded").
, .lcd at http://press.prince
r Cross-tabulatehappy-re with health-re. Insert happy-re as the row
variable and health-re as the column variable (explanatory variable).
- ,: :. : of chaPter1) use the
-nd Use the options that you used for the exercisesat the end of chapter
-:: :.' - :. >rr if i'ou did the chapter1
1. Ask for column percentages,not row percentages(1 decimal point
.: a : -: .xe rci s esher e.T o av o i d re -
is fine for the percentages,except where I note otherwise). Check the
:: .. ::i- ker-commands;I do not
box beside "statistics," and use the drop-down menu to indicate that
i .- : : r . :r-rScsfor r ule 1. Rec a l lth a t
---- r GSS.the most recent GSS you want 3 decimal points for statistics. Remove the check beside
"color coding" and use the drop-down menu to indicate that you
::::.. ) ti t'ou Are working with
':'-::',...;-.r.', want "(no chart)" under "Chart options."
^,ctutt'iil need to use the r If you have followed the instructions properly, using "'Wt for black
':'" t | = '.e,1rsdfter 2004. Also re'
oversample," you should have 4,124 respondentswho are "not too
i.:: :'.,:.t: h.lve changedsomewhat
h"ppy," 18,476 who are "pretty h"ppy," and 10,808 who are "very
, i-: : ::..' specificdirections below
h"ppy."
* :: -::-.; io follow the logic below
:
- - . : : . I ! 1 1 1 ). Question 1. (a) Is the association between health and happinesspositive
and statistically significant?
(b) On the basis of your examination of the cell percentages,how would
. ,, . , . 1. ... i you characterizethe size of the association?
(c) Observe that percent "very happy" increaseswithout exception as you
:::. ::: happv: "Taken all together, move from the poor health category to the excellent health category.This
: , ', . - '. '.,rul d v ou s ay t hat y o u a re is called a "monotonic increase." Is the change also linear-that ls, are
:::' : ' \ote that the questionasks the changes in the percentagesroughly the same between each categorp
-.rr r. : - :>! i n a s pec if icdom ain ,s u c h or is the jump noticeably larger between some categoriesof health than
:. : . . :,tn ci als it uat ion. others?
60 . ChaPter2

the percent "not too happv" de-

(d) What about "not too happy"-does the
from poo' it' excellent health? Does
cline monot<lnically as you move
changeapPearto be linear?
in
with housekeepto test the claim
Next let's cross-tabulatehappy-re same average
women have the
,ftir'.n"0,.. that housewiut' u'iJ*'otking"happiness greater for h'use-
;; r'ariance on is
level c,f happiness, b;
w1V eS .
wrkstat-an eight-
o Create a new variable housekeep b'v collirpsing
(e'g'' full-time wclrker' part-ttme
category measureof rvork status
wor k er , r et ir ed ,t..p i " g h o u s e )-c l o w n to tw ocategori es" ' keepicom- ng
part-time'" Use the recode
house" versus-*'o'kinif"ll-time or
housekeepis the new variable to
mand to .nttop" 'utttst"atas follows'
1" under "Name of cxisting vart-
be created.Insert lvrkstat for "Var
row l under "Output variable"
ables ttl use for the recode"'For
:" .tl rti tg ' . fo r " l a b e l " ' and " 1' 2" for " V ar 1" '
t , v pe" 0" t o. " tn l .,.,"
F or t hes ec or rd ,..' ,.' p ..,1 ' ' " fo r..v a l u e ,' ' ..housekeeper'wrkstat
' .for..l a-
are
iiv"''t"' The remaining categories,of
bel," and "7" f;; has
housekeep
auromaticallutlita as rnissing' The new variable
men end women) and 8'219 house
27,784 workers (this includes
rvell as women)'
k..p.r, (this also includesmen as
e Cross-tabul*.';;;-;; i;;', virriable) u'ith housekeep (columr
"Oversamp-'Weight for black over
vanable). n.rntrn6*'to "lttt
\ /ant-to restrict th'
samples" Uafote.In this case' hon'er't" *t
an<
",
sample ro *o'-'-'i'-t, sex(2) for the "selection filter" box'
"'-in"" for statistics'(Sex is coded 1 for men an
indicate : a..i-li foint'
2 for women')
(a) Dld you reproducethe pe-rcentages "very happi-'-and "nc
Question2. (ver1'happy: 3::'%' for hous
too happy" cit.O ea'litt in the chapter?
wives, 32.2o/,, for working women; not too happy: 13 '4% for housewlve
percentages suggest..that housewrv
10.0% for rvorking *,o#n1. These
;ttty'n"pp'1"-and they are more likely to be "n<
are more likclv to t''"
too haP P Y . " ,----:^.^^^
i- L
greater variance in happine
(b) Give a plausible explanation for tl-re
among housewives'
the di{fererrce
cross-tabulationnecessaryto examine
Question 3. Do the men rvho are in the pa
versus
happinessfor menwho are keepinghouse
iou'"kttp *ith happy-re b;fore' b
labor force. tHint, tt*'-iabulaie ..selectiorrfilter" 1s analvsis
as to restrict the
this time inr"., ,o-.ih-g.tr.
r.,r"t ,"tplt lize should be.13'781' of whom 199 keep hous
men.l Your
resultsfor women'
(a) Compare yo,t tt"'lt' for men with your
 The SecondRule . 67

(b) Give a plausible explanation for why the association differs for men
and women.

Now use the Berkeley SDA web site to determine whether men or women
tend to be happier in the United States.
. Cross-tabulate happy-re (row) with sex (column). If you select
"Oversamp-\Xieight for black oversamples" along with the other
options chosen above, you should get 43,326 cases.Select3 decimal
points for statistics.

Question 4. On the basis of the 3-point happinessscale-coded 1 for not

too happy, 2 for pretty huppy, and 3 for very happy-for the 1972-2004
GSS the average(mean) happinessfor men is 2.198 and the averagehap-
piness for women is 2.207. (This result is found in the row just below the
column total row.)
(a) Explain this difference as clearly as you can in ordinary English. How
huppy are men on average and women on average?How largc is .009
points on this scaleof happiness?
(b) Is this gender difference statistically significant? (In other words, is
there an associationbetween gender and happiness?)
(c) \fith regard to happiness,do you see any differences of consequence
between men and women?

Question 5. (a) Compare the magnitude of the gender difference on hap-

piness with the magnitude of the differences in means across the health
categones.
(b) In comparing averageson this 3-point scale of happiness,what as-
sumptions are we making about the differences between the three re-
sponsecategories(very happy vs. pretty hnppy vs. not too happy)?

Question 6. (a) Rank the following variables in terms of their association

with happiness,from strongest associationto weakest association:college
degree, gender, health (4 categories),and married (dichotomize marital
into "married" and "not married" as in the exercisesfor chapter 1). Use
the recode command to create a new variable college (college degree=1,
otherwise 0) bv collapsing the GSS variable degree.Here is the informa-
tion you need to know to create college:degreeis 0 for no high school de-
gree, 1 for high school degree only, 2 for junior college degree, 3 for
bachelor's degree, 4 for graduate degree. College should have 46,349
valid cases,of whom 19.4 percent have four-year college degrees.
(b) Defend your ranking of the four variables.
(c) Consider the size of the associationsin an absolutesense.On the ba-
sis of your examination of differences in the cell percentages, which
62 . Chapter2

associationsare large, which are medium, which are small, and which
are trivial?

Question 7, Is there an associationbetweenage and happinessin the United

States?Before looking at the data, write down which one of the following
you expect to find: a positive associationbetween age and happiness(older
people tend to be happier), no association,or a negativeassociation.To see
if you are right, recodeage into 20s, 30s, 40s, 50s, 50s, 70s (or older).
(There are somerespondentsin the GSSwho are 18 or 19-code them wirh
the 20s.)Then cross-tabulateage_reand happy_re.'What do you find?

AssrcNnrNr 2. Vrsuer REpnpsrNterroxs

The aim in this assignment is to create pie graphs to visually represent

Americans' changing sexual attitudes over the past quarter century. \7e
will examine attitudes about homosexuality, premarital sex, sex among
unmarried teenagers,and extramarital sex.

Question B. Before doing the analysis, make your best guessabout what
percentageof American adults (1980-2004)think that:
o homosexuality is always wrong
. premarital sex is always wrong
. sex between unmarried teens is always wrong
. extramarital sex is always wrong
(a) \X/ritedown your ranking, from type of sex you think is most frowned
upon to type of sex that is most tolerated.
(b) Then, for each of the above, write down whether you think the per-
centage of Americans saying "always wrong" has increased,declined, or
stayedabout the samesince1980.

Now go to the homepagefor the GSScumulative file on the Berkeley web

site. The relevant GSS variables are homosex, xmarsex, teensex,and pre-
marsx. To find out how homosex is coded, enter homosex in the box be-
side "view" (unless the SDA forrnat has changed, you will find it in the
upper left of the page), and click on "view." Use the recode command
to create four new variables that are coded 1 for "always wrong" and 0
for the other three categories ("almost always wrong," "sometimes
wrong," and "not wrong at all"). You have had enough experiencewith
the recode function that you should be able to do this by yourself. Call
those variables homosexwrong, xmarsexwrong, teensexwrong, and pre-
marsxwrong.

::- 1:: '.:'J happinessin the United
i , i . .,.:rrchor.reof the following
' ',:-,. :::' .rseand happiness(older
. - : .: r iiJtive association'To see
- , . - '. ,;rl s.60s , 70s ( or old e r)-
...-- - \ crr l9-code them with
: L'a!. re. \\'hat do you find?
.:.:ls to visually represent
l.l:t Quart€r century' \tre
-.: L.estguessabout what
-. i t h; lt:
::S
. ,,u think is most frowned
,.:r'ther y'ou think the per-
:'-..sincreased,declined,or
.. hle on the Berkeleyweb
\marsext teensex,and Pre-
vou u'ill find it in the
the recodecommand
:.'r "al rv ay swr ong" a n d 0
.,r','s\\'rongr" "sometimes
j enough experiencewith
, do this by yourself. Call
ia\r\ i,tng. teensexwrong, and pre-
The SecondRule . 5.1
The next step is to use the recode command to create a variable called
decadefrom the GSS variable year. You want to enter three values undEr
"Value" for the output variable: 7980, L990, and 2000. For 1980, enter
"1980s" under "Label" and "1980-1989" under "Var 1." For 1990, en
ter "1990s" under "Label" and "1'990-1'999" under "Var 1." For 2000'
enter "2000s" under "Label" and "2000-2004" under "Var 1." Click
"start recoding." You should have 14,241 casesfor the 1980s, I3,ZZ3
casesfor the 1990s, 8,394 casesfor 2000-2004, and 10,652 casesof "no
data" (theseare the respondentsin the earlier surveys).
The final step is to cross-tabulateeach of the sex variablEswith decadc
'Weight
(column variable). Select "'$7t for black oversamples" from the
dropdown menu, and remove the check beside "color coding." Select"Pie
Chart(s)" under "chart options" and then run the table.
Question 9. (a) Compare the percentageswith your guessesin question 8.
(b) Then summarize the results in one or two paragraphs.There are two
big issuesto discuss:first, the level of Americans' approval or disapproval
of premarital sex, teenagesex, homosexuality, and extramarital sex; sec-
ond, the trends in theseattitudes. The pie charts should be especiallyhelp-
fu l i n seei ngthe trends.
(c) Which resultssurprisedyou the most?
CHA P T E R 3

The Third Rule

BUII,D REAI-ITYCHECKSINTO YOUR RESEARCH

To determine the height of a tree or the distance of a ship from the

shore, we can use a method known as triangulation. Triangulation is
basedon the principle that if we know the length of the baseof a trian-
gle along with the two anglesat either end of the base,then we can cal-
culate the height of the triangle. In other words, we use elemenrary
geometry and trigonometry to calculate an unknown distance from a
known distance.
The term triangulation has beenembracedby social scienrists,who use
it to refer to the practice of using more than one method in a study as a
way of double-checking results.The idea is that we can be more confident
of our resultsif different routes lead to the sameconclusion.
This chapter presents rule 3, the reality check rule. In the chaprer I
discussthe importance of building reality checksmore rourinely into so-
cial research.Triangulation using alternativemethods is one approach,
but it is not the only approach. Someof the reality checksapply to data
preparation, some apply to data analysis,and some apply to checks on
conclusionsafter the analysis.In many instancesthe word "check" would
probably suffice, but I use the term "reality check" deliberately,to em-
phasizethe importance of stepping back from our analysisoccasionally
to seeif what we are doing is sensible.In many insrancesthe reality checks
are commonsensicalor pedestrian,yet evenexperiencedresearcherssome-
times go astray by failing to build checks into their studies. Indeed,
some of this chapter consists of case studies of researchthat has gone
astray (researchillustrating how not to do it) in the hope that as social
scientistswe can learn as much from our mistakes as from our suc-
cesses.I have taken most of the examples from researchin sociology,
since that is the field I know best. Becausethey provide the most arrest-
ing and memorableexamples,I tried to selectlines of researchinvolving
influential studies of important issr.res.In instanceswhere it was appro-
priate to focus on a specificstudy to illustrate a point, I hope authors
understandthat it is the importance and clarity of their work that com-
pels the close scrutiny of it.
 .fhird
The Rule . 65

INrenNar Rrarrrv CHEcrs

Reality Checks on Data-Dubious Values and lncomplete Dat(t

The first step in any analysisof quantitative data is to check means, vari
ances,and correlations for plausibility. Experienced researchersknow to do
this but are often eagerto "run some models" before carefully checkingthe
data. Researchers are especiallylikely to skip this stepwhen using secondary
survey data from large survey firms, where the data presumably are already
clean.It is nonethelessa good idea routinely to look at the means'variAncesl
and rangesfor each of the variablesin our analysis.Are the m€ansand vari-
: ::,':.:l.cof a shiP from the
: : ::!.'-;ri,rr r . T r iangulat i o n i s ancesreasonable?Do all the valuesfall within a plausiblerange?
o f t he bas eof a t ri a n - You should eramine scatterplots for outliers and nonlinearities in the
- -:i :
early stagesof your analysis.For surprising correlations-ones that are
: :::: t ase,then we can cal-
smaller or larger than expected, or not in the predicted direction-
:Js. u'e use elementarY
scatterplotscan revealif there are singlecasesor clustersof casesthat might
inown distance from a
account for the unexpectedresult.
use It is especially important throughout the analysis to keep track of the
:: : - : -: ^-' .o ci al s c ient is t swho,
a studY as a item nonresponses,that is, missingentriesbecausea respondentfailed to
: :..:- :r.' method in
r'\'c more confident answer a question. How are those missing values coded? How do you
:: - ::.:: can be
want to code the responsecategory "ns1 5s19"-is it coded as missing, or
i- : - 1 .l l : 'onc lus ion.
as neutral? Item nonresponsesare a potential pitfall in any analysis, and
. - :: - .:'. -'k ru le. I n t he c haP te rI
inattention to missing values can lead to serious blunders. (It is also im-
tr *r : -:S mor e r out inelyinto s o -
portant to pay arrentlonto missingdata due to nonrespondents-ind:id-
:: ': - : . . : ::l -Ih ods is one aP P r o a c h ,
uals who were chosen for the sample but who fail to participate. This
: : : ::,:.litv checksapply to data
issue is discussedin the next chapter.)
. ,. '.:-j \ome aPPIYto checks on
r.:i-:.--:>tl-rervord "check" would
M ISSIN G D AT A
::-. :'. rieck" deliberately'to em- TRAD IT ION AL M ET H OD S F OR D EALIN G W IT H

j-1. :: ::: (tur analysisoccasionally If you encounter missing entries in your study-as you will sooner or
-.. * -,.::'.. :rslancesthe reality checks later-the first question to ask is how much missing data there is. That
:-,:: : ..:iriencedresearcherssome- question is first because,as Paul Allison (2002) observes,a standard old
s t udies .I nd e e d , method calledlistwise deletion is an attractive option for dealing with in-
- r :, r .> .rro their
- ir-, - : : i ,f res ear c ht hat has g o n e complere data, so long as the amount of missing data is not intolerable.
: : .: in rhe hoPe that as social Listwise deletion (also known as cdsewise deletion or complete cdse
-: ::-tistakes as from our suc- analysis) removesindividuals with incomplete information. Although that
-*
::: .. ::om researchin sociologY, appearsto be very wasteful, listwise deletion has severalimportant advan-
a rre s t-
-r - : : iili \ p ro v ide t he m os t tages.First, it can be used for any kind of statisticalanalysis,from ordinary
: . =- . ; r l i n eso f r es ear c hinv o l v i n g regressionto the most sophisticatedmethods, with no special software re-
. l-. . : stance swher e it was aP Pro - quired. You use the methods that you had planned to use, with off-the-
. . -r: lr.]tea Point , I hoP e au th o rs shelf software, but you include only the caseswith complete data. Second'
:r: ; . . . .:i tvof their wor k t hat c o m- although your standard errors will tend to be larger becauseyou are using
less information, you will obtain appropriate estimatesof those standard
66 . Chapter3

errors. As a result, "you do not have to worry about making inferential

errors because of missing data-a big problem with most of the other
commonly used methods" (Allison 2002, p. 6).
Third, if all the standard assumptions of regression are met) listwise
deletion provides unbiased estimatesof the regressionslopeswhen "miss-
ingness" is caused by independent variables but not by the dependent
variable (Litde 1992). Imagine, for example, that we are studying the ef-
fect of income on happiness, and that some respondents have missing
'We
data on income or happinessor both. might suspectthat the missing-
ness is causally related to income, the independent variable, since those
with more income are lesslikely to answer the income question. That sort
of missingnesswould not bias our estimate of the effect of income on
happiness (see box 4.1, in the next chapter). However, if missingnessis
also affected by happiness (the dependentvariable)-that is, independent
of income level, if h"ppy people are more likely (or less likely) ro answer
the questions about income and happiness-then our estimatesof the ef-
fect of income on happinessare biased when we use listwise deletion.
Other traditional methods for handling missing data are lessattractive.
Researcherssometimeshave filled in missing data by inserting the average
X for caseswhere X is missing. To avoid discarding caseswith missing
data on income, for example, you might simply insert the mean income.
This method is called mean substitution or mean imputation. Mean im-
putation is not to be confused with multiple imputatictn, a method dis-
cussed subsequently.Mean imputation understatesuncertainty and thus
understatesstandard errors; multiple imputation introduces randomness
in order to provide better estimatesof the standard errors (below).r
Another traditional method for handling missing data is pairwise dele-
tion, or audilable caseanalysis. Pairwise deletion takes advantage of the
well-known fact that regression slopes can be estimated from sample
means and the covariance matrix. In other words, you can do linear re-
gressionwithout knowing values on the variablesfor the individual cases,
as long as you know the means, variances,and correlations of the vari-
ables.This suggeststhe following strategy: On the assumption that some
information is better than no information, use all the casesavailableto es-
timate the covariancematrix, and estimatethe regressionslopesfrom that
matrix. Becauseall available casesare used, the correlation of X, and Y
may be based on 300 cases,while the correlation of X, and Y is based
(say)on 250 cases.Despitethe apparent advantagesof pairwise deletion-

I In a more sophisticatedversion of the mean substitution

strateg)',researchersinsert a
conditional mean (from a regressionequation) instead of the overall mean. This srraregyrs
superior to substituting the simple mean, but lt still produces understatedestimatesof stan-
dard errors. Multiple imputation methods were developedin large part ro overcome such
oroblems.
 The Third Rule r (-

-rhoutmaking inferential no information is discarded, and no values are imputed-pairwise dele

* irh n'rost of the other tion generally should be avoided because the standard errors given b)-
-, ; . : standard software will be biased.
.i : :-lrcssion are met, listwise Allison (2002, p. 12) sums up the situation as follows: "All the com
i.. ; t ; - i ;S5l o n s loP eswhen " m i s s - mon [traditional] methods for salvaging informati<-rnfrom cases with
r: 1r :i l'-rt not by the dependent missing data typically make things worse. . . . In light of these shortcom
-: :. -;.rr \\-eare studying the ef- ings, listwisedeletiondoesnot look so bad."
: I :::pondent s hav e m is s i n g [n some instances,however,listwise deletion is unattractivebecauseit
I . -- discards too much data. How much data discarding is "too much"? Un-
-::: iuspect that the missing-
:ir::.-ient variable,since those fortunately, there are no hard and fast rules. Discarding 50 casesis prob-
'," :: : : .:.o me ques t ion.T hat s o rt lematic if you have 100 cases but barely matters if you have 100,000
' -i: : - th e e f f ec r of inc om e o n cases.One point is clear: Other things equal, the more variablesyou have
H,
'r.
o'cr if m i in your model, the more likely you are to face missing data problems.
: - : . . , : r ^ l c -that is , indepen d e n t Supposeyou have twenty variablesin your regressionmodel. Then, as Al-
:: . ;.'. or lesslikely) to answer lison (p. 2) points out, even if you have only 5 percent missing data for
-:rr--:----r our estirxatesof the ef- each of the twenty variables,you can expect only 35 percent of the cases
.:..-.:. ,',: use listwisedeletion. to have complete data on every variable under the assumption that the
-; -' -.:::q Jat r ar e les sat t r ac ti v e . chance of missing data on one variable is independent of the chance of
- .. r - -.ri .-:hr i ns er t ingt he av e ra g e missing data on any other variable. In other words, by using listwise dele-
: : .:-:rding caseswith missing tion you would expect to discard nearly two-thirds of your cases' even
- - , - . .. tn i e r t r he m ean jnc o m e . though each variable has data for 95 percent of respondents!
". . ":.-;': intPutatioz. Mean im- In short, listwise deletion is not always a viable option. What do you
:: . . . : ..:. :'r:p tttat ion,a m et hod d i s - do when you cannot use listwise deletion? That is the topic of the next
-::::.:-i:es uncertainty and thus section.
-:.,: .: ,r rntroducesrandomness
NEWER M ET H OD S F OR D EALIN G W IT H M ISSIN G D AT A
:-. -: , , :-:,rrderror s( below) . 1
r .-- --..:>ir1s data is pairwise dele- In instanceswhere listwise deletion is not viable, you should consider re-
;: :: ::. 'l takes advantageof the cent data augmentation methods that have been developedto filI in miss-
:. -.: :; estimatedfrom sample ing entries in data sets.Modern methods of data augmentation use what
ir:: ,', rrlS.\'ou can do linear re- is called multiple imputation to impute values for individual units. The
: .: : rr.;s for the indiv idualc a s e s , logic is to use information we know (information already in the data) to
:.-:.. ..:i .orrelations of the vari- impute values we don't know. In effect, then, we insert values where none
.: ''' -l'r J s SL l f i t n f i o n t h a t so m e existed before in the data.
: . - : : ,:--rh e c as esav ailablet o e s - Before describing the multiple imputation method, it is important to
::: :..: :-gressionslopesfrom that clarify two points. First, imputation is not fudging the data. At first blush it
,r,:i. r:'; correlation of X, and Y might appearunseemlyor evenunethicalto impute valuesfor individuals-
: ': : -. -r:i ,tn o f X , and Y is ba s e d when you impute values, are you not "making up data" ? In point of fact,
: :- . -:. :.:.ge s of pairwise deletion- however, you are imputing on the basis of information that is already
there, in the data. The aim of social researchis to determine how X is re-
' - -- :r r . . r : r lrrate g y ,r e s e a r c h e r si n se r t a
- ' : : :: : : -: Lr| era l ln . r e a nT. h i s s t r a te g yis
lated to Y, and that is done on the basis of information that is in the data.
- " - .,:: .i.]derstatedestimatesof stan- Becauseimputation, done properly, doesnot distort or add to information
: : : ::j : . lerge p a r t t o o v e r c o m esu ch that is already in the data, imputation is not fudging. (If you are uncom-
fortable with multiple imputation, then you should also be uncomfortable
68 . Chapter3

with mean substitution, since that also "makes up data." The difference
is that mean substitution makes up worse data.)
The second point is that there are proper and improper ways to im-
pute missing values from the data. tWealready noted that the old strategy
of mean substitution is a bad idea. It's a bad idea becausewe end up
turning a variableinto a constant for respondentswith missingvalueson
that variable. As a result, we fail to control for the variable over that por-
tion of the sample where the data are missing. Mean imputation pro-
ducesbiasedestimatesof regressioncoefficients,variances,and standard
erfofs.
Newer imputation methods differ from earlier ones by deliberately rn-
troducing random variation as part of the imputation procedure (Rubin
1987; Schafer 1997\. Multiple imputation (MI) uses Monte Carlo tech-
niques to replace the missing values with m> 1 simulated versions.There
are three steps:imputation, analysis,and pooling.
. ImPutation. Random draws from some distribution are used to fill in
the missing entries. (The distribution is selected to inject the right
amount of uncertainty to produce unbiased estimates of variances
and standarderrors: seeAllison 2002, chap.5.) The missing entries
are filled in z times, resulting in m completedata sets,where m typ-
ically is small (say 3-10).
o Analysis. Each of the m data setsis analyzed,resulting in z analyses.
. Pooling.The m analysesare combined into a final result.

The method is called multiple imputation becausemultiple complete data

sets are produced and analyzed.Single imputation sometimesis adequate
if the proportion of missing values is small. However, single imputation
requires almost as much set-up as multiple implication does (though less
computer time), and single imputation tends to understatestandard errors
becauseit omits the berween-impuration component of variability (Alli-
s on 2002, p. 29 ).
Specializedsoftrvare programs have been developedfor multiple impu-
tation, and more are on the way. The key assumption of these programs
is the so-calledmissing at random (MAR) assumption: Data on Y are said
to be missing at random when the probability that Y is missing for the lth
unit is unrelated to the value of Y for the lth unit, after controlling for the
other variablesin the analysis.
Becausemultiple imputation is computationally demanding, even fast
computers may require severalhours of number crunching to obtain esti-
mates for very large data sets. Other than its demands on computer time,
the downside of multiple imputation is that it yields somewhat different
results each time you do it, so there is no "right answer" that others can
duplicate exactly when applying the same method to the same data.

-.,:f.:: -:: ;-rta." The difference
: :: l
t. : - -. . . * l rl ro ner way s t o lm -
t' , . ::J that t he old s t r at eg Y
, : - : . ::,,. b ec aus ewe end up
..:: :i '.'.'ithrr.rissing valueson
: :. . -lrlilble over that Por-
" 'While
-
. .: : r \1-.rn rnlP ut at ionP r o -
:: . -: . . ":i 'i anc es ,
and s t anda rd
in -
: : : . : : :l e i br - deliber at elY
: - .: - -:.r::(l 1 1pr oc edur e ( Rubi n
- l. l- -:i i s \Iont e Car lo t ec h -
-; - - r ;11,:i.tted versions.There
!: :'.rtion are usedto fill in
: : .<.i €d to injec t t he r igh t
:i:.: iitilrates of variances
. . . ; j . the m is s ingent r i e s
.::: j-tl.t sets,where m tYP-
::. :r-suitingin z analYses'
: --.:nal result.
,,-.-. ::'rultiPlecomPletedata
--.: :r sometimesis adequate
--. .i ;r cr. ringle imPutation
-'.:. .,rtion does (though less
standarderrors
-: : -.,--lc-rStote
.-: rint of variabilitY (Alli-
":::. ::'....rped for multiPle imPu-
:r : : : - . :t:ti on o f t hes eP r ogr a m s
. :-- -: .-.rtl o n:Dat a on Y ar e s a i d
., . .. . .., 1 -i s m is s ingf or t he i th
: i:- -::',:.lfter controlling for the
'..:::. :-.,:i1.. demanding,even fast
:-:-.r:: ;rr-rnchingto obtain esti-
i: . . : - j-':r.tndson c om P ut ert im e ,
::-:: :: -'ields somewhatdifferent
- -:.::: answer" that others can
:.: --.:::'.,1d to the samedata'
The Third Rule . 59
Reality Checks on Measures-Aim for Consistency
in Conceptualization and Measwrement
Inconsistencyin the conceptualization and measurernentof key concepts
can lead to problems that range from confusing presentationsto fatal er-
rors in the conclusions. I begin with the problem of opaque prose result-
ing from fuzzy conceptualizationand measurement.
denseprose sometimesmay be a sign of deep thinking on diffi-
cult issues,more often it is merely a sign of muddled thinking. The point
that I want to stresshere is that clear writing begins with clear thinking,
and clear thinking includes transparency and consistencyin the measure-
ment of key concepts.
CONSIST EN C Y BET W EEN C ON C EPT AN D M EASU R EM EN T
Social scientists sometimes introduce needlesscomplexitn as if to keep
readersoff balance and guessing.Let me provide a simple illustration, As
a journal editor I often receivedpapers where one of the critical variables
was coded in a direction opposite to the concept (for erample, a paper on
poverty that usesincome as the dependentvariable; or the title of the pa-
per usesthe term equality but the study usesthe Gini coefficient or some
other measureof inequallty). This might not seem like a big deal, since
the reader can just remember to reversethe signs of the coefficients, but
why should the reader need to do this? Placing the onus on readersto re-
verse the signs is an unnecessaryburden, particularly in studies where
some of the variablesare coded in a direction consistentwith the concept
and some are not. It is hard for readersto keep this straight-as witness,
for example, instanceswhere a double-reversalis called for (both variables
are coded opposite to the concept), so the original sign is correct. Some-
times authors stumble over their own coding, and in the worst casesread-
ers can't even interpret the reported coefficients because it isn't clear
whether the reported coefficients are intended to correspond to the con-
cept as stated or to the concept as actually measured!
The simplicity of this illustration should not detract from the importance
of the issue.The purpose of a researchreport is to communicateresearch
findings, and the stage for clear communication of findings is set well be-
fore the first word of text is written. If you have trouble keeping everything
straight as a researcher)you can be sure that readers,who are not as famil-
iar with your researchas you are, will be evenmore befuddled.Where there
is fog in the pulpit, there will be densefog in the congregarion.
CONS IST EN C Y IN T H E M EASU R EM EN T OF SIM ILAR C ON C EPT S
Sometimesresearchersbecome so involved in the nuts and bolts of their
project that they lose sight of the big picture. One of the big-picture issues
 70 . Chapter3

is consistency in measuring variables. variables that measure the same

essenceshould be measuredin the sameway. In comparing the effects of fi-
nancial investment in different regions of the world, for example, we
would want to use a consistent measure of investment. Otherwise, if u.e
found regional differencesin the effect of investment,we would not know
whether those differences reflected real differencesin invesrment effects,
or merely differencesin the way investment was measured.
Lest this example seemfar-fetched, it is instructive to consider a boom-
ing line of researchin the 1970s and 1980s on the determinantsof eco-
nomic growth in poor countries. At that time a number of sociologists
and political scientistsadvancedan unorthodox idea: poor countrieswould
be better off with less,not more, economic contact with the richer natrons
of the \7est. The prediction was that poor countries engaging heavily in
economic exchangewith rich countries would, due to exploitation by the
rich countries, tend to erperience slower economic growth than poor
countries not so heavily engaged. contrary to the prediction, however,
empirical studies found beneficial trade effects: Other things equal, poor
countries that traded more with rich countries generally experienced
faster, not slower, growth.
So the focus on this line of researchshifted from the effects of trade to
the effecrsof foreign investmenton economic growrh in poor countries.
Here the findings seemed to supporr the prediction. Literally dozens of
studies concluded that foreign investment from rich countries has short,
term positive effects-so-called investmentflow effects-but harmful long-
term effects (stock effects) on income growth in poor countries. Because
the negativestock effect was judged ro overshadowthe positive flow effect,
some scholars believed that social research had discoveredan imporrant
principle for economic growth in poor counrries:Do not permit foreign
investment from rich counrnes.
It turns out, however, that this conclusion was based on inconsistenr
measuresof investment. Foreign investment and domestic investmenr are
similar conceprs-rhey differ only in the source of the funds-yet thev
were measured in very different ways in this body of research.when for-
eign investment and domestic investment are measuredthe same way, we
find that borh types of investment tend to boost income growth in poor
countries (casestudy, box 3.1). The error was not discovereduntil studies
applied the reality check suggestedhere: use consistent measuresof the
same concept. In other words, a misguided line of research could have
been avoided if researchershad done a simple reality check by asking the
question: what do we find when we measureforeign and domestic invest-
ment the same way?

' . . . : =i th at m eas ur et he s a m e
,,,:', : . 'rmparing the effectsof fi-
': . : -.i ri or1d, for example, we
,-: r-'..lrnlent. Otherwise, if we
I r :-irll€[t, rve would not know
: .r::::-.es in investmenteffects,
-:-: ','.-:-rnleasufed.
i : - : -: -.'tl\.eto considera boom-
'-: r ,, the determinants of eco-
- vestigations; countries, as in studies of the causes of econotnic growth;
:.;: :..:.: .r number of sociologists
rr- "- i .Je.r; Poor countrieswould
'-- , , :':-,..rn'ith the richer nations
: , - - .:::rri e sengagingheav i l y i n
- -. l u - to ex ploit at ionby th e
.,r,:: :'- ,nornic growth than poor
' ' . : : . ' .' rhc pr edic t ion, how e v e r,
: :: ': -: -: Other t hings equal,P o o r
- : -.:.::ies generally experienced
:::: :rom the effectsof trade to
-' - iro\\'th in poor countries.
: : - -i :cti on. Lit er ally doz e n so f
:: ::r rich countries has short-
'. :. :t'iects-but harmful long-
.i.::-..l-lpoor countries.Because
i r....i!r\\'the positive flow effect,
, :,..i Jiscoveredan important
-,:-.::3s: Do not permit foreign
.-. -:. r- ',ias basedon inconsistent
:::,.r: -,.,:i domestic investment are
-.-: : :!-:e of the funds-yet they
- -- : rLrJv of research.'lfhenfor-
:.: :.:':::reasuredthe sameway, we
: -. : .,sr income growth in poor
' : .i.-:-:ror discovereduntil studies
:.: - .r ,(tnsistentmeasuresof the
:-, :: : :ne o f res ear c hc ould h a v e
. . : . - : . . ::.rl i n 'c hec k by as k in gth e
: t . . -- . r' ':ci g n a nd dom es t icin v e s t-
The Third Rule . 71
Realiry Checks on Models-The Formal Equiualence Check
Sometimes-as in the example of the effects of foreign and domestrc
investment-social scientistsuse aggregatesinstead of individuals as their
unit of analysis.The aggregatescould be precincts, as in studies of votingl
censustracts, as in studies of residentialsegregation;classrooms?as in
studiesof teachereffectiveness;school districts, as in studiesof educational
funding and equity; cities, as in studies of crime rate, employrnent rate!
and suicide rate) to name a few; statesor provinces, for any number of in-
and so on.
B ox 3.1
Case Study: The Effect of Foreign Investment on Economic
Growth in Poor Countries
Dozensof cross-countryregressionstudiesin the 1970s and 1980s
concluded that foreign investment from rich countries reduces eco-
nomic growth in poor countries. However, when we replicate the
studies applying the same measuresand the same logic to domestic
investment as to foreign investment, we conclude that domestic in-
uestment also harms economic growth in the long run (Firebaugh
1992).
Becauseno one believesthat countries fare better in the long run
economically by lowering their domestic investment, there is a seri-
ous problem here. To identify that problem, let's look at the cross-
country model used in this line of research.The outcome variable is
rate of grorvth of income per capita for some sample of poor coun-
tries over some time period. The control variables vary somewhat
from study to study, but the investment measuresalways consist of
these three: (1) stock of foreign investmentas of the first year of
the study, measured as the cumulated foreign investment in a coun-
try divided by the size of the country's economy; (2) flow of for-
eign investment over the period of the study, measured as change in
foreign investment stock over that period, divided by the size of the
country's economy; and (3) domesticinvestmentrale, measuredas
the rate of growth of domestic investment over the period of the
study.
In short, the dependent variable is measured as a rate and do-
mestic investment is measured as a rate, but foreign investment is
(continued on next page)
/z . L , n a D t er - ,

Most often the variables that are employed in aggregateanalysesare

compositesof somekind-per capita measures,growth rates,ratios, and so
on. By composite mezsuresI am referring to variablesthat are composedof
two or more separablevariables.Per student expenditure, for example, is
total expenditure divided by number of students.Crime rate is number of
crimes divided by population size. PercentLatino in a censustracr is num-
ber of Latinos divided by total population in the tract. Composite measures
such as theseare standardfare in aggregate-level social research.

(contintred from preuious page)

measuredin a very different way. To appreciatethe problem caused

by this measurement inconsistency, suppose we measure growth
rate as proportional change,that is, as (Yr-Y1)/Yp where the sub-
scripts 1 and 2 denote the initial and final years of the study. (I use
proportional change for convenience;alternativeways of measuring
growth rate yield the same result.) Then the foreign investment
model used by researchersin the 1970s and 1980s is as foliows,
where 1 denotes per capita income, F denotes foreign investment,
and D denotesdomesticinvestment:
( 12-I ) lI 1 = a + 6 (F z - F r) + t F ,+ BI(D
z - D 1)I D l
+ controls+ e (3 . 1 )
This model states rhar a country's rare of growth of per capita
income from time 1 to time 2 is determined by foreign investment
inflow (F2-F1) over the period, by level of foreign investment at
time 1 (Fr), bl rate of domestic investment, and by other factors
captured in the control variables and in e.
The dozens of studies using this model reporr these findings and
conclusions:
o 6>0. Conclusion:In the short run, foreign investmentboosts
economic growth in poor countries.
. y <0. Conclusion: In the long run, poor countries are better
off economically without foreign investmenr.
. P>0. Conclusion: Domestic investment boosts economic
growth in poor countries.

Note that theseconclusions are basedon a model that measuresdo-

mestic investment as the single variable (D2-D1) / D, and foreign

(continued on next page)


:: .-.i;regate analyses are
:\- -- .' :: rates, fatios, and So
' ii 1' :- ::l,lt are COmpOSedOf
: :,.:: j-:'!ire, for example, is
r:rl: - :-::1erate iS nUmber Of
::r- - -: .ensus tract iS num-
:--.i ::-,.:. lomPosite measures
:" : . , - -:. l. ' S e afCh .
:. 'rom preuious page)
problem caused
:reasure growth
. g'here the sub-
rhe study. (I use
rts of measuring
: 'ralqn lnvestment
- -r r. i s as f ollows ,
: ,:arqn lnvestment,
l.l
(3.1)
'.,.lh of per capita
: ,:eign investment
-:rgn investment at
:; bv other factors
: -::he:e findingsand
1\'estmentboosts
I - ,,rntriesare better
:.- . 1.-,,
:- .- . eCOnOmiC
-r, O \ t S
:l.ratmeasuresdo-
'D, and f or eign
,:iinued on next page)
The Third Rule . 73
As a general rule there is no special difficulty in interpreting the results
of studies with composite measureswhen the composites are all formed
the sameway (Kuh and Meyer 1955; Firebaughand Gibbs 1985). An ex-
ample would be studies where all the variables are expressedas percent-
ages, such as percent single-parent households and percent poor in a
neighborhood, or as per capita measures,such as income per capita and
nurses per capita in a country. The denominator, population size, is the
samefor all variables.Becausea researcherusesper capita measuresto ad-
just for the fact that the units (censustracts, schools,countries, and so on)
vary in size, there would be no need for such a denominator if all Lrnits
(continued from preuious page)
investmentas two variables,Fr-F, and F,. To make the measures
consistent, suppose we split the numerator and denominator of the
domesticinvestmentmeasure,entering Dr-D, and D, as separate
variables for domestic investment, iust like they are for foreign in-
vestment. It follows logically that, since B > 0, the effect of the nu-
merator will be positive and the effect of the denominator will be
negativewhen entered as separatevariables (seeFirebaugh 7992 for
empirical verification). To be consistent with the conclusiolls above,
then, we have this new set of conclusions:
. 6 > 0. Conclusion: In the short run, foreign investment boosts
economi cgrow th i n poor countri es.
. y<0.Conclusion: In the long run, poor countries are better
off economically without foreign investment.
. Bnu,,",oror)0.Conclusion: In the short run, domestic invest-
ment boosts economic growth in poor countries.
. Fd",u,rinotor<0.
Conclusion:In the long run, poor countriesare
better off economically without domestic investment.
The conclusion that poor countries would have higher incomes if
they had lower levelsof domestic capital investment is ludicrous, of
course,and it points out the danger of measuringan outcomevariable
as a growth rate and explanatory variables in terms of components
of growth rate. When foreign investment and domestic investment
are both measured as rates (consistentwith the outcome variable),
cross-country regressionsindicate that both types of inuestment are
associatedwith faster income growth in poor countries (Firebaugh
1992).
71 . Chapter3

were the same size. In effect, by using per capita measures,a researcher
uses a mathematical operation (division) to control for differencesin the
populations of units. Alternatively, a researchercould use raw totals rather
than per capita measures,and control for population by adding popula-
tion size as a control variable in a regressionmodel. Whichever method is
used-controlling for population by division (per capita method) or by
residualization2 (regression method)-there should be no special diffi-
culty in interpreting the results.3
Sometimesresearchersperform analyseswhere the numerator of one
variable is the denominator of another variable. In that case, you are not
using division in a consistent fashion to conrrol for population size, so re-
sults are not always easy to interpret. The general point is that the use of
dissimilar composite variables (where, for example, some variables are
per capita measuresand others are raw counts, or where the numerator of
one variable is the denominator of another) in some instancescan produce
resultsthat are tricky to interpret.
Reality checks are particularly important when you estimatemodels
containing dissimilar compositevariables.In the caseof dissimilar com-
posites, it is often useful to try to express the model in alternarive
forms. The aim is to see if there are different ways to express the same
model that will aid your intuition in interpreting the model's coeffi-
cients.
Consider this example, again from the cross-countryregressionliter-
ature. On the basis of an analysis of income growth for about five dozen
poor non-\Westerncountries from 1960 to 1998, Giovanni Arrighi, Bev-
erly Silver, and Benjamin Brewer (2003, hereafter ASB) claim thar in-
dustrialization does not promote economic advancement in poor
countries. "For most [poor] countries," they write (p. 18), "industrial-
ization [has] turned out to be an ineffectual means of economic ad-
vancement." This claim is questioned by Alice Amsden (2003), who
notes the contradiction between the ASB claim and the strong associa-
tion between industrialization and income growrh in poor countries in
recent decades.

2 By "residualization" I refer to the fact tl.ratthe effect (slope) of X on Y controlling

for
p o p u l a t i o n c a n b e o b ta in e d b v r e g r e ssin gth e r e sid u a l(Y-Y) on the resi duel1X -X1, w here
the regression of Y on population and i is from the regression of X on popula-
Lj:t."
sThat is, there is no special difficLrlty arising from the use of composite
fileasuresper se.
\ff/ith aggregate data there ls, however, the very real danger of committir-rg the "ecological
fallacy" of drawing conclr-rsionsabout individual relationships from aggregate-levelrelation-
ships. That practice is called a fallacy becauseit makes the problematic assumption that re-
lationships at the aggregatelevel mirror relationships at the individual level (seethe section
"Context Effects as Nuisance" in chaoter 7).
 The Thir:dRule . 75

I r',: :: ::leasL1res, a researcher Amsden appears to be correct: Using the same countries as ASB, and
: . :: . ;or differer-rces in the weighting by population size, I found a whopping correlation of r=0.98
:--: , user awt ot als r at h e r between industrial growth and income growth over the period 1960 to
-..:
: : : . : ::1'\nbi' adding popul a- 1998. How do we account, then, for the ASB claim that industrialization
' - : \\-hichevermethod is has not benefitedpoor countries economically? Amsden suggeststhat the
: : - r:: ..rp i ta m et hod) or b Y ASB finding must be due to a "bug in their program" (2003, p. 34). In
:t; : . ...,.it'c no special diffi- their reply, however, ASB state that they double-checkedthEir computer
program and are satisfiedthat their finding is not due to a programming
,:i ., : -: :he numerator of one errof.
: :: : -r :hat case,you are not It turns out that ASB appear to be correct on this point: The problem is
--.- ::,rp ulat ion s iz e.s o re - not their computer program. The problem lies in their model. To under-
: i :r: i- -- :o i n t is t hat t he us e o f stand where the confusion comes from, begin with the essential model
r :, . , , r1 1.c.som e v ar iablesare that ASB estimate:
r- : : i. '',','5eret he num er at oro f (Y/P)'=Br+B, (M/Y),'+
e, (3.2)
: . : -.: lllstancesCanPrOdUCe
where Y denotesincome (total national output), P denotespopulation, M
:: 1 : , .-:1 \'OUes t im at em ode l s denotes manufacturing output, prime denotes growth rate, and the sub-
. - - : -. - ;,rseo f dis s im ilarc om- script I identifiesthe individual case(here,the country). Hence (Y/P),'is
::: - . : -.. nodel in alt er nat i v e rate of growth of per capita income in the i th poor country and (M/Y),' is
:::-: '.-:.'! i fO expressthe Same rate of growth of manufacturing output as a fraction of total output in the
r : :: - : : : . :l g the m odel' s c oef fi - i th poor country. For this model, ASB report a slope of zero for fifty-eight
poor countries, L960-1,980, as well as a slope of zero for fifty-nine poor
- - ' . -'-L rl l n tr vr egr es s ionlit e r- countries, L980-1998.
-: :: '.'.ih for about five dozen Now let's rewrite the ASB model. Becauserate of growth of a ratio
: - - - '. C,iovanniArrighi' Bev- A/B is the rate of growth of A minus the rate of growth of B, we can write
(Y/P I ,' as Y l -P ,' and 1M/Y l ,' as M,' - Y ' . Thus w e can rew rj te equat ion
" - : r: , irr:f -\S B ) c laim t hat i n -
.i Jr.rn c em ent in P o o r 3.2 as;
- - .: . ,.:::. rp. 1B ) , " indus t r ia l - (Yi'-P i')=Fo+F.,(M,'-Y,')+e,. (3.3)
:-: ---:, ::1-.111S Of eCOnomicad- 'lfhen
- , . ,- -\msden ( 2003) , wh o expressedthis way, it is clear that this model does not actually esti-
: - -: :.-.,.-rldthe strong associa- mate the economic effect of industrialization in poor countries.In plain En-
-: r - '.',::ti n P oor c ount r iesi n glish, the ASB model statesthat the ertent to which income growth exceeds
(or lags behind) population growth is a function of the extent to which
manufacturing growth exceeds(or lags behind) income growth. Thus the
coefficient F, reflects the effect of segregated manufacturing growth on
:. of X on Y controlling for economic growth in poor countries. Because Fris zero, we conclude that
.: :- : rrhc res i d u a l ( X - X ) , w h e re
the difference between income growth rate and population growrh rare is
' - - ^ . : . : egre s s i o t o
] f X o n p o p ula -
uncorrelated with the difference between manufacturing growth rate and
I aomposlte measuresper se. income growth rate (growth rate of total output)-precisely as we would
. :. . -: . : c omm i t t i n g t h e " e c o l o g ica l expect if industrialization boosts economic performance.Successfulindus-
r: - . .:: : : om ag g r e €a a t e - l erveel la t i o n - trialization means that manufacturing growth boosts not just manufactur-
I ' :- '- : :: ilent at i c a s s u m p t i o nt h a t r e -
ing but other economic sectorsas \.ell, so that manufacturing growth rate
r - r ' : . r : '. -i: ric lu al le v e l ( s e et h e s e c tio n
does not greatly outrun rate of growth for total output.
76 . Chapter3

In short, Amsden and ASB are talking about different things. For the
countries that ASB analyzed, the correlation between industrial growth
and income growth is close to r= 1.0 for the 1'965-1'998period, so Ams-
den is correct in her claim that industrialization benefits poor countries.
Poor countries that are industrializing clearly tend to grow faster econom-
ically than poor countries that are not. Yet it is also true, as the ASB results
indicate, that successfulindustrialization means manufacturing growth
that links to growth in the rest of the economy.
The bigger lesson here is that reality checks are important for models,
especially in the case of models with dissimilar composite variables. So-
cial science researchersshould try to express models formally, and to
search for alternative expressionsthat facilitate interpretation of empiri-
cal results.

Exrnnuer Rserrrv Cnpcrs: VRrroRtroN wrrH OrHER

DRra aNo Mr,rnoos

The reality checks describedto this point are internal, in the sensethat
they can be done by researcherswithout reference to other data sets or
results from other studies.Thus we can check our data for dubious val-
ues, we can check our measuresfor consistencywith each other and with
the conceptsthey are intended to capture, and (sometimes)we can ex-
press our models in alternativeforms to check the interpretation of our
results.
In this part of the chapter I turn to methods for evaluatingthe sensibil-
ity of social scienceresults by comparing them with results drawn from
other types of data and methods. I have in mind purposiue checks for the
sensibility of our results. The checks might involve thought experiments,
simulations, or the direct comparison of quantitative and qualitative re-
search. I give severalexamples below from the research of others. These
examplesdo not exhaust the possibilities but are intended instead to srim-
ulate thinking in this area.
In describing these methods I do not mean to suggestthat current re-
search practice ignores external reality checks. The most obvious reality
check is the comparison of one's results with the results of prior research,
and that is done routinely. Journal articles in the social sciencestypically
include a discussion-of-resultssection, for example, and the comparison
of one's results with the results of other studies is standard fare in that
'We
section. But we can do better. need to devote greater attention to the
development of cleverer and more direct reality checks for our conclu-
slons.
 Th e T h i r d R t r l e ,

using causal-process obseruations to Testprausibility

of Resurts
Henrv Brady (2004) uses a causal-processobservatiors
approach (rris
terms) to provide an exampleof what I mean
by an externuti.otity chc,ck.
A causal-processobscrvation is defined as ,,an insight
or piecc of clata
that providesinformation about context, process,
or mechanisrn,and thar
contributes distinctiveleveragein .ausaiilferense.
a .rrr"i orucessob-
servarl.n . . . is frequentry viewed as an indispe.sable
r"ppt.rrl.rrt to cor-
relation-basedinferencein quantitativeresearch,,
1s."-.rgt t c_otti..
20 04, p p. 277- 781. "',a
In Brady's example a causal-process
observationsanalysis servesto
provide evidenceagainsta claim ,hut
."u, made on the basisof ..g..r-
sion analysisof election data. The election "
in question is the 2000 U.s.
presidentialelection in Florida. on
the basis of his analysisof-ru.nou,
jala all sixty-sevenFlorida countiesin l9Bg, 1992,
lor 1996,a'd 2000,
John Lott (2000) concluded that the nerw'rks, pr._"iur. pru;r*rlun
AI Gore as the winner in Florida cost George of
w. Bush r.ur,'tb-,'oo0vores
in,ten predominantly Republicancou.,tie's ",
in the panhandre of Ftorida.
(The ten panhandle counties are in the
central time ,or-re, it .-pro,..-
rlon was made when the poils were stiil open in "rrJ
those counties but crosed
in the rest of the state.) The votes were lost,
then, by d"-p.Jng uo..
turnout in countieswhere Bush was popular.
As Lott explains (quotedin
Brady 2004, p. 268):
B.y pr-ematurely declaring Gore the winner
shortly before poils had
closedin Florida'sconservarive westernpanhandle,in. up
suppressingthe Republicanvote. . . . An -.dru'.nded
examination of past Republi_
can presidential votes by county in Florida
frn- tggs tr'r0-0; shows
that while total votes declined, the Republican
v.ti'g ,"* i,-,ii. *.r,-
ern Panhandle was significantly ,uppressed
relative to the non_
Republicanvote. The 4 percent gr."r..'..,iuction
in n.pulli."n uo,",
averagesabout 1,000.vo.tes.percotrnty,
lyieldingl f O,OifOfi.frUn."r,
votes for all 10 counties in the western ianliandle.
Lott baseshis conclusion on a crifference-of-difference
anarysisof turn'ur
data for the four presidentialelections.The
prematureprolec-
tion of Gore as the winner is perceived 'etworks,
ki,r.loi"lir.ri*o.ii*.r,
"r,
where ten counties received tire "treatment,,
and th. oth..-fiity_r.u.',
countiesdid not' The basiclogic for the anarysis
is this: Sincethe ,1.*p.r-
iment" occurred-onlyin the i000 erection,
compare county turnout dif-
ferencesin the 2000 erectionwith county
differencesi, ,h; ;;;;earlie,
elections to esrimatethe effecr of th. n.t*orks'
projection. In effect, data
on pasr turnout is used to project the number
of additional vot"s gush
78 . Chapter3

would have received in the panhandle counties under the counterfactual

condition that the networks had not projected Gore as the winner. Lott's
answer: 10,000.
There is nothing wrong with Lott's method per se; difference-of-
difference models are standard in many social scienceapplications. But
the 10,000 figure invites a reality check. As Brady (2004, p. 269) ex-
plains, there are ways to test the plausibility of the 10,000 figure: "Is
Lott's estimate reasonable,given the numbers of voters who had not yet
voted when the media called the election for Gore? How many of these
voters heard the call? Of these, how many decided not to vote? And
of these who decided not to vote, how many would have voted for
B us h?"
To addressthese questions, Brady first determined the precise time the
media declaredGore the winner: 6:50 pm central time, ten minutes be-
fore the polls closed in the panhandle. Data from 1996 on time of voting
indicate that no more than 1112 of the voting in Florida occurs during
the last hour, and interviews with election officials indicated that there
was no last-minuterush to the polls in 2000. The assumptionthat voters
go to the poll at an even rate during the last hour implies 116of the 1172
total voters went to the polls in the last ten minutes. This figure is about
4,200 voters for the panhandle (1./72of the approximately 300,000 pan-
handle voters that day). If all of those voters were Bush voters and they
all gave up and returned home when the media called the election for
Gore, that would imply 4,200 lost votes for Bush. As Brady goes on to
observe, however, even the 4,200 frgure is implausible since only a frac-
tion of the 4,200 would-be voters were likely to have heard about the
projection, and not all of them were Bush voters in any case.On the ba-
sis of plausible assumptions about what fraction of the 4,200 voters
heard the announcement and, of that group, what fraction were Bush
voters who would have been dissuadedfrom voting, Brady (p. 270) con-
cludes that the upper bound for Bush's vote loss was 224-a far cry from
10,000.
As Brady (p. 270) observes,although his detectivework usesquanri-
tative data, it uses methods typically more characteristic of qualitative
research. "It tries to approach the problem in severaldifferent ways,
cross-checkinginformation at every turn, and asking if the posited
causal effect is probable, or even possible, given what we know from
many different sources."The lessonfor quantitative researchers,Brady
(p.271) writes, "is the necessityof paying attention to the causal pro-
cessesunderlying behavior. Otherwise, regressionanalysisis likely to go
off the rails."

.-' . :: i:rJer the counterfactual
- Using Etbnographic Data to HeIp Interpret Suruey Results
:-i: I r.r rf e as the winner. Lott's
: r . : . . . : Pcr s e; dif f er enc e -o f-
' :
- .: .,icnce applications.But
. - -, :, ,.J. \2004, p. 269) e x -
-.
i - : '" ': :h e 10, 000 f igur e: " Is
. - r : r: - ',oters who had not yet
. : .: rsi How many of these
- , . * .;.Je d not t o v ot e? A n d
'i - ,: , s'ould have voted for
_.
- -. ,
"- I 'r'
.-..-rlu (r r L - y^' - ^is e t im e th e
r LLr
: : : . - ::-r1 ti m e, t en m inut es b e -
." .: : : . :.. \9 96 on t im e of v ot i n g
: : :: :--Florida occurs during
: : :j::-rls indicated that there
' -: ts'u11P 1is n t hat v o te rs
::r i -.: implies 116of the 1.11.2
:: :-.:'.,::es. This figure is about
r - : , , . r r,,\i mat ely 300, 000 p a n -
: : : : :i .re Bus h v ot er s and th e y
::.: : ::r-,. called the election for
:: i --,:'h. As B r ady goes on to
: : .'.r.-'.-rSible since only a frac-
-: r: i r) hav e hear d about th e
-:. - . : r:! rn anv c as e.O n t he b a -
-:: : : : --::1 )no f t he 4, 200 v o te rs
a: ,.:. .''h.rt fraction were Bush
: : :" ' :rrrg.B r ady ( p. 270) c o n -
. r: . :: ir'.rs)24-a far cry from
- - . *.::.ri re uor k us esqua n ti -
-. i: - r .rr.rCteristicof qualitative
: ::- I seYeraldifferent ways,
-i- - . ',"J .rsk ing if t he pos i te d
i. :. : . $ 'hat we k now f r o m
--r;n
: : -^-::.:::Jti\-e researchers,Brady
r t . . : : : :-:tti on t o t he c aus alp ro -
: :. : : : : : L rn analy s isis lik ely t o g o
The Third Rule . 79
Social scientists often specialize either as quantitatiye researchersor as
qualitative researchers.(There are further specialtieswithin each, but thc
quantitative/qualitativedistinction is the rnajor divide: SeeBrady and Col-
lier 2004, and King, Keohane, and Verba 1994 for recent discussion.)As
a result, studiesin sociologyand the other social sciences tend to be either
quantitative or qualitative. Although mixed qualitative-quantitative stud-
ies are becoming more common, they remain the exception in the social
sctences.
Very often, however,a full understandingof some phenomenonof inrer-
est to social scientistsrequires both types of data. To advanceknowledge
significantly,we might needfieldwork as well as surveydata, ethnographies
and casestudiesas well as censusdata. In that regard, there are large-scale
projects under way in the social sciencesthat involve teams of specialists
collecting both qualitative and quantitative data. One example is an inten-
sive study of the effectsof welfare reform on the well-being of children and
families in Boston, Chicago, and San Antonio-called "Welfare, Children,
and Families:A Three-City Study" (http://www.jhu.edu/-welfare)-which
features an ethnographic component (in-depth interviews and participant
observation)as well as a longitudinal survey.
These big studies aside,the typically more modest social sciencestudy
involving a single researcher or a small team of researcherscould also
benefit from the juxtaposition of quantitative and qualitative evidence.In
this section I provide an example of how censusand survey data, the sta-
ples for nonexperimental quantitative research in the social sciences)can
benefit from the insights of ethnographic fieldwork. Of the many exam-
ples that could be given, I focus on the important and timely issue of
racial and ethnic self-identity in the United States.
Race and ethnicity continue to serve as important sources of eco-
nomic and social cleavagein the United Statesand elsewhere.Consider
again the hotly contested2000 presidentialelection in the United States.
Although George W. Bush and Al Gore each received48 percent of the
total vote nationwide, that overall equality masks sharp differences
across racial and ethnic groups. \White voters, for example, favored
Bush by 54 percent to 42 percent for Gore. By contrast, Asian Ameri-
can, Hispanic, and African American voters all favored Gore: Gore
received54 percent of the Asian American vote,67 percent of the His-
panic vote, and 90 percent of the African American vote (cited in
L u h man 2002, p. v).
'While
scholars have long recognized the presenceand importance of
racial and ethnic divisions in the United States,recent researchunder-
scores just how difficult it is to capture the complexity of race and
80 . Chapter3

ethnicity in a censusor survey. Individuals'racial and ethnic identities

can be very fluid. Multiracial individuals might view themselvesas
white in some contexts and as nonwhite in other contexts, for example.
Mary'Waters (2002) cites the example of one of her undergraduatestu-
dents at Harvard who had been told by her mother that she is an Amer-
ican Indian with lrish, Scottish, and African American ancestry also in
the mix. In filling out applications for college admission she had routinely
checked all the boxes that applied. Upon her arrival at Harvard she be-
gan to receivemail from the African American Association.By contrast
her identical twin sister,who had checkedthe same boxes she had, was
being lobbied by the Native American StudentsAssociationto join their
group. How could she and her twin sister,she asked, have different racial
identities?
How indeed? Yet that is precisely the sort of situation many Ameri-
cans face. The situation has important implications for censusand sur-
vey data on race and ethnicitv. Consider the case where adolescents
report their racial identities differently at home and at school (Harris and
Sim 2002). In survey analysisthis type of inconsistencymight be viewed
as misclassification-that is, as error in the data. In the caseof racial and
ethnic identities, however, it is more fruitful to view such inconsistencies
not as error but as raising issuesto be probed further.
The issuesto be probed involve the rules governing racial self-identities.
'What
rules do multiracial individuals use to choosetheir racial identities?
Why do children's racial self-identities sometimes vary from context to
context, and why do they sometimesdiffer from the racial self-identitiesof
their biological parents?And what does this tell us about race and ethnic-
ity in the United States?Both quantitative and qualitative evidenceis criti-
cal in our attempt to understandthesevexed issues.
"Currently the social rule in the United States governing racial and
ethnic identity is self-identification," writes 'Waters (1998, p. 29), and
"For a large portion of the population, this involves some degree of
choice." How large is this portion? For that question it is useful to turn
first to survey data. David Harris and Jeremiah Sim (2002) analyze data
from the first wave of the National Longitudinal Study of Adolescent
Health (Add Health) for their study "\fho Is Multiracial? Assessingthe
Complexity of Lived Race." The Add Health data set is ideal for such a
study becauseit provides self-reported race data for parents, for adoles-
cents at school, and for adolescentsat home. By comparing parent-based
race, school race, and home race, Harris and Sim find considerablefluid-
ity in racial identity. For example, on the basis of responsesadolescents
give at school, 6.8 percent of them are multiracial. On the basis of re-
sponsesthey give at home, only 3.6 percent of youth are multiracial.
 Thc Third Rule . 81

overall, while 8.6 percent of adolescentsgave a multiracial responseet

rnd ethnic identities
ther in the home survey or in the school survey' only 1.6 percent gave a
: r'ierv themselvesas
multiracial responsetn both conrexts (and, of the 1.6 percentwho idcn-
' . : :: -o [t€x t s r f or ex am p l e ' tified themselvesas multiracial in both contexts' only 1'1 percent se-
: :'.crundergraduatestu-
lected the same combination of two or more races at home and at
:-.<fthat she is an Amer- s c h o ol).
'.;rri an anc es t r yals o i n
Ethnographic evidenceis often vital for understanding patterns such as
.....i nn ' he had r out in e l Y
these. consider, for example, waters's (1999) ethnographic study of the
:::-..riat Harvard she be-
By co ntr as t
racial and ethnic identities of immigrants from the West Indies (mainly Ja-
-:r -. . . . i' slt lciatio n' maica, Guyana, Trinidad, and Haiti). Her study is strategic in under-
.:l i : :-i: l1e bo xe s she h ad ' w as
their standing racial self-identitiessince West Indians, though generally viewed
\ssocia tion to ioin
::-:: : ' : .
racial as blacks by most whites in the United States,do not necessarilythink of
.-.-.*- ..'...1. have different
- themselvesas black Americans.In the early 1990s'Watersbegan a study
of !(est Indian immigrants and their teenagechildren in New York city.
many Ameri-
': ilttlatlon She was particularly interested in how these immigrants would balance
sur-
.,.:arnsior census and their identities as
'WestIndians and as Americans, and in tracing intergen-
:::. ;.tsg rvhere adolescents
(Harris and erational changes in racial and ethnic identities for the immigrants and
:-..: '-ndat school
-t'-.,rr.,-t.u their children. From in-depth interviews with second-generationadoles-
, might be viewed 'Waters (2004) identified three
racial and cents from various West Indian groups,
-.:.,...tn the
case of .,paths of identity development" rhat underlie 'west Indians' re-
.' ie\\' such inconststenctes broad
sponsesio questions of racial identity: some identified themselvesprima-
: -.rth er . rily black Americans, some as ethnic Americans (with some distancing
- :ri rrg.rocial self-identnies' "s
from black Americans), and some as foreign-born individuals not'West be-
.'..,h . ir . r ac ialidenti ti e s ? ethnic In
categories. her interviews of
to holden to American racial and
::-:: ::les varv from context Indians she also found, however, that "The situation determinesthe iden-
---,-.. ,l-t. racial self-identities of
tity chosen by most people, and there is a great deal of ease in moving
. :.- us about race and ethnic-
is criti- back and forth betweendifferent identities" (1999,p.62). Shefound, for
:-.: :.relitative evidence example, that most
'West
Indians think of themselvesas black when they
i : . 1 i S,
I
and feel threatened by whites, or find themselvesin the minority. Or West In-
:i )r'1te5governing racial dians might adopt a black identity to conform to the majority, as in the
( 1, 998, P . 29) , and,
, =. n "r..i caseof the woman who recalled "being black by day and being west In-
some degree of
::'.:. irN'olves dian by night" (Butterfield2004,p.298). This sort of detail is not easily
it is useful to turn
:---.:qllestion obtained from standard survey data.
-'-'
:,:--.-ltl Sirn (2002)
;n flrrr \,vv!/
analYzedata
*-'*^

1.,:,,drua1StudY of Adolescent
the Research
Is ,lt.tltiracial? Assessing Other Examples of Multiple-Method
--i set is ideal for such a
, J.,,
Multiple-method studies employ more than one type of analysis.often
... i;;;.; parents'for adoles-
these studies draw on data from more than one source as well. The kev is
.l-. st comParingParent-based that the multiple data setsand methods are used in a single s/zdy. Multiple-
fini considerablefluid-
,,'.i si,-,-, merhod studiesdiffer from what I call replication studiesin rule 4. Replica-
: r,.:.sisof resPonsesadolescents
tion studies apply the same metbods to different samples. In replicarion
= On the basis of re-
',.',,-rito.iul. studies the aim is to see if identical analysesyield similar resuits for
..,...t of vouth are multiracial'
82 . Chapter3

different samplesof people.In multiple-method researchthe aim is to seeif

different methods lead to the sameconclusions.
Multiple-method research has become more common recently in the
sciences.This trend likely has been encouragedby the migration of
s<-rcial
social scienceresearchfunding from proiects done by single investigators
to projects involving multidisciplinary researchteams. Whatever the rea-
sons for its growing popularity, multiple-method researchpromises to be
better research,if only becauseit permits more reality checks throughout
the researchprocess.
Many combinations of methods are possible, as illustrated by research
appearing in recent issuesof the American Sociological Reuiew (Jacobs
2005). A study of Americans' changing perceptions of PresidentAbraham
Lincoln as a leader, for example, draws on data from diverse sources-
surveys, the writings of leading historians, treatments of Lincoln in his-
tory textbooks, and even statuesand memorials (Schwartz and Schuman
2005). The researchersconclude that, while Lincoln's reputation as a
great leader has remained intact, the popular basis for that reputation
changed with the civil rights movement, from "Lincoln as savior of the
Union" to "Lincoln as the great emancipator."
In other studies qualitative and quantitative methods are employed in
sequence,with one method building on the results of the other. Thus, for
example, Brian lJzzi and Ryon Lancaster (2004) interviewed a small
group of lawyers and clients before beginning their study of social ties
and pricing in U.S. law firms, and Rodney Benson and Abigail Saguy
(2005) interviewed journalists and others before conducting a statistical
analysis of media coverage of social problems in the United States and
France. Sometimesthe quantitative analysis comes first. In a recent study
of physical and sexual abuse (Cherlin et al. 2004), for example, re-
searchersfollowed up a survey of over 2,000 families with a seriesof re-
peated open-endedinterviews with 256 of the families over a period of
twelve to eighteen months. Reports of abuse surfaced increasingly in the
subsequentinterviews. In this case, then, the follow-up qualitative com-
ponent was key to obtaining accurate information on the phenomenon of
interest.

CoNcI-uorNGREMARK

In his critique of current methods in the social sciences,David Freedman

(1991, p. 358) concludesthat what we need is "reality tests insteadof I
tests." I agree,and in this chapter I describedsome possibilities.I do not
have ail the ansrvers, and the examples I gave are intended to suggest
'We
and to provoke. not to be exhaustive.I do know this: need to think
 l he ThrrrlR rrlr ' \

seriously and creatively about how to incorporate reality checks mor.-

:: - ,: r: ;:'-:;h rhe aim is t o s eei f
routinely into our research.I want this chapter to serve as a push in thar
:-
direction. The next chapter pushes even further, by suggesting that vre
'. : : :111on recently in the
- should strive to make replication-the identical anaiysisof parallel data
)' :i . '.:;J b v t he m igr at iono f
sets-the rule rather than the exception in social research-
-. -. - ^ r' ;i ngle inv es t igato rs
- -::::l i , What ev ert he r e a -
-it i-
-':l- - ::sc.rr c hpr om is est o b e
- . . :.-.--!.'c hec k st hr ougho u t

. : lustratedby research
qii,tl Reuiew (Jacobs
.i fPre s ident A br aha m
::, 'nr diversesources-
' : - : : : . . :.-.:n tso f Linc oln in hi s -
- : : : :;hs-artz and Schuman
as a
' - : - :-,,l n's r eput at ion
r :- ',.:. for t hat r eP ut at i o n
. -ol n as s av ior of th e
i:a
':r: , : .'..:hodsare emPloYedin
- : i : : -- :. ,ri the ot her .T hus , fo r
: .: -' i n ter v ieweda s m a l l
: : - : :...1r studYof s oc ial t i e s
-
:r. l: i.>Lrl t and A bigail S ag u Y
-: :,:: :: .!jnducting a statistical
r :.-.- - rhe United Statesand
r : - :-.:! irst. In a recent studY
:'. :. l .j+). for example, re-
s u'ith a seriesof re-
:..::.r.ie
: : : :-t:rilies over a Period of
: ,. . , . .::-.i ;cdinc r eas inglyin t h e
' . - -: - .. \\-up qualit at iv ec o m-
: i -',:: :. (ln the Phenomenonof

: ''. -:. :.:-:l.es, David Freedman

r , : : i : "::l l i i l ' t es t sins t eado f I
: :: : i :re possibilities.I do not
r . : , . . : ,l re i n t endedt o s ugg e s t
j : r-. ,\' this: We need to think
A P P E NDI X

on Rule 3
StudentExercises

This chapter has stressedthe importance of checking and double-checking

your data, measures,and results. You should look for ways to cross-check
your results both internally-using other information in your data set-
and externally-against other data sets.You should check for consistency
betrn'eenconcept and measurement, and for consistency in the measure-
ment of similar concepts.N{ost basic of all, you must be sure that your
data are appropriate for the question you are asking or the theory you are
testing. In this assignment we rework data from an earlier study where
there is an apparent disjuncture between theory and data. The study in-
volves differencesin nations' income inequality.a

NerroNRr Drr,rEnrNcns IN INconr, INEquRrrrv

There are severalwell-established facts about the degree of income ine-

quality within nations. One is that income inequality within nations gen-
erally does not change precipitously. Income inequality might rise or fall,
but change tends to be glacial and may take severalyears to be noticed.
(Exceptions occur in the wake of radical economic transformation, as we
saw with the sharp rise in inequality in the 1990s following the fall of
communism in East Eurc'rpeand the former Soviet Union.) Second,differ-
encesin income inequality from nation to nation are, by contrast. quite
noticeable. Some nations, such as Brazil and South Africa, have stub-
bornly high levels of income inequality, whereas other nations such as
Sweden are characterizedby much lower ievels of inequality. Third, the
causesand consequencesof national differencesin income inequality are
hard to pin down. Economists, for example, have long debated rvhether
income inequality tends to have a positive effect, a negativeeffect, or no
effect crneconomicgrowth (Kaldor [1,9561argued that inequality boosts
growth, but later evidence suggeststhe opposite: Aghion, Caroli, and
Garcia-Peiialosa1,999;Benabou 1.996),and political scientistsand sociol-
ogists continue to argue about the nature of the association between
democracy and income inequality (Bollen and Jackman 19951'Muller

a I thank Shawn Dorius for assistancein designing this exercise.


. - : -. - :.-,rrci double- c hec k i n g
: ., : : 1\'av st 0 Cf O S S - Cl fe C k
-. : ...1:.'n in vour data set-
: ;heck for consistencY
: , :'.- -::naI in the measllre-
... :'.:st be sure that Your
:: : , : . :'.:,l r th e t heor y y ou are
:; : : , . .. - e ar liers t udYwhere
: : : : . ',:l j dat a' T he s t udYin -
_ :, - -:
- , - - : '- Jcg re et t f inc om e ine -
: - . - . , ::. $ i t hin nat ionsgen -
-: : - -.,'.irtt'might rise or fall,
.:: -: :i -tL ve ar st o be not ic e d .
:, :' -- - irJnsformation,as we
r t : ', -. ' . tol l or v ingt he f all o f
:: : .: .-i.iion')Second,differ-
- .- - .-.r.. l.) contrast. qutte
:: - ' -.:h -\frica' have stub-
.: . : : : : : -: r)th ernat ions s uc h a s
r : :.: : incquality' Third, the
: - : - - : i ::- :ncc im einequalit Ya re
: :. ' : . ieng debatedwhether
: : .:,.. -: negtttiveeffect, or no
- , , : . . .:j rh et inequalit yboos ts
: . : .:-: -\ghion' Car oli, an d
*: : s s oc i o l -
::::l scient is t and
-:: : :ll. ,.lssociation between
:: . , : : T:ckm an 199- 5;M ull e r
:-
The Third Rule . Ei
1995). One of the few associationsthat virtually everyoneagreesabout is
that income inequality tends to be lower in communist countries than in
capitalistcountries.Apparently communist societiesare successfulin re-
ducing economic inequality.
These facts provide the background for a reality check on rhe findings
reporred by StevenStack (1978) for a cross-sectionof thirty-two nations
circa 1960. The aim of his paper is to test "the Keynesiannotiofl that the
degree of direct government involvement in the economy should reduce
income inequality" (abstract). Stack's central finding is a correlation of
r=-.74 between level of income inequality in a nation and tlre nation)s
level of "direct government involvement" (DGI) in the economy. In other
words, Stack reports that more than one-half (r2=.55) of the variancein
income inequality from nation to nation is associatedwith variance in the
level of government involvement in the economy, where government in-
volvement in the economy is measuredby expendituresof the central gov-
ernment as a proportion of total gross national product. The startling size
of this associationsuggeststhe need for a reality check, since it is difficult
to find anything that accounts for more than one-fourth, much less more
than half, of the nation-to-nation variance in income inequaiity.
Note that what we are interested in here is national differencesin in-
come inequality among nations with market economies,not nations with
centrally planned economies,sitrceKeynesiantheory bearson how govem-
ment spending and manipulation of the money supply affects economic
performance in capitalist systems. Quoting Robert Jackman (1980, p.
1,32): "If it is possible not to overstate the ob'r'ious, the Keynestan
perspective addressesgovernment intervention in market economies, as
opposed to the central planning that is typical of the Soviet and East Euro-
pean economies."If this is a test of Keynesiantheorn then, the r=-.74 cor-
relation between DGI and income inequalit;' reported by Stack (1978,
table 1) ostensibly appliesto market economies.The text of Stack's article
reinforcesthat impression by emphasizing,for example, that the irnpact of
government spending on income inequality should be felt through "the
multiple reinvestment of monev obtained from higher lel'els of consump-
tion in job creativeventuresinthepriuate sector" (p. 883, italics added).
In fact, however,eight of the thirty-two countries used in the study had
centrally planned economies in 1960: Bulgaria, Czechoslovakia.East
Germany, Hungary, Poland, Romania, the Soviet Union, and Yugoslavia
(Stack 1,978, appendix). This is an instance, then, where the data do not
match the theory becausewe have casesin the data that do not belong
(J a ckman1980).
What we want to know is whether the results Stack reports were bi-
ased by including eight casesthat do not belong in a test of Keynesian
theory. Perform this thought experiment. Imagine there is little or no as-
86 . Chapter3

sociation between DGI and income inequality among market economies'

so in a scatterplot relating income inequality (Y-axis) to DGl (X-axis) the
regressionline would be horizontal, or nearly so. Now add the eight non-
'We
market economies. would expect now to seea negativeassociationbe-
tween DGl and inequality, since the nonmarket economies all fall in the
southeastquadrant of the scatterplot (high DGl and low inequality).
It is quite possible, then, that the strong negative association between
DGI and inequality would disappear,or nearly so, if we removed those
eight nonmarket countries in the southeastquadrant of the scatterplot.
Your assignment is to find out if removing the nonmarket countries
indeed removes the negative association between DGI and income ine-
quality. Table 3.1 gives the data you need to complete the assignment
(N=24 market economies).They are the data Stack used in his study.

1 (ron rvEnvoNl)
AssrcNrvrE'Nt

Use your favorite packageprogram (SPSS, SAS,Minitab, Stata,etc.)to do

the analysis required to answer these questions. Recall that variable
names are in boldface in the computer assignments this book.
in

Question 1. Vhat is the correlation between DGI and Gini for these
twenty-four market economies?[ANswrn: r=-.16.] Doesthat correlation
provide strong support for the argument that government intervention in
the economy is associatedwith lower levels of income inequality in capi-
talist countries? Defend your answer.

Question 2. For the eight centrally planned economies included by Stack

(1978), Gini for 1960 is estimatedto be 0.17 for Poland,0.20 for Czech-
oslovakia and East (iermany, 0.21 for Bulgaria, 0.23 for Hungary and
Romania, 0.26 for Yugoslavia,and 0.28 for the Soviet Union (Paukert
7973). The mean for these eight countries is 0.2225. On average,then, is
income inequality larger or smaller in these nonmarket economiesthan it
'What
is in the twenty-four market economies? is the difference in the av-
erageGinis for the two types of economies?

Question 3. Use your program to produce a scatterplot for the twenty-

four market economies,where the Y-axis is Gini and the X-axis is DG/.
If possible, use your computer program to add the regression line to the
scatterplor. Either way*with or without the regression line-print the
scatterplot. (There should be twenty-four dots, one for each country; and
the regressioniine should have a downward slope, sincethe associationof
DGI and Gini is slightly negative.)
 The Third Rule . 87

... . ; . . . .: t (-rngma rke t econ omie s ' Thnrn3.1

:, : -.:si-r to DGl (X-axis) the Data on IncomeInequalityand DGI for
:. - , \ort' add the eight non- Twenty-fourMarket Economies, circa 1960
i:: -r :rr'gativcassociationbe-
1960 7 L5) 9
lr.;l:.;. :vonQmiesall fall in the Gini^ DGlb
- - . .'-r.l lorv inequality).
i I r: i.iri\ e as5ociationbetween 0.42 0.170
Australia 0.30 0-180
,.:.... . :,,r.if rve removed those
:: --.-:,:lr of the scatterplot. Bolivia 0.53 0.76r
- Brazll 0.54 0.t37
. .. : . - :l :c n onm ar k et c ount ri e s
Burma 0.35 0.291
. : . . , . ; : :r DGI and inc om e in e - Denmark \t.t / 0.214
:: * r -,,mp l e t et he as s ignme n t Finland 0.46 0.301
:::.-- ::.r;k usedin his study. France 0.50 0.400
Greece 0.38 0.236
India 0.33 0.139
Iraq 0.60 0.325
Israel 0.30 0.3B 2
. > . : . \Lnrrab, S t at a,et c . ) t o d o Italy 0.40 0.342
Japan 0.39 0.345
: - : i: . : '.\. Rec ali t hat v ar ia b l e
Netherlands 0.42 0.370
. : - :: ': : ::> i n this book .
Nor*'ay 0.35 0.286
Peru 0.61. 0.131
:,. .: : - DGI and G ini f or t he s e Philippines 0.48 0.092
- = - - -.l Doesthat correlation South Africa 0.s8 0.311
-
: -- - :-l tment int er v ent ioni n Sweden 0.39 0.391
- - 'Ir( inequalir yin c a P i - United Kingdom 0.38 0.388
United States 0.34 0.210
Venezuela 0.42 0.272
'West
: j :- r- '::tresincluded bY Stack Germany 0.45 0.306
- : : P,,l.rnd,0.20 for Czech-
'The Gini is an index of inequalit.v that ranges
: - -.. .... r.l-l for Hungary and from zero for complete equality to 1.0 for maximum
. r . ' : :ttri e t Union ( P auk e rt inequality. D.tta source: Paukert (1973).
bDirect government involvement in the economy,
. i - -l ;. On av er age,t hen , i s
measured as proportion of total GNP. Data source:
::: - r r---iiketeconomiesthan it Russett et al. (1964\.
-"r,- ,- : :r.' differencein the av-
a-l

Now placea "C" on the Y-axis where Gini=O.2225 (the averagedegree

--: ., :,-,-::-rplotfot the twentY-
. .. C, :,ti .i n dth e X - ax is is DG I.
: .::: :r- regressionline to the
-r i -: r:i i i Ssi on line- P r int t h e
: : :.. :-.-lor each country; and
:: : . : :. \i nce t he as s oc iat io nof
of inequality for the eight nonmarket or communist countries). Similarly,
place an "M" at the proper location on the Y-axis to indicate the average
Gini for the twenty-four market countries.
(a) On the basis of the relative locations of C and M on the scatterplot,
and on what you see in the rest of the scatterplot, approximately hou'
large would you guessDGI needsto be for the eight nonmarket countries
to reproduce the r=- .74 correlationobtained by Stack (1978)? (Don't
BB . Chapter3

try to do any calculations-iust make a reasonableguess,based on rvhat

you seein the scatterplot.)
(b) To determine the accuracy of your guess, add the eight nonmarket
countries to the twenty-four market countries above, and calculate the
correlation between DGI and income inequality for the thirty-two coun-
tries together. For the eight nonmarket countries, insert the Gini figures
given in part 2 above, and the DGI figures that you guessed.How close
did you come to reproducing the r=- .74 correlation when you insert the
new data?Stop, if your correlationwas betr,veen r=-.72 and -.76; other-
wise, insert another value for DGI {or all eight nonmarket countries and
keep trying until you produce a correlationbetween-.72 and -.75.
(c) Do your findings cast doubt on the conclusions of the original 1978
study? Explain.

AsstcNuENl2 (ron sIUDENTSwITH A sTRoNGER

BAcKGROUND
I N S T A T I S T' ICS)

'$7hile
it is ciear that income inequality was (and is) higher in Latin Amer-
ican countries such as Bolivia, BrazII, and Peru than in Scandinaviancoun-
tries such as Norway and Sweden, estimates of incorne inequality for
1,960arc far from perfect.To gaugethe consequences of this unreliability
for correlations involving Gini, usethe random number generatorin your
computer program to create a standard normal deviate R (that is, a nor-
mally distributed random variable that has a mean of zero and a standard
deviation of 1.0). For tbe tu,enty-four market econornies,urseR to simu-
late the effect of measurement error on the Gini by creating three new
variables,as follows:
c Newginijl = Gini(l + .01R)
. Neugini2S = Gini(l+.25R)
o NeuginiS0 = Gini(l+.50R)

Note that each of the Netugini variables adds random error to Gini by
multiplying Giniby 1+ftR, where A is a constant and R is a random vari-
able with a mean o{ zero and a standald deviation of 1.0. You are adding
error randomly to each of the twenty-four countries individually. Con-
sider Argentina, for example. You are multiplying the stated Gini value
for Argentina (0.42) by (1+AR), where R is a normally distrihuted ran-
donr variable rvhose averagevalue is zero. Hence you expect the Netugini
value for Argentina to be larger than Gini half the time and smalier than
Gini half the time. The same logic holds for the values of Newgini versus
Gini for the remaining tu'enty-three countries.

: . - - , -.; :ues s .bas edon r v h a t
-: :he eight nonmarket
- : : . 1 ^ '.i . and c alc ular et h e
. : :fre t hir t Y - t wo c ou n -
.,.,,
,- : ::. :.rfrl the Gini figureS
:i t - ti -: guessed,Florv close
t-: :-... hen vou insert the
-
- .76; other-
r.. :: - .- ) ar-rd
-
: : . .-.i:t.rrketcountriesand
I a/
: : '. --' - anCl-./O.
-, -i :'. o i t he or iginal 197 8
:; '. : . : 1f K G RO U N I )
:. : - . :isher in Latin Amer-
'' ::' re. t ndinav ianc ou n -
- -r: - : :lJolTletnequalitY for
- -: - .::-..:i of this unreliabilitY
- - - - " .':l 'crq ener at or in 1' o u r
-....ri c R r t hat is , a no r-
:- : '-: , .-. ,i ze r o and a s t and a rd
, , ':. - ' : - ' ': , rit ie s , U S e R t O S i m U-
' , : | ;ttii i. ' c r e a t i n g t h r e e n e lv
jom error to Gini by
r.r:-.
. : .r'rJR is a r andomv a ri -
,,f 1 . 0. Y ou ar e add i n g
. :'::re s i ndiv idually .Co n -
::: rhe stated Glai value
... :mallv distributed ran-
:.
-: !)ir expect the Newgini
: :.-.:lime and smallerthan
- . '. '. : ..:l ueso f Ner ugini v ers u s
TheThi rd R u lv . ):
BecauseR is distributed symmetricallyaround zerct,Newgini should
be greater than Gini for about half the countries, and snraller than Gini
for about half the countries. For all twenty-four courltries, then, we ex-
pect the mean for Newgini to be approximately the same as rhe mean for
Gini,thotgh the two meanscould divergequite abit due tct cbance,The
more measurement error, the greater the possibiliry that the tr,vo means
will differ substantially.
The size of A determines the degree of me;.rsurementcrror. Thus
Neugirri}l adds a trivial amount of rneasurementerror to GinirNewgini2S
adds n-roderateto large measurementerror, andNewginiSO adclsquite large
rneasurementerror.
With theseobservationsin mind, complete the following exercises,and
answer the following quesrions,involving correlations(not just means):
Question 4. (a) Correlate Netugini0l with DGI for the rwenty-four mar-
ket econornies.Is the correlation larger or smaller than it was using Ginl?
(b) Repeatnineteenmore times, using a new R for each trial. Calculate
the mean of the twenty correlations o{ Newgini}T with DGL Is the mean
of the Nerugini}l-DGl correlations similar to the original correlation of
GiniwithDGI?
Question 5. Norv introduce greater measurementerror on the income rn-
equaiity measure by using Netugini2S. Repeat the procedures you used
for Neugini0l to generaretwenty correlations of Newgini2S with DGI.
(a) Is the mean of the Newgini2S-DGI correlations similar to the original
correlation of Gini with DGl?
(b) What about the uariance of the correlations: Do the correlations with
DGI vary more with greater measurementerror (Netugini2S) or lessmea-
surement error (Neargini} 7) ?
Question 6. Generatetwenty new correlations,this time using NewginiS\.
(a) Is the mean of the NetuginiS)-DGI correlations similar to the original
correlation of Gini with DGl?
(b) What about the uariance of the correlarions: Do the correiations with
DGI vary more using NewginiS} than they did using Newgini2S and
Newgini0T?
Question 7. (a) Random measurementerror tends to reduce correlations
in the bivariate case.we expect that, the greater the error, the greater the
bias toward zero. Is that what you found in your simulations?
(b) we also expect that random measurementerror wiil increasethe va',
ance in our estimates.Is that what you found in your simulations?
CHA P T E R 4

The Fourth Rule

REPLICATEWHERE POSSIBLE

Rule 4 is rhe replication rule. The replication rule is a natural follow-up to

rule 3, "Build reality checks into your research." Rule 3 advisesyou to
look for ways to cross-checkyour results both internally-using other 1n-
formation in your data set-and externally-using different methods and
data sets.In multiple-method research,as described in the previous chap-
ter, your aim is to seeif different methods and different sorts of data lead
to the sameconclusions.
Rule 4 advises replication-the identical analysis (same measures,
models, and estimation methods) of paraliel data sets (different samples
of the same population)-to see if you obtain similar results.In replica-
tion studies your aim is to seeif identical analysesyield similar results for
different samples of people. Hence rule 4 can be thought of as a special
type of reality check, one that involves the parallel analysis of multiple
data sets.
Note the qualifier "where possible" in rule 4. Often it is difficult to
find one data set that meets your needs,much less several.For that reason
it is still relatively rare in the social sciencesto find single studies that re-
port analysesof multiple data sets.Hopefully there will be more examples
to cite in subsequenteditions of this book.
Hence the chapter is forward-looking, focusing more on what can be
done than on what is being done. The issues discussed should become
more relevant in the future as the stock of good data sets increasesin the
social sciences.There is reason for optimism. Just a few decades ago
cross-sectionalanalysesprevailed in political scienceand sociology, and
time-seriesanalysesof small sampleswere standard fare in economics,
even though everyoneknew that large longitudinal data setsgenerally are
better suited for answering the sorts of questionsthat we ask as social sci-
entists. Today cross-sectionaland small-N time-series studies have been
largely replaced by studies using larger samples and better longitudinal
designs,such as the panel and repeatedcross-sectiondesignsdescribedin
chapter 6. In the future we might expect the analysis of parallel data sets
to become more routine as large representativedata setscumulate.

- -. : ' -. :lstural follow-uP to
.. : : - , - R,rl e 3 adv is esY ou t o
' . i - . . :: --11l r' - us ingot her in -
- , - : -.:i crent m et hodsand
::*- - -.: :r th e pr ev iousc haP -
i - * rr:r::lt Sortsof data lead
- . (Same measUrest
-: .-, ., )iS
, : - : . '.:..:-l srd if f er ents am pl e s
: j - - -.:-'-rre s ult s .I n r eP lic a -
:..t -:- ::iJ sin-rilarresultsfor
- : :..,uql- r tof as a s pec i a l
-: -
-: - j:.:. . ,lnalvsisof multiPle
' : - . - ,r:ten it is difficult to
- * - : -- -:' rr.1 l.F or t hat r eas o n
:\ : - : .r,rqlestudiesthat re-
, -: -: ' :l l be m or e ex am P l e s
' . , -, - c an b e
- :]l (rr eon what
- i - . .. - .,..ri r'd s hould bec ome
' : - -....r -*ersincreasesin the
- -- :.: r ferv decadesago
- - r - - :l -.i a nd s oc iologYan , d
r: -l - *-,:': fare in economtcs,
:: - : , j .'.tas et sgener allya re
:.:: - . ,. -:.:rr'eask as social sci-
'i I r--: .:':i l S St udieshav e be e n
- - - :: - ...::;het t er longit udin a l
- - - --: , :: ,r Je s ignsdes c r ibedi n
:- : , , . . ..rr o f P ar alleldat a s e ts
::: : -. 1-. i >et S C U m U l a t e .
The FourthRule . 9I
This chapter also servesto reinforce one of the major themes of the
book-that progress in social researchwill come mainly from better rc-
searchdesign,not from statisticalwizardry (seerule 7),THe parallel analy-
sis of multiple data sets opens new possibilities, and we should be
thinking now about how best to exploit those possibilities in social re-
search.
rN SocIer RBsnaRcn
Souncns oF UNCERTATNTy
It is easy to be lulled into a false senseof security about uncertainty rn
'We
social research. are very consciousof the possibility of sampling error
and generally devote a great deal of attention to it in our research.Stan-
dard statistical packages all report standard errors, and we dutifully
gather that information and report it in tables in our researchreports.
The false sense of security arises when we forget about the other
sourcesof uncertainty and assumethat by ruling out sampling error we
have ruled out all uncertainty. In many instances sampling error is not
even the most important source of uncertainty in social research.Indeed,
for very large data sets-which are becoming more common in the social
sciences-sampling error becomesrelatively less in'rportant as a source of
uncertainty. Nonsampling error, by contrast, generally is not relieved by
larger samples.
I want to make two main points in this chapter. The first is that we
should take a holistic approach to uncertainty in social research.Robert
Groves et al. (2004) describe the move toward a total suruey error para-
digm in survey methodology. The virtues of such a paradigm are apparent
for all types of observational studies in the social sciences,not just survey
analyses.We should aim to developand refine atotal uncertainty perspec-
tive that is based on all sources of error, not just sampling error. To be
sure, sampling error receivesa lot of attention in social research.\(e give
'We
it a lot of attention in part becausewe can: have methods for cstimat-
ing the uncertainty arising from the use of a probability sample. Other
sourcesof uncertainty are harder to quantify, but they are often no less
important than sampling error. We need to find better ways to gauge the
effects of these other sourcesof uncertainty as well.
The second big point in this chapter is that replication prouides a
meansfor gauging toldl uncertainty.This is the triangularionpoint again.
except that here the triangulation involves the same methods applied to
different data sets.The juxtaposition of results from identical analysesof
parallel data setsin a single study should provide more realistic appraisals
of the uncertainty in our conclusions than do significancetests alone. If
92 . Chapter4

so, then comparison of results across data sets might become a star-rdard
feature of articlesin social sciencejournals in the future, iust as significance
tests are today,
It is useful to begin with an inventory of the sourcesof uncertainty in
observationalstudiesin the social sciences.Clifford Clogg and Aref Daiani
(7997) identify six separatesourcesof uncertainty in the modeling of social
statistics."Our main point," they write (pp. 12-13' italicsadded),"is that
we must seriouslyconsider the uncertainty in inferencescreated along the
way by the data collection processesso common in social research." Their
thesis,and mine, is that uncertainty "crops up all along the way" (p' 15) in
data collection due to numefous problems that are encountered,and deci-
sions that must be made,in the processof collecting data. Thus evenif there
were no samplingerror, the use of the samemodels and estimationmethods
acrossdifferent data setsmost likely would deliver somewhat different re-
sults. Any given data set, then, is subjectto uncertainty about how much (if
at all) the conclusionswere affected by the data collection methods'
'$(here possible,then, it is imporrrnt to compare results across data
sets.Of coufse,we hope to find consistencyin our results,since similar
rWhererve
results across data setsincreasesconfidencein our conclusions'
find inconsistent results, we must temper our conclusions accordingly, or
find reasonsfor the divergence.In some instanceswe might be successful
in pinpointing the major sources,and in other instancesw-emight not be
able to do so.
\(hat are the sourcesof uncertainty associatedwrth data coiiection?1
For convenienceI focus on survey data, but the generalprinciples apply to
other types of social sciencedata as well. My accounting schemerelies
heavily on discussionsin Kish (1987), Clogg and Dajani (1991), and
.We want to focus on sources of uncertainty other
Groves et al. (2004).
than sampling error since methods for estimating and evaluating sampling
error are alreadywell known. In typical studieswe know much lessabout
the narure of the other sourcesof uncertainty arising from the data col-
lection process.
Let me make the point as cleariy as I can. Imagine we have two iarge
probability samples(say of 100,000 caseseach) of the same population.
The samplesare collected by two first-rate survey firms. l)ue to the size of
each sample,samplingerror should be minimal in either case.Yer rhe tyr,o
data sets could yield different results even if they used exactly the same
questionnaire.\ilihat might account for the differences?
I The p;oblen of uncertaintv due to data collectior.ris inclependentof other major prob-
lens q.e face in social research,such as the problem of omittecl variables bias {discussedin
chapter 5). So erlrr-free data n'ould not solve all the problems n'e face in social research.
Even n,ith error-free data, for example, lve rvould still need to find rvays to compare Jike
u ' i t h l i k e n ' h e n m a kin g ca u sa lin fe r e n ce s( r u le .5 ) .
 The FourthRulc . 93

Ouetuiew: From Population to Sample and Bach. to Population

Consider two simultaneoussurveysof the same population. An example

would be a Harris poll and a Gallup poll just before an election in the
United States"Even if both polls used the same questions and procedures,
they might yield somewhat different percentages.We want to locate the
sourcesof that divergence.By thinking through the processby rvhich data
are obtained in this simpleexample,we can better understandthe sourccs
of uncertainty in sociai researchmore generally.
The Harris and Gallup results differ, obviously, becausethey collect
different samples.(For norv we assumeno measurementerror. Measure-
ment error is addressedlater in the chapter.)So iet's look more closelyat
the processby rnhich a sample is extracted from a population. The ex-
traction of a samplecan be thought of as a successive processof exclu-
sion, or winnowing. The population is rvinnowed in three steps(6g, a.1).
As figure 4.1 indicates,error can be introduced in each step.I will refer to
the sum total of this error as exclusion error, sinceit results from the ex-
clusion of individuals as you mor-efrom the target population down to
the collectedsample.
The target population is the universe\-ou \\-ant to srud.', :uch as all
U.S. adults. The frame population is "the set of target population mem-
bers that has a cha,nceto be selectedinto the surve.vsample" (Groves et al.
2 004,p.45) . I n t he c as eo f U .S. a d u l ts ,fo r e x a m p l e .a sampl ecol l ected
by telephone excludes some members of the target population because
they don't have a telephone.Likewise, a survey conducted only in English
excludesSpanishspeakers;a survey of citizensresiding within the coun-
try excludesmilitary personnel and others living overseas;and so on.
The next step, the selection of the sample from the frame popula-
tion, is by far the most dramatic of the winnowing processes.In the
case of the General Social Survey and similar surveysof the U.S. adult
population, for example, millions of casesare reduced to a sample that
typically numbers in the thousands. Because most of the exclusion
takes place in this step, you might suppose that most of the exclusion
error also is introduced in this step. As noted in chapter 1, horvever,
how individlrals are selected is more critical than how mttny indlidu-
'lfith
ais are selected. the use of probaL'ility sampling methods, exclu-
sion error actually can be less problematic for this step than for the
other two.
Finally, there is the samplethat is actually coliected.Becausesome ln-
dividuals in the selectedsampie may not respond-perhaps they cannot
be located, or they decline to participate if they are located-the collected
samplediffers from the selectedsample.(Eventhe U.S. CensusBureau is
unableto elicit 100 percentcooperation.)
94 . Chapter4

Target Population

[coverageerror]
i
Frame Population

Isamplingerror]
+
SelectedSample
I
[nonresponseerror]
V
CollectedSampie

Figure4.1. Sourcesof
ExclusionError in Data
Collection

There is the possibility of error at each step, so each step adds uncer-
tainty about your results. 'We examine this uncertainty for each of the
stepsin turn. It is convenient to consider the steps in reverseorder, begin-
ning with selectedsample -->collected sample, since as analysts our infer-
encesmove in opposition to the direction of the arrows. In other words,
as analysts,we move in reverseorder to the winnowing process,using ob-
servations based on the collected sample to draw conclusions about the
target population.

S E LF ,C T E D S AM PLE .+ COLLECTED SAN{ PLE: ERR OR D U E T O U N IT N ON R ESPON SE

Typically the sample you obtain differs from the sample you selectedbe-
cause some individuals who are selectedare never contacted (they might
be difficult to locate, for example), and some of those contacted decline to
participate. The failure of some selectedunits to respond (for whatever
reason)is called unit nonresponse(Groveset aL.2004, p. 45) or total non-
response(Kish 1987, fig. 2.1,.1.).Itis important to distinguishunit nonre-
sponse from item nonresponse-the failure of participants to answer
particular questions-a subject addressedin the discussionof missing val-
ues in chapter 3. Although both involve missing data, item nonresponseis
often addressedusing data imputation methods, as describedin chapter 3.
In the caseof unit nonresponse,however,there is no information about the

..:;h stePadds uncer-
:::,.rinlv for each of the
begin-
order, heoin-
: .: reterse^.1o.
:. ,r. analYstsour infer-
: --. '\\ 5. I n ot her wor d s ,
'.,,:rg process,using ob-
the
-,,nclusionsabout
.
. \II N ON R E S P ON S E
:-i:rple 1'ou selectedbe-
:r -(ntacted (theYmight
ii ,i;' iootacted decline to
::rpond (for whatever
+. p. 45) or total non-
jrstinguishunit nonre-
: --: : a.lrilclpants to answer
- - r -. : :..;uss ion of m is s ingv a l -
"- :.i i .t. i tem nonr es P on sies
-.
- Indeed, it is fair to say that error linked to sample size (sampling error)
-::: ::. ,-.. describedin chaPter3'
tnformationabout the
. :-::: . :l,l
The FourthRule . ej
person (with the possibleexception of some basic information such as the
person'sgender and address)from which to impute values.
The most obvious problem with unit nonresponseis reduction in the
size of the sample.This problem is easily remedied,though, by selecrurg
someoneelse.In the caseof limited funds, replacementmay not be possi
ble, so the final sample is smalier. If that is all there is to it-that is, if unit
nonresponse results merely in a smaller sample-then the problcm is
fairly minor for studieswith good responserates:At worst, unit n{)nre-
sponseinflates standard errors somewhat due to a smaller N.
But that is not all there is to it. Unless unit nonresponse is random*
which is unlikely-the collected sample will fail to constitute a random
sample of the selectedsample.As a result, unit nonresponsewill intro-
duce an unknown amount of bias in the collectedsample.This bias can
affect means,variances,and associationsamong variables(box 4.1).
The most serious problem with unit nonresponse,then, is that it tends
to change the composition, and not just the size,of the sample. Change in
the samplecomposition due to unit nonresponseresultsin what is called
nonresponseerror (Groves et al. 2004\. Becausenonresponseerror is the
result of bias in the sample-the collected sample is not a random subset
of the selectedsample-the solution is not a larger sample. Unlike sam-
pling error, nonresponseerror does not diminish with sample size.
The solution lies instead in reducing the rate of nonresponse.The
lower the response rate, the greater the potential for nonresponse error,
whatever the sample size.That is rvhy survey researchfirms try so hard to
raise responserates, and why prudent researcherstry to gauge the repre-
sentativenessof samples by comparing sample data with census data on
basic demographic characteristics(age, gender,levei of education, etc.) of
the target population.
In sum: Unit nonresponsecontributes to uncertainty in the results of
social research.Becausepeople can decline to participate, the problem is
inherent in the collection of data on human subjects. Because nonre-
sponseerror existsindependentof samplingerror, it is neither reflectedin
significancetests nor remedied by larger samples.
FRAM E POPU LAT ION - + SELEC T ED SAN {PLE: SAM PLIN G ER R OR
As noted earlier, the selectionof the sample would appear to provide the
greatestopportunity for exclusion error, since most of the membersof the
frame population are excluded from the sample selected.So long as exclu-
sions are determinedprobabilistically, however,statisticalsignificancetests
apply. ln this case,then, we at least have direct methods for estimatingthe
amount of uncertainty introduced at this stagein the exclusion process.
tends to be lessformidable than error linked to responserate (nonresponse
96 . Chapter4

error). As a practical matter, sampling error becomes less worrisome in resp0nserateSCOritr:: --:
the social sciencesas we accumulatelarger data sets.Nonresponse error is ticipate that s;rrnpi;:^:,
a different matter. Responseratesappeafto be decliningin the United States data ser s,hr r r , r li, ', . -
and elsewhere(Groveset aL.2004, chap. 6). Large samplesor not, nonre- mofe s ev erer c l u€ tt, ,-.

sponseerror is likely to become more problematic if, as seemspiausible, for s<.rcial

research.

B ox 4. 1 c Lineclr /-sg' _.-

When Does Unit Nonresponse Produce Bias? asedest im ; : : .
causailr - r ei: : .
Unit nonresponsecan affect estimatesof descriptive statistics,such is related or-..'.:
as means, as well as analytic statistics, such as regression coeffi- asedest im al. .
cients. The conditions are different for means and regressioncoeffi- To assisr, . , .
cients, so I consider them separately. alize holr- .1 :.-:::
a scatterplc)i
e Means. The mean of some variable X is understated when To rnake rhl. ,
those with above-averageX values are underrepresentedand pinessis the I -.:
the mean of X is overstatedwhen those with below-average come on hap:
valuesof X areunderrepresented. So estimatesof the mean of a SSU m eth e .. .-
X are biasedwhen unit nonresponseis relatedto X. If, for ex- population .
ample, the rich are lesslikelv to participate ir-ra study, then in- nonresponsa
come estimates are biased in a downward direction; on the bias our €strl-.,-,
other hand, if the poor are less likelv to participate, then in- To ansrir:. -
come estimatesare upwardly biased. plot. Suppos. : ,
If unit nonresponseis random, then we expect respondents come and oi : ,: -
and nonrespondents to be the same, on average)for all the scatterpiot u: : :
variables.In that h"ppy casewe do not expect the means to be t om , so r he l. - : . .
biased by unit nonresponsefor any of the variables.Typically, nonresp0lse J :
though, nonresponse is not random. In the case where unit efficier-rtrn rlt;:
-
nonresponse is not random, the estimatesof means and per- Now im acr : :
centagescould be either unbiased or biased for a particular t hinning is n, , -
variable, depending on whether those who participate' and if the rich ar'. .
those who do not, differ for the variable in question. Suppose, thinning of ri. ,
for example, that married people are more likely to participate r ight ) . Does : : - : :
in a study, but that political party identification is unrelated to The ansq-erd.:.
response rate. In that study, then, estimates of percentage and unhapp,, ;.
married will be biased by unit nonresponse, but estimatesof words, the a:--.''.
political party identification will not be biased by unit nonre- causallvrelar.; :
sponse. nessdoes noi : . -
-
(cctntinuedon next pctge)
 The Fourth Rule . 97

t r- Ic s : \ \ or r is om e l n responserates continue to decline. In the foreseeablefuture, then, we an-

:r \Llnf eSPOnSe efror iS ticipate that sarnpling error wili become less probiematic due ro larger
: :-J ln the United states data sets, but also that the problem of nonresponse error will become
. ..,. .ples or not . nonr e - more severe,due to declining responserates. That is not a good rradeoff
, :. rs seemsPlausible, for social research.

(continuedfrom preuiouspage)

Linear regression coefficients. Unit nonresponse leads to bi-

ased estimatesof regressioncoefficients when nonresponse is
causally related to the dependent variable. Nonresponse that
is related only to independent variables does not result in bi-
- :.::i\ . e statisticsrsu ch
- ....regression
coeffi- ased estimatesof regressioncoefficients.
coeffi-
- r:J regression To assistour intuition on thesepoints, it is helpful to visu-
alize how a scatterplotis "thinned" by nonresponse.Imagine
a scatterplot of the X-Y relationship for a target population.
To make this concrete, supposeincome is the X-axis and hap-
- -- understatedwhen
and piness is the Y-axis. 'Wewant to estimate Fy;,, the effect of ir-r-
-.::jerrepresented
.; ith belou'-average come on happiness.To isolate the effect of unit nonresponse,
". assumethe selectedsample is a random sample of the target
:.:'.ltes of the mean of
population (so any bias in our estimateof B* is due to unit
:. .:ied to X. If, for ex-
nonresponse).Under what conditions will unit nonresponse
:.,:( tn a s t ud1.t hen in-
.:-; Ji rec t ion; on t hc bias our estimate of 8""?
- :.l rri c iP at e't hen in- To answer,cclnsiderhow unit nonresponsethins the scatter-
plot. Supposefirst that unit nonresponseis independentof in-
'.'.: e\PeCtrespondents come and of happiness.In that caseunit nonresponsethins the
. r a\.erage,for all the scatterplot uniformly from left to right and from top to bot-
r :nlcCt the meansto be tom, so the regressionline is unaffected. In other words, unit
' tl., nonresponsedoes not affect the estimate of the regressionco
-. . : r. rrla b l e S . I Y P l C a l . / ,
efficient in this case.
-:: the casewhere unlt
:--::i> 0f means and Per- Now imagine that nonresponsels related to income, so the
: rr-'.sedfor a Particular thinning is no longer uniform from left to right (for example,
:: -.rho participate, and if the rich are less likely to parricipare in surveys, then the
: .1 g ues t ion.S uppos e . thinning of the scatterplot increasesas you move from left to
:: ,re likelv to ParticiPate right). Does that affect our estimate of the regression slope?
r:::;.rrion is unrelatedto The answer dependson whether, at a given income level,happy
and unhappy people are equally likely to participate. In other
---:.r'rJtesof Percentage
::,rnse. but estimatesof words, the answer depends on whether unit nonresponse is
r: 'r.i.tsedby unit nonre- causally related to happiness,the dependentvariable. If happi-
ness does not causally affect response-so huppy people and

on next Pdge)
cr,tntinued. (continued an next page)
98 . Chapter 4

(continuedfrom PreuiuusPdg()

unhappy people are equally likely to respond at each income

level-theo the greater thinning of the right side of the scatter-
plot does not affect the slope: Nonrespondents (the missing
poi.rtt in the scatterplot) are equally likely to have come from-
o. below the regression line. Although our estimate of
"borr.
the overall mean of Y is biased, at euery income leuel our estr'
mate of the mean of Y is unbiased.
Suppose,on the other hand, that happinessis causallyrelated
to unit nonresp onse-happiness independently affects participa-
tion. Perhapshappy people are more cooperativeregardlessof
their income,for example.If so, we will tend to overestimatethe
mean of Y (happiness)at euery income leuel. Moreover, unless
to X; that
Fyx=o, overestimationof the Y-mean will be related
ir,tt. Ui", will be more severe at higher (or lower) income ievels.
Flence the slope of the regression line is affected'

The sameprinciples hold for bias due to coverageerror' Coverage

error leadsto biasedestimaresof the mean of a variableX when non-
coverageis relatedto X, and noncoverageleadsto biasedestimatesof
norrr.rin regressioncoefficientswhen noncoverageis causally related
rc the deqendent vatiable.

TARGET POPULAT'ION -+ FRAME POPULATION: COVERAGE ERROR

The frame population is only as good as the list from which it is derived.If
we used u pno". book to selectour sample, for example, we would miss
those who do not own phones as well as owners whose numbers afe not
listecl.This would not pose a problem if our researchwas on behalf of the
phone company, and they were interested only in the characteristicsof
iustomers with-listed numbers. As social researchers,though, we typically
are interestedin a more generalpopulation, such as all adults in some re-
gion, not just peopie with listed phone numbers.
Exclusion error due to incomplete sampling frames is called couerage
errol (Gtoves er al. 2004). Lists are never complete, and even fairly com-
plete lists are soon out of date, particularly in geographicallymobile so-
cieties. Sometimes exclusions are dictated by the nature of the
population. Consider,for example,the U.S. adult population (age eight-
e..r older). At first blush U.S. adults might appearto be an easypop-
"r-rd
ulation to demarcateand sample. But some adult U.S. citizens afe ln
relativelyinaccessibleplaces,such as in prison, or serving in the military

:.1 iront PreuiousPdge)
:ri rflrl at each income
of the scatter-
- .: srde
.- :r.jcnts (the missing
:.. :rr l-iale come from
of
. -r orlr e\tilni.lte
: -' . ' iI('/t'r'g / o ur esti-
)- - ;.:. :.; is c;utsallYrelated
. ,:: .,: '- -::l : r' trffectsParticiPa-
::ritive regardlessof
:"J IO O\'€testimate the
. : .'.. -\loreover'unless
:'r relatedto X; that
: -t,\\er)incomelevels.
::::ted.
. ::-ige error. coverage
'..r:i.rbleX when non-
. : , niasedestimatesof
:,.:r is causallYrelated
: ER R O R
:: :l n"hich it is derived. If
:\.1mple' we would mrss
. s hose numbers are not
-..;h u-ason behalf of the
:ir the characteristicsof
-:rs. though' we tYPicallY
-. ;. all adults in some re-
:rrnes is called couerage
:-c-te.and evenfairlY com-
:.'osraPhicallY mobile so-
-
hrl the nature of the
. - : ,:;.ilt PoPulation(age eight-
: : - : - . l PPe art o be an eas yP o P-
.: , - . .rJul t U . S . c it iz ensare i n
: :: . i .-. o r ser v ingin t he m il i ta rY
The FourthRule . 99
overseas.Some do not speak English. Some are homelessand have no
postal address.Hence obtaining a perfectly representativesample of the
U.S. adult population is more complicatedthan it might at first appcar.
Survey firms have limited budgets. To get the rnost for the dollar, com-
promises must be made. The frame population for the U.S. General
Social Survey, for example, consists of the noninstitutionalized L,r-rglish-
speakingadult population. (Spanishinterviews were added in 2006.)
In some instances,then, the exclusion is purposive, arising from bud-
getary limits, as in the caseof the absenceof institutionalized adults in the
GSSframe population. In other instancesexclusions occur becauscof dif-
ficulty in identifying the target population very precisely. Consider polls
before elections,for erample. The target population here is voters, since it
is voters who will determine the outcome of the election. But voting has
not yet taken place, so it is impossible to identify the members of the tar-
get population with complete precision. For our frame population, then,
we want to use those who are most likely to vote. How do we determine
that? One possibility is to use voter registration lists. But even if we had
accessto up-to-date lists, the lists are rarely complete, and some registered
voters don't vote while others who aren't registereddo.
In short, each of the steps from target population to frame population
to selectedsampleto collected sampleexcludespeople,and erclusion adds
uncertainty. Some individuals are excluded becauseof incomplete lists for
the frame population (coverageerror), some are excluded because they
weren't selectedfor the sample,and some exclude themselves(unit nonre-
sponse).
\fhile all exclusions add uncertainty, some exclusions are more prob-
lematic than others. The key is whether the consequencesof the exclusron
can be modeled, as in the caseof random exclusion or some other kind of
exclusion that relies on the laws of probability. Probabilistic exclusion
results in uncertainty, but a type of uncertainty that can be estimated-
that's what significancetests are all about. We could say, then, that prob-
abilistic exclusion results in "manageableuncertainty" in the sensethat
we can at least quantify our level of uncertainty. Coverageerrors and unit
nonresponse errors, by contrast, are likely to be nonprobabilistic and
thus more elusive.With nonprobabilistic exclusion it is more difficult to
bound our uncertainty: Sfe are lesscertain about the level of uncertainty.
Becauseconventional standard errors do not reflect nonprobabilistic
exclusion error, conventional confidence intervals understate the uncer-
tainty in our inferences(Clogg and Dajani 1,991),sometimesdramatically
so. From a total error perspecrive,it is important to go beyond sampling
error to try to gauge the level of uncertainty. That is why replication is
valuable. It is perhaps the best tool we have for providing a sober assess-
ment of the total level of uncertainty in our inferences.
100 . Chapter4

short-sightedio {ri.:.f::
Measwrement Ertor as ct Source of Uncertainty tempt s t O Calt ur r l
In addition to exclusion efror, there are other sourcesof uncertainty that errof.
'Within
can arise in the processof data collection. surveysthemselves,the I t is t inr c nu\ \ lL, : . . .
greatest source tf uncertainty involves question wording' An apparently
Lrno..n, change in wording sometimes prompts a notable change tn re-
sponses,so survey ,.r."r.h.rs are loathe to modernize the wording of a Ilr,uslRartc-lN:Tu o \.l"
'lrhen wording is
question that has a long tradition in prior surveys. new
To ilir r st r at et hc sr r r . -
introduced (for erample, changing the word "blacks" to "African Amer-
cussecl.I dcseril'c .; . -
icans',) the time-seriesis usually spliced by using two forms of the survey
volves the pressrnL.
in the transition survey,with respondentsrandomly assignedto one of the declinrng.[r rurns , .:: :
rwo forms. In that way it is possible to determine if the wording change sure d.The decline . -
itself has affectedresponses. ci ti z enswho ar e p, , , :
The concern about precisewording is fueled by a number of famous ex- ci on dar a r han ir i. . . ,-
amples in the history of tutu.yt. ln 1'992,for example, a stir was created from householdsur , - . .
*hen a Roper poll commissionedby the American Jewish committee was In t his case,t h. 'n. . : - .
said to haue fouttd that22 percent of Americans thought the 1939-1945 i cs seekst o under r l. , : . _
Holocaust neverhappened.Here is the question: "Does it seempossibleor 2003; Tr iplet t lqq- . :
does it ,eem imposslble to you that the Nazi extermination of the Jews changein world po\ .---,
never happenedi" A subsequentpoll, using clearer language, found that sual obser vat ionr crr . , ,
not
fewer than one American in 200 said the Holocaust "definitely" did abjectconditions.Tl:; :
happen, and about one in 50 said it "probably" did not happen-
off mar er iallyt h. r nr - - -
pr.r..rring a far different picture from the earlier 22 percent figure' coun t r iesis a ver eJ . . .
'
Anothir potential source of uncertainty involves the general content are those r hat havc . : -
and ordering of questions in the survey. For example, a question about cons um e.Poor eor r : : : .
tolerance to*"rd atheistsmight elicit somewhat different responsesif em- val u ablecom m odir ir . .
bedded in a general survey on tolerance than in a survey on religious be- How do we lr ( . r : . : -
liefs. Moreover, respondents' answers to a question might be influenced pl i fy. ler 'ssuppo\ c \ , , : -
by the immediately preceding quesrions. In the 1972 and 1985 General goods and sen. ices. l: : -.
Social surveys, for example, a question about general happinesswas pre- w ei ghr dif f er enrgo, , - . .
ceded by a similar question about the respondent'smarital happiness,and estimatethe total tj/t.;1:
Smith (1990) found that this ordering may have influenced responsesto ti on in a socier r '.O : , '
the general happinessquesrion-a survey-context effect. In some in- retail side, eir her I . . . -
stanJes,then, two ,,1ru.yr with the same question may eli"it somewhat have pr oduced. or h. . .
different results becauseotber questionsin the survey are different. have sold. Alt er nau. . . .
Responsesfor some questions might be particularly sensitiveto world
ev enr sat t h e ti m e o f th e s u rv e y ,re s u l ti n gi n anothersourceof uncertai nty' rOf c ours e.s etri rre.r.-..-

Responsesto questions about suicide, for example, might be sensitivein reki ng . Fi rs r.qual rrr r.r.--- ..
\ame qual i rv .Ir i ' harJ - .
the ihort term to highly publicized suicides, such as Marilyn Monroe's
l i rrn;] O.l aafor s ome l , . .
suicide in August t962.This is also a type of context effect, but the con- i ererrc esi n rel ari r.ep-r-,.
text here is time, not the other items in the questionnaire' di ffere nc esi n qual i rr. \l -.
In short, there are many sourcesof uncertainty in social researchother pri cer. W e ex pec t hl rr.rr:.,.
than sampling error. Thus a preoccupation with sampling error alone is checp in c ountri esrr hc rc -

! --:rls of uncertaintythat
'- r : -r\ c\ s t hem s elv est h
,e
',,rrding,An aPParentlY
.. r,utablechangein re-
- t he wor ding of a
-:::i i ze
- . .. \\'hen new wording is
'- ^ ':Jii" to "African Amer-
: r :.,',(l iorms of the surveY
: --. 1'.lssignedto one of the
i:i-- -.: ri rhe rvording change
-,: : - I .: number of famousex-
l. : : .'..'-:lPle.a stir was created
'--- was
- - - .: .l rrv is hCom m it t ee
-:: ,.,.:-::notight the 1939-1945
"D oes it s eemP os s ib l eo r
-: : : : -:
, \ .: of t he Je w s
-:'::rmi nat ion found that
- -: - r-::-r language,
-: :-. -.1-ist "definitely" did not
*
: r: - ..:.r'" did not haPPen-
: :t- :: il Percentfigure'
-. - .'es the generalcontent
---: .'..-,:rlPle, a question about
--: ' t----::::ferent responsesif em-
. .: :,. -.--: surve,von religiousbe-
r i -,-::ilofl rnight be influenced
. : : -. i 9-2 and 1985 G en e ra l
: r . -. . :;:l e ra l haP P ineswas
s P re -
-
1'-; ::.-.:'smarital haPPiness, and
- : .",.: ttrt1r-renced resPonses to
In some in-
-:.: ', :-.:-xt effect'
.-: i-. : -i l ,rn m aY elic it s om e w h a t
' : : : . : :'rr!e ]' ar e dif f er ent '
i :": r,1-:i-JLliarl1' sensitiveto world
'.- .r. .:.:' :her sourceof uncertainty'
; 1 ...--.;:-.rle. might be sensitivein
Monroe's
- - ::-. .:ch as MarilYn
:-";,. I JLrflte \t effect, but the con-
1 :: . : - , .:si i onnalr e.
researchother
-.,..-.-..t'.r.'ln social
'::. r ''.r:h samPlingerror alone is
The FourthRule . 1 I
short-sighted; a grearer premium should be placed on research rhar at
tempts to caprure the total uncertainty arising from multiplc sources oi
error.
It is time now for an examole.
IrrusrRerroN: Two Mnrnoos ron Esnl.errNc Groeer, povp,Rry
To illustrate the sorts of exclusion errors and measurementserrors just dis-
cussed, I describe a current example from economics. The examplc in-
volves the pressing question of how quickly the world's poverty ratc is
declining. It turns our that the answer depends on how income is mea-
sured. The decline in the world poverty rate (the percentageof the world's
citizens who are poor) is steeperrvhen you measureincome using produc-
tion data than it is when you measure inco're using consumption data
from household surveys.
In this case,then, measurementmatters a lot. Recent work in econom-
ics seeksto understandwhy (Deaton 2005; Milanovic 2005; Ravallion
2003; Triplett 1997). The issue is more than academic. Measuring the
change in world poverty is an urgent task for the social sciences.Even ca-
sual observation revealsthat many millions of the rvorld's citizens live in
abject conditions. The poorest poor in the world are immeasurably worse
off materially than the richest rich. Yet comparing living standards across
countries is a vexed issue (seeFirebaugh 2003, chap. 3). Rich counrries
are those that have an abundance of valuable material commodities to
consume. Poor countries by contrast are those that have a scarcity of
valuablecommoditiesto consume.
How do we measure material differencesacross countries?To sim-
plify, let's supposewe can establishstandard international prices for all
goods and services.2That takes care of the problem of how ro value or
weight different goods and servicesdifferently. \flhat remains then is ro
estimatethe total quantity of goods and servicesavailablefor consump-
tion in a society. One approach is to focus on the production side or
retail side, either by surveying producers to determine how much they
have produced, or by surveying retailers to determine how much they
have sold. Alternatively, you could focus on the consumption side. In
2 Of course, setting a standard international
price that {its all countries is a heroic under-
taking' First, quality might vary across countries. Not all watches or shoesor radios are the
same quality. It is hard to take quality into account when attempting to arrive at an interna-
tional price for some good. Differences in relative prices are not a reliable guide, since dif-
ferences in relative prices across countries might reflect differences in preferences, nor
differencesin quality. Moreover, climate, location, and natural resourcescan affect relative
prices. \ffe expect bananasto be cheap in countries rvhere bananasare grown and coal to be
cheap in countries rvhere coal is abundant.
1,02 . Chapter4

this caseyou would survey consumefs to determine how much they have mofe recent purcir,r:::
c ons um ed. foun d t hat a r eeai_: - .
using eirher approach, we find that living standards are improving for crea sedr epor r eJc\ ; : . . -
the worid as a whole, as well as in most regions of the world (sub-Saharan perc entf or all pr r ". - : .
Africa is the notable exception in recent decades).Typically, however, the sum pt ion som ef in- - . '
production-basedestimatesare higher than the consumption-basedesti- tinguished.In orhe: ,,, '
mates: Production data generally suggestsomewhat higher levels of wel- surv ey,Sur ver s, r l -
fare and somewhatmore rapid growth in welfare than do the consumption householdm cr r r l'<r ,. : .
data. To shed light on the possible sources of these discrepancies,Iet us permit t edr o spc. r k -
consider the consumption and production data in light of the exclusion affecf resulrs.r\ft'!
'We
errors and measurementerrors described above. begin with the con- Even wit h p€r f uJr i. :
sumption data. exclusioner r or . \ \ : - , .
national exper r Jr rr r - . . .
N A T I o N A L E X IENDIT URE ( co Nsu u lr lo N) su R vevs
country, dfld somr . . -
One way to estimate consumption is to ask people about their expendi- Hence there is rhe : ..
tures. The number of such surveys available to the research community hous eholdlisr . . r . r , - - - - -.
has mushroomed in recent years. For example' Angus Deaton (2005' ous e xclusionpr ul. . : : - - .
table 1) locates 557 surveys fuom I27 different countries that can be used is especiallyproblei.r..:: -
t o es t im at ea v e ra g ec o n s u mp ti o n o r a v eragei ncome per person.There the leasr iikelr r o : , - .
were only three such surveyscollected in 1979, representing9 percent of R ava llion 2001, r . i"- .
the world's population;in 1,998there were 57 surveys,covering53 coun- great erit em nor r r c. : . .
tries that are home to 70 percent of the world's population. that m ost househ, , l,: , . .
One source of uncertainty is measurementerfor in the household sur- In shor t .m osr ( \ : : - : .
veys. In contrast to the highly scripted sufveys of national production, ti mat esbasedon h, , : . - '.
there is no strict protocol for data collection for national expenditure sur- l i kel y t o f or ger ir cr : - . : -
veys. Hence discrepanciescan arise due to measufement differences. In cl ai m it em s r har r hr . . '
some surveys respondents keep a diary of their expenditures. In other stateincom e and r h. . :
surveysrespondentsare asked to recall expenditures.The length of the re- ternat iveest im at crr '- ' ,
call period might matter since respondents are more likely to remember N at ional pr odu, l -
tw o d at a set spr ot iJ- . : . :
In addition-and most important-a country's price strl-lctufefor goods and servicests re-
isolatethe effect of i-..::
lated to the country's level of economic developmentitself. Labor is abundant and cheap in samein each c, r se . r , .
poor countries, so labor-intensive goods and services (e.g., haircuts) are cheaper in poor measur et he r r r ur cr i, r,:, -.
countries than in rich countries. Capital, on the other hand, is scarce in poor countrie\, so coincide.Yer rhe tl.u; .
goods that require more capital than labor tend to be relatilely expensivein poor countrles.
R ava llion ( 2003. p. br -
Becauseforeign exchangerates are basedlargely on the trade of capital-intensivegoods that
disfavor poorer countries, exchange rates cannot be relied upon for setting international
on ho useholdexpen; : . -
prices to compare living standardsacrosscountries. As a result, scholarshave developedspe- they a r e obr ained. "T- . -
cial currency indexes, called purchasing power parity (PPP)measufes,based on prices from lustrate in concrctc r,,.,
an extensivemarket basket of goods and services,not just on prices of goods that nations can re su lt in d iscr cp. i.:. --
trade (Nuxoll 1994; Summers and Heston 1991). Almost all studies now use PPP measures
for comparing material well-being across countries, in line with instructions in the United N AT IONA L P R O D U C TI O \ .:
Nations Systemof National Accounts (7993,para.1.38): "Vhen the objective is to compare
the volumes of goods and servicesproduced or consumed per head, data in national curren- In the t it le of an ar r : ; . ,
cies must be converted into a common currency by means of purchasing power parities." R avallion ( 2003r r . x. '
 T h e F 6 r :rthR ul c o l rti

more recent purchases.In India, for example, a split-sampleexperinlcrrt

found that a recall period of sevendays as opposed to thirty days rn-
creasedreported expendituresby about J0 percentfor food and about l7
percentfor all purchases(Deaton200-5,p. 16). Surveysof householdcorr
sumption sometimesdiffer on the number of separateitems that arc dis-
tinguished.In other words, there is no uniforln list of items used irt evcr.v-
survey.Surveysof householdconsumption might also differ on whethcr
householdmembersare questionedindividually, or whether one person is
permitted to speak for everyone.These arc design differencesthat could
affect results,especiallyin richer and more diversehouseholds.
Even with perfect measurement,we would still face the problem of
exclusion error, wl.rich in many studies is just as great a threat. Some
national expenditure surveysavoid remote or dangerousregions of the
country, and some exclude rural householdsaltogether (Deaton 200-5).
Hence there is the possibility of substantialcoverageerror. Incomplete
householdlists exacerbatethe coverageproblem. Probablv the rnost seri-
ous exclusion problem, however,is unit nonresponse.Unit nonresponse
is especiallyproblematicsincethose rvho consumethe most (the rich) are
the least likely to participate in an expenditr:resurvev (Deaton 200.5;
Ravallion 2003); irnd among the rich who do participate we expect
greater item nonresponseas well. Thus there is good reason to believe
that most householdexpendituresurveysunderstateactual consumption.
In short, most expcrts believethat there is nontrivial bias in income es-
timatesbasedon houscholdexpendituresurveys.Becausepeopleare more
likely to forget items they purchased(or werc given) than they are to
claim items that they do not possess,expendituresurveystend to under-
state income and thus overstatepoverty. It is important, then, to find al-
ternative estimatesof income and poverty.
National production data can be used for alternativeestimates.The
two data sets provide the basis for just the sort of comparison we need to
isolatethe effect of data collection methods.The target population is the
same in each case(all the world's citizens).The objcctive is the same (to
measurethe materialwell-beingof the world's citizens).The key concepts
coincide.Yet the data collectionproceduresare quite different; as Martin
Ravallion (2003, p. 6a6) observes,data on national production and data
on householdexpenditures"could hardly be more different in the way
they are obtained." These data present,then, a strategicopportunity to il-
lustrate in concrete ways how measurement errors and exclusion errors
can result in discrepantfindings acrossdata sets.

N AT I O N AL PR O D U C T IO N DATA

In the title of an article about measuringwelfare in developingcountries,

Ravallion (2003) asks "How well do national accounts and sun'eys
104 . Cha p te r4

ap5ree?" The term "national accounts" here refers to production data and
the term "surveys" refers to consumption data from household surveys.
Consumption data are collected from consumers. Production data are
collected from producers, or from retailers. There is no standard con-
sumption survey used by all countries, nor do all countries field con-
sumpuon surveys on a regular basis. By contrast, virtually all countries
attempt to estimate their total annual economic output using highly
scripted methods developedby the United Nations. National accounts are
compiled following the protocols spelled out in the 1993 version of the
Systems of National Accounts or SNA93 (United Nations 1993). To
the extent that national accounting practices differ acrosscountries' then'
the fault lies with uneven implementation, not with uneven standards,
since the standardsare set internationally.
Household or private consumption is estimated from national account
data as a residual (Deaton 2005; Triplett 1'997). Total domestic produc-
r ion is es t ima te dfo r e a c hc o m m o d i ty ,th e n governmentconsumpti on.i n-
vestment, intermediate consumption, and net exports are subtracted out
to estimate private consumption. Domestic food production is estimated
by weighting the acreageof land under cultivation (from agricultural cen-
suses)by an estimate of average yield per acre (from crop-cutting sur-
veys).This figure is important becausethe food that is produced for home
us ec ons t it u te sa l a rg ep o rri o n o f th e to ta l consumpti oni n poor countri es.
The key point to note here is that there are ample opportunities for er-
ror in each step of the estimation process(Deaton 2005). Let's begin with
the estimation of total production. As defined by SNA93, production in-
cludes all goods whether they are exchangedor not' so food produced for
your own consumption counts. In the caseof services,however, exchange
or work within the household, such as care of children, typically does not
count. As societiesbecomericher, within-household exchangesthat didn't
count before might be replaced by market exchangesthat now count (for
example, someone is paid to care for children), so the rate of welfare
growth is likely to be exaggeratedby national account estimates.
There are other issuesthat make it difficult to estimate national pro-
duction, even for statisticians who are highly trained in following the
U.N. protocols. One is the problem of economic activity that is concealed
from authorities to avoid taxation or regulation (or prosecution, if the ac-
tivity is illegal). Becauseproducers typically have greater incentive than
consumersto concealsuch activity, concealedeconomic activity is expected
to bias production estimatesof income downwardly more than it does ex-
penditure survey estimatesof income. This differencebetr,veennational ac-
count (production) and household survey (consumption) estimates of
income is likely to be more than offset, though, by the failure of household
surveysto include the imputed rent of homeowners'Homeowners of course
 The FourthRule . 1i,{

avoid rent, and this important dimension of consumption is captured br

-: - , prodLlction data and
production data but generally not by household survey data.
, r: ,:11household surveys'
The take-home point of this example is that the national account and
-. Pro eluc t iondat a a re
household survey methods-irnperfect as they are individually-provide
: : is no standard con-
strategicindependentestimatesof economic well-being in the world, If the
...1countries field con-
data yield consistentfindings, that consistencycannot be easilydismissedas
..;. r rrtL r allIall c t - r un tri e s
- ,- output using highlY an artifact of correlated errors linked to the way the data were collccted.
Given the differences in the data collection procedures, the results on
-. \ational accountsare
- ::c i 9 93 v er s ionof th e world trends are somewhat reassuring,since both methods point to rising
-"- -.',i L7 7 J l' . T
1993) r wo
incomes, and declining poverty rates, in most regions of the world. As
. a - J ^\at
\ alions
l o n s'
j rr:f .lCfosscountries' then,
noted earlier, however, production-based estimatesgenerally (but not al
-- : '...ith uneven standards,
ways) indicate faster rates of income growth than do household-survey-
based estimates. The actual rate of growth in economic well-being is
probably somewherebetween the two (Deaton 2005), For such an irupor-
:rr '.',t:i from national account
',-'- , Ttrtal domesticProduc- tant question, it is useful to have independentestimates,if only to give us
rn- some senseof the degreeof uncertainty surrounding the estimates.
-.::. : .::1:'!1fI1tCOnSUmPtlOn'
' i : : l : \l rl fts a r e s ubt r ac t edo u t
:r - r ; rroduction is estimated
Toweno e SorutroN: IosNtrcer ANervsEs oF PARALLEL
Dere Sprs
. -.: ..:: 'r from agriculturalcen-
:- - : . .- -: i rom c r oP - c ut t ings u r- In this chapter I have argued that social scientistsshould place a greater
-: : : ::..rtis Producedfor home
premium on research that attempts to capture the total uncertainty aris-
c ount r i e s '
- - - - -.':'l ti o n in P oor ing from multiple sources of error. That is not an easy task. The most
:: ::: .:-::-:leopportunitiesfor er-
promising long-run strategy appears to be identical analysesof parallel
.. - .:..: :..1005). Let's begin with
data sets.
r:r- :t .'" :\A9 3, P r oduc t ioni n -
Imagine we had 1,000 data sets drawn from the same target popula-
-::: : LLrt.Sofood Producedfor
tion. (Usually we are fortunate to have two data sets, much less 1,000,
..,. - -:ir' iC€s,however,exchange
but the thought experiment here is useful for clarifying key concepts.)
.: : : : :r.i dre n, t Y P ic allYdoesn o t 'We
analyze all 1,000 data sets in exactly the same way-same variables,
- ; -..31,,|Jexchangesthat didn't
same models, same estimation methods, and so on. So we have 1,000
:,:: : , - iiiqcs that now count (for
replications-1,000 estimates of the same parameters. If all 1,000 data
. )o the rate of welfare
sets are randomly drawn from the same frame population with 100 per-
:;. . :.... -,---ountestlmates'
cent responserates and perfectly reliable measuresfor each sample, then
:.-, -: : :(l estimatenational Pro-
our results will differ only becauseof sampling error. From the Central
: : ::. . :rained in following the
Limit Theorem for means we know that the mean of the sample means
:- .:... .rctility that is concealed
provides an unbiased estimateof the population mean. In other words the
-. -...: .t or prosecution,if the ac-
samplesthemselvesare unbiased (in the sensethat they provide unbiased
:.-: . I-i\'e greaterincentivethan
estimatesof the mean), so differencesin sample estimatesof the mean re-
.:r :- :.,'nomic activity is exPected
flect only sampling error. The same principles hold for other population
:, ',;,:",';.-:rdh' more than it does ex-
-, i : ::r:ence between national ac- parameterssuch as regressionslopes.
of Now consider the more realistic casewhere responserates are lessthan
-:-,:: .rrnsumption) estimates perfect, frame populations vary acrossdata sets,and there are survey con-
::. '-::. rv the failure of household
'::. 1.,:rers. Homeownersof course text effects and other types of measurementerror. In the (unlikely) event
106 . Chapter4

that theseexclusion errors and measurementerrors are random, samplees- procedures,so thar -rr', - i-:
rimates will provide unbiased estimatesof population parametersbut the errce sur t hc dat u. r r ul , - . ; :
variance of the sample estimateswill increase.Even in the best of circum- coursc.exact ly id. r r r : -,
stances,then, r.l'eexpect exclusion errors and measurementerrors across Hence compromises :---.. .
data sets to add uncertainty to our results-uncertainty that is not cap- searcherhas control rr',:r r-
tured by the usual estimatesof standard errors (Clogg and Dajani 1991.). As noted above. hr,',', :. : .
This is the sort of uncertainty that we want to gauge by replication studiestypicalll' anal. zr
across data sets within a given study. The explicit purpose of such inter- rcsult sucr osst laI u : r r r , '.=
nal replication is to provide a method for gauging total uncertainty aris- nal rcplicalioz \ l r , r r r J\ .
ing from the data collection process itself. \7e want to compare results l n t hc caseol elr r r
from the identical analysisof different data sets.'Wewant the replications data point and useCoD',:11
to be done in parallel, by the same researchers,to make the analysesas dency and disper sior r
similar as possible. beenformalized Lrsins-r ii-:r-
Internal replication is especiallyuseful in the caseof large data setswhere anal ysis.M oder n m cr . ; - . :
sampling error is not a major problem but other types of exclusion error sucha s G eneG lass 1*- - . - -
could be. Examples of internal replication are nonethelesshard to find. mul tiple st udiesr o r r r .-
Replication acrossdata setsrequiresdata setswith overlappingcontent. Un- mi gh t want t o kr r , . . . . - -
derstandably, funding agencieswant to support new and unique data, not achievement,and u'h.::-.: - ,
repetitivedata sets.As a result the overlapof content acrossmaior social sci- The lit er at ur er c\ : ; . - -
ence data sets is often lessthan ideal for extensivereplication work. can seenas a kind oi : : . , - :
One solution might be to piggyback on surveys done by independent much more standirrdrz.* .- -
polling organizationssuch as Gallup and Harris. Ideally this would be done l i terat ur er evier r s.h. r '. : : - -
when different polling agenciesare in head-to-headcompetition to make on met a- analyt icnr . r h - . .
accuratepredictions about a common target population, as occurs, for ex- narrat ivelook at ) on] . - . - - - '
ample, in the polls leading up to an election. By enabling identical analyses sul tsfr om a neu' r t r r i.
of parallel data sets,the addition of economic and social indicatorsto these sti tut et he dat a.Thu. : . - . . : - ,
simultaneous polls should provide leverage for investigating the effects of as si z eof t he sam plq.: - .- - . :
the nonprobabilistic exclusion errors discussedin this chapter. are used. qualit r or : - .- . . -
As it stands now, however, it is hard to find studies that perform paral- reviewed journal, u'h.:.-.: :'
lel analysesof different data sets as described here. Hopefully this mode sourceas the independ.':.: ,:
of researchwill become more common in the future. In the meantime, it studiesthemselves ore r:; ,: -
is instructive to consider a more common mode of research, called meta- pri or st udies.or ar lc. r . : - ,
analysis, which also uses multiple data sets in a single study. The aim in ro rest on much lnorr - "
meta-analysis,however,is not the parallel analysesof a limited number of fi nds in m osr ot her r r : . . ,- .
data setsbut the averaging of effects acrossmany data setsto obtain more Met a- analysisis h. . : , - :
dependableestimatesof effect sizes. the i s sueof nonr esiJr ' .
evi dencer hat a f ar hcr . : . -
hi s ch ildr en'sacaden: : . , - -
Mr.re-ANervs r s: SvNlrrnsrzruc Rnsurrs FoRMALLyACRoss Srurrss hi s vi sit swit h his chr i- : , - - -
l ated issues,Paul \ r : . : : : - .
To gauge uncertainty, ideally we want to analyzedifferent data sets rn a anal y sisof 63 st uJi. . -.
single study becausein a single study we can make every effort to run qual if y: "To be ir r . lu- =, . - : -
identical analysesacross the data sets.Our goal then is identical analytic i n fat her - absent
hou. . :
 The FourthRule . 1'-,-

procedures, so that any differencesin results can be attributed to differ-

encesin the data, not to differences in the way the data are analyzcd. Of
course, exactly identical analysesmight not be possible across data sers.
Hence compromises may be inevitable. Importantly, though, the re-
searcherhas control over those compromises.
As noted above, however, this sort of internal replication is rare; single
studies typically analyze only one dara set. Very often, then, to compare
results across data setswe must compare results acrossstudies-an exter-
nal replicatioz strategy.
In the case of external replication we might think of each study as a
data point and use conventional methods for summarizing the central ten
dency and dispersion of the effects of interest. Exrernal replication has
been formalized using a method (or a family of methods) known as meta-
analysis.Modern meta-analysiswas developedby educational researchers
such as GeneGlass(1976,1977) as a merhod for combining the resultsof
multiple studies to arrive at a more accurate estimate of effect sizes.\)7e
might want to know, for example, the effect of class size on student
achievement,and whether the effect varies across contexts.
The literaturs lsyisw-x longstanding tradition in research articles-
can seen as a kind of precursor to meta-analysis.Meta-analysis involves
much more standardizedand rigorous procedures than found in standard
literature reviews, however (seeCooper and Hedges 1994 for a handbook
on meta-analytic methods). Whereas literature reviews generally present a
narrative look at some subsetof prior studies as a context for viewing re-
sults from a new study, in meta-analysisthe prior studies themselvescon-
stitute the data. Thus the studies are codified according to categoriessuch
as size of the sample, the use of randomization, whether control variables
are used, quality of the study, whether the study appeared in a peer-
reviewed journal, whether the dependent variable came from the same
sourceas the independentvariables,and year of data collection. Becausethe
studiesthemselvesare the data, data collection involves locating all relevant
prior studies,or at least the high-quality studies.Hence mera-analysestend
to rest on much more comprehensivereviews of the literature than one
finds in most other research studies.
Meta-analysis is best understood by looking at an example. Consider
the issue of nonresident fathers and children's well-being. There is some
evidencethat a father's payment of child support has a positive effect on
his children's academic and social well-being, but that the frequency of
his visits with his children has little or no effect. To addressthese and re-
lated issues, Paul Amato and Joan Gilbreth (1999) performed a mera-
analysis of 63 studies. The first step is to demarcate rhe studies that
qualify: "To be included, studies had to (a) include a sample of children
in father-absenthouseholds,(b) present quantirarive data on a measureof
108 . Chapter4

paternal involvement and a measure of child well-being, and (c) present

enough information to allow the calculation of an effect size summarizing
the strength of the association between the two variables" (p. 561)' The
next step is to determine a search procedure to locate the studies that
qualify. Amato and Gilbreth used three computerized data bases(Socio-
dle, Psychlit, and Family Studies Database), and they supplemented the
computer search with the literature review sections of the articles uncov-
ered by the computer search. They excluded conference papers and dis-
sertations,as well as studies published in non-English publications.
Becausesome studies reported data separatelyfor boys and girls, the
63 studiesincluded data from 100 independentsamples.Measuresof the
dependent variable, children's well-being, were categorized as school
achievement(for example, grades),erternalizing problems (for example'
misbehavior at school), and internalizing problems (for example, low self-
esteem).Measures of nonresident father involvement were categorizedas
payment of child support, frequency of contact, feelings of closeness,and
authoritative parenting (for example, listening to children's problems,
monitoring school performance, helping with homework, providing ex-
olanations for rules).
Amato and Gilbreth used the product-moment correlation coefficrentr
as their measure of effect size (partial r for the studies that used control
variables). Some studies reported effect sizes in terms of other statistics'
such as regfession coefficients or logits. In those instances Amato and
Gilbreth used formulas from Hedges and olkin (1985) and Rosenthal
(1994) to translate the reported statisticsinto rs and partial rs. It is a sim-
ole matter then to calculate the mean and variance of the associationsof
iach of the three dependentvariables with each of the four independent
variables. The associationsshow that three of the dimensions of father
involvement-payment of child support, feelingsof closeness,and author-
itative parenting-are associatedwith positive child outcomes, with au-
thoritative parenting having the most consistentpositive effect. Frequency
of visitation, by contrast, had little or no associationwith child outcomes.
Amato and Gilbreth employ several tests to gauge the robustness of
their results. One concern is publication bias, or the "file drawer prob-
lem" (Rosenthal 1.979)-the problem that occurs when researchersput
their negativeresults in a file drawer rather than submitting them for pub-
lication. In the caseof father involvement and child outcomes, one might
expect rhat researchersare less likely to submit their findings when there
is no association,and editors in turn might be lesslikely to accept a paper
with null findings. The result then is publication bias favoring studies that
find correlations between involvement and child well-being.
If only the largest associations end up being published, then meta-
analyses of published studies will overstate effect sizes. To address the

' i , ' : -rri n g! and ( c ) P r es e n t
ijr
: . '. . . . : ct q lT eS um m ar iZ i n g
': :, ..::ta bles "( P . 561) . T h e
r-r: i . ,;lte the studies that
,- :. : : :t?3d dat a bas es( S oc i o -
r: .-- .;-.c.supplementedthe
" - .: . ; . :'.:,,t th e ar t ic lesunc o v -
j:- , -rlic'Il.e PaPerSand dis-
- - - : : : :.r Public at ions .
:,:-.',: :,,r bo)'s and girls, the
': :- : - '-::l l l es. M eas ur esof t h e
:. .' ::: -.lregorized as school
-: .--:-- ::rrblems (for examPle,
: : - .: :t : :or erample, low self-
.. : -, -: .:::i rn gs of c los enes san
, d
,i :r .r - :Lr chi i dr en' sP r oblems ,
,;"-- :,rne\\'ork, providing ex-
:r : -.: ;orrelation coefficient r
-.
: : .... .:'.rdiesthat used control
::- . :irns of other statistics,
.: :: .;- instancesAmato and
.r , ' r,r-. 1985) and Rosenthal
: ..:.:-:..-..e of the associationsof
i: :-,--:' of the four indePendent
:r: : :he dimensions of father
::: :-.:>of closeness, and author-
':.: :; ,l.rild outcomes,with au-
i i:::': :r)siti\-eeffect.FrequencY
::: ,,.r:r,.-)r1 rvith child outcomes'
::::: :a) Saugethe rObustnesS of
:-.:>. or the "file drawer Prob-
:.j: ,-;l,rs .r'hen researchersPut
=: :::,..:tsubmittingthem for Pub-
.: ::-.i ;hild outcomes'one might
;-r::rii their findingswhen there
.:.: :. ...sslikell'to accePta PaPer
..:-:,':t b,iasfavoring studiesthat
:: , - .. . j u 'ei l -b eing'
meta-
-: ::r:rs published,then
:: : : : : : :e .t si z es T
. o addr es sth e
Thc Fourth R ule o l r)9
issue,Amato and Gilbreth report "fail safe N values" using procedurcs
suggestedby Rosenthal (1,979).This statistic refers ro the number of nex'
studies that would be required to reduce a significant mean effect size to
nonsignificanceif all the new studies had null results. Amato and Gilbreth
also compare the effect sizesfor large and small samples.This servesas
another check for publication bias, as follows: Supposeauthors are reluc-
tant to submit (and editors are reluctant to accept)articles in which results
are not statistically significant. If so, it follows that published correlations
should tend to be larger in small samplesthan in larger samples,srncea
larger correlation is required to attain statistical significance in a small
sample. Thus larger effect sizesin published studies with smaller samples
may indicate the existenceof publication bias.
Quality of the study might matter, of course, and we would want to
give more weight to the findings of the better studies.Amato and Gilbreth
test for the effect of study quality by coding each study as 0 (no) or 1
(yes) on the following characteristics:whether families were selectedran
domly, whether the sample size is at least 100, whether the dependent
variable had a reliability coefficient of 0.8 or grearer, whether the study
employed control variables, and whether the study used different infor-
mants for the independentand dependentvariables.By codifying the stud-
ies in this way they were able to assesswhether the quality of the study
moderated the strength of the association between paternal involvement
and offspring outcome. Our confidence that the effects are real is en-
hanced if we find that the better studies find the srronger results.
Finally, it should be noted that meta-analysisis not useful where a lit-
erature has gone off the rails. Recall, for example, the fatally flawed
cross-country studies of foreign investment effects described in chapter 3
(where foreign investment and domestic investment were measured very
differently). Meta-analysis would not have helped in that case,since con-
clusions were basedon a fundamental misinterpreration of the model be-
ing estimated.A meta-analysiswould simply have averagedthe erroneous
interpretations.
SuuueRy: Youn CoNrTDENCEINrr,Rvars Ans Too Nennow
One key to the scientific method is the validation of one's results with re-
peated trials. In the case of nonexperimental social research, "repeated
trials" often means different samples.Yet it is rare to seethat type of val-
idation in sociological research articles (though the findings might be
replicated in subsequentresearch),and in my reading in the other social
sciencesit appearsto be relatively uncommon in those other disciplinesas
well. I am hard-pressedto find examples of substantial literatures where
110 . Ch a p te r4

parallel analysesin a single study have become an integral part of the re-
search process. If my observations are correct, we will know that social
researchersare coming to grips with the uncertainty issuewhen the report-
ing of identical analysesof parallel data sets becomesas commonplace in
researcharticles as the reporting of significancetests is today.
Identical analyses(samemeasures,models, and estimation methods) of
parallel data sets(different samplesof the sametarget population) can yield
divergentresultsdue to exclusion error and measurementerror. Of the "big
three" erclusion errors-coverage error, sampling error, and nonresponse
error (error due to unit nonresponse)-only sampling error is reflected in
conventional standarderrors. And measurementerror is not reflectedat all
in conventional standarderrors. As a result, confidenceintervals understate
the true level of uncertainty.
Unlike sampling error and measurement error, which often can be
modeled using probability theory, coverageerror and nonresponse error
generallycannot be modeled or are very difficult to model. To gauge total
uncertainty, then, we must often proceed inductively, that is, by examining
more data. That's where internal replication-the identical analysisof par-
allel data sets by the same researchteam-comes in. Realistically,replica-
tion may not be possible for most social researchtoday. But that should
not prevent us from being alert to the possibility. Rule 4 is a call for social
scientists to become more opportunistic regarding replication. Perhaps
internal replication will become a hallmark of social research later in the
twenty-first century, replacing today's norm of relegating replication to
follow-up studies.
A PPEN DI X

StudentExerciseson Rule 4

The exercisesbelow use data from the American National Election study
(ANES). ANES data are collected every rwo years around both presiden-
tial and midterm elections. Each survey is carried out in two waves, one
wave before the election and one after the election. Much like the GSS,
the ANES contains a multistage probability sample of noninstitutional-
ized individuals in the United States, age eighteen or older on Election
Day of each national election year.
To do the exercisesbelow,3 go to the web site http://sda.berkeley.edu.
As with previous exercises,the instructions below should generateal1the
statisticsyou need to answer the questions, making hand calcr-rlationsun-
necessary.Again, remember that variable names are in boldface. Note
well: The student exercisesat the end of chapters 7 and 2 use rhe same
web site. The instructions below include only the key commands; I do not
give step-by-stepinstructions. The fomrat in the sDA rveb site rnight have
changed somewhat since these directions were written, but ,vou still
should be able to follow the logic below to obtain the tables vou need ro
answer the questions.

AssrcNmeNt 1: SsrprrNc Supponr FoR THEEquer Rrcnrs

Aur,NlnENr (ERA)?

ln 1972 congress submitted the Equal Rights Amendment to the state

legislaturesfor ratification. It read, in part, "Equality of rights under the
law shall not be denied or abridged by the United Statesor by any State
on account of sex." Approved quickly by several state legislatures,the
amendmenr seemedheaded for certain ratification. But in the mid-1970s
conservative activisrs capitalized on growing opposition to federal gov-
ernmental acrion and concern with the health of the traditional family
to prevent ratification by the requisite 37 state legislatures.They did so by
stoking fears that "equal rights" would further erode traditional gender
roles, subject women to selective service registration and higher insur-
ance premiums,and outlaw sex-segregated restroomsand organizations.

3I thank Matt Schroederfor

assistancein designing the exercises.
1I2 . Chapter4

One might think, therefore, that public support for the ERA declined Wc f ind sim ilar r : ,
as well in the face of these arguments. But the data are lessthan definitive trenr,Js. Let's brgin ri :-
on that issue,as we seefrom thesetrends in Gallup and Roper polls span- quer ied r espor r dcn[ :. . .
ning the period March 1976 to June 1982. v83 3) : "An ef ior r i. :
According to the Gallup polls, the percentagein favor declined from sti tut ionwhich u, - u - ' -
70 percent in 1,976 to 62 percent six years later. On the basis of the sex,I)o yt-luirppr()\t'
Gallup series,then, it appears that support for the ERA in fact eroded Constitution?"
over the late 1970s. But the Roper polls indicate no such erosion of sup- . From the SD\ -
port for the amendment (and perhaps even a slight increasein support).
ur chive" lr nd r i. :
r You will r r eetri , ' - .
-
Question 1. The question wording differs for the Gallup and Roper polls. f r om your anir '. ': .
Observe that the Gallup poll says that the Equal Rights Amendment
sponsesam or 't !: - :
"would give women equal rights and responsibilities." The Roper ques-
t r eatt hoseii'h,, -
tion says nothing of equal responsibilities for women. Do you think the -.
ing.
addition of "and responsibilities"in the Gallup question is the decisive . Fr om t he r eeoc- . , '
difference that explains the divergencein the Gallup and Roper trends, or f or "Nam e ior : .
are there other differencesin the polls that you think are more important? "Nam e( s) t - l1y\ . - : :
Defend your answer. lect "yes" inr '-
t ion. For r oir I
"approve" for " .--
PublicOpinion Surveyson the ERA, 1976-1982 t ype "2" f or ". : .
I . " Under "\ \ h. : :
Gallup Polls
ables( if an\ - . : " : .
Have you heard or read about the Equal Rights Amendment to the Constitution
( This will aLr r ( , : '. i
which would give women equal rights and responsibilities?Do you favor or
oppose this amendment? ing. ) Then e1r . ^
shouldhave- {.i. . -
3/1976 5/1978- 7/1980" 7/1981* 12/1981 6/1982* o To cr oss- t abul. r :.- .
Favor aio/
65% 65% 66% 63% 62% menu ar rhe rop : '
Oppose 30% 35% 35% 34% 37Y" 38"/" cies clr cr ossr . r b: :. :
var iableol' ir r r r : r -
"Question asked only of those who had heard or read about the ERA.
st udy)as t he . , ' - -
Roper Polls cent age5. 5elc. r "\
The various State Legislaturesare now voting on an amendment to the United Under "Char r , ': :
StatesConstitution which would assurewomen equal rights under the law. As char t " {unlet r t
I'm sure you know, there is a lot of controversy for and against this amendment. f ault " m ode. . r . '.
How do you personally feel about it-are you in favor of the Equal Rights
t he r able" r o , , - - i. :
Amendment or opposed to it?
valid cases,oi r '.: -
1 2 /1 9 7 7 7/1978 10/1979 12/1981 in opposit ion. 'l- -
In Favor 67% 69% 69% 71% aSome readersmigh: .,.
Opposed 33% 3 r% 31% 29"/o
for ANES. As discusse.i:r :
Note: Only valid responsesare shown Data provided by the iPoll Databank of the Roper neces s aryto rec ons truai:a:
Center for Public Opinion Research. oversampled.(Recall rh.r: ::

- . r: -. :,,r the E RA dec lin e d
: : -. ,: I -.-:r'lessthan definitive
- : : .'.rJ Roper Polls sPan-
j: - "'-: '' fl vor dec linedf r o m
:: L)n the basis of the
::- FR-\ in fact eroded
- .uch erosion of suP-
. ....rc J s ein s uP P or t ).
. . : ':..i,luPand RoPerPolls.
: - : : , ..-.. R ight s A m endme n t
- : j. .;:;:'-i . "T he Roper que s -
. : - ,. r'.in, Do you think the
:: , r .i ,.i stion is t he dec is i v e
, ..: lnd Roper trends,or
: -...i Aremore imPortant?
-.nr to the Constitution
:.i Do 1'oufavor or
1211981 6/ 19 8 2 r
62%
63%
37% 38%
: ::r. L,RA.
::'..-;rdmentto the United
::.ihts under the law. As
: .rglinst this amendment.
: ,,i the Equal Rights
I ; t 19 7 9 1 2/1 981
tr9oo 7r%
-l 1oo 29%
, P rll D a r a b a n ko f r h e R o Pe r
TheFourthRule . tt:
\7e find similar inconsistencieswhen we compare the ANES and C,\S
trends. Let's begin with the ANES. ln 7976,1978, and 1980 the A\ES
queried respondents on their support for the ERA (the variable label is
v833): "An effort is being made to passan amendmentto the U.5. Con-
stitution which would guarantee equal rights for all citizens regardlessot
sex. Do you approve or disapproveof the Equal Rights Amendment to the
Constitution?"
o From the SDA homepage (http://sda.berkeley.edu), select "SDA
archive" and then 'ANES Cumulative Datafile 1948-2000."
o You will needto recodev833 to exclude the "Don't Know" responses
from your analysis. The ANES has included the "Don't Know" re-
sponsesamong the valid responsecategories,and researchersusually
treat those who decline to give a firm answer to the question as mrss-
tno
o From the recode screen(found under "Create variables"), type in era
for "Name for the new variable to be created." Type in v833 under
"Name(s) of existing variablesto use for the recode." Be sure to se-
lect "yes" for the "Replace that variable,if it already exists?" op-
tion. For row 1 under "Output variable," type "1" for "value,"
"approve" for "label," and "1" for "Var 1." For the secondrow,
type "2" for "valuer" "disapprove" for "labelr" and "5" for "Var
1." Under "\7hat to do with unspecifiedcombinations of input varr-
ables (if dny)," make sure "Convert them to MD code" is ielected.
(This will automatically code the "Don't Know" responsesas miss-
ing.) Then click on "Start recoding" to convert v833 to era. You
should have 4,589 valid cases.
e To cross-tabulatethe recoded variable with survey year, return to the
menu at the top of the home page. Select"Analysis," then "frequen-
cies or crosstabulations." Becauseera is the outcome or deoendent
variable of interest here, enter era as the row variable and v4 lyear of
study) as the column variable, and be sure ro ask for the column per-
centages.Select"No weight" under the "'Weight" drop-down menu.
Under "Chart options" use the drop-down menu to indicate "no
chart" (unless you want a chart). Leave the other options in "de-
fault" mode, as you have done in previous assignments.Click "Run
the table" to obtain the cross,tabulation.You should have 4,589
valid cases,of whom 3,215 are in support of the ERA and 1,,374 are
in opposition.a Print out your results.
aSome readersmight wonder why they
are instructed to choose the "no weight', oprion
for ANES. As discussedin the section on sampling in chapter 1, reweighting of the data is
necessaryto reconstruct the overall means and varianceswhere minority populations uere
oversampled.(Recall that the GSS oversamplesblacks in some surveys,for example.) In rhc.
114 . Chapter4

GSS Resrzlrs

Next let's seewhat the trend looks like using the General Social Survey.In
1977 and 1982 respondentswere asked "Have you heard or read about
the Equal Rights Amendment?" For those who answered yes, the GSS
variable era asks simply, "Do you strongly favor, somewhat favor, some,
what oppose, or strongly oppose this amendment?"
. You will notice that there are four responsecategories.To allow for
an easiercomparison with the ANES, collapse the era variable into a
dichotomy scored 1 for "Strongly favor / Somewhat favor" and 2 for
"Somewhat oppose/ Strongly oppose."
o From the recode screen,type in era2 for "Name for the new variable
to be created." Type in era under "Name(s) of existing variablesto use
for the recode." For row 1 under "Output variable," type "1" for
"valuer" "favof" for "label," and "1 -2" for "Var 1." For the second
row, type "2" for "valuer" "oppose" for "labelr" and "3 -4" for "Yar
1." Make sure to select "yes" for the "Replace that variable, if it al-
ready erists?" option. Then click on "Start recoding" to convert era
to eta2. You should have 2,765 valid cases.
r From the "frequenciesor cross-tabulations" screen,enter era2 as the
row variable (dependentvariable) and year as the column variable (in-
dependent variable). Select "oversamp-weight for black oversam-
ples" in the "lWeight" menu and retain the same options as in previous
examples.Then click "Run the table" to obtain the cross-tabulation.
You should have a total of 2,025 respondentswho favor the ERA and
749 who oppose it. (If your numbers differ, you probably selecteda
different weighting option. )

Your results should show different trends for the ANES and GSS data.
On the basis of the ANES data you would conclude that support for the
Equal Rights Amendment declined sharply after 1,976,from 80.8 per-
cent approval in 1976 down to 61.3 percent approval in 1980. On the
basisof the GSSdata, by conrrasr,you would conclude that support for
the ERA did not change from 1977 to 1982. (Note that the Chi-square
value indicates no association between year and era2 in the GSS data-
in other words, there was no statistically significant change in era2 from
1977 t o 1, 9 8 2 .)

case of the ANES data eramined here, there are no oversamples,and the weighting scheme
is intended instead to weight respondents as necessary to obtain a sample that looks in the
aggregdte like the U.S. popularion with respect to age, educational artainment, and geo-
graphical location. Becausethe provided weights adjust for nonresponse,using them would
take away some of the heuristic value of this exercise.

i. : i ir: Ge ner alS oc ialS ur v eYIn
'
or r ead ab o ut
, -i: - . r. .'ou h ear d
' :: ,r,1\-r.lnsweredYes,the GSS
a , :-.1 ,:. somewhatfavor, some-
_ _ -_----r: "
::i: arL. categories' To allow for
i.. - ..:lse the era variable into a
::.' . flttlervhat favor" andZ for
! \i.
:-l : : "\ame for the new variable
\, :': r of existing variablesto use
-,-\--:r',rtlariabler" type "1" for
- i -1" :or ''Var 1." For the second
r- : : "irbel." and "3 -4" for "Yar
:-.: '-it:place that variable'if it al-
r ">:.:rr recoding" to conveft era
screen,enter era2 as the
::-.: r.rr ls the column variable (in-
..;1-;-11'sight for black oversam-
::.1 ::- sJmeoptionsas in previous
: : : .rt'tain the cross-tabulation'
::: r.;-:'rtsrvho favor the ERA and
:.:. :-::er. r'ou ProbablYselecteda
=.... :,-rrthe ANES and GSS data'
for the
-.,: ;onclude that suPport
-::: '. -.rfrer1'976, ftom 80'8 Per-
:. : - . : t aPPro v alin 1980' O n th e
for
- ,,. :iJ concludethat suPPort
1-. L \ote that the Chi-square
.. \.3r :r-rdera2 in the GSS data-
i .'. ..:iificant changein era2 from
:::.lllrples, and the weighting scheme
' :: - - ::. , obt ain a s a m p l e t h a t l o o k s in tb e
:: : .t::. .ducational attainment, and geo-
- - :- .::: iL)rnonre S p o l l s eu, s i n g t h e m wo u ld
The FourthRule . I 1.5
'$7hat
accounts for the large discrepancy between the GSS and ANF,S
results? Two possibilities can be ruled out at the beginning. First, sam-
pling error is highly unlikely. For samples this large, it is ertremeiy un-
likely that differencesthis big are due entirely to sampling error. Second,
year of melsuremenl is also unlikely. Consider the differcnces in the ap-
proval rates basedon the ANES in 1980 and the GSSin 1982; 61.3 per-
cent versus 72.8 percent Although historical eventscould account for a
shift of this magnitude over a two-year period, that appearsunlikely here,
since the Gallup results (above) show no evidence of radical swings in
public opinion on the ERA from 1980 to 1982.
Question 2. (a) Describe the critical differences in the wording of the
questions in the ANES and GSS.
(b) Describe the differences in the population actually surveyed. (Recall
that the GSS question uses the filter, "Have you heard or read about the
Equal Rights Amendment?" whereas the ANES does not.)
(c) Either difference-the difference in wording, or the difference in the
population surveyed-could plausibly account for some or all of the dis-
crepancy in the two trends.'S7hichdifference do you think is more critical
here? Defend your answer.
AssrcNurNr 2: Estruerno Vorr, TunNour UsINc Drrrtnr,Nt
Sel.plrNc Fneuss
In 2000 the ANES experimentedwith drawing respondentsfrom two dif-
ferent sampling frames. In the first method (referred to here as the FTF
frame, for "face-to-face"), a traditional multistage probability sample de-
sign was used to select respondents for home interviews. In this design,
large regions are selectedprobabilistically, then subregions are selected
within the chosen larger regions, and so on, down to the household level.
In the final step, a single respondent (an adult) is randomly selectedfrom
the chosen housing units. The FTF sample includes 1,001 respondentsin
the preelection wave. All FTF interviews were conducted in respondents'
homesin face-to-faceinterviews.
In the second method, random-digit-dialing (RDD) was used to select
respondents,and all interviews were conducted over the telephone.In the
caseof RDD, then, the frame population excludesindividuals who do not
have telephones.The 2000 RDD sample includes 806 respondentsin the
preelection wave of the survey.s
5For further explanation of the RDD and FTF designs used in the 2000 ANES, go to
ANES 2000, click on "codebook" at the top of the SDA analysis page, then click "Intro-
ductions" in the left frame. Finally, click "Sample Design" from the introduction menu.
116 . Cha p te4r

You lvill be investigatingthe consequencesof the different sampling tech-

niques fcir estimatesof voting turnout. 'Wewant to seeif sampling method
and mode of interview (face-to-face versus over the telephone) matter.
That is:
. If the same question is asked at the same historical time point to dif-
ferent samplesof individuals, to what degree do the results differ if
the respondents are selected using RDD and interviewed over the
telephone as opposed to being selectedusing a multistage area sam-
ple design and interviewed in person?
The ANES asks the following question, denoted vl24l: "In talking to
people about elections, we often find that a lot of people were not able to
vote becausethey weren't registered, they were sick, or they just didn't
'Which
have time. of the following statementsbest describesyou:
One, I did not vote (in the election this November);
Two, I thought about voting this time-but didn't;
Three, I usually vote, but didn't this time; or
F our , I am s u reI v o te d ? 6

Use the Berkeley SDA web site to examine the differences in voting
turnout between the two sampling frames describedabove.From the SDA
homepage, click on "SDA archive," then go to the American National
Election Study (ANES) 2000 data and follow these instructions:
o Recode vL24I into a dichotomous variable, coded 1 for respondents
who voted and 2 for respondentswho did not vote, as follows: From
the recode screen,type in vote2000 for "name for the new variable
to be created." Type in v1241 under "Name(s) of existing variables
to use for the recode." For row 1 under "Output variable," type "1"
for "valuer" "voted" for "label," and"4" for "Var 1." For the sec-
ond row, type "2" for "valuer" "did not vote" for "labelr" and "1-3"
for "Var 1." Then click on "Start recoding" to convert v1,24L to
vote2000.You should have 1,554 valid cases.
o Cross-tabulate vote2000 with the sampling frame variable, denoted
v4. Insert vote2000 as the row variable (dependentvariable)and v4
as the column variable (explanatory variable). In the "'Weight"
drop-down menu, select"No weight." Use the options for previous

6The wording may seem a bit lengthn but this is designed to avoid what
survey re-
searcherscall "social desirability effects." Becausepeople often feel ashamed to admit that
they failed to vote, this rvording aims to avoid falsely affirmative answers from rhese non-
voting respondentsby giving them other, less stigmatizing reasonsrhat they might not have
voted. And by calling attention to a specific election, they attempt to avoid memory recall
errors by forcing respondentsto think carefully.
 The FourthRule . 11-

:'!erent samPlingtech- exercises.Ask for column percentagesand not row percentages

if samplingmethod (one decimal point will suffice unless otherwise noted). Check the
-:;
:.-.: telephone)matter. box beside "statistics," and use the drop-down menu to indicate
that you want three decimal points for them. Remove the check be-
side "color coding" and use the drop-down menu to indicate that
.,.r1time point to dif-
you want "(no chart)" under "Chart options." Finally,click "Run the
: , thc results differ if
raD l e.
: :rrerr"iewedover the . If you have followed the instructions properly, you should have 1,554
:rulristagearea sam-
valid cases,of whom 1,182 voted in the 2000 election and 372 did
not vote.
r 1 l -l 1: " ln r alk ing rt.r
:.ople were not able to Question 3. (a) Supposeyou want to use the ANES to determine voring
t hey ius t didn ' t
-.. or rates in America-the proportion of adults who voted. What do you dis-
J.scribesY ou: cover in the cross-tabulation of voting rate by method that complicates
your task? In a paragraph, describehow the two samplesdiffer in their re-
ported rates of voting. In your description, be sure to note both thc sub-
stantive and the statistical significanceof the difference.
.
(b) \hich method-FTF or RDD-would you suspect gives the more
accurate estimate?Explain.
-. ..-..-iifferences in voting
the SDA Question 4. Observe that the target population is the same for both FTF
-: :i .rbole.From
: :::- -\merican National and RDD-voting rates for Americans-so we can rule out differencesin
l-: : : l l l St l U C tl O[S : the target population as an explanation for the observed difference in the
FTF and RDD results.'$fhat,then, doesaccount for the higher reported rate
:. - ,led 1 for resPondents
of voting among those interviewed by telephone?Let's proceed systemati-
j -. : \ ote. as follows: From
cally by considering differences between the FTF method and the RDD
"-:..''.; for the new variable
method with regard to the frame population, the representativeness of the
. .r'.' s of existing variables
' collectedsample,and the likely measurementerror. Consider each in turn:
--::',rrvariable,"tYPe"1"
: : ,: "\ar 1. " F or t he s e c - Coverageerror (differencesin frame populations)
:: ' i t)f " labelr "and " 1-3 " The RDD sample excludesthose without telephones,who are probably
: r-. to convert v1241 to poorer, lesseducated,and more transient than people with telephones.
_:::r.
-
They would be included in the FTF sample.
: :rame variable,denoted (a) Is this type of exclusion likely to bias the RDD sample results on re-
:.1::'ndentvariable)and v4 ported voting in a positive direction (over-reporting on voting) or n a
. . -,::-'.b l eIrn. t he " ' W eig h t" negativedirection (under-reporting of voting)? Explain.
'
- : :hc opr ions f or P r ev i o u s Unit nonresponse(representativeness of collected sample)
The RDD sample has higher nonresponse-it's easier to refuse a
- - i :- : : -. i t o a v o i d w h a t s u r v e y r e - phone call than someone at your door. (The FTF sample had a re-
: ,,: : : :::1 ieel ashamed to admit that
sponse rate of 64.8 percent in the preelection wave, while the RDD
:'- i - -: : : \ e J n s w e r sf r o m t h e s eno n -
-: sample had a 57.2 percent responserate.) Supposethis excludesthe
- - ]r I - . : : ': 'rns t h a t t h e y m i g h t n o t h a ve
-. :' : ,.::::t.lPtto avoid memory recall poor and less educated(thosewho are less likely to vote) more than
the FTF sample does.
118 . Cha p te 4r

(b) Is this type of exclusion likely to bias the RDD versusFTF results
on reported voting in a positive direction for RDD (higher reported
rates of voting in RDD) or in a negative direction for RDD (lower
reported voting in RDD)? Explain.
(c) AdditionallS the FTF sample may find it harder to reach the rela-
tively well-off and older people who live in gated communities, further
differentiating the composition of the two samples.This type of exclu-
sion is likely to exacerbatethe difference in the RDD and FTF reported
rates of voting. Explain why this is the case.
Measurement error
(d) Peopleinterviewed over the telephone may give less thought to the
question, or it might be easierto lie over the telephone. Is this type of
measurement error likely to bias the RDD versus FTF results on re-
ported voting in a positive direction for RDD (higher reported rates of
voting in RDD) or in a negativedirection for RDD (lower reported vot-
ing in RD D )? E x p l a i n .

AssrcNupNr 3: CoupenrNc RDD AND FTF Seuprrs ro CENSUS

DArA

In assignment 2 we hypothesized that a sample based on random digit

dialing is likely to over-representthe more educated. If so, our estimate
of vote turnout is likely to be inflated or "upwardly biased" in the RDD
sample, since there is a strong posirive associationbetween education and
voting in the United States.The FTF estimate of voting rate may well be
upwardly biased as well, since Americans are reluctant to admit that they
did not vote. But we expect the upward bias to be greater in the RDD
sample than in the FTF sample.
Now we test those hypotheses.Let's begin by comparing the propor-
tion of college graduates in the FTF and RDD samples.If both samples
are representative,then of course the proportions should be roughly the
sarne.Are they?
o Using the ANES data file for 2000, first recode v913 (the ANES vari-
able for highest degreeearned) into a dichoromous variable coded 1
for respondentswith a college degree and 2 for respondentswithout
a college degree.From the recode screen,type in college for "name
for the new variable to be created." Type in v913 under "Name(s) of
existing variables to use for the recode." For row 1 under "Output
variable," type "1" for "value," "college degree" for "label," and
"6-7" for "Yar 1." For the secondrow, type "2" for "value," "no
collegedegree" for "label," and "1-5" for "Yar 1." Then click on
 T h c FourthR ul c .
1,I9
"Start recoding" to convert
v9r3 t. co'ege. yo' should
valid cases,of wh,om 555 have havc r,g00
. Cross_tabulate ..fi.*. degrees.
collegc with v4 (;;;;;rg
frame). Enter collegeas
row variable and v4 as the the
col,mn variabre(explanatory
Use the sameoprr.onsas above. variable).
B;;";. ro specify ..No rveight.,,

Question 5' (a) A higher p,roporti.cn.f

responclentsin the rclcphone
r' ii'ir**i;;;"i'crue
rosampring
error?
(use
sanr-

4 1;;ffn:ffix;'
(b) Give le"rt riu,,
"t
*llljT#:H"ffi'.1?,,:il.j:::i*:tri;llti?
rares, *.",,ffi ',i:h:',ili :::iJ"l "', dirr...;;.;;i,'.,0o,,,.
and/or
th.high.,;;r,."i.i;;;;;l;;.ffi i::,:Tili:J:X',?X:n":
the argumentsyou maclern
assignment2.

Question6' As noted in.thrschapter,you

can assessthe representatlveness
o f a samp lehv c om par ing y g y i
r;;r;i . i rr, * ,,r, c e n sus
hrgh-qualitydata) on key variables. data 1or .rher
e..orairrg.ro the U.S. C"nrurirr.ur,
about22'3 percenr
of tlir u's. fili;;;threen
possessed
collegedesreesin zdod, while vearsof ageor
'rder
Jout 67.4 percentofeligible
Americans
voredin the2000 pr.ria.nriri.i..,ion.

|x:ii"l,':rlX3tJffif*;: *' eii "'i'i'DD'.,,r,,tothe

census
ng-
(b)!'ere you surprisedth"t?e
RDD estimatesare farther
FTF estimates?Did vou exp-e*the FTF unJ off than the
noo estimates
to be closerto
;ff:i:?,T:';::il"* ih.y;;;;;,?il,ou ."0..,them
,obeeven
CHA P T E R 5 Third, the searchioi ji.
the soc ialscier r , , es,
.\. : .
is causing the rise in r:-..*
The Fifth Rule policymakersas u-ell: ,,,:.-
consequences rather tn...l l
COMPARE LIKE WITH LIKE haveto ols t hat can bq. , . . .
Fou r t h,I agr een ir ' - '
nati ve s r o t he st nnJ. r : - :-
methodsI desr , r il. ct l : ^ -
qui re p ar t icularr r pr , .
This chapter examines causal inference, an issue of rekindled interest
lately in the social sciences.The relevantliterature on causal analysisin cateds t ur isr icllr echnr l : . ,
nonexperimentalsocial researchspansstatistics,economics,political sci- rhat as social scient ist .. . .
ence,and sociology.This literature is far too extensiveto cover completely analysisin which \\'e ir-.:-:
gressi onequat ion.anu .
in a single chapter, and the flowering of recent work in this area makes it
hard to keep up (see"selected further reading" at the end of this chapter W e can do ber t er .. : - -
magic bullets for esttr:-;::
for a few useful references).It is possible,however,to spell out the funda-
a cardinalr ule: Cont p- ; - .. .
mental logic and principles of causal inference.That is my goal in this
the standard regressic,,^: :.
chapter.I
standar dr egr essioni. . : : , :
I want to stressfour points at the outset. First, the term causeis used in
The nexr secr ioner plr : : . . : :
a probabilistic sensewhen applied to human outcomes. In this chapter I
'W'hen for causalinference.
use the example of smoking and lung cancer: 'uvesay that srnoking
causes cancer, we mean that smoking increasesthe risk of contracting
lung cancer. We do not mean that all smokers contract lung cancer, nor
ConRrrerroN AND C.rr ..-
do we mean that smokers are the only ones who are susceptibleto lung
cancer,
The first law of causairi'.::
Second, the search for causal relationships is not the only reason
for doing social research. The objective of social research may be cl assesis "No caus. r r i, . :
correlation.But the re\-::r
descriptive-to get the facts right. Even if our ultimate objective is to esti-
i4147-((Csllslation do.. :-,
mate causal effects, accurate description is vital, since questions of wbat
w arned ;"Thou shalr ni) : r r :
conre before questions of why or how. Often half the battle involves de-
Many homespun €-\.iii t :
termining precisely what it is to be explained. Y/ith regard to rising in-
exampl e,a near lr per f t i: -
come inequality in the United States, for example, is the dilation of
and left lcgs.Yet u'c ri , .. *
incomes occurring primarily at the top, at the middle, or at the bottom of
causesthe lengrh of the l.::
the income distribution? Policymakerswant to know the answersto these
absenceof gray hair is ;.:-:.
and other questions, if only to fashion policies to ameliorate the conse-
observet hat wom en u i: ^ . :
quencesof the trends. If incomes are skyrocketing for the rich and stag-
women with gray hair.
nating for everyoneelse,that knowledge can be useful for fashioning tax
policies, even if we do not fully understand the reasonsfor the rising ine- The classictextbook ;r.,
quality. positive associationl-rer'.,. :::
In one ver siont her e i. . , _
rTo simplify the discussionin this chapter, I assumethat causal effects are the same Ior tw een the pr evalence, , : : ,
everyone.See $Tinship and Morgan (1999) and Moffitt (2005) for good oveniews of litera- bi rths i n villagesin sour : . . : -
ture that relaxes the constant-effectsassumotion. rvi th a n ewbor n ar e I r r c: ; : - :

-- -; oi rekindled interest
' - -. , In c r us al analy s i si n
'. :.!l nom ic s , polit ic al s c r-
::-.>:\'eto covercompletely
' :i i n t his ar ea m ak e si t
..: :ie end of this chapter
: :r. i0 spellout the funda-
. lr.rt is my goal in this
:. ::; term cduseis usedin
..:,,rme sI.n t his c hapte rI
: '.::: \\e s av t hat s m ok i n g
. -. ;1.'el'l-sftof contracting
nor
- :-L!ractlung cancer,
r-. -rresusceptibleto lung
- :- r: :! not t he only r ea s o n
: : .,,ci .1 1r es ear c hm aY b e
: ...:.::rlteob jectiveis to esti-
:', . .ince questionsof what
:- : : '-.r.: the battle involves de-
.; ,\ irh regard to rising in-
::.:.::tPle.is the dilation of
: : : :-..:Jle.or at the bottom of
.: -.'. i:1(r\\' the answersto these
- :: :(r amelioratethe conse-
,:.:i 1! for the rich and stag-
' '. .r..fu l f or f as hioningta x
': r:rSotlSfor the rising ine-
: .:: -.r,.rslleffects are the same for
-
. :or good overviews of litera-
The Fifth Rulc . 721
Third, the search for causal relationships is a reasonableenterprise in
the social sciences.As social scientistswe would like to understand what
is causing the rise in inequality, for example. Causes are important for
policymakers as well; otherwise, policies are resricted to addressingthe
consequencesrather than addressingthe causes.And social scientistsdcr
have tools that can be effecrive in investigating causal relationships.
Fourth, I agree with those who argue that it is rime to consider altcr-
natives to the standard regression approach to causal inference. The
methods I describe in this chapter are widely applicable. They may re-
quire particular types of data, but generally they do not require sophisti-
cated statistical techniques to execute. Their practical simplicity suggesrs
that as social scientistswe can strive to do better than business-as-usual
analysis in which we measure all the variables we can, put them in a re-
gression equation, and hope that we haven't omitted anything important.
rWe can do better, and we should
do bemer. Although there are no
magic bullets for estimating causal effects in obseruational data, tbere is
a cardinal rule: Compare like with like.It is time to employ alrernativesto
the standard regression approach to causal inference precisely because
standard regressionis a relatively blunt tool for comparing like with like.
The next section explains the logic behind the compare-like-with-like rule
for causal inference.
ConnErerroN AND Ceuserrry
The first law of causality that students are taught in introductory starisric
classesis "No causation without correlation." Causality, then, implies
correlation. But the reverseis not true, as students learn from the second
l4v7-"Q6r1slation does not prove causation." Thus students are sternly
warned: "Thou shalt not infer causation from correlation.,'
Many homespun examplescan be given of the secondlaw. There is, for
example, a nearly perfect correlarion between length of individuals' right
and left legs. Yet we would not conclude that the length of the right leg
causesthe length of the left leg, or vice versa.Nor would we infer that the
absenceof gray hair is causally related to having babies, even though we
observethat women without gray hair are more likely to have babies than
women with gray hair.
The classictextbook example of a noncausal correlation is a reported
positive associationbetween presenceof storks and presenceof babies.
In one version there is said to be a significant positive relationship be-
tween the prevalence of nesting storks and the prevalence of human
births in villages in southern Germany (the explanation being that houses
with a newborn are preferred by nesting storks becausethey tend to be
722 . Chapter5

warmer). It is hard to know whether or not such accounts are apoc-

ryphal, since they very often are inadequately documented (an excepuon
is Matthews [2000], who finds a highly significant correlation between
stork populations and human birth rates across Europe). Apocryphal or
not, the accounts illustrate the point that two variables could be corre-
lated becausethey share a cause rather than becauseone variable causes
the other.
Many other examples could be given as well, such as the positive sta-
tistical correlation between number of churches and number of bars in a
city. Before trying to determine whether it is bars that cause churches or
churchesthat causebars, it is important to note that larger ciries naturally
have both more bars and more churches.The point is that the correlation
between number of bars and number of churches in a city reflects a com-
mon cause, population of the city. Figure 5.1 represents this situation,
where P-+ Y means P causesY, P -->X means P causesX, and the ss are
error terms reflecting the other causesof X and Y. The absenceof an ar-
row from X to Y indicatesthat X doesnot cause Y (nor does Y causeX)-
bars do not causechurches, nor do churches cause bars.
By the phrase "X causesY" I mean that individuals (or firms or nations
or whatever you are studying) with higher levelsof X have different levels
of Y than they would have had otherwise (with lower levelsof X). This is
known as the counterfactual approach to causal inference (Morgan 2001;
Morgan and'$Tinship2007; Winship and Morgan 1999) becauseit concep-
tualizescausality in terms of a counterfactualsituation: Would Y differ for
individuals with X=1 if they instead had value X=0? In the languageof
experiments, the question is whether Y would differ for the treatment group
(X=1) had individuals in the treatment group been in the conrrol group
instead(X = 0).
Consider the issue of whether cigarette smoking causes lung cancer.
SThen we say that smoking causes cancer, we mean that smokers have
higher rates of lung cancer than if they had not smoked. Evidence dating

<- ex

Figure 5.1. Correlation of X and Y Is Due to Common

CauseP
 The FifrhRule . 121
from nearly a century ago
finds a correrationbetween
cer. In the New nlgjanilour"oi"i sm'king and can-
M;,;;,.ne in 1928, Drs. Herbcrt
bard and Carl Doenng report Lorn-
results fi

#1,:::T,:[ir1,-u,.h"].;;;;;i;;;;:iy,,]lrH,; ll.:"11;
th..nn,..g;;;;""#ilil:?*:","JrTii:tri,i,}fl *:*".,,x",:,il1
berweenthe cancer-ana
.""i."r gl3rir',"i,r-' regard
to thcir contactwith
,t['f,",*:::.ff:ilru1*l-m'.1..';;;;;',it".r'"can ca.ncerts spread by creatures
such as ratsithe ,";;;; ;;^^:::'^il|'rner,
theconstipa,,"" of
,',ijill llt:tt ,cancer)'
'rttit '*lii"iuntu., 1r,,,.r,
;,T,*i"Tff '_I!:'JJ';"fi::';ff ;,'i:l';ff
i'k::;::i::]
wasnot,t",i.ri."rif'r:;'J#fri:;f: sroup,b,t thedirference
on
rvinrerm
sor,'"okin rT""'
gu.r.,','
r,..r."ll"i'iT'iol
iT'lii'iilfiil,Xl
udethat" h""; ;;;'i' -.,n..,,,,
J.tfl,j;.:'"cl somererario,,o
"1i.,
A secondlandma_rkstudy
on the.possibledang.erof
a decaderater in science. smoking appeared
R"y-o.,d'p-.-""ri a medicar
Hopkins, had carefully _ri*lr.J statisticiJn at
h;;il records of hundreds Johns
lies in the Baltimore area. of fami_
Unlike the retrospective
study, the pearl study ."", L.mbaril-Doering
p.orf..ii"",'foffo*ing
time (Pearl 193S). Thus pearj 6,g13 subjecrsover
;;;;;lr'
jlfit'ftIl:.*l:
$li;:lll*l"ll.,,rsmokers,-r;;;::[Tu':,i
smokers r'"ar"*"'i:ffi'Tl*:iill?l*""1- 'r'i'ii" 'iI""rnr"""'
had lower survival rates smokers,.who rurn
rhan nonsmokers]::*a.
r-uk.r,li,,.d
o"rrrr"'rir,, ;-o"lxT* ;:*ll ffiT:
,"jff:lJ,.
The pearrstudy was folrowed
by two rnuchlargerprospective
of- smokingand lune t:n:..., stuiries
onein Englandand o'e in
In Engrand'Richarl the U'itecJsrares.
iJ a'ratTa'itt, sentquesrionnaires
9^"1
smokinghabitsto all 6Q000 about
ing repliesfrom about ,he medicarprofession, recerv_
-.-1..r'"r
10,0.09.ii;;#;data from tire British,"g,r.ro-
:ff ffif;'j;,.*:, f,,eddearh,,;.,
;;;^ r""g.n;._.#',_,,i1^,
"o
*-i*l
mond and Daniel Ho* fi #rTi;*?lr.:: **#;:r unrted st-ates,cuyler Ham-
so-G
eo,,-r*'
-*:,:i..ll-?Tr
i",r,:il #'iffi',ili :'ffi41.;
..ti",-^,:;!^':;_:',:

; l,**:: ;:l;:;
reportedresurts
vervsim'ar ,o ,rr. o"iiiitiir-,rrarn*r,
over thenextfour
jfn r"lrg.,*",'* ^,'r.,.*,i,*, r,igrl..
f';i: jltif ;:.T:., ro..i.
"u,
f24 . Chapter5

With the mounring evidenceof a large and robust statisticalcorrelation

between smoking and cancer, in 1,952 the Reader's Digest published
"Cancer by the Carton," an influential article warning of the dangers of
smoking. As other periodicals followed suit and the public began to ques-
tion whether smoking is as benign as previously thought, tobacco compa-
nies responded with "A Frank Statement to Cigarette Smokers" that
appeared in the Neerr York Times and over four hundred other newspa-
per s on J a n u a ry4 ,1 9 5 4 (s e eb o x 5 .1 ).

Box 5.1
A Frank Statementto CigaretteSmokers

As it appearedin the New York Times and

over 400 other newspaperson January4,1954

Recent reports on experimentswith mice have given wide public-

ity to a theory that cigarette smoking is in some way linked with
Iung cancer in human beings.
Although conducted by doctors of professional standing, rhese
experimentsare not regardedas conclusivein the field of cancer re-
search.Horvever,we do not believethat any seriousmedical research,
even though its results are inconclusive, should be disregarded or
lighdy dismissed.
At the same time, we feel it is in the public interest to call atten-
tion to the fact that eminent doctors and research scientists have
publicly questioned the claimed significanceof these experiments.
Dis ti n g u i s h e d
a u th o ri ti e sp o i n t o ut:
1. That medical researchof recent years indicatesmany possible
causesof lung cancer.
2. That there is no agreement among the authorities regarding
what the causeis.
3. That there is no proof that cigaretre smoking is one of the
causes.
4. That statistics purporring to link cigaretre smoking with the
diseasecould apply with equal force to any one of many other
aspectsof modern life. Indeedthe validity of the statisticsthem-
selvesis questioned by numerous scientists.
rWeaccept an interest in people's
heath as a basic responsibility,
paramount to every other consideration in our business.

(continued on next page)


ir statisticalcorrelation
.;:!er's Digesr published
...-rrningof the dangersof
: pearing in today's leading media, evenwere such advertising allowed.
'- - :1i public beganto ques-
! :.rolrsht,tobacco compa-
i r {Jfett€ Smokers" that
,.: lur-rdred other newsPa-
-< Ttntes and
:--::ri ' 4, 1954
:,rr e given wide public-
:ome way linked with
:::!roflx1 standing,these
,^.:he field of cancer re-
i::rous medicalresearch,
-. ,:ld be disregarded or
'..: interestto call atten-
:-.earch scientistshave
, ,i theseerperiments.
rdicatesmany possible
.luthorities regarding
is one of the
:.:-1:.rre smoking with the
,: :, i .1n\-one of many other
-,...::n.of the statisticsthem-
i -: i :tiis t s .
basic responsibility,
business.
tcontinued on next pdge)
The Fifth Rule . 725
In light of what we know now about the close link between srnoking
and lung cancer,it might be hard to imagine such a "frank statement" ap-
'We
must remember that, a half century ago, the case against smoking was
based almost entirely on statistical associationsthat were open to alterna
tive interpretations. Hence tobacco companies were not out of line in
(continuedfrom ltreuiouspage)
'We
believethe products we make are not injurious to health.
\7e always have and always will cooperate closely with those
whose task it is to safeguard the public health. For more than 300
years tobacco has given solace, relaxation, and enjoyment to
mankind. At one time or another during those years critics have
held it responsible for practically every disease of the human
body. One by one these chargeshave been abandonedfor lack of
evidence.
Regardless for the record of the past, the fact that cigarette
smoking today should even be suspectedas a causeof a serious dis-
easeis a matter of deep concern to us.
Many people have asked us what we are doing to meet the pub-
lic's concern arousedby the recent reports. Here is the answer:
'We
1. are pledging aid and assistanceto the research effort into
all phasesof tobacco use and health. This joint financial aid
will of course be in addition to what is already being con-
tributed by individual companies.
2. For this purpose we are establishinga joint industry group con-
sisting initially of the undersigned.This group will be known as
Toeecco INoustny RsseeRcH CoumrrtrE.
3. In charge of the researchactivities of the Committee will be a
scientist of unimpeachable integrity and national repute. In
addition there will be an Advisory Board of scientistsdisinter-
ested in the cigarette industry. A group of distinguished men
from medicine, science,and education will be invited to serve
on this Board. These scientistswill advise the Committee on
its researchactivities.
This statementis being issued becausewe believethe people are en-
titled to know where we stand on this matter and what we intend to
do about it.
(continued on next page)
126 ' Chapter5

1954 to note that "there is no proof that smoking is one of the causes[of
lung canccr]." As late as 1.959the eminent and colorful statisticianSir
Ronald A. Fisher published a book emphasizing that, in the absenceof
randomized experimentswith human subjects,the correlation of smoking

(continuedfrom preuiouspage)

Trrs AnsnrcaN Toeacco CoupeNv, INc.

Paul M. Hahn, President
BuRrr,yToeecco Gnowtns CoopERerrvEAssocrRrroN
John \W. Jones,President
Pnrrrrp Monnrs sr Co. Lro., INc.
O. ParkerMcComas, President

BENsoN& HEDGES
JosephF. Cullman, Jr., President
Lenus & BRoTHERCouneNl INc.
\7. T. Reed,Jr., President

R. J. RTvNoLDSToBACCoConpeNy
E. A. Darr, President

Bnrcnr BErr \TenenousE AssocrATroN

F. S. Royster,President
P. Lonrrreno Cotrpeny
Herbert A. Kent, Chairman

SrnpueNo Bnorunns, INc.

C. S. Stephano,D'Sc., Director of Research

BnowN g WrrrrenasoN Toeecco Conponer.r<tN

Timothy V. Hartnett, President

MenryeNo Tosecco GRowEns AssocretroN

SamuelC. Linton, GeneralManager
Toeecco Assocrarr,s, INc.
(An organization of free-owned tobacco growers)
J . B . H u ts o n , P re s i d e n r.
BuRrey AucrroN rWRnsnousl, As socretrou
Albert Clay, President

UNrrEn Srerns Toeecco CoupeNy

J. X7. Peterson,President
 The Fifth Rule . 1ll
and cancerdoesnot prove that smoking
causeslung cancer(Fisher1959,
cit e d i n Bro wn r 972) . F is h e r' ,e _ s s e ,,ri a r" a rH u rre n t
rsc a p rrrrcdi rr ri pi trr.e
5. I
above:Some hereditaryfactor p might prcdisp.sc
some individuarsborh
to smoke and to c()ntractlung .a'.Jr.
of course,the searchlbr p (herecJ-
itarv predisp.sitio^ toward smoking nna
tung.rn.;i;;;;;;"d erusiue
in the decadessince Fisher, .rori almost all agreethat the evidence
is
that smoki'g "nd in fact is a causal fr.tn.;tr[ ;;il to tung
:;';::nst-trg
consider again the counterfacruarapproach
ro causarinf'ercncein Iighr
of the smoking-cancerconrroversy.
whar we really want to kn'w, f'r
causalitl', is lvhether smokers would havc
had rower canccr r"i", t.,na,n"y
not sm.ked (and whether nonsmokers
would h"u. h;; ;i;;; cancer
rateshad they smoked).Again, the
counrerractualapproach is probabilis-
tic: It doe.snot say that if you smoke,
you will lrrrg ;"r.;;: ;;; does it
rule out lu'g cancer among nonsrnokers. S.,
Rat'er, to say trrat srr'ki.g
causeslung canceris t'say that smoking
increasesthe risk of lung sancer!
so individualswho smoke are in greare.le.,p^rdy
,,r.nnr.n.rinf ri,rg ."n-
cer rhan if they had not smoked.
To take another example,-ifcollegeeducatio'has
a positrvecausaref-
fect on earnings,then.thecoilege-ed'.c"r.d
hou" higher'l"velsuiJrrn,ng,
than they would havehad if they were nor
c'llege-educated.Note the dif-
ferenceberweenthis sratementand the
finding thlt th... i, u-po.r"riir..or."_
lation between college education nr-rd
.".ningr. In the correration berween
college and earnings we "college_
co_mparingi-he e"rnings .i ;;
.are
educatedwith the non-college-.du.nt.dl"s
opposed to comparing the
earnings of the college-educated with their earningsif they had not
tended college. at-
Thetwo comparisons
wourdu.,rr. in*.-""ir'iii".
assume that the non-college-educated ."rra
earn the same afirount as the
college-educated would have had ,h.y noi un.nded colrege.
reasonableassumption unless (as That is not a
describedbelow) individuals arc ran_
domlv assignedto colrege, so that
tt-t. in,tirt differences between those
wh3.So to collegeand those who do .ro,
on average,zero.
The fundamental concept here_that ".., -do",
correlation not prove
causality-holds the key to understanding
why it i, i*;;;;";;'ro .u__
pare like with like. Br:iefly,the logic
is thii ln social researchwe do not
directly observecausesat work, b"ut
only th. ..rult, of the causes,that is,
r As I write
these rvords, the longstanding controversy
over the fluoridation of tap warer
has bee. reignited by Harvard profi*o,
cn.rr.. o..,gi"ss,s resrimony to the
scarch council that there is no significan, National Re-
r,nt rr.,*"L fruoride and osreosarcoma,
but deadly form of bone cancer.According,o a rare
rh. h"nJro*icologist f.r a \x/ashington-bascd
envrronmenral group, ..His conclusion
th"i th".e is no link i, ii.,.
a ne t hi cs c omplain t a g a i n s t l ) o u g l a s s( r e p o r r e d lu ,tr .t.p *- n "r 2" g ,2 0 0rrJrf_,.'r..r.p.i",
Sissueof fif"a
uard Crimson, online edition, p. I TheH ar-
).
128 . Chapter5

we observe correlations among variables. To infer causality from those

correlations, we must determine in general why it is that correlations do
not reflect causality, so that we can know when a correlation between
two variables would reflect a causal relationship between the two van,
ables. Put simply, correlations do not reflect causality in observational
data becauseindiuiduals are not assignedrandomly to cduses.If individ-
uals were assignedrandomly to causesin the real world, then our work
as social scientistswould be easierbecausewe would be much closer ro
comparing like with like. It would be much easier,for example, ro deter-
mine the causal effect of a college education on earnings if individuals
had no choice in the matter but were randomly assigned,some to attend
college, some not. By removing individual choice from college atten-
dance,random assignmentto collegewould reducenot only the problem
of obvious initial differences(suchas socioeconomicdifferencesberween
those who attend college and those who do nor) but also the problem of
more subtle differences,such as selection-on-Yu'here individuals self-
select college in part on the basis of how much they are likely to benefit
from it (resulting in what is known as selection bids in observational
s t udies ) .
That is why random assignmenris universally used in controlled ex-
periments with human subjects.Random assignmentis designedro com-
pare like with like-we expect rhe treatment and control groups to differ
only on the treatment X-so with random assignmentto X we can more
readily demonstrate causality from correlation. In other words, random
assignmentis the tool social researchersuse in controlled experiments to
try to thwart the second law of causality ("correlation does not prove
causation").
Suppose,for example, that we have all power and no scruples,and we
randomly assignedeveryoneeither to a smoking group or to a nonsmok-
ing group. No one is allowed to smoke before age sixreen. On your six-
teenth birthday some random method is used to determine whether or nor
you will smoke. If you are placed in the smoking group you must begrn to
smoke one pack of cigaretteseach day whether you wanr ro or not; like-
wise, those in the nonsmoking group are prohibited from smoking,
whether they want to or not. (\Wecould of course vary the level arrrong
smokers-some are required to smoke one-half pack each da% some
smoke one pack, some smoke two packs, etc.-on the erpectation that
heavier smokers are more likely to contract lung cancer than lighter
smokers. The samelogic holds. For simplicity, we focus on rhe two-group
case.)If we find over a long period of time that the rates of lung cancer are
higher for smokers than for nonsmokers, we would claim that smoking
'We
indeed causeslung cancer. would base that claim on the assumption
that the cancer rate for the nonsmoking group is a reliable measure of

:: . . .:rusality from those
:": -. I l: rhirt correlationsdo
- :.. : correlation between
: , : :-I\\ ' c en t he r wo v a ri -
r obs er v at i o n a l
t : - : . . ,.:-..:,'. litin
: , ' . . "':.'. r( ) ( aus es , I f indi v i d -
- i - - ':.1 ',1ririd, then our work
-- - - : i 'rLd be m uc h c lose rto
- -- - : ,- ::. ior examPle'to deter-
-- :: - : if indiv id u a l s
i-: ':- '-:)lgnedr -rrningssome to attend
:-, ,: ri e fr om c ollege at te n -
'- : r : :,.-: l l o t onlY t he P r ob l e m
,- :- ::1. differencesbetween
: . -'ut also the Problem of
: - - - : '* here individuals self-
:- -:- ;. ': L,i;tsin observational
- .,'- -. .:-.:ntis designedto com-
: .- :-': ..-.J;,rnt r ol gr oupst o d i ffe r
t - ::- t o x we c anm o re
-:l :l l ent
-: .' :-. Ii-i other wordst random
- :rrrolledex P er im e n tsto
,: -. , ::elation does not Prove
: . , :: ,..ndno scruples,and we
i :::' , .-.: irouP Or tO a nOnSmOk-
: r:r r: -lge sixteen.on your stx-
: , -: - i ieterrnine whether or not
: . . :-: SroupYou must begin to
.; :::-.:: \ ou want to or not; like-
-: .. . : ::,,h i b it ed f r om s m o k i n g ,
,-:rscvary the level among
- : : ,
:: :'.-11fPack each daY, some
'.:
i-r -. : i-.-o n t he ex P ec t at io nth a t
lighter
- :::-:,: iung cancer than
r ..:'.. s e focus on the two-group
: ^ . :: :he ra t esof lung c anc e ra re
-:
:r:. I : r"'ould claim that smoking
-r , : r' i : Jl ri m on t he as s um p ti o n
:: . --: is a reliable measureof
The Fifth Rule . I29
what the cancer rate would have been for the smoking group had they not
smoked. That assumption is plausible since by random assignment we
have alleviated the effects both of selectionbias3and of initial differences
between smokers and nonsmokers that might account for subsequentdif-
ferencein cancer rates.
To summarize: The problem in estimating causal effects is that we can-
'$fhat we
not simultaneouslyobserveX and not-X for a given individual.
would like to have is a reverse-Xuniversein which, for example, smokers
in our universe are nonsmokers in the other universe. In the absenceof
such a universe we attempt to simulate it. The most obvious simulation
method is the controlled experiment, where the researcherrandomly as-
signs subjectsto treatment and control groups. But there are other meth-
ods, and the remainder of this chapter focuseson them. The idea in each
method is to compare like with like.
TVpssop Srnerr,crns FoR CoMpARINGLIKE wITH LIKE
The oblective in causal analysis is to estimate a causal effect from an ob-
served correlation. To estimate a causal effect from an observed correla-
tion, we must compare like with like to rule out other explanations for a
correlation between X and Y:
1. Omitted-variables: Individuals' differenceson X are related to other
causesof Y.
2. Selection:Individuals selectX (or others selectX for them, as in the
caseof tracking in schools) on the basis of expectations about Y.
In the classic experimental design we avoid bias, first, by random
selection of units before the treatment X (to avoid selection bias) and,
second, by random assignment of units into the treatment (to avoid
'We
omitted-variables bias). also would like a large N, to guarantec that
our chances of sampling error are vanishingly small. \fhat we want, in
brief, is a researchdesign that makes indiuiduals in treatment and control
groups indistinguishable from each other except for (a) tredtment status
and (b) completely random shocks to the outcome uariable Y.
So there are three key ingredients in causal analysis:random selection
of units, random assignmentof units (to a purported cause), and sample
size. Chapters 1 and 4 addressedthe issue of random selection of units,
3Selectionbias here rvould mean that individuals basetheir decision to smoke in part on
their actual propensity to contract lung cancer, u.hich seemsfarfetched. In many other in-
stances,however, selection bias is very likely in observational data, as in the case of college
attendance(choosing to go to college is based in part on how likely you are to benefit from
co lle se).
130 . Chapter5

and the sample size issue is addressedin those chaprers and elsewherein
the book. In this chapterI focus on avoiding omitted-variablesbias through
the random assignmentof units, or through some alternative strategy that
mimics the effect of random assignment.Randomization is the means to
an end; the end here is the comparison of like with like. Once the subjects
are selected,how do we craft our study to try to make individuals in the
treatment and control groups indistinguishable except for treatment sta-
tus and random shocks to Y? The most common method is regression.
Regressionis a parametric approach, meaning that it tries to compare like
with like by using estimated paramerersto adjust for differencesbetween
treated and nontreated individuals.
Although regression aims, in a very general sense,to "match" on the
confounding variables,other methods do so more explicitly by the pairing
of similar units. The aim is to compare individuals in the treatment and
control group who have the same or very similar values on confounding
variables. One way to match individuals is on the basis of measuredvari-
ables,using either eract matching or matching with propensity scores(de-
scribed later). A second method is to match a unit on itself through
multiple observations (for example, two or more observations per indi-
vidual for investigationsof causal individual-levei effects, or observations
on two or more family membersfor causal family effects).For this match-
ing method, unlike the first method, we do not need to measure all the
causal variables in order to compare like with like.
A third way to match is through random assignmenrto the treatment
and control groups. Although random assignmentis not always included
in discussions of matching methods, the matching or "balancing" of
treatment and control groups is precisely what random assignment aims
to do. Indeed, random assignmenris a particularly attractive method for
matching becauseit harnessesthe power of probability in the service of
matching. To be sure, as researcherswe are not always able to randomly
assignunits. Nonetheless,as we will seesubsequently,it is sometimespos-
sible to exploit naturally occurring random assignment.

MercnrNc vERSUS
LoorrNc ron DrppnRr,NcEs

Before we consider various matching methods, it is important to note

that, depending on our objective, we might not want to match casesin
observational studies. If our objective is to examine a range of potential
causes'then we want to compare individuals who are dissimilar, not sim-
ilar, on those hypothesized causes.After we identify likely causesin this
manner, then we estimare the effect of a cause using matching methods.
Until we discoverthe likely causes,however,we generally want to resrricr
the matching to nonmanipulable variables such as age and sex.

i l r: lh.rprersand elsewherein
..r": -..ned-r'ariables biasthrough
,-:.-. : .le alternative strategy that
is the means to
:, i..:-.;,rtt.iizatien
: r : ...rrh like. Once the subjects
: ::., :() make individuals in the
, r: :1 . a\cept for treatment sta-
-: - - 'r,)n me r hor lis r egr es s i o n ,
:rl r-r :irar it tries to comparelike
- .,; .l:t for differencesbetween
i r :: : - t - se nseto, " m at c h" on th e
: : ,re explicitly by the pairing
, :.-: .,:iu.rls in the treatment and
: . :. - : -rl -rrra l u es on c onf oun d i n g
: .: :. :he basisof measuredvari-
:ir :'i ',r'ithpropensityscores(de-
-.:-.: .r unit on itself through
: r'rrrr observationsper indi-
:*,,.-.:i. cl effects,or observations
.... :.. :- -r' cffects).For this match-
: i :- ,r need to measureall the
: \ -- Ng ,
' : ' -.-..ignmentto the treatment
.i . i i: r -n t i s not alway sinc lu d e d
: - : : - -.-::h i n go r " balanc ing " o f
- , : :.rndomas s ignm enta i m s
::.: --...rrll attractivemethod for
:: : ::,rb.rbility in the serviceof
: ::: : ',i irlrvaysable to randomly
-- . ^-: --l sntl r. i t is s om er im esp o s -
: :: -:s!lqnment.
:r:::. ,Js. it is important to note
:.::.: not want to match casesin
: : :\.lI-nine a range of potential
-.. , - ri h o are dis s im ilar ,not s i m -
:: .'.; iJentify likely causesin this
-i -'.ir-i:' using matching methods.
:. J: . ( gener allywant t o r es tri c t
"\
: - -. : . :r .1\a geand s ex .
The Fifth Rulc . l.l I
Consider again the 1928 Lombard-Doering study of the determinanrs
of cancer. That study compared a control group of 217 individuals
without cancer with 277 cancer patients, where the two groups werc
matched on the basis of age and sex. Note that, contrary to the standard
experiment, the "control group" in this caseis a category of the outcome
variable Y, not the treatment variable T. Moreover-and more to thE
point about looking for differences-the cancer and control groups were
not matched on the basis of hypothesizedcausesof cancer.At that trmc it
was thought that cancer might be linked to constipation, carried by ver-
min, or spread from victim to victim. If those theories are correct, we
would expect to find differences between the two groups with regard to
laxativeuse, housingconditions,and contact with cancervictims, respec-
tively. To bring data to bear on the theories, then, we would not want to
remove the differencesa priori by matching on those characteristics.
The overarching point here is that we need to understand whether our
obiective is to discover the causesof an effect or to estimate the effect of
a cause.If we want to know if T, U, and V are likely causesof some out-
come, then we cannot match on T, U, and V. To locate the possiblecauses
of lung cancer, for example, we want to know how lung cancer victims
differ from those who do not have lung cancer.This harks back to rule 2,
"Look for differencesthat make a difference," as well as to the first law
of causality, "No causation without correlation." In exploratory studies,
we do not want to match on potential causes.
If, on the other hand, our aim is to estimate the effect of a cause,then
we want to match or "compare like with like" to eliminate the confound-
ing effects of other causes.In the social sciences,regressionanalysis is by
far the most common strategy for eliminating the effects of other causes.
'We
begin, then, r,vith the regressionstrategy.
THn SreNoaRDREGRESSToN
METHoD FoR CoupenrNc
Lrr<nwrtn Lrrs
The standard regression method attempts to compare like with like by
including the causesof Y as independentvariablesin a regressionmodel.
Imagine, for example, that Y has two causes,X and Z. To estimate the
causal effect of X on Y, we regressY on X and Z. AlternativelS we could
estimate the causal effect of X on Y in two steps: First regress Y on Z
and X on Z to create i and X, respectively (values of Y and X for indi-
v i d u al i , predi ctedon the basi sof Z): rhen regressY ,-Y ,on X ,-X, . Be-
cause the residuals Y,-Y, and X,-*, are unconelated with Z, ihe effect
of X,-*, on Y,-t, ir' noi due to Z. in other words, linear regiessionhas
l.een used as a tool to, in effect, "match" X and Y on Z. By including Z
,:-sa regressor, we attempt to compare like with like on Z through
132 . Chapter5

regression-basedstatistical adjustment (as opposed, for example, to a

strategy that relies on random assignmentto X to remove the confound-
ing effect of Z).
Now imagine that Y has many causes)as is generally the casein social
research. Consider the standard linear regressionmodel in the social sci-
ences,with measured causal variables denoted by X, unmeasured causal
variables denoted by W, and random disturbancesdenoted by e:
Y,=a+ a,+ pXr+ yW1 a, ( s.i)
Y is the value of the outcome variable for the ith unit (individual, coun-
try, etc.), and eis a random error term that has a mean of zero and is
uncorrelated with the Xs and V/s. The rerm d is an overall constant and
ai, a constant for the ith individual, is a term that captures differences
between individuals that are stable over time and not otherwise ac-
counted for by the Xs and V/s. (Although a and e, are conflated in
cross-sectiondata, they are distinguishablewith panel data, as we see
subsequently.)A bolded Greek letter denotes a row vector of parame-
ters and a bolded Roman letter denotesa column vectors of variables.
T hus B X , is a s h o rth a n d w a y to w ri te FrX r,+ B rX r,+ ... .+ B pxpi ,
where P is the number of measuredcausal variables,and yW, is short-
hand f or 7 ,.W ,.,+ fz Wz ,+ ' ...+ y o Wo , w here Q i s the number of
-v a rl a b l e s . Y Y'
unm eas ur ed ' c a u s a l
The key feature of this model is the distinction between measured
causes,the Xs, and unmeasuredcauses,the Ws. The model is very gener-
ally applicable to regressionwith observational data since we rarely can
measure all the causesof Y. Almost invariably, then, vou cannot estimare
equation 5.1, and you might resort to estimating the truncated model:

Y,=So+QrXr,+Q2Xr,+. . . '+ QrXo,+u, (s.2)

Equation 5.2 representsstandardlinear regression(SLR). The idea is
to estimatethe causesof Y by trying to include as many of those causesas
possible in the model. SLR is the canonical method in mosr of the social
sciences;its use is ubiquitous in nonexperimental social research.

Critique of the Standard Linear Regression Strategy

The problems with the standard linear regressionstrategyare well known.

\7e rarely know all the causesof Y, and, evenwhen we do, we might not
be able to measure all of them. By regressing Y on the measured causes
(the Xs) while ignoring the unmeasured Ws, we in effect estimate rhe
wrong coefficients.Instead of estimating Fp the causal effect of X, on y,
we estimate dr; instead of Brwe estimate dr; instead of B, we estimate @,;
and so on. Using a regression method that yields unbiased estimatesof

:: 'red. for example, to a
.,. :., removethe confound-
: r:rerallv the casein social
: - :. -\, unm eas ur edc a u s a l
. : ..,: s Jcnot edby a,
( s.1 )
: - : : : :::. .tn it ( indiv idual,c o u n -
- :-,,.: .-.-::,-imean of zeto and is
. .rn overall constant and
.tf J apt ur esdif f er e n c e s
: :rnd not otherwise ac-
-:.. , -::1da, are conflated in
. : ,..::1p :r neldat a, aS we s e e
- i:. -:-ro\\- \'ector of parame-
- - -.:::i lectors of variables.
. -: 'r - B,X r i+ ' ' ' ' + F p X p p
. ,,: .:i l es , and y W , is s h o rt-
' -::: Q is t he num be r o f
: -:.: i:lon between measured
'.1...
The model is very gener-
- succeededas engines for discovering causal relationships [in social re-
:: .:. i:ta sincewe rarely can
- . : . : . n. \ OU Ca nn ot estim at e
:: :-: :he truncated model:
/s 7)
:. : =: : : . :.ssi o n ( S LR) .T he ide a i s
-.- ,.:: ,:-imanv of thosecausesas
:. - : : :ri o d i n m os t of t he s o c i a l
:r: : ::-: -: social reseafch.
-,ri
-<51 if7-7fgg1r
':ir::: 1 Striltegy are well knOwn.
-. :,:r '.i hen u'e do, we might not
::: : i . r : )' on the mea su red caus es
':: '.1.:. rie in effect estimate the
: - . ::.i c.tusal effect of X, on Y,
i: r.; ,.-.:lr.1dof B, we estimate d3;
::.:: '.:-iJs unbiasedestimates
of
The Fifth Rule . l.lf
the @sdoes not help us much, since the gls and the ps generally differ
(dr* B, etc.)when the Xs and V/s are correlated.
So when the measured and unmeasured causesof Y are correlated, as
is almost always the case with observational data, the SLR approach is
problematic. The standard practice of regressingY on its measuredcauses
is, we say, subject to omitted-variablesbias, This is all very familiar to re-
searchers:We know that we are all too often estimating truncated models
such as model 5.2, and we know that this isn't ideal since our estimates
are subject to omitted-variables bias when the Xs and Viisare correlated.
The SLR model continues to dominate many fields of social sciencede-
spite classic warnings about its limitations. Notable critics of standard
regression analyses in social research include econometrician Edward
Leamer (1983), who writes about taking the "con" out of "economet-
rics," and sociologistStanleyLieberson(1985, chap.2),who warns about
the dangers of the usual control variable approach in regressionmodels.
As Lieberson notes, adding control variables willy-nilly or ritualistically,
with little thought given to how their addition changesthe interpretation
of other variablesin the model, can have disastrousconsequences.
Some observersgo further in their criticism of standard regression in
social research. Statistician David Freedman, for example, argues that
regression-though useful for summarizing linear associations among
variables-is ill-suited for finding causal relationships in social research,
no matter how much we dress up the method. He writes: "I see no cases
in which regressionequations, let alone the more complex methods, have
search]. . . . As I read the record, correlational methods have not deliv-
ered the goods. nfle need to work on measurement,design, theory. Fancier
statisticsare not likely to help much" (1997, pp. 1I4, 757).
I agree about the "need to work on measurement,design, theory" (the
theme of chapter 7),but not with the implication rhar we would do better
to abandon regression methods altogether in causal analysis. The prob-
lem lies not in regression itself, but in reliance on regression control
(residualization) as the primary or only method for matching like with
like. The crux of the matter is what to do about the "W's problem" in
equation 5.1. There are two general approaches or grand strategiesfor
trying to minimize omitted-variablesbias:
1. Add-a-regressor approach: Move variables from W to X, that is,
measuremore of the causal variables.
2. Design approach: Reduce the correlation between the V/s and Xs
through researchdesign.
Until recently observational research has relied most heavily on the
add-a-regressorapproach. In this chapter I suggest that we follow the
734 . Chapter5

lead of recent statistical and econometric work and try something else:
ti rne.Th e subscr ipt/ is r : : - : : -
Let's focus our attention on the creation of research designsthat reduce first term in the panel rirr:::
or eliminate the correlation of the Xs with the Ws. On this I think Freed- measurementpoints rha: .:: -
man would agree. Yet, as I show in the examples that follow, regression rviseascountedfor br ti.; .'.r ,
may still play an integral role in causal analysesin the social sciences, is not needcdfor cross-ir:: . i
since research designs developed to eliminate the correlation of the Xs ro Y , X a nd W, t hc p; r ,
with the Ws very often still rely on regressionto estimarethe key causal pendentvar iablest o cha. i: ', : '
parameters. Importantly, though, this more tailored use of regression rc B and 7t he m odel ai- , '. '. : - .
analysis plays to a strength of regression-summarizing the information time-
in data-rather than to a weakness-matching like with like. To i l l u st r at ct hc hr ir . ; - ' - -
A major shortcoming of the conventional add-a-regressorapproach is ' *
X s at two point s in r ir r ';
that we never know how successfulwe have been in identifying ancrmea- . - : -
i 5.3) w e can wr it e our : : . -
suring all of the causes,or at least most of the important ones. For causal
time 2 as follows:
analysis, then, we would prefer the second option above, the design ap-
proach. Random assignmentto X is one such design method. Yet random Yl = ar + a- + B, X, , - , r , .\ \
assignmentis not the only possibiliry and, when random assignmentto X
Y,r= arl a,l p.X,. - ,r\\ , -
is not feasible-as is frequently the casein anthropology, economics, po-
litical science,and sociology-social researchersare prone to scurry back The critical advantag.- .. ::
too quickly to the comfortable add-a-regressorapproach. A central mes- individual changeover :r:-: - ,
sageof this chapter is that we need to think more creatively about how to from Y,r:
design our researchto compare like with like when classicrandom assign- Y ,r-Yu=@z- a, ) - P. \ . - p
m ent is out o f th e q u e s ti o n .
=( ( r z- or t - B. \ _ - )
Social researchershave at their disposal a number of strategiesthat,
whether they realize it or not, are rooted in rule 5. 'We begin with the +02- 7r \ \ ' -
most obvious: Compare individuals with themselves. E qua r ion5. 4 st at ei r r . : : - - . -
or), bY change in ) "s : : - . . . , . - - .
W ,r-W ,r . r espect i\ el\. : -
Collpenruc Llxe wrrrr Lrrn tnnoucH Flxno-ETFECTS
METHoDS unmeas ur edcausest F. - F . '
term (e,r - qr ) . Fr or r rce; . . :
First-Difference Models: Swbtracting Out the Effects of the W s disappearin t hr - - . - - :
ConfoundingVariables
r the Ws ar e consr r r - - : l
The first-difference method compares like with like by matching individ- c rhe pffcrtc of rh- \\ . -. -
uals to themselves.To match individuals to themselves,we musr have
measuresof individuals at two or more points in time, that is, we must l n shor t ,a sim pleiir - : - - - - . - .
have panel data (more on panel models in chapter 5). Consider the elabo- confounding effectsot ..--...:.., -
ration of the standard regressionmodel to permit measuresof individual ables that are const2lnt: ' -, :
units (people,corporations, nations, etc.) at multiple time points: effects over time. Thi. :. ,. :
unmeasuredcausesver\ i: :-
Y i, = a, + d i + F } i r+ y fl ,r+ e o (s.3
) or approxim at elvso. \ , ', : : : - :
- .- : : ' -
i ncl uded if f icult - t o- r nr . i)
Note that the panel form of the regression model adds the subscript I
work ethic, and sense(-\:- .- j
(for "time") to each term in the linear regressionmodel (5.1) excepr a,,
cri mi nal just ice.cr r m in. - - : -
which needs no subscript / becauseit reflects effects that are stable over individual differencesr:r :: -': -
 -l'heFifrh Rrrle . 1-l-i

tirne. The subscripr t is neededto capture possrbrecrra'gc

over fime. Thc
first term in the panel model, or, .npr.'.., diffcrenccs
.regression bctwcen
measurementpoints that are the same for all individuals
and not other_
wise accountedfor by the Xs and ws, or "secular change." (The
a, rerm
is not neededfor cross-sectiondata, ofcourse.) By adding the
subsc'pt /
to Y, X and w, the panel model allows values of the independcnt
and de-
pendentvariablesto changeover time. Similarly,by adding the
subscriptr
tcr B and 7 the model allows the effects of rhe Xs ancl wJr.
chanse .ver
t r me .
To illustratethe first-differencem'del, supposewe rleasure y
ard the
Xs at two points in time. Using the notarion .f the general
pn*i rr_,ud.l
(5'3) we can write out the separateregression.quutinn,
foitime 1 and
time 2 as follows:

Yi t= dt+ a. + B r X . r + ! 1W ,r+ e ,,

Y,r= a.r a,+ BrX,r+ TzW;z*siz (s . 3 )a

The critical advantageof a panel data set is that it allows us
ro lneasure
individual changeover time. Let's exploit that fezrtureby subtracting y,
fro m Y,:

Y,.- Y,r= ( o z- a,) + (F

rX,, - B tX i1)+ ( yzw iz- z, Wi, ) + (e,.- e,)
= ( d: - a, ) + B r ( X ,r-x i t)+ (p 2 -
B,)X ,, + y 2 (w ,2_w ,r)
+ ( / z - y r ) W - r+ (e ,r-e ,,) (.t.4)
Equation 5.4 statesthat changein yis determinedby a constant
\az_
a , ), bI ch ange in Y ' s m e a s u re da n d u n me a s u re d .u u r ., (X ,r_X ,
and
W,r-W,,, respectively), by change in the effects of y,s
mellured and
unmeasuredcauses(P2- B1 and yr-
T1), and by a random disturbance
,ltT_-!",..-e,,). From equation 5.4 *e seeimmediately
that the effectsof
the Ws disappearin the first-difference modcl whcn:
. the lVs are constant over time, that
is, W,r=W,r, anrl
. the effects of the Ws are constant
over tin'.re,th"t ir, Tz=Tr.
In short, a simple first-differenceor "change-score,, model removesthe
confounding effectsof unmeasuredvariablesthat are stablc,
that is, vari_
ables that are constant for a given unit over time, and that
have consranr
effects over time. This result is promising, since the most
troubresome
unmeasuredcausesvery often are the unit-specifictraits
that are consranr,
or approximarely so, over time. For researchon individuals,
this might
include difficult-to-measureenduring individuar traits
such as ambition,
work ethic, and senseof_confidenceand personar efficacy.
In the field of
criminal justice, criminologistshave devoteda great deai
of attenrlon ro
individual differencesin propensityro commir crime or "criminality.,,
yet
 136 . Chapter5

criminality remains an elusivetrait that is difficult to measure.For research

on corporations, there is much talk of the importance of differing "cor-
porate cultures," which presumably are relatively constanr, yet difficult to
measure. For cross-country research, difficult-to-measure traits include
constant or slow-changingnational characteristicssuch as a country's cul-
ture, climate, topography, geographical location, accessto seaports,and
so on. First-difference models remove the effects of these enduring char-
acteristics when the effects of the enduring characteristics are constant
over time.
The first-difference method can be viewed as one type of a more gen-
eral method known as fixed-effecrs (Allison 2005; Halaby 2004; Hsiao
2003). consider how the fixed-effects method differs from the usual
cross-sectionalanalysis where Y is regressedon Xs. To avoid omitted-
variables bias in the cross-sectionalregress-Y-on-X method, the unmea-
sured causesof Y must be uncorrelated with the measuredcausesof y. In
the case of first-difference models, by contrast, omitted-variables bias is
avoided when the unmeasuredcausesand their effects are stable over the
measurementinterval. As a general rule, then, first-difference models are
better suited for alleviating omitted-variables bias. \we might call this the
first-difference advantage:

First-differenceaduantage.The first-difference model replacesthe typ-

ically unrealistic assumption that the measuredand unmeasuredcauses
are uncorrelated with the less restrictive stability assumption that the
unmeasuredcausesare constant and have constant effects.
It would be difficult to overstatethe importance of the first-difference
advantage. Omitted-variables bias is the bane of nonexperimental social
research.To avoid this type of bias in standard r.gr.riion analysis,we
must assume either that we have measured all the important causes,or
that the causeswe have omitted are uncorrelatedwith the causeswe have
included. very often those are heroic assumptions. The first-difference
model, by conrrasr, assumes only stability in the level and effects of the
unmeasured causes.In many instancesthe assumption that unmeasured
causes are constant over time within units (recall the examples of the
criminality of individuals, the culture of corporations. and the top.rg.a-
phy of countries) is more defensiblethan the assumption that thoseiauses
are uncorrelated r,r'iththe measuredcausesacrossunits. This feature gives
fixed-effects models a significant edge over y-on-X regression models in
much social research(Allison 2005).
Moreover, first-difference models permit us to test whether the causal
effect of X has changed from time 1 to time 2. If we regress y,.-y,, on
X,r-X,, and X,r, then the coefficient for X,, estimatesthe diffeience be_
tween Brand B, (equation 5.4). lf rhe two parametersare the samewe can

- : - --, : : \ freas ur e.F or r es ea rc h
:-: . . - -:.i nceof dif f er ing " c o r-
: : : : , : . -Lrfl S t loty, et dif f ic ul t to
- :-:- -.:-: -rle.rsuretraits include
Iil :- : t -. -'.tahas a c ount r y ' sc u l -
,- : . . --..r;ces st o s eapor t sa, n d
-, : : : --i .,i thes eendur ingc h a r-
- *r: : -'- .::-larcristicsare constant
: ,i: - . :: ,ni tfpe of a m or e g e n -
. - - i : Halaby 2004; Hs i a o
-::- ; iiffers from the usual
i::::i :-..\s. To avoid omitted-
.:: -- r - ::-.\ method, the unmea-
' , :- : -: :re asu r edc aus esof Y . In
. - : : - : : :. ,mi tte d- v ar iablesbia s i s
-: . .: : ::lects are stableover the
::.i -d i ffer enc em odelsa re
-:- r - . .
:: :. r .:.. \\e might call this the
: : : : r : : . -: r'nodelr eP lac est he t YP -
-:, r-: : J .rn du nm eas ur edc au s e s
j time, we add change in X over time for individuals lxiz-X)
r' : ,:.-,;:::tt' assumption that the
- : , : - :'.>fAnt ef f ec t s .
::;e of the first-difference
: rt-: : nonexperimentalsocial
:. :- i- . .:i regr es s ionanaly s is w
, e
important causes, or
-::- ..-. :he
::;..,::; .r'ith the causeswe have
:! ! -. -.:: , ,ns. The first-difference
.: - :::. leveland effectsof the
:- : : : : -::l p ti o n t hat unm eas u re d
-r : : : ::a l l the ex am plesof th e
: . r ::trions,and the topogra-
: ::.: .::: -.:trPtionthat thosecauses
r:- - , - .. Lrn i tsT. his f eat ur egi v e s
..: : - ::-,\ regressionmodels in
-:--.: ,.: :,r rest whether the causal
. ,: : : Ii u'e r egr es sY , r - Y , , o n
: ,'r. :>:rmatesthe difference be-
:.::l:---ters are the samewe can
The Fifth Rule . '1
-l-
speak of a constant causal effect, but if B, + Brthen the causal effect itself
changesover time. This might complicate our conclusions, to be sure, but
the point is that with first-difference models we can at least test for timc-
varying causal effects (more on this in chapter 6).
It is surprising, then, that social scientistsdo not use first-difference
and other types of fixed-effectsmodels more often (Allison 2005; Ha-
laby 2004). One likely reason is habit. Until the Iast few decades
cross-sectiondata setswere the norm. As panel data sets becornernore
routine, first-difference and related models should become mare stan-
dard as well.
As with all methods, the first-difference method is no panacea; there
are costs involved. In the first place, first-difference models remove only
the stable effects of unchanging Vi/s.So you cannot rely on differencing
as a foolproof method for sweeping away the effects of all types of
unmeasured causal variables. First-difference models are designedfor in-
stanceswhere there are important causesof Y that are stable and hard to
measure.
A second cost is loss of variance on the measured causes. First-
differencing (and other fixed-effects methods, such as adding dummy
variables for individuals) reduces the variance in X. When we regress y
on X for a cross-sectionof individuals, we use the variance in X acrossall
individuals. \fhen we measure the same individuals at a later point in
to the vari-
ance in X. Thus when we regressY on X for a panel of individuals mea-
sured at two points in time, the variance in X consistsof change in X over
time for individuals as well as variance in X across individuals. By re-
moving the variance in X across individuals, first-difference models trun-
cate the variance in X. Truncating variance on independent variables
typically is not beneficial since,as noted in rule 2, we need variance to ex-
plain variance. Yet truncation may be beneficial here becausewith first-
difference models you generally have less, but "higher-quality," variance.
The variance is higher quality becauseit has been purged of the enduring
causal effects of unmeasuredfixed traits.
Let me elaborate.Becausean individual contributes to variance in X. -
X,, only if X changesfor that individual, causal results for first-difference
models depend heavily on how much Y changesfor individuals whose X
changes over time. So the practical implication of variance truncarion
here is that results for first-difference models depend heavily on how
much Y increases(declines)for individuals whose X increases(declines)
over time. Consider, for example, the effect of southern residence on
racial attitudes. Differencing means that our estimate of the causal effect
of southern residencedependson how much (and in what direction) atti-
tudes changed for individuals who moved into or out of the South.
I -l8 . (.hapter-5

In short, first-difference models might severelyreduce the number of

casesin the samplethat contribute to variancein the causalvariableof in-
terest. ln the limiting case,xrr-x., is zero for eueryonein the sample.
This occurs when X is a fixed trait. Ironically, then, the chief virtue of
first-difference models-that they eliminate the effects of stable causes-
is also onc of its weaknesses: First-differencemodels eliminatethe effects
r.tfntcasured,as well aS unmeasured, causesthat are constant over time.
Hence first-difference models are problematic if one's obiectiveis to esti-
mate the effect 9f country of birth, for example, since "chalge in country
of birth" is always zcro.a
Consider,then, three generalclassesof variables.At one extreme are
ascribedtraits that are constantfor individualsthroughout their lives.You
cannot use the first-difference method to estimate causal effects for those
variables,though you can use the method to determine lvhether the asso-
ciation of X and Y has cbanged (shown in chapter 6). At the other ex-
treme are individual traits such as agethat changefor everyoneover tlme.
Most variablesfall somewherebetu'eenthe two extremes.Income tends
to fluctuateover time for most individuals,so differencingis often useful
for estimatingincomc effects.Differencingis not as useful for cstimating
education effects for adults, since formal education is fixed for most
adults after age thirty. Thus a large samplewould generallybe required to
provide enough varianceto estimateeducation effects for adults with a
?irst-difference model, and in any caserhe varianceyou do observeis likelv
to be besetwith measurerlenterror.
In comparing first-differenceand other fixed-effectsmodels with al-
ternative methods, then, it is important to pa.Yclose attention to the
tradeoff of quantity versus quality of variance. Under first-difference
models you have lessvariance,but it may very well be more telling varr-
ance, since the variance that remains has been purged of the enduring
effects of unmeasured(and often hard-to-measure)fixed traits of indi-
vi d uals .

Special Case: Growth-Rate Models

ve don't usually think of using growth-rate models to eliminate con-
founding effectsof unmeasuredvariables,but growth-rate models bear a
formal similarity to first differencemodels. BecauselogY,,-logY,' is the
rate of growth of Y (where log is the natural logarithm), growth-rate
models can be expressedas differencemodels, where the variablesare

a T h e p o i n t m l v be m o o t if, a s Pa u l Ho lla n d ( 1 9 8 6 ) argues,i t makes no senseto tal k

abour crrusaleffecrs for variables, sr.rchas countrv of birth, that cannot be rnanipulated lbut
s e e\ l o f f i t t 2 0 0 - 5 ,p . 1 0 5 , fo r a co n tr a r y vie w) .
 T h e Fi fthR ul e . t i9
logged' To simplify norcrion'
supposey is causedhy just rw.
and w' where as before X is ,n."sur"a variahlcs,X
,nJ'w is \we assunre
that the effectsof [oeX and logw 'ot. furrher
o;l;; are linear. wc have thcn
model for time t: this

logY,,=a,+ a + B,logX y,logW,,*

,+ o, (s . s )
It follows that changein log y from
tirne I to trnre2 is;
Io gY,,-logY , = ( az - + ( Brl o g X,,_ Brk tg X,r)
. d)
| ( T r lo g Vl ,r- ^ /,\o g W ,)+ (r,z _
u ,)
= ( ar - . a) +
B .(l o g X r_ l o g X ,r) + (F ._ ,6 ,) l ogX ,
v r'
+ y , ( log ,W r_ l o g V/,,)+ Oz _
T ) l o g \y ' ,
+ ( e, :_ r,,)
(5.6)
Observe that equation 5-.6,a growrh
rate model,_isequation 5.4
plied to logged u",i"br.r. H.";.';;; ap_
modelsalsoapplyhere:Gr.wrt _.*. -i,,n.turio.,, about Iirst-clifference
_oJ.l, ,.,;.;.;;Jrr.rillrr..,,
consranrunmeasured .r
acrossunits.In .rorr-.ou,r.ry.._
,:l.ur., *,uf "i.r.
that would incluie
ii:Sf:l:xamp]e, the steacly oi n"J.,un,.y,,
tocatlon, topography,history,mineralresources, "ff..t,
cal and legaisystems(to the exrent,trniit"r. of'cial tn.gu;g., politi_
sysrems don,ichinge;, ac_
cessto seaports'and so cln. In
researchon individunlr,thot w.ulcl
includeenduringindividualtrrit,
*hur. .ir..r, areconstarltovertlme.
Applicationsof growth-raternodels -.qrutio.,
,o-.ti-., diffcr from
5'6 by regressinggrowth rateof v g.";th rateof x .,"i*"", ircrucring
log X at time 1. These,studie, "r
rhat log X is not neededin
the model.To assumethat log X"rru_{-,n.n,
i, ;r;;;;ed in the growth_rare modelis
to assumethat the rate of growth
of X hasrhe sameeffecton rhe
growth of yregardlersoflh. i"ittal rate of
,ir. of.X. H;;.;;l;0"r"_.*.
the effectof growthof X on gr,,*,i zu.
ol;is rhesamewhetherwe esrimare
the effectof growthratefo^iard
i; ;-; 1iogy,.-rugy,) or backwarcr
we in
esti
mate
t-,"..l
*";a'i
il['"[ ::,?; i:: k,lt "r-' i',r'."".J, g,o*,r'
log{, - logY @,-
2= _
.ar)1 F rlosX,.,-logX,2) + @1 B2)logX,,
" IL
* +0t_ 4) log:f,r-,
7JloSW.,,-log\X/,)
+ 1e,r-e,r)
(5.7)
The effectof the Browth ,*:
on the growth rate of y is
:f J
tion 5'6 andB, in equation5.7. Brin equa_
Because
X has no effelt in ihe growth;ruatiJrilB7^:d4 nr. J.-rn.#r*i.,-' ro*
growth ratemodels,we ar€testing ih.n, by including log X in
wheth.. o. ,rut the effectof the iate
growth of X on the rateof growtlicrf y;;;;."r"-e .i
forwardor backwardin time. whether we estimare rr
140 . Chapter5

'We
can difference our :-::-
Sibling Models
analogousto the diffe:.r.
Sibling models attempt to compare like with like by matching individu- data. Alt hough t he . l. r : . ,- .
als to strategic other individuals-their siblings-who are likely to be ual tra ir sr equir esp; r nc.- . :
similar in multiple (and often unobservable) ways. Sibling fixed-effects acteri st icsr equir e' t ih. : .
models, or simply sibling models, are used to remove family environ- S upposeour sam pl. -
ment effects that are the same for all children in a family. A sibling l i ngs.Call t hesedlt r r . : - '
model can be thought of as a contextual model where family is the con- of the siblings( r and, 'i . -
text. Contextual models assume that the characteristics of contexts- call thesedata sampleB. ','.
families, neighborhoods,schools,and so on-matter, just like individual (Note that the sibline :- :.
traits do. By collecting data on individuals with shared environments- which we can general:2. .
children in the same family or in the same neighborhood, for example- the subscript i indere s -..-.:
we are able to difference our consranr unmeasured effects of the shared l i ngs a r e denot edr . 1 . i: '- :
environment. A pplyingequar ioni':
Becausesibling models are a type of fixed-effect model, the effects of
Y ,o =ao+ a- u+Br \ . - ; .
stable confounding variablescan be removed without measuring the vari-
ables.In light of that feature, Guang Guo and Leah Vanlfey (1999) use Y,o=a"t a,r+ B"X.u - ? ,
sibling models to determinewhether the inverseassociationbetweenfamily
N ow subt r agrY. , t : . .
size and children's intellectual development is causal. In brief, therr an-
swer is no, it is not; the "dumber by the dozen" concern (Zalonc 1975)ris Y ,o-Y. o=( a, - a
ID 1A Ib :1
- lJ
without merit (see Guo and Van'Wey 1999 along with commentary by (l,ruW,,- yA\\',{ - :...
Phillips 1.999 and by Downey et a|. 1999). (As we note subsequently,Dal-
ton Conley and RebeccaGlauber [2005] investigatethe same issue using Keep in mind that )-. - I
an instrumental variables approach. Interestingly, Conley and Glauber the differencesfor the :..:.
find causal effects that are smaller than those found in traditional regres- in the same familr-.
sion approachesbut nonethelessare not zero.) \7e have then a s:::-.: ,
To see how the differencing method works for sibling models, con- di fferencem odels de: - : .: :
sider again the standardlinear regressionmodel (equation 5.1). In addi- gui sh f am ily- levelc. r u. . -
tion to distinguishing observed from unobserved causes, we also equation 5.4 abor.e.re\,.:.:
distinguish family characteristicsfrom ir-rdividualcharacteristics,so the
Y ,o-Y, n=@i, - o. . . - 13
sibling model consistsof four classesof causal variables:measuredand
unmeasured individual traits, and measured and unmeasured family +( t n- 71 \ \ . .
characteristics.Of course,if the context were neighborhood instead of
family, we would substitute measured and unmeasured neighborhood - r \ Bor t n- O ''
characteristicsfor measured and unmeasured family characteristics. + t ^t t A t t t _. i
' I rB ; I
Our essentialconclusions about sibling models apply to other types
of contextual models (neighbor models, school classmatemodels, etc.) From th e lasrlinesof : - - . :
as well. unmeasuredfamilv ch,::.:, .
\X/eadd the superscript Fam to distinguish family variables and coeffi- model when:
cients from individual traits and coefficients. Thus the seneral sibline
model is: 'To s i mpl i fy equl rrnr ' .
Iecred.If that a> s ul rrprr,' - . '
Y,=o+ a,+ BXr+ yw,+ BF"-X,F"-+ 7 F"''wiF"'+ q (s.8) conclusions.

i.' L.vmatching individu-
r - r . -:-\r'ho are likely to be
: :: ,,:r.. Si bling f ix ed- ef fe c ts
. :-nto\ e f anr ily env ir o n -
-- : -
:- * r:-. i11a family. A sibling
i - -: ',,,here family is the con-
. - : . - .r-.,tEri s t icof s c ont ex t s -
- ---'.,-i cr. j us t lik e indiv id u a l
. ., :: .h.rred environments-
: -: - -.orh ood, f or ex am pl e -
--:' .: -.rr.deffectsof the shared
:- . . : - . : :-.t model, t he ef f ec t so f
, : - , riLrur measuringthe vari-
.,--; :rh Van!fley (1999) use
. ::ir ',ri,tcietionbetweenfamily
:-r : -.r'-iSil.In brief, their an-
: r -',nCern( Z ajonc 1975 ) rs
'-- - :-,: u'ith commentary by
--: ' : iiot€ subsequentlY,Dal-
- . : :.1 :cthe s am eis s ueus i n g
: :: r i: ' j -r. Conley and G lau b e r
-. : -. rd in traditional regres-
: I7e have then a simple first-difference model, similar to the 6rst-
.. , i: . . rL)rsrb lingm odels , c o n -
- ' : . e quat ion5. 1) . I n ad d i -
- . - r)rfVed c aus es , we a l s o
.-. . :,.,-.1 char ac t er is t ic ss,o th e
,r -- .,. ...lriables:measuredand
-
:- ': - -ti tJ u nm eas ur edf ami l y
,: . i : : , ::i ghbor hood ins t ea do f
:- - -.:::llL'i.lsured neighborhood
-=- -. . - =; i e rn ily c har ac t er is ti c s .
-,: -' *.,s aPPl,vto other tYPes
i. i- - . Jl ass m at em odels , e tc .)
.- ,- :-'.:irilvr-ariablesand coeffi-
:- . : : . . Th Lrst he gener als ib l i n g
,.-.-
i '--\\' ( 5 .8 )
",
The Fifth Rule . 1,11
'We
can difference out family characteristicsfor sibling pairs in a manner
analogous to the differencing out of individual characteristicswith panel
data. Although the data demands differ-the differencing out of individ-
ual traits requires panel data, whereasthe differencing out of family char,
acteristicsrequires sibling pairs-the logic is the same.
Supposeour sampleconsistsof i=7,2, . . . , N children who have sib-
lings. Call these data sample A. Imagine we collect the same data for one
of the siblings (randomly selected)for each of the children in sample A;
call thesedata sampleB. !7e have, then, N matchedpair:sof (A, B) siblings.
(Note that the sibling model rules out only-children-so the population ro
which we can generalizeis children with siblings, not all children.) Now
the subscript I indexes a sibling pair, not an individual; the individual sib-
lings are denoted iA and iB.
Applying equation 5.8 to samplesA and B in turn, we have:
Y,o= cro* a,o+ BoX*+ yAW iA+ BAt"-X,ou"- * Zo F,-W.^F"- + a^
Y,u= ar+ c.i|+ PBXIB+TsW,r+ Br.u-X,uuo- * 7, Fu.W,nFu-+ e,u (5,9)
Now subtract to from Y,r:s
Y,"-Y io=(a u- a;a)+ (Buxru- FaX,o)+ (f, t"-X,ut"- - fo Fu-xiAFu-)
+
(y"w - zow,o)+ (yro"'w*u"- - 7ot"-w,ou"-)* (e,u- e,x) (5.10)
"
Keep in mind that Y,r- Y,o is the Y difference for the ith sibling pair. Thus
the differencesfor the family variablesdenote differencesbetween siblings
in the same family.
difference models described earlier, but with additional terms to distin-
guish family-level causes from individual-level causes.Proceeding as in
equation 5.4 above,rewrite (5.10) as:
Y,u- Y,o= la,u- a,) + ft(XiB - X ,o)+ (Br- Fo)X,o + yr(W.u - W^)
+ (Tn- Ta)w ro+Bur"-(x.BF"- - xiAFu-)
+ (B sF^ - F etu')X,ooo-* zrFu'(w*Eu* -'wiAFun')
+ (yBF" - zoF"-)W,oFu'+ (e,o- t,o) (5.11)
From the last lines of equation 5.11 we seeimmediately that the effects of
unmeasuredfamily characteristics(the W,tu') are eliminared in the sibling
model when:
5To simplify equation 5.10, we assume aA=aB since samplesA and B
are randomlv sc
lected. If that assumption is false we could restore the term ar-aA without affecur.rgour
conclusions.
142 . Chapter5

o W is the same for the sibling pairs, that is, 'W ='1X/
rF"- ,o'^^, ard
. the effect of V/ is constant over time, that is, y"r^- = yor^ .

In short, sibling models are most effective at removing confounding

family effects when the relevant unmeasured features of the family envi-
ronment are constant, or nearly so, in their levelsand effects. In practice
the constant-levelscondition that W,uF"--1y,oF"- is the bigger concern
than the constant-effectscondition that yuF^ = 7oF"-. This is the case be-
cause,assuming the siblings were selectedrandomly, there is no reason to
expect big differencesin 7s for the A and B sibling samples.This is not to
say that siblings respond the same way ro family characteristics.To the
contrary, some children might be more sensitivethan other children are
to parental conflict, for example. But under random selection of siblings,
those differencesshould even out for samplesA and B, so 7uF"-- ZoF"- is
gener allya r e a s o n a b l e
a s s u mp ti o n .
The more problematic assumption is that W,uF"-=1y,oF"'. Families
change over time, so children of different agesare exposed to a "different
slice" of family history. Some important but difficult to measure features
of the family environment, such as parents' commitment to children and
how highly they value academicsuccess)might be relatively constant over
time, and thus approximately the same for siblings regardlessof age dif-
ferences between them. Other unmeasured features of family environ-
ment, such as how well parents get along, might change over time, or
parents might treat their children differently, so in that way different chil-
dren in the same family are exposed to different family environments (see
Conley 2004). Sibling models eliminate only the featuresof the family en-
vironment that are constant across siblings. In evaluatinghow well sibling
models remove the confounding effects of unmeasureddifferencesin fam-
ily environments, then, researchersneed to consider whether the features
of the family environment that matter are constant (or nearly so) within
families.
'What
about the confounding effects of unmeasured indiuidual traits?
Cons ider t he te rms y B (W i B* W ,o )+ (1 " - 7o)W ,o i n equati on 5.11,
where the Ws refer to individual characteristics,not family characteris-
tics. Becausewe expect Ta and 7u to be roughly the same, the second
term should be close to zero. It is the sibling difference in unmeasured
causes,WiB-WiA, that we are more concernedabout. Two observations
about W,u-W.oare relevant.First, siblings tend to differ lesson key in-
dividual traits than two randomly-selectedchildren do, so the use of
sibling models typically reduces the variance of the unobserved
individual-level causes.Second,researchersoften use sibling models to es-
timate the effects of family characteristics, and it might be reasonableto
assumethat sibling differencesin unobservedpersonaltraits (W*-W,o)
 The Fifth Rule . 143

are uncorrelated with sibling differences in family characteristics, the

causal variables of interest. We might expect siblings' personality differ
ences,for example, to be independent of siblings' differences in level of
family income when they were growing up. Sibling models can be useful,
then, in alleviating the confounding effect of unobservedfamily and irudi-
uidual characteristicsin investigationsof the causal effect of fanily enui-
ronment.
To give substantiveflesh to the model, and to illustrate its potential ancl
its limitations, we now considera study that usessibling models.

EXAMpLE : T H E E F F E C To F F AM IL y INCOM E o N cHIL DREN' s scttoot-tttc

In a researcharticle published in the June 1998 issue of the American So-

ciological Reuietu, Greg Duncan, Jean Yeung, JeanneBrooks-Gunn, and
Judith Smith use sibling models to addressthe question "How much does
childhood poverty affect the life chances of children?" As they note, a
number of studies have documented an association between family in-
come and children's life chances:For example, children from poorer fam-
ilies tend to do worse on standardized tests of ability and achievement;
they tend to drop out of school earlier; and teenagegirls from poor fami-
lies are more likely to experiencea nonmarital birth.
The stubborn question is how much of the observed income associa-
tion is causal. To be sure, earlier studies of the income effect typically had
included a number of demographic and other control variablesin their re-
gressions.Yet skepticsof this prior cross-sectional research(for example,
Mayer 1.997) remain unconvinced, given the inherent difficulty of mea-
suring all the important family differencesthat might be related both to
parental income and to children's outcomes.As Duncan et al. (1998,
p. 409) explain:

A persistent concern with these kinds of analysesis that the estimated

effect of income might be spurious, caused by the mutual association
that parental income and the outcomes for children share with some
unmeasured "true" causal factor. Suppose,for example, that the men-
tal health of parents is the key ingredient for children's successand
that measuresof parental mental health were not included in the mod-
els. Becausepositive mental health in parents is likely to make parents
more successfulin the labor market as well as to lead to fewer prob-
lems with their children, the absenceof adjustments for differencesin
parental mental health may produce a seriousoverstatementof the role
income plays in causing children's success.

Observe the policy implications here: If income effects are not causal,
then raising the incomes of the poor will be ineffective in improving their
children's life chances.So it is important to know whether familv income
 114 . Chapter5

is causal, or whether it is merely correlated with differencesin famiry en-

vironment that we have not measured.
Duncan et al. use sibling models to alleviate the problem of unmea-
sured differencesin family environments. Their data consists of 328 sib-
ling pairs in the PanelStudy of Income Dynamics. The dependentvariable
is years of schooling completed, estimated using this model:
A ScnoorrNcr= Bo+ BrA FeNrrrylNco^as.
+B, A CurroTherrs+q (5.12)
where A refersto differencesbetweensiblings.Thus A ScsoouNc is the
difference in the number of years of schooling complered for the iih sib-
ling pair, which is assumed to be a function of the difference between
siblings on a number of individual traits, denoted A CnrroTnerrs.. and
the difference between siblings on the family's level of income when they
were ages0-15, denotedA Fal.rrvlNcolre. In the caseof twins there is
no difference in family incomes, of course, since twins are the same age.
For other siblings, though, A FeurryINCoME generally is not zero, and
we must assume that fluctuation in family incomes over time produces
enough variance in family income between older and younger children
for reliable estimates of the income effect (more on this assumption
below) .
Note that, aside from family income, the model has no measure of
f am ily env ir o n me n t.T h e a b s e n c eo f fa m i l y envi ronmenrmeasuresi s nor
problematic if in fact differencing on siblings has eliminated the effect of
relevant differences in family environmenrs (except for the effect of dif-
ferencesin family income, which is in the model). As shown above, dif-
ferencing on siblings does eliminate the effects of unmeasured causal
family characteristicswhen WiBFU--WiAF"-and when 7rF".- ZaF"- (equa-
tion 5.11). In other words, with sibling models we do not need ro mea-
sure causal family characteristicswhen those characteristicsare the same
for both siblings and have the same averageeffect on y for both samples
of siblings.
'Sfith
regard to unmeasured family characteristics,then, we must as-
sume that the family environment was the same for both siblings as they
were growing up. As regards the family characteristic of interest, how-
ever, the family environment could not be the same for both siblings, or
there would be no variance on the causewe want to estimate.In the Dun-
can et al. study of income's effect on children's schooling, for example,
we assumeone feature of the family environment, income, changed sig-
nificantly over time, whereas other causal features of the family environ-
m ent did not .
Suppose some important unmeasured causal family characteristic-
let's call it Q-did change, so siblings were reared in different family

:* ,. :ir Jifferences
in family en-
r :", :.r: rhe problemof unmea-
- - ..: J.rtaconsistsof 328 sib-
-.-,':* -) The dependentvariable
'- -' .: :hrsrr.rodel:
j ( 5. 1 2 )
' r:, I rus -\ S c uoor , r Nc ,is th e
^ .nl cr edf or r he r r h s i b -
-:: - : ihe difference between
-- j:- i:J -\ CutroTnetrs,, and
r- , - ...rl of income when they
,.r: -: :::e caseof twins there is
-:: :u'ins are the same age.
-:- r
ri:
-- -::rerdllv is not zero, and
' i ., ::las over time produces
::- -::: and younger children
' -: -- i r o o t his as s um pti o n
--
" i-: r-. iel has no measure of
' - . . .- nme nt m eas ur esis n o t
-:: -.:r :liminated the effect of
-:: :r.-:rr for the effect of dif-
:: -' *: . -\s shown above, dif-
: : r r : -t : rti u nm eas ur edc au s a l
-,'. -
"- :: : hen yrF"' zaF"' (equa-
ir '*: : -.'.edo not need to mea-
- :: ,..-r:.icteristics are the same
-1i. :r:r,': on Y for both samples
-:: - ': -: : ::si i cs,then, we m us t a s -
-: :', '- :rrr bclth siblingsas they
- - . . , , -::ri sti c of int er es t ,ho w -
rtr :-: !.ii'nefor both siblings,or
.i: .,.-:,:-.: !o estimate.In the Dun-
' . . :: . : . '. .chooling,f or ex am p l e ,
: -': -.::.:.i nco m e,c hangeds i g -
' : i: . . -:. of the f am ily env ir o n -
- - :- i. r f.rmi l v c har ac t er is ti c -
,,.::: ::,lred in different family
The Fifth Rule ' l4-5
environmentswith respectto Q. Then the causalmodel for A Scsoorrxc
would be:
A ScHoouNc,= Bo+ B, A Fel.trvlNcoMEi+4 A CnrroTnaIrs,
+ 7A Fel -rrvQ,+ a, ( - 5. 131
where A FerltrvQ, is sibling difference on family characteristic Q. tsl'
omitting A Fer,rrrvQ,in equation 5.13, our estimateof the family income
effect will be biased when A FamtrvlNcoun is correlated with A F.q'rtI-
rvQ,. In general, the greater the correlation, the greater will be the bias.
The sameprinciples apply to the biasir"rgeffect of unmeasured indiuidual-
leuel causaleffectsthat do not "difference out," that is, WiB lWiA in equa-
tion 5.11. There may be unmeasureddifferencesin the personaltraits of
siblings that affect their differencesin years of school achieved (perhaps,
for example, we have not measured motivation adequately).If those un-
observeddifferencesin siblings'personaltraits are correlatedwith differ-
ences in their exposure to different levels of family income-which
Duncan et al. (p. 411) say is unlikely-then our estimateof the family in-
come effect will be biased.
It is time now to summarize. The key idea in sibling models (as in stan-
dard first-difference models) is to difference out the effects of confound'
ing causeswithout differencing out tbe effectsyou want to inuestigate-So
you want siblings to be alike and different: alike on unobserved causes,
and different on the causesyou are invesrigating.Thus sibling models are
most effective when your purpose is to estimate the causal effect of some
feature of family environments and:
I . Fami l y envi ronmentschangefor the causesyou are i nvest igat ing.
Becauseyou cannot explain a variable with a constant (chapter 2),
siblings must be exposed to different family environments for the
causesyou are tryi ng to esti mate.
2. Key unobserved features of farnily environments are stable over
time so siblings are exposed to the same featureswhen growing up.
The sibling model will difference out these effects (W.BF"--'WiAFu'').
3. Siblings are randomly assignedto samples A and B so we expect
that the average effects of unobserved causes will be the same tn
both samples(yn=laand yuu^^=I4F"-)and that sibling differences
in unobserved personal traits (W*-W,o) will be uncorrelated with
the family-level characteristicsof interest.
4. Sibling differencesare the samewhether you have one sibling or five
(or, if sibling differencesdo depend on family size,the effect can be
modeled).
When these conditions are met, sibling models provide traction in the es-
timation of family-level causal effects.
146 . Chapter-5

CoupnntNc l-txE.wttrt l.txr tsRoucH MercntNc

oN MtesunEP VeRtasLE,s
use of a fixed-effects
One wa1' to compare like with like is through the
is through matching on values of measuredvari-
model. i r..ond-*ay
ables.
variables' One
There are two common ways to match on measured
from the treatment group with individuals
method matchesindividuals
group who zrrcidentical on the measured confotlnding
fr<tm the control
generally feasible
causes.Although stra'ightfor*a'd, exact matching -is
th. nu-b.r-of nontreated units is large relative.to the number
o.,ry
-rr." units, and rvhen the characteristicsthat needto be matchedare
of tr.at.d
fail to meet
relatively ferv. Becausemany studies in the social sciences
methods have been developed that attempt only
thesc conditions, other
of the covariates across the treatment and
io .q.t"1lr. the clistribtiions
methods make use of propensity scores.to "bal-
control groups' These
confounding vari-
ance" the treatment and control Flroupson measured
I describe both approaches, beginning with exact
ables. In this sectton
matching.

Exact Matcbing
observationaibio-
The textbook example of exact matching comesfrom
a clinical study where an lnvestlgator wants to
medical research.Imagine
assessthe efficacy .ri rr."r-."t X. for alleviating some symptom Y'
accessto two
There is no randomizedtrial, but the investigatordoeshave
profiles. One set consists of the demographic charac-
sets of individual
(X,), health histories (X'), and.current measurement of symp-
i.rirti.,
y were treated, and the other set consists .f the same
tom fo. tii,rr. who
lvho were not treated. Matching consists then of
information for those
from among the control group to provide matches
strategicsubsampling
The treatment ef-
on X.-and X, fu iniividuals in the treatment group'
calculating the average difference on y for the
i;;, i: ..ti_"t.d by
matched units.
I nex ac t m at c lr i rrg re s e a rc h e rs s i ftth ro u g h thecontrol stofi ndi ndi vi d-
cases
uals who match those in the treatmentgroup. Becausenonmatchlng
matching is most commonly used where there is an
are discarded,exact
a bundanc eof uni ts i n th e c o n tro l g ro u p fro mw h i chttl formmatches
the re-
with units in rhe treatmenrgroup. In prospectiveclinical studies
set through the discarding of control units can be cost
duction of the data
permits investigators to follow onll' the control units
effective becauseit
data
that matter (that is, the ones that match) rather than collecting
controls. In those types of studies researchers
over time on all the original
 T h e Fi fthR ul e . l ,+ i
with limited budgets might decide to usc one-to-one
exact .ratc'i'g.
w here eac h r r eat edunit i s rn a tc l .reto
c r e ra c tl y ,.rr" .o r.,tr,rr.' ," i i . r"
eral, however,it is better to use a[ control units * ." -
that n'o,.t , .in.. ,t.'^,
techniquereducesthe varianceof estirnates.
Neither method-exact_matching nor one-to-onc
exact matching-is
very common in the social scierces,where the
curse dimensio,aliry
makesexact matchesproblernatic.(The te'n ..curse 'f
of clime'si<.r'ality,, i'
statisticsrefers to the<lifficulty of fitting a modcl
*rth ,,.r,rf folrrut. .,._
planatory variables.)with regard to thJiss,,e
of matchi'g, ,tr. rrro..
ables you have to match on, and the morc categories "".r-
there arc for each
variable' the more difficult it is to find exact
,naicrres.rypi."rii we want
to march individuals simurtaneouslyon multipr.
uariabrei,uni ort.n tn.
matching variableshave multiple categoriesjthe
confoundirrg uorr"bl.,
might be conrinu'us, for e"arnple;.As"a result,
..r."..h.., _i!nr.na up
discardingunits in the.treatedgi,rup, as well as
unirs ir ,rt" ."riror grorp,
for lack of a match. The-discaidingof .ur., rs
so severein many applica_
tions that onlv a small fracti<rnof the data can
be used and, if treated
units are among those discarded,the treatmenteffect
of inte..st might no
lo.ger be the trearmenr effect being estimated.
In other words, the
wh.olesalediscardingof casescould result in b.th
inefficientand l:iased
estimatesof the treatment effect.
mirtching does not require the exact matching of
-Fortunately, individ-
ual units. It sufficesinstead ro march ihe distributions
as.tor?tf porrr
ble, as we now see. ",

Prop ensity- Score Meth od

The pr<rpensiry-score method (Rosenbaumand Rubin 19g3,19g4,

r9gs;
Rubin and Thomas 1996; seeBeckerand Ichino
2002 for an ovcrviewl at-
tempts ro compare like with rike by rnatching
individuarson prrpensity
scoles-summary scoresbasedo. observed
variables.In effect tir. nr.th,r.l
attemprs to do by matching on observableswhat
random
tempts to do probabilistically-equalize or "balance,'
th. "rrigr-"r, ",
t.."1-.rrt u,-,d
control groups at the outset, so any subsequcnt
diffcrencc in ,t g.u.,p,
can be attributed to the treatment itself. The propensity-score "
mcthod is
appropriate for estimating the causal effects
of .nt.go.i."t unri"rrt., ,r,",
are dichotonous (treatment versus control groups)
or that can be di-
chotomized (there is stilr work to be done to
."t.nd th. prop.'rlity-score
method to multivarued Xs)..The aim is to equarize
on c khe;;rrio,1na..r)
individuals who differ on X (the rr.",-.n,).
Becausethe goal is to barancethe treatment
and c.nrrol groups on the
confounderswithout the benefitof rand.m
assignment,*."-uri1i., .nrr-
trast to the fixed-effects approach) have data
on the confounders. In this
148 . Chapter5

respect propensity scoring is similar to standard regressioncontrol with David Harding (201)-1.
: ^
covariates.Flowever, becausethe use of propensity scores permits us to neighborhoodeffecr.,
achieve balance by trial and error, whether or not we know the proper
functional form, the method offers an attractive alternative to standard The solr r t ionr r sL
J . -. .
regressionin instanceswhere there is uncertainty about the proper func- mQre control sul.i.,:,
tional form of the relationship between Y and its predictors (Harding idenrical to the cor::
2003). Hence the approach is less subject to bias arising from confound- treatment.. . . Tre.::::'.:
ing variables that are included, but in the wrong functional form. subjectchar acr er ir : :--
The general procedure is to summarize all the measured confounders treat m entand cor r r : . _
by a single variable called the propensity score and then to match the To use propensrlr':, ::
treated and nontreated units on the basis of the propensity scores.There di ffere ncesbet wecn. : . . - :
are four basic steps:(1) use logistic regression(or a similar method) to children in other neign: :-
regress the binary treatment variable X on the confounders to create a example, since childre:: ,,.
variable X; (2) match treated subjectsto controls on rhe basis of the esti- l i kel y t o live in po( ) r : . : r '
mated propensity sco.esX, for each unit (in "nearest-neighbormatching," likely to finish high scir,
for example, each treated unit is matched to the control unit with the mediate the effect of r..:-
most similar value of X); (3) verify that rnarching has balanced the treat- how ever ,sincet hat r vr'. : . -
ment and control groups; and (4) draw conclusions about causal effects effect.)Then we use log:.: .
on the basis of the differences on Y between the treated and control tor to regressthe "trear::-:
groups for the matched data. denotedX) on f am ilr i: : . -
If step 3 telis us that we have not balanced our data, then we try a poor neighbor hood.
again with a different specificationof our regression.That is the cardi- The regression of ,\ :
nal virtue of the propensity-scoremethod: With propensity scoreswe nei ghb or hoodyieldsi. . ' -
can try increasingly elaborate specifications (interaction terms, squared di stri bu t ionsof * lr he - . : .
terms, and so on) until we find a specificationthat balancesthe data. As distribution ofX for th.
-
Daniel Ho et al. (2007) point out, the specification works when it most instancesit makc. ..:
works, and when it doesn't work we keep trying. This is in contrast to for X where the treatnrr:.: ,
standard regression-control-by-residualization, where we are beholden overlap,estimatesof rh. ::.
to unverifiable assumptions about proper functional form for balancing tions about functional r, :-:
our data. to avoid by using prr)f.. ) :-
To flesh out the propensity-score method with an example, consider with covariates.
again the effect of childhood poverty on later educational attainment, ex- Next we match trearr.:.'
cept this time think about the effect of living in a poor neighborhood i ng one of a num ber , , i :
rather than the effect of living in a poor family. rWeknow that children Morgan and Winship. -
who grow up in poor neighborhoods are lesslikely than other children to for discussionof softii e:.
complete high school. \fhat we want to know is how much, if any, of desi redbalanceon . or ^: - - :
this association is causal, that is, how much the graduation prospects can use a quantile-qu,t:.:.:
would improve for children in poor neighborhoods had they grown up in confounders.Good b.r1::-.
nonpoor neighborhoods.
As noted earlier in this chapter, the problem is that we cannot observe
5 Where rhe pool ot .:. -
the counterfactual, since a single individual cannot simultaneously both f
R uhi n and Thomc s (l qq^. - - '
experienceand not experiencegrowing up in a poor neighborhood. Our
he rel evan t.A gai n. rari .rl .i . :: j
alternative is to have control subiects serve as the counterfactual. As di ri oni ng o n the c on:equ.; ...

i:: - : : - : :Jg re s s ionc ont r ol wi th
:' - ' .) .'ores permits us tt-r
. : - .-..)r\\e know the proper
=:-...- , .: -rlternative to standard
'.-:* . r i'. rt out the proper func-
- | : . . - r:s p r edic t or s( Har d i n g
-: : - .:. .--risingfrom confound-
: i i ir :-:ncfl O nal f O r m .
-: i :--:' nlriSUr€d confounders
- -i.' -, -..nilthen to match the
i ' t - : :ro pens it y s c or es T . he re
.. : : : . . -. ,; a s im ilar m et hod) to
- . . -r;,rn follnder st o c r eatea
' - : .,:-!:-n er ghborm at c hin g ,"
-:- i ::.- control unit with the
- : : - - :'- i rl s balanc edt he t r e a t-
-- .--. ,:r: about causal effects
-
-r :, . : : ::a tre at ed and c ontro l
-. i . i- -r: ,ru r dat a, t hen we try
, , - - : : : ::>i Lfrr. T hat is t he c ard i -
-. - 'i. :l Fropensity scoreswe
'.:::;r.ti o n t er m s , s qua re d
. :.-.-:lbalancesthe data. As
-::
- : - . : -...;.]ti o n wor k s when i t
::: : : :-.:.Th is is in c ont r as t to
: : : : -.'.,.her ewe ar e behold e n
,: : : - . , :.,,:trl for m f or balanc i n g
:: 1 r '. ,.::1a n ex am ple,c ons i d e r
.::: :: * j.ltional attainment,ex-
: :-.: :1 a poor neighborhood
: :.-:--... \\e know that children
: . : -: . . .:.r th an ot her c hildr e nto
r r ',.. ri how much, if any, of
----:. :re graduation prospects
.' - : - J' had t hey gr own up i n
: r. that we cannot observe
:-::rot simultaneously both
": ' :. roor neighborhood.Our
n:: : -'.i rhe counterfactual. As
The Fifth Rule . 1'19
David Harding (2003, p. 684) explainsin his propensity-scoreanalysisof
neighborhood effects:
The solution used here is to march each treated subject with one or
more control subjects such that the treated subjects are, on average,
identical to the control subjects on observablecharacterisricspnor ro
treatment. . . . Treatment and control groups are well matched when
subject characteristicsthat affect the outcome are "balanced" in the
treatmentand control groups.
To use propensity scores,we first must find measuresof the pertinent
differences between children who grow up in poor neighborhoods and
children in other neighborhoods.'Wewant to measure family income, for
example, since children who grow up in poor neighborhoods are more
likely to live in poor families, and children from poor families are less
likely to finish high school. ('Wedo not wanr to conrrol for variablesthat
mediate the effect of neighborhood poverty on educational attainmenr,
however, since that would bias our estimate of the neighborhood poverry
e ffect.)Thenw e use l ogi sti cregressi on
or someother appropri a r eesr im a-
tor to regressthe "treatment" (growing up in a poor neighborhood, here
denoted X) on family income and other variablesassociatedwith living in
a poor neighborhood.6
The regression of X on the relevant predictors of living in a poor
neighborhood yields X, which we use to match units. We also use the two
distributions of X lthe distribution of X for the treatment group and the
distribution ofX for the control group) to check for adequateoverlap. In
most instancesit makes senseto restrict the analysisto the range of values
for -* where the treatment and control groups overlap. \7here there is no
overlap, estimatesof the treatment effect depend on unverifiable assump-
tions about functional form of the effect-the very situation we are rying
to avoid by using propensity scoresinstead of standard regressionsontrol
with covariates.
Next we match treatment and control subjectson propensity scoresus-
ing one of a number of possible matching algorithms (see chapter 4 of
Morgan and \Winship,2007, for examples;seeBecker and Ichino 2002,
for discussionof software). To verify that our matching has produced the
desired balance on confounders across treatment and control groups, we
can use a quantile-quantile plot (Ho et al., in press) for each of the
confounders. Good balance is more critical for variablesthat have a larse
6Where the pool of potential control casesis large relative
to the number of treated cases,
Rubin and Thomas (1996, p.253) adviseresearchersto include all variablesthat are likely to
be relevant.Again, variablesmediating the effect of X on Y should not be included, sincecon-
ditioning on the consequencesof X can severelybias our estimateof the causal effect of X.
150 . Chapter5

effect on rhe outcome variable than it is for variablesthat have a small ef- andW inship,2007t.\f -, .
fect. If necessary,as noted above,X can be reestimated,using a different data do es not r ule oui ; . . s - - '
now see.
functional form for the confounders, until balance is achieved'
The final step is to estimatethe averagetreatment effect for the ueated
(or ATT) for each of the matched groups of individuals as the difference Matching as a Prcprocd-i-rrr:i)
between the treated and control subjects within each of the groups. To Mod"el Dependence
calculate the overall effect, it is usually a simple matter to take a weighted
averageof the treatment effects for each matched group' -Vodel dependenceeoir. ' ::
The propensity-scoremethod has severaladvantagesover standard re- causal ef f ect sin soci, r r : - : -
gression. First, as I have emphasized,successfulbalancing of the data on since estimatesof cau.., : :-
the basis of propensity scoreseliminates the need for strong assumptions searchersmake abour i-,: i -
about the functional form of the relationship between the outcome vart- tional form to use,rhe i.: : :
ables and the regressors.Second,inferencesare less likely to be made on so on, As a r esult ,t hr . i. r - ': - ,
the basis of extrapolation from treatment and control casesthat do not articlesgenerallyrepr.i: . .
overlap (note that extrapolation is especially dangerouswhen we are un- matcspossiblet r r r de r ....:-..-:
certain about the functional form of relationships). One of the virtues of esti mat esar e m odel- dci; . - . - ; tt .
the propensity-scoreapproach is that it brings the nonoverlapping cases l ectto r he suspicionof '. ; - . : .
problem out of the closet (in contrast to standard regression analysis, suspicionthat researchl:i , :
genialwith their favore; .. . ; .
where the problem generally is well hidden). Checking for adequateover-
genialwith their data.
lap of the distribution of the treatment group and the distribution of the
B eca usef avor ed- hr r r - : : -
control group is standard procedure in propensity-scoreanalysis. Under
search.r educingm odc. - : : : - -
regression,researchersmight or might not perform the necessarychecks.
researc hdesigns.To r . i . - . -
Third, comparing similar groups of treated and control casesis a more
intuitive method for control than the residualization method used by re- that researchersuse prL,:::- :
rrep, which Ho and
gression. This can be an important advantagewhen presenting results to -, ::.-.
a lay audience. Finallp propensity scores can result in smaller standard propensit yscor esr o r s: '. i_-: - :
errors despitethe removal of nonoverlapping cases(seeSmith 1997 for an possi ble;r hen, r o ohr . r : - -
example), and in some instancespropensity scorescan be useful for test- parametrictechniqueth-.. -,:
ing sensitivity to bias from unmeasured confounders (seeHarding 2003 gressi o n,logit , st r uct ur . i.. - _ _-
for an example). Preprocessing with F:, :: : -
It is important to stress that neither method-propensity scoring nor nri ni sh ingt he im pact ( ': : - '- .
regression-solves the V/s problem.'Whetherwe attempt to match by resid- er al., 2007). The balan;, = :
ualization on confounders, as in regression,or by balancing the distribu- sel f) req uir esf ewer r ssl- : -: :
tions of theX in the treatment and control groups, as in the propensity-score cessingthe treatment qr,,--: ..
approach, we can match only on observables.Becauseskepticscan always eonfoundingvar iahiecf . : . -
point to potentially confounding variablesthat you did not include in your parametric adjustments.::.r. _
analysis, both methods require strong theory. implicarionis that r\\ o ir. . : -- _ _
The most obvious limitation of propensity-score analysis relative to to addre sst he sam er esr - lr - . -, , :
regression is that the propensity-scoremethod yields no estimatesof the resul tsi f t hey successiu. .:-,. =-
confounders' effects on the outcome variable. (Fixed-effectsmethods are Jo not p r epr ocess t he d: : : . : - :-
subiect to the same criticism.) This limitation is more apparent than real
-
for causal analysis,however, since there is emerging consensusthat causal H o an d c o l l e a g u e s n o l c : - : - :

inferenceis best accomplishedby focusing on one causeat a time (Morgan D v k ( 2004 ), R o s e n b a u m a n c i . _ : - -

 Thc Fifth Rule ' 1'51

scores.to balance
and Winship,2007). Moreover, the use of propensity
use
data does ,-rtt ,,rle out the subsequent <lf anall.sis,as u'e
regrcssi<ln
now see.

Matching as a Preprocessing Strdtegy for Reducirtg

Model DePendence
estilnatts of
Model dependencecontinues ro vex social research.Most
are model dependent t() s()me degree,
causal efflcts in social research
usually vary depending on choices re
since estimatesof causal effects
variahles to include, the func-
searchersmake about their model-the
include.and
tional form to use,the period to analyzeor subsctof casesto
afe prcsented in published rescafch
so on. As a result, the estimatesthat
articlesgenerallyrepresenta selectsubset of the much larger set of esti-
(realistic) model assumptions' Because
mates plssible under alrernative
estimatesare model-dependent, publishedfindings nearly always are sub-
ject to the suspicion of^fauored-hypotbesis bias (Firebaugh, in press)-the
suspicionthai researchers have settled on the estimatesthat are most con-
eenialwith their favored hypotheses rather than thosc that are most con-
g en i a lwi rh t heir dat r r .
Beca us ef av or ed- hy p o th e s i s b i a s p e rv a d e s o b s e rvati onal soci al re-
primary obiectivein our
search,reducing .r-to,leidependenceshould be a
researchdesigni. To reduce model dependence, Ho et al. (2007) suggest
that researchersuse propensity scores to preprocess their data.- The first
call nonparametric PreprocesstnS" uses
step, which Ho and colleagues
and control groups as similar a
propenslty scoresto render the treated
'porriUt., researchers can use whatever
ih.r-r, ,n obtain final estimates,
(linear re-
p"r"-.r.i. technique they had planned to use in the first place
e
gressi o n.logit . s t r uc t ur a le q u a ti o n s , tc ' )'
" p..p.o."Ising with propensityscoresreducesmodel dependenceby di-
minishing the impact of th. we make about our model (Ho
"tt,rtt-tptions
data set (that is, the preprocessingit-
et al., zi}l). The balancing of the
self) requiresfewer assumftions than regression does' and after prepro-
cessing ,h. t."ur-"nt group is much closer to being independent of the
confofnding variablesihan it was before preprocessing' Any subsequent
parametric idlurr*.ntr, then, are less consequential for inferences.The
implication is ihat two investigators who use the same data and variables
the sameresearchquestion are more likely to arrive at consonant
to
"dd..r, data by preprocessing than if thcy
results if they successfullybalance the
do not preprocessthe data. And, when we read an article where balancing

7Ho and colleaguesnote that their approach is similar in spirit to ideas in Imat and van
T h o mas (2000)' among others'
Dy k (200 4 ) , R o s " J a . t m a n d Ru b in ( 1 9 8 4 ) , a n d Ru b in a n d
I52 . Chapter5

has been achieved,we can be more confident that small changes in the be able to incorporate::-.-:--
authors' model specificationwould not greatly alter the paper's empirical tinely in our researchci. :-
conclusions.

INstRuuEr'ital- VAI{I.\8 . -
CoupenrNc Lrrt wrtn Lrr<r rnRoucH NATURALLYOccURRING R.q,Npor,r As sr cx r t r r-:
Reunotvr AsstcNl-ENt
The i nst r um entvcr i, r l', .
The purpose of random assignmentin social researchis to create matched a sort of par t ial r r r t Jor ' . -
groups, so that we can compare apples with apples,like with like. Social causei n t he lV nr et hr , ; :
scientists in every field understand and appreciate the power of random self, that is randomizec.
assignment in controlled experiments. Individuals in an experiment are To fix ideas.supfr,-i '.
assignedrandomly either to a treatment group that receivesX or to a con- assignedto X. Supposc:-- :
trol group that does not receive X. The notion is that we want to vary X that is, X and Y each [-.-r -..;
to seewhat happens to some outcome Y. Of course, lots of things can go Due to random assignnr.:-:
wrong in an experiment with human subiects becausewith humans it is X nevertheless is corr..,:.=*
hard to hold constant all factors except those under investigation' Even is depictedin figure -i.1. .' ,
under laboratory conditions, factors that we don't want to vary might denotcsall the othcr c-:...:-
vary anyway during the course of the experiment (seeCampbell and Stan- causei t showsall t h. . "- -
ley's 119661classic statement on threats to internal and external validity Our o bject iveis t , r r - : "
in experiments). In an ideal experiment, though, X varies, other causesof of usr two var iabler\ . ; . -
j
Y do not vary, and-due to random assignment-we expect no initial dif- Y when (as here) the r',:: -.
ferences of consequencebetween the two groups' so we can infer that al ent to est im at ing' \ . , '
subsequentgroup differences on Y are due to the introduction of group pri nci plesof pat h . r n. r.. - -
differenceson X. and Y i s a f unct ion of r
\fhy do we expect random assignmentto produce initially equivalent j ust one link. X- - + ) '. r : -
groups? The answer lies in probability theory. In controlled experiments, correl a t iondoes r cf lc. . ,
we attempt to harness the power of probabillly to produce indistin- causality.
guishable groups. The point I want to make in this section is that random In ob ser vat ion. r. l.
l r:...
assignment is also found in nature, so in some instances observational to X, so in obsert',tti,,:.: -
studies can also harnessthe power of probability to compare like with than X-+ Y. When lhr:r :
like. correl a t iondoes nor i: ': :
Until recently, discussion of the random assignment of subjects has Fi gure5. 3 depicr ' t h. i: -
been largely relegatedto experimental research,since observational stud- 5.2), but conf oundinc' r -
ies in many areasof social sciencerarely consider how to incorporate ran- X , si nceC causesX. I r : - -
dom assignment in the research design. This state of affairs might have ( l ) the c ausalef f ccrt r ' - '
come about becauseof a misimpression that random assignment is out- both X a nd Y. As . r r c: . . . .
side nature's domain and must be manufactured by researchers:Either we
can randomly assign individuals, in which casewe do an experiment; or
we cannot randomly assignindividuals, in which casewe don't give the con-
cept of random assignmentfurther thought. As a result, nonexperimental
social researchhas been dominated by lessreliable methods for removing Fi o rrre i I Fi -- -

the effects of confoundins variables.It is time to consider how we misht Effect of ,\ :


: : : - : :r-i t sma ll c hangesin th e
.:::: , ,-::r rhe paper'sempirical
'- r. :. r.-:-)'OccunnrNc
- r r: :: '-i.h is to creatematched
- - : : r'. h ke wit h lik e' S oc i a l
rr r i: - .::. the P ower of r ando m
-- . --,.- rn a n ex per im enta re
t: ,-r r- -:i receivesX or to a con-
- 1 :. .: :i,ti \1-ewant to vary X
: : ,.:).. iors of things can go
- :, . . -.r.r\r r r it h hum ansit i s
:- -: -.:-.jerinvestigation'Even
i: .. j :' : \\'ant tO vafy might
' , :: - : . : .re Cam P belland S t a n -
r - i : i:- ..iI and external validity
:: -: : . \ r.rries ,ot her c aus e so f
'- -'- --'... e expectno initial dif-
-: -.:>. so we can infer that
:-: . ::-- i n tro duc t ionof gr o u P
-: : : : :.l ce i nit ially equiv al e n t
- . : -:' ,ntro lledex per im e n ts ,
-,
:' -: :.:,', ro pr oduc e indisti n -
tr: i r i .r>i e ction is t hat r and o m
. - : : :nsta nc es obs er v at io n a l
:: - ; r. . ,:i to c om par e lik e wi th
,-, - ,.-:.iitnlent of subiectshas
:' :: : - -. .:n ceo bs er v at ionals t u d -
- : - :,: ilLl\\'to lncorporate ran-
, - . -:,r:<oi affairs might have
' ::.:: :..:-.iotl assignmentrs out-
:::-::: :'' researchers: Either we
-.- - . . , . .i I Jtr an ex per im ent:o r
',i : -. ,,rseu'e don't give the con-
- x ----------------*
:-: 1.: ,,-:esult, nonexperimental
::: :: .-::remethodsfor removing
: :.:'.: : , considerhow we might
The Fifth Rule . l-53
be able to incorporate naturally occurring random assignmentmore rou-
tinely in our researchdesigns.
IN srnulrENrAl VARIABLEs: Merc ntNG THRouGH Penrrel
ReNoon AssrcNtl.pNr
The instrument variable (IV) method for estimating causal effects involves
a sort of partial random assignment.The random assignmentis partial be-
causein the IV method it is incentiuesto take tredtment, not treatment it-
self, that is randomized.
To fix ideas, suppose X is a cause of Y, and individuals are randomly
assignedto X. Supposefurther that X and Y are standardized variables,
that is, X and Y each has a mean of zero and a standard deviation of one.
Due to random assignment,X is uncorrelated with the other causesof Y.
X neverthelessis correlated with Y, since it is a cause of Y. This situation
is depictedin figure 5.2, where X-+Y denotesthat X causesY and er+ Y
denotes all the other causesof Y. Figure 5.2 is called a path diagram be-
causeit shows all the links or paths from X to Y.
Our objective is to estimate B, the causal effect of X on Y. In the case
of just two variablesX and Y, the slope B equals the correlation of X and
Y when (as here)the variablesare standardized.So estirnatingB is equiv-
alent to estimating r(X,Y), the correlation of X and Y. According to the
principles of path analysis (Alwin and Hauser 1975), the correlation of X
and Y is a function of all the links between X and Y. In figure 5.2 there is
iust one link, X+ Y. So for the model depicted in figure 5.2 the bivariate
correlation does re{lect the causal effect of X on Y: correlation implies
causality.
In observational data, however, individuals are not randomly assigned
to X so in obseruational data we expect links between X and Y other
'When
than X-+ Y. there are links between X ar-rdY other than X+Y,
correlation does not imply causality (recall the secondlaw of causality).
Figure 5.3 depicts that situation. In figure 5.3, X causesY (as in figure
5.2), but confounding variable C also causesY, and C is correlatedwith
X, since C causesX. In figure 5.3, then, the correlation of X and Y reflects
(1) the causal effect from X to Y and (2) the effect of the confounder C on
both X and Y. As a result, in figure 5.3 (unlike figure 5.2) the correlation
Y
Figure5.2. ExampleWhereCorrelationr(X,Y) Is Causal
Effectof X on Y
l -54 . Chapter5

y€- tv
rhroughout
rhis discus.:,
rhat our object iveis t o es: . -
rhe correlarions of Z :. :-
,.t,1,1,) =n.
In shor t , if we can t lr : : . : -
hrrr has no dir ccr cl lc. : l
nd r(X , Z) pr ovides ili_ : - - -
,\- Lr
can si mply divide r he . , : : . . '
tnd Z: F- r ( Y, Z) / r lX. Z -- :
Figure5.3. Example'Where
Correiation
r(X,Y)Is Nor Causal
Effectof X on Y ' tandard ized)t he I Y eiir lr r : :
u'ith the covarianceof -\ :.: : _-
:ot,(Y,Z)/cou(X,ZJ=[i,,: .,. _--.
of X and Y gives a biased estimareof B, the causal effect of X on y.
B,r.=F when cou(Zl s. = :-
(Note that if B=0 then figure 5.3 reducesto figure 5.1. So figure 5.1-
F{enceif we can 6nd ., . .-- ,.
"correlation of X and Y is [entirely] due to common qasss"-is a special
:hn cou( Y, Z) / cor 'qX. Z: : _:
caseof figure 5.3.) ,,n Y .qThat is r he lc. , qr,. : : . . . '
Random assignmenton X alleviatesthe problem by attempting ro re-
move the empirical link betweenX and C. If the link is removed so rhat
.\.{M PLE: E FFE C T O F 5 . \ \ 1 E -. : ' .
X and C indeed are independent,then r(X,Y)=B. The difficulty in ob-
servationalstudiesis that X is not randomly assigned.The instrumental Intui ti ve ly,inst r um en[ . r ' : - . -
variables strategy is to try to find a randomly assigned variable Z that rr.rsby using onlr p. r r r , : : . .
causesX but not Y (except indirectly, through X-see figure 5.4). Be- ,orrel atedwit h r he coni, : - -_
causeZ is randomly assigned,the correlation of Z with the confounder .imilar to the fixed effe.:. .-: -
C is zero (in expectedvalue), so the correlation r(X,Z) is the causal ef- Consider again the .r,.'.
fect Z-+X (denoted a in figure 5.4). BecauseZ causesX, we have the .ehool . In dozensof , r , : ; . .
causal chain Z-+X-+Y. That is key to the IV strategy, since in the :r.rtel l ect ual advant ages,: - .
causal chain Z -+ X -+ Y the correlation of Z and Y is aB, the product of 5ated whether childrcn ..':
the causal effect of Z on X and the causal effect of X on Y (we assume :.reed i n cognit ive der . . : . - -
:hat l arge f am ily size h- . . .: . : : -
Jer.el opmentis well e\ f r : : : : _
Y(__r 1975 Psychology Tod,i.. :.:.
p.769) advisespar enr i: "- r : -
:mportant to you, the n: *:
:h:rn tw o , , , becauser hr . . , :
,f i ntel l e ct ualf unct ionin: , '
R esear cherconsir s t enr . . - .
-;ore l ow er on t esr sof c : -
j :uer
x (€x yea r s of schoolin, :- - ,

\l n statis ti c alparl anc e.

I\' ...:
\:ueger 200 1). A n es ti matorr: :,: _ ,
L ::,)n parameteraSthe s ampl e!r:: I -
-
" Becausethe IV method r: _.._r,:
Figure5.4. InstrumentalVariableMethod for EstimatingB - , rhod gqng l ,rl l , doe' nor \\,,:. ..

, tu
J
. ', : l. -\r.,rCausal
-- : -: . -l usa l ef f ec t of X on Y .
:: I . . :u re 5.1. S o f igur e 5' 1-
: , ::'.i'l1on g3u5g"-i5 a special
' : : - rl cttt b\ at t em P r ingt o r e -
-
.: :lc link is removed so that
.,-.: = B. The difficulty in ob-
: .. , -: . stgned.The ins t r um en ta l
: - ... -rssignedvariable Z that
:,: -.:i't X-see figure 5.4). Be-
ri. r' ,rf Z with the confounder
: ., : - : :r:1r\X,Z) is t he c aus ale f-
:-= -. <rZ ca l rs esX , we hav e t h e
:-: I\- strategy, since in the
: Z .:rr.l Y ts aP, the Product of
,: . :: : rct of X on Y ( we as s u m e
t'
2;, +--
a
":: : Es timat ingB
The Fifth Rule . 155
throughout this discussionthat the variables are standardized).Recall
that our objective is to estimate B, and note that we can estimate p frorn
the correlations of Z with Y and Z with X, since r(Y,Z)=aB and
r(x,z ) = a.
In short, if we can find a randomly assignedvariable Z that affects X
but has no direct effect on Y, then r(Y,Z) provides an estimate of aB,
and r(X,Z) provides an estimate of a. For an estimate of f, then, we
can simply divide the correlation of Y and Z by the correlation of X
and Z: F=r(Y,Z)/r(X,Z).ln general (whether or not the variables are
standardized)the IV estimator is the ratio of the covarianceof Y and Z
with the covarianceof X and Z, that is, B,r= cou(Y,Z)/cou(X,Z). Because
cou(Y,Z)/ cou(X,Z) = fcou(X,Z)B + cou(2, e")f/cou(X,Z), it follows that
Frr= F when cou(Z, e") =0, as we assumewhen Z is randomly assigned.
Hence if we can find a variable that is randomly assignedyet affects X,
then cou(Y,Z)/cou(X,Z) provides a way to estimate the causal effect of X
on Y.8 That is the logic of the IV strategy.
EXAM P LE: EF F EC T OF SAM E- SEX VER SU S D IF F ER EN T .SEX SIBLIN GS
Intuitively, instrumental variables solve the problem of omitted variables
bias by using only part of the variance in the cause-the part that is un-
correlated with the confounding variables.In that sensethe IV method is
similar to the fixed effects method.e
Consider again the erample of family size and children's successin
school. In dozensof studies sinceFrancisGalton (1874) proclaimed the
intellectual advantagesof being eldest born, social scientistshave de-
bated whether children who come from large families are disadvan-
taged in cognitive development and academic achievement.Concern
that large family size has detrimental effects on children's intellectual
developmentis well expressedby the famous title of Robert Zajonc's
1,975 Psychology Today article: "Dumber by the Dozen." As Zajonc
(p. 169) advisesparents: "If the intellectual growth of your children is
important to you, the model predicts that you should have no more
than two . . . because the larger the family, the lower the overall level
of intellectual functioning."
Researchersconsistently find that children from larger families in fact
score lower on tests of cognitive abilitS do worse in school, and attain
fewer years of schooling than do children from smaller families. The
8In statistical parlance, IV estimates are consistent but not unbiased (see Angrist and
Krueger 2001). An estimator is said to be consistentwhen estimatesconverge to the popula-
tion parameter as the sample slze lncreases.
eBecausethe IV method truncates the variance of the causal variable of interest. the
method generally does not work well with small samples.
156 . Chapter5

inverse association between family size and academic performance per- of familyvulues.
rh.,r' -
sistsevenwhen socioeconomicstatusis controlled (Blake 1989; Downey n o m i c sta tu s, i n te l l r g c- ,;,
1995. t able 3 ). of Y ).
The question,of course,is whether the associationsare causal. Skep- The causal leveragc::
tics can point to any number of W-variablesthat are difficult to measure, conditions: First, ser r::-. -
or to measure adequately, in standard regression approaches (Guo and must aff ecracader nr . . . . . .
Van'Wey 1999). Some potentially confounding factors, such as parents' first condition can be i..i-r ;
values and the intellectual climate of the home, are difficult to measure on the basisof theorr. \\ :
completely. If parents' values and intellectual climate of the home jointly to childbearing.Coniri. :
determine children's academic performance and decisions about family ported in Angrisr and f ', :-,
size, the observedassociationsare likely to be spurious. and one girl,37.2 perc.:.: -
Becauseof the difficulty of measuring confounders, researchersin or tw o g ir ls,43. 2 per , r : . : -
this field have sought alternativeways to relieve omitted-variablesbias. restrict the analysisro :::::
One possibility is to use sibling models (Guo and Vanlil/ey 1999), as censusfiguresfor 199i,,.-.
describedearlier. Another possibility is to use instrumental variables,as boy and one girl, 34.-1i.:,.
describedhere. boysor t wo gir ls, 40. - ; . - .
The trick for the instrumental variable approach is to find a ran- marri edwom en t he dir : =: - -
domly assigned Z such that Z -+X-+ Y. Here Y is school achievement The secondcondiric,-....
and X is family size. \7e want to know whether the observedeffect of controver sialbecausc: : *'-
X on Y is causal. To be more concrete, let's suppose Y is measured by si der,for exam ple.J. . - - ,
'What
failing a grade in school. we want to know is whether children in mix of children as an i:-.:: -:
large families would be less likely to fail a grade if they were in small ing (X) on women's l.rl : :
families. arguethat t her eis no. : : :
The problem is that family size (X) is not randomly assigned,so larger to affect labor force fr:: . :
and smaller families are likely to differ on any number of characteristics bearing, Mark Rosenz...,.-
other than family size. As a result we cannot infer a causal effect of X on coul d b e r he caset hr r r : : : - :
Y from an association of X and Y. To make causal inferenceswe seek a costsfo r sam e- sexanJ : . - - .
random Z that affects family size yet has no direct effect on a child's dren might be cheap. r r . - -
probability of failing a grade. share a bedroom), and :: ,
What we would like to find, then, is some sort of nature-provided ran- the mother works.
dorn assignment mechanism that affects family size but not children's A s th e Rosenzweigr : . ,
academic success(except indirectly', through its effect on family size). variables strategy is nor -'.:.
Conley and Glauber (2005) suggestthat the sex mix of children provides is the difficulty of findi:: :
such a random assignmentmechanism Z. Among women with two chil- causalvariableof interc::. '
dren, about one-half will have same-sexchildren and one-half will have omitted variablesbias a:.
mixed-sex children. Importantly, the "assignment" of same-sex versus It i s nor alwayseasvr o : : * '
mixed-sex children is random. So we expect women with two boys or two anisms that provide rhe s ::
girls to have the same preferencesin general as women with one boy and casefor an instrument.rl ',--,
one girl, with one important exception: If there is an underlying preference fects Y only t hr ough ir . - - '
for having at leastone child of eachsex,then women with same-sexchildren one should abandon rhc -.-
will be more likely to want a third child. In other words, among women need to do a berrerjoh : -
with rwo children, the sex mix of children-a random occurrence-affects ever it may be found in n.::-
decisions about childbearing (so Z-+X) but otherwise is independent forces researchersto rhr:.-. :

.-- ..-..;sr]]icPerformancePer-
- :- :: Bl ak e 1989; Down e y
-- -..,.uonsar e c aus al.S k e p -
:-..: -lrc difficult to measure,
.: ,:r eppro2ches(Guo and
:: :-rctors,such as parents'
^ :. -rrc difficult to measure
- .:rJIe of the home iointly
.: :.; decisionsabout family
: i-- r f i O L l S .
- - .rrrJ er s ,r es ear c hersi n
:', : Lrmitted-variables bias.
.i- ::.rd Vannfley 7999), as
i: 1:irulrefltal variables,as
r r : ', rrrr)Jch is t o f ind a r a n -
: :: : : : t i s sc hool ac hiev eme n t
, -: r-:r th e obs er v edef f ect o f
. :i : : ,.:Fo se Y is m eas ur edb y
' : . .. ','.'is rvhether children in
-: ', ::.:ie if they were in small
: :.,.:-.i,rnlr'assigned,so larger
- - '. ::.: e causaleffect of X on
-.-r : -.'...>.rlinferenceswe seek a
-r : - :l re ct e f f ec t on a c hi l d ' s
,,,- r : :: of nature-providedran-
:: r : . - ', .i ze b ut not c hildre n ' s
,--- :: r'ifect on family size).
:-.: -: '. :rir of children Provides
- ----. :-* \\'omen with two chil-
, :- r::':r lfld one-half will have
t:: l--rll:Ilt" of same-sexversus
'.-: ,,. :::<nwith two boys or two
'::: :. '.','omen with one boy and
r i--:r: : :n underlying preference
-:- '.,. :r':r ri'ith same-sexchildren
: .. :t--r \\'ords,among women
::.- -,-:.:r.iJom occurrence-affects
'. r--: ,,:henviseis independent
The Fifth Rule . 157
of family values, the intellectual climate of the home, parents' socioec<.r-
nomic status, intelligence, and so on (so Z is unrelated to other causes
of Y).
The causal leverageprovided by sex mix of children dependson two
conditions: First, sex mix must be related to family sizel second,sex mix
must affect academic successonly through its effects on family size. The
first condition can be checkedempirically whereasthe secondis determined
on the basisof theory. \fith regard to the first condition, sex mix ls related
to childbearing. Consider these figures from the U.S. censusfor 1980 (re-
ported in Angrist and Evans 1998, table 3): Among women with one boy
and one girl,37.2 percent had a third child; among women with two boys
or two girls,43.2 percent had a third child, a differenceof 6 percent.If we
restrict the analysisto married women, the difference is 6.8 percent. U.S.
censusfigures for 1,990 show the same patrern: Among women with one
boy and one girl, 34.4 percent had another child; among women with two
boys or two girls, 40.7 percent,or 6.3 percentmore, had another child (for
married women the differenceis 7 percent).
The second condition, that Z affects Y only through X, is often more
controversial becauseit must be establishedon the basis of theory. Con-
sider, for example, Joshua Angrist and William Evans's (1998) use of sex
mix of children as an instrumenr to estimatethe causal effect of childbear-
ing (X) on women's labor force participation (Y). While Angrist and Evans
argue that there is no compelling reason to expect the sex mix of children
to affect labor force participation decisionsaside from its effect on child-
bearing, Mark Rosenzweig and Kenneth Wolpin (2000) are skeptical. It
could be the casethat there are important differencesin the childrearing
costs for same-sexand different-sex children (for example, same-sexchil-
dren might be cheaper becausethey can wear hand-me-down clothes or
share a bedroom), and those costs might affect decisions about whether
the mother works.
As the Rosenzweig and \Wolpin critique underscores,thc instrumenral
variables strategy is not a panaceafor social research.The chief problem
is the difficulty of finding randomly assignedZs that are related to the
causal variable of interest. Studies using instrumental variables to relieve
omitted variablesbias are only as convincing as the instruments they use.
It is not always easyto find naturally occurring random assignmentmech-
anisms that provide the sort of leverageone needs to make a compelling
casefor an instrumental variable Z that measurably affects X and that af-
fects Y only through its effect on X. That difficulty does nor mean that
one should abandon the attempt, however. In observational research we
need to do a better job of harnessingthe effects of randomization wher-
ever it may be found in nature, and the IV strategy,fallible as it is, at least
forces researchersto think more concretely about how one mieht be able
158 . Chapter5

to employ randomization principles to alleviate the inherent limitations of

observationaldata.

MetcnrNc rHRoucH Natunerrv OccunnrNc ReNron AssrcNunNl

To rHE Tnr,errvrnNtGnoup

In the caseof instrumental variables, the cawseof tbe treatment is ran-

domly assignedin nature. We would get even more leveragefrom ran,
dom assignment if the treatment itself were randomly assigned,as in
controlled experiments.This section describesthat possibility, which I
call naturally occurring random assignment to the treatment growp. By
the term "naturally occurring" I do not mean that nature necessarily
made the assignments,but that the assignmentswere not done deliber-
ately, for the sake of randomization, as one would do in a controlled
experiment.
This type of matching strategy is best described by illustration. David
Freedmantells the story of John Snow's demonstration, in the 1850s, that
cholera is a waterborne infectious disease.Freedman 11991,p.29a) places
Snow's achievementin historical context as follows:
To seehis achievement,I ask you to go back in time and forget that germs
cause disease.Microscopes are available but their resolution is poor.
Most human pathogenscannot be seen. The isolation of such mlcro-
organisms lies decadesinto the future. The infection theory has some
supporters,but the dominant idea is that diseaseresultsfrom "miasmas":
minute, inanimatepoison particlesin the air.

Cholera, which had first appeared in Europe in the early 1800s, came in
epidemic waves. Victims became sick suddenly and often died. From his
clinical study of the course of the disease,Snow conjectured that cholera
was caused by a living organism that humans contracted through food or
drink and then passedthrough the body to infect new victims through the
water supply.
Snow did a great deal of detectivework to find evidencefor his theory.
If miasmas in the air causethe disease,then propinquity alone should ac-
count for the way the diseasespreads.But if the diseaseis waterborne, he
reasoned that the use of a common water supply is the key. It turns out
that London at the time presented a naturally occurring random assign-
ment mechanism for testing his theory. Some areasof London were served
by more than one water company, so in some districts next-door neigh-
bors receivedtheir water from different sources.Of two major water com-
panies in London, the Lambeth company drew its water upstream in the
Thames River, above the main sewagedischarge points. The Southwark

! :\ . R \\DoM AsstcNrytnur
--,r:::- ,/ tlte treatmenl is ran-
:'. rl lll(lf€ leveragefrom ran-
,, : : : : . li td ol rl l Yas s igned,as in
I
-:. :: ) : h at p os s ibilit Y ,whic h
. ,:. l:e
: : treatment group. BY
:-.:.'.--- rhat nature necessarily
Ll':l-i: \\'ere not done deliber-
: : . , , ,'rrl l ldo
d in a c ont r oile d
t'::.:t litl bv illustration. David
::-. r::rltion, in the 1850s,that
i:. =::i..tt.r| 1997, P. 29 4) Places
: I rr| S :
:.:\ .r. :rr.neand forget that germs
' : - .i rlicir resolution is Poor.
:.. l^. i.olation of such micro-
I . .:tfection theorY has some
...::.:.>:resultsfrom "miasmas":
: l:.
r : ': fh c earl Y 1800s ,c am e i n
::-:'..',.-rnd often died' From his
. i: ,i ;on;ectured that cholera
-:-: , ,nlr;1ctedthrough food or
.. : : r: n ew vi c t im st hr ough t h e
r. i i:rd evidencefor his theorY.
:: :: ,rinquity alone should ac-
: .: :.-.:Jiseaseis waterborne,he
:r :-^rrlr is the keY.It turns out
- - ': . i ;trrri n g r andom as s ig n -
of London were served
-:-:.,1s
-,.
. :-: .listricts next-door neigh-
,-:-::. [)f ttvo maior water com-
. J::'," lts \\'aterupstream in the
:-:,:.::e Points. The Southwark
The Fifth Rulc ' 159
and Vauxhall company, by contrast, drew its water below the sewagc
discharge points. In his own words, here is how Snow described the ex
periment:
Each company supplies both rich and poor, both large houscs and
small; there is no difference either in the condition or occupation of the
persons receiving the water of the different Companies. . . . As there is
no difference whatever in the housesor the people receiving the supply
of the two'$fater Companies, or in any of the physical conditions with
which they are surrounded, it is obvious that no experiment could have
been devised which would more thoroughly test the effect of water
supply on the progress of cholera than this, which circumstances
placed ready made before the observer.
The experiment, too, was on the grandestscale.No fewer than three
hundred thousand people of both sexes,of every age and occupation,
and of every rank and station, from gentlefolks down to the very
poor, were divided into two groups without their choice, and in most
cases,without their knowledge; one group being supplied with water
containing the sewage of London, and amongst it, whatever might
have come from the cholera patients, the other group having water
quite free from such impurity. (Snow 1855, pp. 74-75, quoted in
Freedman 1.991.,p. 297)
Snow's example is particularly compelling becausehe was able to find full-
blown "natural" random assignmenton the cause of interest-a rarity in
the social sciences.Social researchin this century will become more pow-
erful to the extent that we can more routinely follow Snow's nineteenth-
century example in our investigationsof human behavior.
CoupenrsoN oF STRATEGTES
FoR Coru.peRrucLrrn wrrH LrrB
Confounding variables are the bane of causal analysiswith observational
data. The methods described in this chapter attempt to deal with the
problem by comparing like with like. To evaluate the methods, we con-
trast the strategiesthey use for comparison.
In this section the key distinction is between X, the cause of interest,
and the Cs, the other causesconfounded with X. Earlier I distinguished
measuredcauses(Xs) from unmeasuredcauses(V/s). Confounding vari-
ables can be either measured or unmeasured,which I will denote C, and
C, respectively,where boldface is used to denote a vector of variables.So
there are three categories of variables to keep in mind: X, the specific
causeof interest or "treatment variable"; C* measuredcausesof Y other
tban X; and C* unmeasuredcausesof Y.
160 . Chapter5

T.q,srp
5.1.
y{-€y Comparisonof Strategies
for -\.

C:
in Observational
Studics

Strategyand Model

R egressi on w i th c ontrol v arr.ri - ..

t I'-V'): a+P 8,-X,)t r. \\ i :':
Model
Figure5.5. ConventionalMultivariateRegression Y andX arelinearfunction: .:; {

Propensity scores
Now consider figure 5.5, a stylized example of the confounding vari- B= tt, lt, where p, is thc :.r:.r
'Wewant to estimate the (average) Y for the treatment group -\ =
ablesproblem in observational data. B'
p . . .i s t h e m e a n o f Yfo r th e ;, ::-
causal effect of X on Y. The regressionof Y on X does not yield a good
g r o u p ( X =0 )
estimate of B since that method fails to separatethe causal effect of X on
Y from the causal effects of C on Y, where C is bolded to emphasizethat Instrumental variables
it denotes a vector of confounding variables (virtually always there ts Y , = a + P *, + e ,, w h e r eX i r .r '.:
more than one confounding variable). Figure 5.5 is a generalizationof fig- of Z.
ure 5.3 to the typical case of multiple confounding variables. In the gen-
eral case X and C are correlated, as indicated by the curved arrow
between X and C in figure 5.5. The curved arrow means that there are F i r e d - e f f e c t sm o d e l s( i l l u 'tr .r t- - - ,
causal effects from C to X, or common causesof X and C, or both. ;ifferencing method)
The key to estimating B lies in removing the confounding effects of the \Y,= a+ p AX,+ q, rvhere:
variables in C. The standard regressionstrategy is to regress Y on X and Y,r.-Yn for the itl.' L:::.:
log{r-log{, for rhc ::
C* the measured variables in C. Alternatively, we obtain the same re-
and X on Cn4,to create i in growth-rate moJ..
gi!tti"" coefficientsby ( 1 ) regressingY on C1u1,
Y,o-Y,o for the lrl: srb
and -*,-values of Y and X for individuals, predicted on the basis of the
models
measuredCs-and (2) regressingY,-t,onX,-*,. Thus the standardre-
AX. is defined in the same ri'.:.
gressionstrategycan be thought of as removing the effect of the measured
Key: B i s the caus aleffec t of \ ,.' : :
confounding variablesthrough residualization. Becausethe residuals {-
X i s the c aus alv ari abl eoi r.:r::.'
Y, and X,_X; are uncorrelatedwith those confounders, any associationbe-
Y is an outcome variabie,
t-ee" Y,-f, andX,-*,could not be due to the confounders that we have Z is an instrumental vrn.l. -
measured. So if we can accurately measure all the confounders' we can C is vector of confounder \ -:r : :.
use standard regressionto obtain an unbiased estimateof the causal effect C y i s vector of meas ured., :: . -,
of X on Y. That is the basic logic of the standard regression approach
( t able5. 1) .
In the case of propensity scores, as in standard regression, we must B eca usem easur ing. r . - -
measure confounders to control for their effects. Instead of controlling alwaysfeasible.one of ::: -'
through residualization,however,we control through matching on propen- to developalternatives::.::;a
sity scores,where propensity scoresrefer to the predicted probabilities of observational studies. O:.: ,
selectioninto the treatment group. Propensity scoresprovide an attractive random lyassignedr ': r : : ,
alternative to residualization because propensity scores typically elimi- i ndi rect ly,t hr ough ir s r : : - :
nate the need for strong assumptions about the functional form of effects w hereX is a linearir r ni: . -
and very often result in smaller standard errors despite the loss of cases. pendent of the orher ;:--::

\ lodel
, r^-r : 'i the confounding vari-
i"j.-: : .srimateB, the (average)
: i :: .\ does not yield a good
\: : r :- , : : :i :c ca us alef f ec tof X o n
:: : f , :.,L Je dt o em phas iz et h a t
: r: : i ',:rtually always there is
i-:: '- '- .> r generalizationof fig-
^ : -: *::rg va r iablesI.n t he g e n -
-- .- , . . :-i b v the c ur v ed ar r o w
:-, : i . : ::,)\\'me anst hat t her e a re
- r - -:: - .\ and C, or both.
- :-: ,, ,nfoundingeffectsof the
-
:ir:l:i' :i tO regressY On X and
--.1:.:... rve obtain the same re-
- f ,. .-.rdX on C* to create i
: :. . : : i:.ted on t he bas isof th e
',. - \ . Thus the standard re-
- AX, is definedin the sameway.
:::. : B:c.rusethe residualsY,-
| , -: ,.:-.Jers, any associationbe-
: ; . : -: -,,r-rfounders that we have
:-i: ': :te Confounders,we can
: : : : : .:: i]1t1te
of the causal effect
-: ::.: -i-rrd regressionapproach
. .:,. - : -rrd regression,we must
: r :rr: -i !. Insteadof c ont r ol l i n g
.: : -: :gh matchingon propen-
--. 1i .' r ednr nhabilit ie so f
:r: I :- ':esprovide an attractive
' r r : ': r'' scor est y pic allY eli m i -
, '-: : : :,:nctionalform of effects
:: : . :.'p i te t he los sof c as e s .
The Fifth Rule . 161
There5.1
Comparisonof Strategies
for Alleviatingthe Effectsof Confounders
in Observational
Studies
Strategy and Model Cent rul Assunrption(s)
Regressionwith control variables Cu=Ctthus({-i)and (X,-X,)are
(!,-t): a+P 1x,-x,7+au
where uncorrelated'lvithC, so ar-rv
Y and X are linear functions of Cy e f f e c r , , l t X , - X , r o n ( Y , - i,, i r ,,u ,
due to Cl.
Propensity scores
F= ltr- t"r, where pc, is the mean of That treatment and conrrol €lroupsarc
Y for the treatment group (X= 1) and balanced on C.
/16.is the mean of Y for the control
group (X=0)
Instrumental variables
Y,= d+ PX,+ e,, where X is a function That Z is correlated with X but not
of Z. with C nor with the error term of Y
(thus Z has no direct effect on Y).
Fixed-effectsmodeis (illustrated here with
differencing method)
LYi-a+p AX,+a, where: That the confounders are constant (over
( Y,r- Y,, f or rheith unit in changemodels time for change models, across sibling
ny . = log Y , r - l o g Y , , f o r t h e i l b u n i t ( n a t i o n .e r c . ) pairs for sibling models, etc.) andhave
|
{ in growrh-raremodels constant effects on Y.
t,- :-
I Y,r- Y ,, for thel rDsi bl i ngpai ri n si hl i ng
I models
Key: B is the causal effect of X on Y.
X is the causal variable of interest.
Y is an outcome variable.
Z is an instrumental variable.
C is vector of confounder variables,measured or not.
Cy is vector of measuredconfounder variables (a subset of C).
Becausemeasuringall confounders,or eventhe most crucial ones,is not
always feasible,one of the most pressingissuesin applied statisticshas been
to develop alternative strategiesfor alleviating the effects of confounders in
observational studies. one strategy is to use an instrumental variable-a
randomly assignedvariable Z that affects X but not C and affects y only
indirectlS through its effect on X. In the IV strategy we regress Y on *,
where X is a linear function of Z (tat:le 5. 1 ). The effect of X on y is inde-
pendent of the other causes of Y becauseX is a linear function of Z,
162 . Chapter5

where Z is independent of the other causesof Y (that is, the causesof Y correlationis all lr,eha,r .,
other than X). Moreover, because Z has no causal effect on Y except rhischaprer hcshecn:: :
t hr ough it s effe c to n X, th e s u h s ti tu ti o no f * for X doesnor bi asour esri - correlationdoesimpir ..
mate by mixing together the causal effects of X and Z. Therc is no inde- rhosecondirionshr -
Dendentcausal effect of Z to mix with the effect of X. p r o d u c ed r a r r .l o n r ,l 5 \l - ..- .r
In principle the instrumental variable approach is a very attractive alter- urul l y o ccr r r r ingr , r r . l' - .
native to standard regression control, since it suggestsa way to avoid matching.
omimed-variablesbias without the needto measurethe confounding causal C on sir , lsruglir r r h.
variables. (In that sense,the promise of the IV approach is similar to the $7hen we say r hlr "r : '
promise of controlled experiments and of fixed-effects models.) In prac- al l smo ker swill be \ i! . r ,
tice, though, it is often difficult to find defensibleinstrumental variables. avoi d i r . Rar her .r he - :
Another strategy for alleviating the effects of unobserved confounding smoker s ar e m or e ir k; :
variables is to use change models, growth-rate models, or sibling models. smoked and, sim ilar i'. _: : -
I group these three methods becausethey can be thought of as belonging cancerrhan if t hev h. r r - *
to a classof models called fixed-effectsmodels. Fixed-effectsmodels pro- ing on lung cancer.rhr:-....,.
vide traction for eliminating the effectsof confounding variablesby provid- rate w o uld have becn : : . .
ing two or more individual observationswhere X varies but C is constant. smokersif r hey had : r : . . - _
'When
we know that two or more observationshave the samevalues on C served,so they musr l.e :.: :
we in effect can match individuals on the confounding variables (for ex- to use the observedcr:.,:: :
ample, two siblings are matched on family environment since they have for smokershad ther : : .-
the samefamily environment; or the same individual at two points in time cer rates for srnoksr: . . -
is matched with herself on permanent individual traits). By using subtrac- smoked. I n ef f ecr \ \ '; . . ) : .
tion or some other method (Allison 2005) we can eliminate the confound- smokersas proxies it,: .:--
ing effect of unmeasured variables that are the same for the matched viously dependson r:: ::
observations(table 5. 1 ). smokersand nonsnrc.,k;:..,:
Becausefixed-effects models can alleviate the effects of unobserved smoki ng)r har leadr t , . . : - _-
confounding variables, they are attractive in a number of social science In the labor at or \ .\ \ - : -
applications. Indeed, as Charles Halaby (2004) argues,fixed-effectsmod- distinguishablear rhe tr,::..:
els are probably underutilized in social research.It is important ro note, u n its are ran dr_tntlt,r_. _.;. ,;. .
however, that the fixed-effects strategy assumes that X differs for the ment an d som e don'r . . , 1-
matched units (the matched units are siblings in the case of sibling mod- other is solely on the b-.-.:.
els, individuals measured at two points in time in the case of panel mod- researchertsypicallr Li. .
els, and so on), but that the variables in C are the same for the matched the caus eis r ar elr 'f ut r : - :
units. The fixed-effect method generally is most effective, then, in appli- ternati v em et hodsr o ln : : . _
cations where the causal variables of interest change over time (or differ common m et hod is r c! - . r .
across siblings, etc.) while the unmeasuredconfounders are stable. tempt to removethe cor:.::
set of ca usalvar iablesr , , : - .
both X (the causallari.r:- .
CoNc r us r oN the other causalvariab,lr... ,
si mi l ar b y m odelingr he
The idea in causal analysis is to find ways to circumvent the second law distinguishthem. The i,:--: -::.-
:.,
of causality, that correlation does not prove causation. 'Wewant a design w e can discer nt he ot he: . . . . . ,
or method that permits us to infer causality from correlation because l i abl y; ev enif we can r : . . , , , .

.:.:- : i :l-ratis, the causesof Y
ir - - ':::.l i e f f ec t 0n Y ex c e p t
'. - : .\ .'loes not bias our esti-
- \ : ' -:,.idZ.There is no inde-
--, -
'r
\-
. : : - . - l' :s :r very attractive alter-
' -- : r':i9$eStS a WaY tO avoid
- 'i . -..:. rhe c onf oundingc au s a l
.- : . '. .:.rFro ac his s im ilar t o th e
: - , . . :--i i e cts m odels . )I n P ra c -
: :-- : r -tlstrufitentalvariables.
'i:-i- : ,:nobservedconfounding
:--- ,: : :-..1l l els or, s iblingm od e l s .
,.,- -:- :irought of as belonging
'- .j: : f .red-effectsmodels Pro-
' -: , :-.::nqvariablesby provid-
-
i -:r: \. r'.rriesbut C is constant'
':'. i'- ',':',.e the samevalueson C
-: - . r ,::rJingvariables(for ex-
- :' . .:,rnfi]€rt since theY have
' : - .:-:.r].rrt r v o P oint sin t i m e
,: - .., :r.iits). By using subtrac-
' ., ,.,:-.elin.rinatethe confound-
-
:: :- : : :. sam e f or t he m at c h e d
:- ,:: :r- effects of unobserved
: ., :--!rnlberof social science
: ,::gues,fixed-effectsmod-
-
r:': . : , r. It i s i m P or t antt o no te ,
:-- . -:i rhat X differs for the
r - -- '. lhc caseof sibling mod-
: . : r:' the caseof Panelmod-
- [ .:. ::e same for the matched
: -' .i :ffective, then, in appli-
:::: -: - :'.-lnqeover time (or differ
:: - , . -: ',i n der sar e s t able.
: 1 i : -.:.umvent t he s ec ondl a w
r . : - , --i -rti o n. ' $7e
want a de s i g n
-: : t ::,)rn c or r elat ionbec a u s e
The Fifth Rule . 163
correlationis all we have (we cannot observecausalitydirectly). That's what
this chapter has been about. \Wediscovered that, under specific conditrons,
correlation does imply causality. The trick in causal analysis is to create
those conditions by comparing like with like-through investigator-
produced random assignment(as in a controlled experiment), through nar,
urally occurring random assignment, or through some other means of
matching.
Consider again the question of whether smoking causes lung cancer.
\il/hen we say that "smoking causeslung cancer," we do not mean that
all smokers will be victims of lung cancer,nor rhar all nonsmokerswill
avoid it. Rather, the claim "smoking causeslung cancer" means that
smokers are more likely to contract lung cancer than if they had not
smoked and, similarly, that nonsmokers are less likely to contract lung
cancer than if they had smoked. To investigatethe causal effect of smok-
ing on lung cancer,then, what we really want to know is what the cancer
rate would have been for smokers if they had not smoked, and for non-
smokers if they had smoked. Both of these rates are, by definition, unob-
served,so they must be estimated.The first strategy that comes to mind is
to use the observed cancer rates for nonsmokers to infer the cancer rates
for smokers had they not smoked, and similarly to use the observedcan-
cer rates for smokers to infer the cancer rates for nonsmokers had they
smoked. In effect we use smokers as proxies for nonsmokers and non-
smokers as proxies for smokers. The reliability of the proxy method ob-
viously depends on the reliability of the proxies-we must assume that
smokers and nonsmokers are alike with respectto the factors (other than
smoking) that lead to lung cancer.
In the laboratory, we try to make the treatment and control groups in-
distinguishableat the outset through random assignment.In other words,
units are randomly assigned to the cause units receive the treat-
ment and some don't, and the placement of units inro one group or the
other is solely on the basis of chance.Outside psychology,however, social
researcherstypically use observational data, and random assignment to
the cause is rarely found in observational data. Thus we need to find al-
ternative methods to investigator-producedrandom assignment.The most
common method is regression using covariates; that is, researchersat-
tempt to remove the contaminating effects of other variables by adding a
set of causal variablesto the regressionequation for Y, thus residualizing
both X (the causal variable of interest) and Y (the outcome of interest) on
the other causal variables.In effect we try to make dissimilar individuals
similar by modeling the effects of the causal variables (other than X) that
distinguish them. The limitations of this merhod are well known. Even if
we can discern the other causes,we might not be able to measurethem re-
liably; even if we can measure them reliably, we might not be confident
164 . Chapter5

about the proper functional form of their effects on Y (for example, A P P E ND IX

whether the effects are linear); and even in the caseof perfect knowledge
and measurement,we might need a large sample for reliable estimates,es-
pecially when the other causesare numerous and highly correlated. StudentExercisc.
In light of the problems with standard regression analysis, modern
causalanalysisin the social scienceshas focusedon the searchfor meth-
ods to supplement or replaceregressionas a tool for estimating causal re-
lationshipsin observationaldata. The aim in each caseis to compare like
with like. One general strategy is the matching of units exactly, or match- Manrt.q,rSratus.Hr,..
ing on the basisof propensity scores.Another strategyis matching through V E R SU SFIxTp - L i I .
repeated measurefrent: \7e use repeated measuresof the same unit over
time, or the samemeasuresof similar units, such as siblings.A third strat- Theassignmenr ior *::.,:-
egy is to use naturally occurring random assignmenteither on the treatment statuson happine...R=,r
itself or on a causeof the treatment. erci se sat t he end uf . : - :
Bv using one or more of the analytic strategiesabove-regression, ex- questionsin the eariie: :-,
act matching or matching with propensity scores,fixed-effectsmodels, or The assignm enrf . : - - .
methods that exploit naturally occurring random assignment-our aim is models.\ Wecould f lor . : : - :
to reduce the uncertainty of causal inference from observational data. In C S Sco nsist sof r epc. r : . - -
addition to the threats posed by sampling and nonsampling error linked use this time is callei .- : :
to data collection (discussedin chapter4)-problems endemicto experi- vi ded b y sociologir r .I -
ments as well-unmeasured confounding variables greatly complicate i s a nat ionalpr ohr hr '. . . .
causal inference from observational data. As our data sets become larger, samplingof t hoser i i:
sampling error contributes relatively less to uncertainty, and the other respondent sar e Jqc - , -
sources contribute relatively more. Hence forward-looking social scien- 1995 a nd r eint er vicr , *. .
tists will shift their attention to the other threats to the accuracy of infer- The dat a ar e ar . l; . - : . .
ences in social research. Omitted-variables bias and selection bias are P ol i ti caland SocialRc: : . , :
chief among these other threats. IC P S R is a t r em endo! . >: : .
Finally, observe that the strategiesfor employing rule 5 generally in- this assignmentis tc'r::-::
volve better researchdesign.This observation underscoresa central theme has i ts own SDA sir e . '. . .
of this book: that, to improve social research,we need to focus on better S D A sit e, so you do r : - .
conceptualization and design of our research,not on more sophisticated plete this assignmenr.
statistical estimation methods. Indeed, I would argue that causal analysis Y ou can accessl( P>:
in the social sciencesis largely an issue of proper research design. If the si ty thar is a m em hcr
design is faulty, powerful estimation methods cannot be counted on to you do not hat e ACCe.:. .
salvagethe project. Sophisticatedstatistical estimation methods may re- abl ero com plct et hii . i. . - -
sult only in more preciseestimatesof the wrong thing (a theme that I will To accessr he . \ sO r
take up again in chapter 7). In addition, I want to stressthat, although .i cpsr. um ich. edu, cnr . - -
rule 5 itself is universally applicable,there is no single all-purpose method Thi s will br ing up r r . : -
(for example, fixed-effects)that is appropriate for all causal analyses.The name. CI ick "onlin. . . '
methods describedin this chapter are tools. As with all tools, the value of Subset-Study No. -i : :-:-
the tool dependson the skill of the worker in knowing when and how to A nal ysisUsing SD\ " : '- -
use the tool. interface.Before the :Ir.-.
 : - : '1 :i l r o n Y ( f or ex am p l e '
- .- : , .,s tl f per f ec tk nowled g e
: * .. : , : re i i ablees t im at es re s -
, - : ', - - -::hl r c or r elat ed.
.-: ' :- i i ])r)n a naly s is rm O de rn
'' r - -: - I rh e s ear c h f or m et h -
- r - : 'r cstimatingcausalre-
- - : : . .-. i .i sc i s t o c om par elik e
-- i -::'ritsexactly,or match-
--
,: : : - i: : '.:.fY i s m at c hingt hr ou g h
- : . . . ' .. ,'i the s am e unit ov e r
:: !--- :.-.t.'itheron the treatment
-- - : -: _ . . ..1 .r-1
s - lggr es s ion, e x -
- r r: . . t re d-effectsmodels,or
- ': - - ' .'...i {n t nent - our aim i s
:-,-: - -:-.observationaldata. In
. ::san.rpling error linked
: :- -
- - : ' '.::11\ endem ict o ex P e ri -
-: . .-.:.esgreatly complicate
:-- - : i.rra sets become larger,
-: i ..: a:rt:IintY, and the other
,-: r -'.r.:':J-lookingsocial scien-
- r - -: :t - :, , the accuracyof infer-
- :- - ,: ,lrld selectionbias are
n : -: . :nS rule 5 generallYin-
:: - .--J;rscoresa centraltheme
;::- -. .t.: r'Lr-ed to focus on better
:::- -. : on more soPhisticated
"r ,- : .,::Llethat causalanalysis
: :: ::: researchdesign.If the
:!- j- :.trlllot be counted on to
r'- i : i, ::.rti o n m et hodsm ay re -
: . i : - : :rl n g ( a t hem et hat I w i l l
-- - .,:': :() stressthat, although
:: - - . rsle all-PurPosemethod
,::-.:: : : .tll causalanalYses'The
- --- .,.::h all tools, the value of
. :r - ::-.,,u'ingrvhen and how to
AP PE N D IX
StudentExercises
on Rule 5
MenrtRr Slatus, HEerrH, AND HAIITNESS:CRoss-sEcrloNAL
VERSUS Frxpo-EpFscrs Moosrs
The assignmentfor chapter 5 investigatesthe effects of health and marital
status on happiness.Recall that we also examined those effects in the ex-
ercises at the end of chapter 2. You should review your answers ro the
questions in the earlier assignmentbefore beginning this one.
The assignmentfor chapter 5 differs becausewe use fixed-effects (FE)
'We
models. could not use FE methods in chapter 2 becausethe 1972-2004
GSS consists of repeated cross-sectional,not panel, data. The datasetwe
use this time is called Aging, Status,and Senseof Control (ASOC), pro-
vided by sociologistsJohn Mirowsky and CatherineRoss (1999). ASOC
is a national probability sampleof American adults in 1995 with an over-
samplingof thosewho were sixty yearsor older in 1995. (Nearly half the
respondents are age sixty or older.) Respondents were interviewed in
7995 and reinterviewedin 1998 and in 2001.
The data are available through the Inter-University Consortium for
Political and SocialResearch(ICPSR)at the University of Michigan. The
ICPSR is a tremendous resource for social research, and one purpose of
this assignmentis to introduce you to this resource.Moreover, ICPSR
has its own SDA site with an interface identical to thar of the Berkeley
SDA site, so you do not need to learn a new computer program to com-
plete this assignment.
You can accessICPSR data only from a network provided by a univer-
sity that is a member of ICPSR. Many universities belong to ICPSR. If
you do not have accessto the network, then unfortunately you will not be
a b l eto compl erethi s assi gnment.
To accessthe ASOC data, go to the ICPSR homepage at http://www
.icpsr.umich.edu,enter *3334" in the "Search" box, and click "Search."
This will bring up the dataset, with a list of oprions underneath the
name. Click "online analysis." When the page labeled "Analyze and
Subset-Study No. 3334" comes up, go to the section titled "Online
Analysis Using SDA" and click on rhe link indicated to bring up the SDA
interface. Before the SDA interface appears,you must log in and agree to
166 . Chapter5

a confidentiality statement. (You will need to create an ICPSR account if Now r:ecodemarst3r-
you do not already have one.) variablecalled married
-.
The directions for recoding variablesshould be familiar by now; if you (code das l) . Your enr : . : :
need a refresher,refer to previous chapters. As before, all variable names marri e d,l; 0, not nr . r : :. *
appear in boldface. to MD code,and clic\ '. -
Finally, recode healih
goc,d.guod. si. r [ islJ, t: .
Cross-sectional Relationships between Marital Statws, Health,
heal th_r eso t h; r hr l: , : . -
and Happiness (1995 Data)
the " Recoding,Rulc: ' .
Our dependentvariable,huppy, is a count of the number of days in the past satisfactory,3; 2. pc,,,:.-.
week respondentsfelt happy. Note that huppy refers to the L995 data, i s eonv er t edt o \ lD i *. .
happy2 refers to the 1998 data, and happy3 refers to the 2001 data. The is an ordinal varial.lt. ,',.
other variablesare similarly named. This protocol is important for the sub- pl i ci ty. You shouldh. :. .
sequentFE analysisbecauseit enablesus to create change scoresfor indi- 'Wecan now run rl--:
viduals simply by subtracting variables(e.g.,happydiff =happy3 -huppy). regression" from the "
-
The frequency distributions of huppy, happy2, and happy3 show that dent" box, then married ,.
ASOC respondentsare generally a happy bunch. In all three years the vast "
" R un regr ession. \ , , - : - -
majority of respondents report being happy all seven days of the past
week, and only a tiny fraction say that they did not feel h"ppy at all over Quest ion1. ( a) Do m . r : : .
that period. ple of mostly older -\:::.:
We first estimate a model using the 1995 cross-section.In this analysis (b) D e scr ibehos or ' : : , -
we regresshuppy on marital status and health. Strictly speaking, huppy causalef f ect sof nr . r : - . - .
is not a continuous variable since it is bounded by 0 and 7.The more im- or more om it t ed \ '. r :. : : . : -
portant problem, however, is that the distribution of huppy is heavily
skewed-as just noted, the vast majority of respondentsare in the highest C ri ti cs m ight poinr : : - .
category, "h"ppy all 7 days." So ordinarily we might dichotomize huppy abl es:whet herbec. r r r . -
and use logit or probit regression. For the sake of simplicity here, how- effectsof ear lv chilJl: -
ever,we use ordinary leastsquares(OLS) regression. "naturally" happr' .rr-: :-
\7e need to tidy up happy a bit before we can use it, since the variable happier people Are n-. ::
treats missing-datacodes (98 and 99) as valid responses.Becauseit is im- mai ntaina sr ablenr J: : , r .
possibleto have been happy for 98 or 99 of the past 7 days, we need to re- Our cr oss- see r iur r . r-. . - - ,
code this variable to eliminate those values. Select "Recode variables" w e ccn t ake advant . r : .
under the "Create new variables" menu, enter happy-re as the name of changein m ar it al st . r : - : .
your new variable (make sure that you selectthe option to replaceany ex- By restricringrhe .rrr.r. - .
isting variable with the same name if it already exists), and enter happy as el i mi na t er he ef f eer , , : - '
the name of the variable to recode. In the first row enter "0" for "Value," to be happierthan ori:':,
"no happy days" for "Label," and "0" for "Var 1." (You need to fill in
rows 2-8 for values 1-7 only if you want the value labels on your print-
Maital Status, Healtl:, .;,:
out.) Select"assign the value of input variable #1" in the "'S7hatto do with
unspecifiedcombinations of input variables(if any)" option (this is impor- For the 1995 ASO L u , : . : .
tant). Then scroll down to the "Optional specificationsfor the new vari- cises),we find thar -\:::.: ,
able" section, where you should enter "0" in the "Minimum valid value" Americanswho are fl1,r. :l
box and then "7" in the "Maximum valid value" box. Click "Start Recod- thosewho ar e in poor : . =
ing." You should now have 1,130 valid cases,with a valid range of 0 to 7. the curr ent chapt er . . r . - -

:. ...1ICPSR account if
-,..:rliar by now; if you
j married, 1; 0, not married, 2-6.Make sure that everythingelseis converted
:.. all variablenames
t-:r : -:.' i :-;rtt,s.Health,
: : - : - -.::'.:.rof daYsin the Past
l^il,:; -: ::s ro the 1995 data,
:' , l - : r: :- :', the 2001 dat a' T h e
': i ,- : tllPoftdflt for the sub-
r --- r: .r.rllge scoresfor indi-
: :,:::', Jiff = happy3- h"ppy).
::::., 1. ..rd happy3 show that
: -- - -.-.-:ll tl.rree,vearsthe vast
::: ,- ::',en da,vsof the past
': - - : :eel h"PPYat all over
.- -:Jf i o n. I n r his analY s i s
.::-;rlv speaking,h"ppy
rn J - . T he m or e im -
: :: of h"ppy is heavily
: -:-nrs are in the highest
. :lr dichotomizehappy
, : .inrplicity here, how-
,- -.-- it. sir-rce the variable
: :rses.Becauseit is im-
. -: : ::.: - Javs,rve needto re-
"R ec ode v ar iabl e s "
- :- ': :-:
:':::- l.lpp\'-re aS the name Of
: - i t - : t::rrrl t o r eplac eany e x -
:: ,: : '..:i ! . and ent erhappy a s
- - -- - ' ::t!c' r" 0" f or " V alue ,"
- : -', .r: ,." \ bu needt o f ill i n
i r - : . -.. l rb els on Y our P r i n t-
i - : = . :--:he "What to do with
,'pr ion ( t his is im po r-
-t:- -- t- ,n \ ior t he new v a ri -
- " -' -\l i n im um v alid v alu e "
: --: .' t. Click "Start Recod-
:;.:.. i ::. .,.r'alidrangeof 0 to 7.
The Fifth Rule . 167
Now recode marstat-the mariml status variable-into a dichotrxrulus
variable called married denoting whether a respondent is currently rnarried
(coded as 1). Your entriesinto the "Recoding Rules" boxes should read: 1,
to MD code, and click "Start Recoding." You should have L,1,42valid cases.
Finally, recode health ("In general, would you say your health is very
good, good, satisfactory,poor, or very poor?") into a new variable called
health_re so that higher values represent better health. Your: cntries into
the "Recoding Rules" boxes should read: 5, very p;ood,1; 4, good, 2; 3,
satisfactory,3;2, poor, 4; 1,,very poor, 5. Make sure rhat everythingelse
is converted to MD code, and click "Start Recoding." Although health_re
is an ordinal variable, we will treat it as an interval-levelvariable for sim-
plicity. You should have 1,138 valid cases.
'We
can now run the regressionon the data for 1995. Select"Multiple
regression" from the "Analysis" menu. Enter happy-re in the "Depen-
dent" box, then married and health_re in the "Independent" boxes. Click
"Run regression."You should have l,l)2 valid cases.
Question 1. (a) Do marital statusand health affect happinessin this sam-
ple of mostly older Americans?If so, describethe direction of the effects.
(b) Describehow omitted variablesmight be biasingtheseestimatesof the
causal effectsof marriage and health. Be specific-in each case,list two
or more omitted variablesthat are likely to have confounding effects.
Critics might point to the potential endogeneityof the independent vari-
ables: whether becauseof genetic inheritance or becauseof the enduring
effectsof early childhood socialization,perhapssome individuals are just
"naturally" huppy and others are naturally unhappy. And perhaps these
happier people are more likely to have had the efficacy and energy to
maintain a stable marriage and good health over their lives.
Our cross-sectionalregressioncannot rule out this possibility. However,
we can take advantageof the 2001 follow-up wave to investigatewhether
change in marital status or health is associatedwith change in happiness.
By restricting the analysis to within-individual variation in happinesswe
eliminate the effect of any stable underlying tendencyof some individuals
to be happier than others.
Maital Status, Health, and Happiness: A Fixed-effects Model
For the 1995 ASOC data, as well as for the GSS data (seechapter 2 exer-
cises),we find that Americans who are married tend to be happier than
Americans who are not, and that those in good health are happier than
those who are in poor health. However, as noted in chapter 1 and again in
the current chapter, association does not prove causation. Perhapsthere
168 . Chapter5

are hidden variablesthat causeboth health and happiness,or both marriage should have 1, 1l6 r - . .r : -
and happiness.Or perhaps there is reversecausation, as just noted in the as for marrdiff,
discussionof endogeneitybias. N ow r un t he f ir e; - . : : .
With FE methods we can gain leverageon the hidden variablesissue as der th e "Analy si<" n: . . .
well as on the reversecausation issue.The cost of FE models is that they and nolongermarried. n..
truncate variance becausethey use only the within-unit variance. In the vari ables.Click "Rur : . - :
caseof panel data for individuals, "within-unit variance" means individ-
ual change over time. Here individual change over time refers to change Question2. O r r r lr r 1. . -
in marital status and health status among Americans, with an oversam- (a) D o cs healr hsr r ll | . -
pling of those age 50 and older. Becausethe possibility of failing health pi ness?Conr par er hc I : .
and the death of one's spouse is greater among the elderly, this sample one sl openot ablr sr ir : : - .
likely provides more over-time variance on the key variablesthan would, reassuring?
for example, a sample that focused on adults who are age 30 to 50. (b) D oes m ar r iagesr : . . - .
The first step is to alter the variablesin the 2001 wave as above.The re- happi ness? Do vor r f : - '
coding and computing techniques are the same; only the variable names effectf or nolonger m ar r ,:.
differ. Recode happy3 into happy3_re (you should have 1,128 valid pi rressin t he cr oss- 'c. : . '
cases),marstat3 into married3 (you should have 1,118 valid cases),and a few sentencesof pl,:-:-.
health3 into health3_re(you should have 1.,137valid cases). riage on happiness.
Next, compute difference scores for happiness and health. Click on (c) R e f er hack t o ) o. . - .
"Compute a new variable" under the "Create variables" menu. In the omitted variables rhr:
"Expression to define the new variable" box, type happydiff=happy3 herealleviat es1or i. r r , - :
_re-happy_re. Selectthe option to replacethe variable if it already exists, (d) On r he basi. of , r l- . .i-
make sure that the "No" option is checked beside "Include numenc of mar it al st ar us. r nJ : : '
missing-data values in computations?," and leave everything else the
same.Click "Start Computing." You should have 1,114 valid cases.Re- Question3. Comp.rr.:-.
peat this procedure for marrdiff=married3-married (this should give How do you accounr:, : :
yor 1,116 valid cases)and for healthdiff=health3_re-health_re (this
should yield 1,131 valid cases).Keep a copy of the output becauseyou Quest ion4. ( a) Vhar : . : - ,
will need it for question 4 below. reported health from 1-- i
Observe that marrdiff has three values: -1 for those who losr a spouse or get worse?What f.:-;:
(due to death or divorce) from 1995 to 2001; 0 for those who did not port worse health:
change marital status (that is, remained married or unmarried); and 1 for (b) rXlhat percentage , : :
those who gained a spouse.To make that clearer in the analysis,let's cre- from 1995 r o 200 | : \ \ : , :
ate two dummy variablesfrom the marrdiff variable-one for being mar- ri ed? What per cenr Jss. . . .
ried in 1995 but not married in 2001 (nolongermarried), and another for (c) D o t he r esult sior - . .
being married in 2001 but not married in 1995 (newlymarried).Our ref- spondents?Explain.
erencegroup is those whose marital status did not change.
To create the dummy variables, return to the "Recode variables"
screen.Recode marrdiff into nolongermarried: you should recode scores Opl ronel Fr xso- Er r r ,
of -1 on marrdiff to scores of 1 on nolongermarried, and then marrdiff WHo Use SAS
scores of 0 or 1 to nolongermarried scores of 0. Then create newlymar-
ried: codes of 1 on marrdiff should remain 1 on newlymarried, but codes Thi s ex er ciseis desigr '.-
of 0 or -1 on marrdiff should become codes of 0 on newlymarried. You son' s e xcellentbook I : . .

:- - .::.:iss. or both marriage
:l- ls just noted in the
..:Jen r-ariablesissueas
' !-E nodels is that they
:---rnitt ar ianc e.I n t h e
'r. ':rr' r m c ansindivi d -
: -lnc refersto change
-..:.>.\ \ it h an ov er s a rn -
., .-:1 i r.of f ailing hea l th
:-.. elderlr',this sample
. '.:ri a blest han woul d ,
.:.:iJs e 30 t o 50.
, .,.-rre as above,The re-
'- ', rhe variable names
-..j h av e 1, 128 v a l i d
-.l l Sv alidc as es ) , an d
.- !- !.1 \15/ .
. .:nd health. Click on
..-::-rl.les" menu. In the
.:: happydiff=happy3
.::ie if rt alreadyexists,
..:.ie "]nclude numeric
,'.: evervthing else the
r_: - -. . 1.11+ r - alidc as es R . e-
i,"i: - :-l :rried (this should give
r.. tj _ -. r
:h I re - healt h- r e ( t h i s
.: :r'-. orltput becauseyou
- : :,,r th os e who did n o t
-- i - : * : u nm ar r ied) ;and 1 f o r
r - : r -: - ,: the analYsis,Iet's cre-
::l -,- ..: .-one for being mar-
r'ur,ci;::r.r,1rriedt. and another for
.: - * -: ; hange.
* : : ::-. "Rec ode v ar iable s "
irrrJ -- .hould recode scores
:c':.::r.-rrried. :rnd then marrdiff
--. : . Tren createnewlymar-
. '. n.'*'lvmarried, but codes
,l:- : ,rr.rnewlymarried. You
The Fifth Rule . 159
should have 1,116 valid caseson both new dummy variables-the silme
as for marrdiff.
Now run the fixed-effectsmodel. Click on "Multiple regression"un
der the "Analysis" menu, then enter happydiff as the dependenrvariable
and nolongermarried, newlymarried, and healthdiff as the independent
variables.Click "Run regression."You should have 1,091 valid cases.
Question 2. On the basis of your results for the FE model:
(a) Does health still have a statistically significant, positive effecr on hap-
piness?Compare the FE slopewith the slope in the cross-secrionmodel: Is
one slope notably steeper,or are they roughly the same size?Is this result
reassuring?
(b) Does marriage still have a statistically significant, positive effect on
happiness?Do you find a positive effect for newlymarried and a negative
effect for nolongermarried? Given the positive effect of marr:iageon hap-
pinessin the cross-sectionanalysis,are you surprisedby the FE results?In
a few sentencesof plain English, summarize the apparent effect of mar-
riage on happiness.
(c) Refer back to your answer to quesrion 1(b), above.For each of the
omitted variables that you listed, explain how the use of the FE method
here alleviates(or fails to alleviate)their confounding effects.
(d) On the basis of the FE results, are you more confidenr rhat the effects
of marital statusand physical health on happinessare causal?Explarn.
Question 3. Compare the R2 values for the cross-sectionand FE models.
How do you account for the fact that the FE model explains lessvariance?
Question 4. (a) What percentageof the respondentsexperiencechange in
reported health from 1995 to 2001? On average,did the health improve
or get worse? What percentagereporr better health? \X/hat percentagere-
port worse health?
(b) \Vhat percentage of respondenrs report a change in marital status
from 1995 to 2001 ? What percentagechanged from married to not mar-
ried? What percentagechanged from not married to married?
(c) Do the results for (a) and (b) seemplausible,given the age of the re-
spondents?Explain.
OprroNer Flxro-Eprscrs ExERCTsE
FoR STuDENTS
Wuo Usr SAS
This exerciseis designedfor students who have accessto SAS. Paul Alli-
son's excellent book Fixed Effects Regression Methods for Longitudi-
170 . Chapter5

nal Data Using SAS summarizes fixed-effects models and how to estr- A P P ENDI X
mate them with SAS.Chapter 2 of Allison's book contains a running ex-
ample of the use of fixed-effects methods to estimate the effect of family
poverty (a dichotomy) and child's self-esteem(a scale)on antisocial be- SelectedFunher R
havior (a 7-point scale)using a sampleof 581 children who were inter-
viewed in 1990,1.992, and 1994.

Assignment: Read chapters 1 and 2 of Allison. Then replicate the results

reported by Allison in outputs 2.1,2.2, and 2.4 through 2.9. (SeeAlli- On the cortnterf.lctu,i. .;: t -
H ol l and,PnulI ( . | , r r r ' l'-
son's page 11 for the web site for downloading the data. He provides SAS
S t , t !i sl i L o r l Assu t ': . '
l .t/;
codes in chapter 2.) Does family poverty appear to affect children's anti-
M o r g a n . Ste p h e r rI .. :- _
social behavior, controlling for child's self-esteem?
Causal Infcrorcc: \'., :
C a m b r i d g eU n r r e r i r . . -
\7inship, Christopher. ...::: -:
effectsfrom ohscn rr

On cxperintentsatr,l t:,.:, -
C a m p b e l l , D o n a l d 1 .. _
L x p er i m e nt a l P4 - r 1 - 3 ,'.'
G h o s h . S., a n d C . R . R '
Analysis oI Exptr:..: .':

On fixed-effects r trrt,i, .: :
A l l i s o r r ,Ptu l . 2 0 tt i . F:r ' r : - -
i r g SAS.C a n . \t : \.:. - .
Halaby, Charles N. I , - -,
p r a c ti ce ."An n tttl R , . .. ..

On instrumental t',tri.;:.. ,
A n g r i s t. Jo sh u a D .. .r r .i .
s e a r ch fo r i d e n r i ti ..r :: .
Joutnal of Ecort,,ntt.' . -
On matching metho.l::
B e c k e r. S. O., a n d \. .- .
h r s e d o n p r o p e n \i r \'.. - ..
H o . D a ni e l E., Ko r r r k. I:- '
r n Pa, s n o n p a r a l n e r r r :-
. -:.- -
r i c c au sl l i n fe r e n ce .' . . '
M o r g a n. Sr e p h e nL .. .r n .,.
e f f e c ts: Pr o sp e cr ..l n J : :- .
and RcsearchJ i; i-a
R u h i n , Do n a l d B. l e - l . . -
t o r e m o veh i a s i n o h ..: .
l 9 - 9 . "U si n l tr :.:-
t o c o n tr o l b i a s i n ,,1 ....
A s s o ci a ti o n - 4 :i l q - l :
 ;:ls and how to esti- A PP E N D IX
' - :1t.1ifls a runnlng ex-
:'.-.rr.rhtseffect of farnily
:.-.le on antisocialbe- SelectedFurther Reading
^ . .i ren rvho were inter-

- I^:i-. replicatethe results

' On the counterfactual approach to causal inference:
,- : - : ::r:ough 2, 9. ( S eeA ll i - Holland, Paui \7. 1986. "Statistics and causal inference." Journal of the Arnerican
:: -. : r- j rta . He P r ov idesS A S Statistical Association 8l :94 5-7 0.
1 : :': : . . :, -tffe c tc hildr en' san ti - Morgan, Stephen L., and Christopher Winship. 2007. Counterfactuals and
.:..:: Causal Inference: Methods and Principles for Social Research. Cambridge:
Cambridge University Press.
V/inship, Christopher, and Stephen L. Morgan. 1999. "The estimation of causal
effects from observational data." Annual Reuiew of Sot:iobgy 25:659-706.

On experimentsand quasi-experimen!s:
Campbell, Donald T., and Julian C. Stanley. 7966. Experimentdl and Quasi-
Experimental Designs for Research.Boston: Houghton-Mifflin.
Ghosh, S., and C. R. Rao (editors).1996. Handbook ctf Statistics13: Design and
Analysis of Experimenls.Amsterdam:ElsevierScience.

On fi x ed- effe cts models :

Allison, Paul. 2005. Fixed Effects RegressionMethods for Longitudinal Ddtd Us-
lzg SAS. Cary, NC: SAS Press.
Halaby, Charles N. 2004. "Panel models in sociological research: Theory into
practice." Annual Reuiew of Sociology 30:507-44.

On instrumental uariables:
Angrist, Joshua D., and Aian B. Krueger. 2001. "lnstrumental variablesand the
search for identification: From supply and demand to narural experimenrs."
Journal of Economic Perspectiues15:59-85.
On matching methods:
Becker, S. O., and A. Ichino. 2002. "Estimation of average rreatmenr effects
based on propensity scores." The STATA lournal 2:358-77.
Ho, Daniel E., Kosuke Imai, Gary King, and ElizabethA. Stuart. 2007. "Match-
ing as nonparametric preprocessingfor reducing model dependencein paramet-
ric causal inference." Political Analysis 15:199-236.
Morgan, StephenL., and David J. Harding. 2006. "Matching estimatorsof causal
effects: Prospects and pitfalls in theory and practice." Sociological Methods
dnd Research35:3-60.
Rubin, Donald B. I973. "The use of matchedsamplesand regressionadjustment
to remove bias in observational studies." Biometrics 29 185-203.
1979. "Using multivariate matched sampling and regression adiustmenr
to control bias in observational studies." Journal of the American Statistical
A ssuciat ion 74:3 18-28.
CHA P T E R 6 1.972 . Fr oml! / 1 16 1: - :
rng ycar s) isin, . cl9q l
sentat iveof t hc ch. r : - -' -
The Sixth Ruie samplef or each nq\ \ >. . -
tracking changein L- . r -
USEPANEL DATA TO STUDY INDIVIDUAL CHANGE AND 1972 G SSsam plc\ \ c: : : -
i n 195 4. So if t he G SS , ,
DAIA TO STUDY SOCIAL CHANGE
REPEATEDCROSS-SECTION
w oul d consistonlr , '- - -- :
B y select ingJ f r c. r . . : -
ci fi c i ndividualsor cr : ". .
moni r or ingindividu. : - -
Chapter 5 focused on causal effects-what they are, and how we might and l ar ge paneJd. r r . r. . : .
estimate them as social scientists.Investigating causesis not the only aim research.
of social research. Sometimes the primary aim of research in the social In view of the limi:--. 1
sciencesis to describekey featuresof the social world, and how those fea- designs,the bestd;1r.r.=:.
tures are changing, This chapter focuses on the question of how to study data setsroutineh' .ldi :: -
change. Although researchersfrequently study change at the individual ti ve sam plelsom er t f . r : -
level in order to investigatecausal relationships, the study of broad social the respondents.1rcI
changemost often is descriptive.Generally the aim in aggregate-levelstud- orher hybr id desic. : r ...
ies is to describe broad change and its correlates-to get the basic facts secti o nr "I am r eiar f : : : :
right. Both types of studiesare needed.Indeed,I seedescription and expla- study p r im anlr r cr r: . -
nation as complementary, since we need to know what there is to explain each time? The be.: ..:: : :
before we set about to provide an explanation. ten rely on panel J.r:.r
Rule 6 statesthat you should use panel data to study change at the indi- so there is good rc.r. - -
vidual level and repeatedcross-sectiondata to study broad social change, you are not interestal .: :-
such as change in societies.An example of the repeatedcross-sectionde- This chapter preSr:ri:
sign is the repeatedsurvey, where the same questions are asked from sur- pl oi t the panel and r : : . . :
vey to survey,but a new sample is selectedfor each survey.Many repeated availablein the socir. , -
surveys,such as the American National Election Study, are spacedat regu- of the m odelscan L. - : .
lar intervals (usually every year or every other year). Other repeated sur- The next sectionsers-.: -
veys,such as the Gallup and Roper polls on the Equal Rights Amendment, d if ferencebetr,recn F .. :
are spacedat irregular intervals (seeassignment1 in chapter 4).
Unlike repeatedsurveys,panel surveysfollow the sameindividuals over
time. Although repeatedsurveysand panel surveysboth enableresearchers A N eryr t c Dlr r ent st . - .
to study change over time, they are designedfor different purposes (Dun- C no ss- sr clor D'. . .
can and Kalton 19871. Panel surveys track how individuals change over
time-for exarnple, whether an individual has switched her political alle- S uppo sewe want t o : : i : i'i
giance from Republican to Democrat, or vice versa-whereas repeated on X a nd Y f or a \ . lr - ': :
surveys track aggregate change over time-for example, change in the ti me 2. What analrr i. . , -
proportion of Americans who identify themselvesas Democrats. ual sover t im e, a: ir r : : . . : . .
Repeatedsurveysand panel surveyshave become staplesof modern so- The panel dcsisn : : . - .
cial research.One of the best-known repeatedsurveysin the United States cross-sect ion design.I . : . : .
is the General Social Survev. an omnibus survey of attitude chanse since change;t hat is, \ \ 'q . . - - "

. i: : - : 1 ::. -]fld how we m igh t
.: - - - , -.::: 1Snot the only aim
' I -' : :::c.lrCh in the social
.\ - -.:, rnc'lholv those fea-
- i': - ,-:::-()nc ) fhow t o s t ud Y
:,-- - - ,-.':c at the individual
- - . -. , -.: >:;d l of br oad s oc i a l
: - : , - :-..l s gr egat e- lev els tu d -
. : r --:,) get t he bas ic f ac ts
-:
r:-:i , -.: .:.lcriPtioil and exPla-
.',:-.-iithere is to exPlain
.a-
::: - : -: -.'lt change at the indi-
. I - -: ..- . -.roild social change,
- rr : i -: : -l:ed cross-section d e-
: - - : - . .-.: ere a sked from sur -
- - : ,-- r--ive)'. -\Iany repeated
: :- : " ' ' : , -J] . . Jre Sp acedat reg u-
rr-: - : .,. . Other repeated sur-
- : -: : Amendment,
- --.,.Rights
' - :-' : ;h;Pter4).
- . : .,rmeindividualsover
-,:- - i .' :rirellt PurPoses(Dun-
- -- , , :--red her political alle-
: -r :':-:'-11'l-ttttasrepeated
change in the
-r...I.nple,
: - -: r. .:: D em oc r at s .
: - : - .i -.;>tJP les of m oder n s o -
:: , - :- - -:,:r's i n t he Unit ed S t a te s
j
.rrti t udec hanges in c e
The Sirth Rule . L73
1.972.From1972to 1993 the surveywas done annually (with a few miss-
ing years); since 1994 it has been done every othcr ycar- To remain repre-
sentative of the changing U.S. adult population, the GSS selectsa fresh
sample for each new survey. Otherwise the GSS would lose its value for
tracking changein U.S. society.The youngestrespondcntsin thc original
1,972 GSSsample were eighteenyears old in 7972; that is, they werc born
in 19 54. So if the GSSwere still basedon the 7972 sanple, the 2004 GSS
would consist only of individuals who are age fifty and older.
By selectinga fresh sample for each survey,the GSScannot follow spe-
cific individuals over time. As a result, repeated surveys are ill-suited for
nronitoring individuai change. That's what panel data arc designedto do,
and large panel data sets are becoming increasingly prominent in social
research.
In view of the limitations of pure panel and pure repeatedcross-secrion
designs,the best data setsoften contain elementsclf each. Thus some panel
data setsroutinely add individuals to the samplc to maintain a representa-
tive sample; some repeatedsurveyscontain a panel con-]ponentso some of
the respondentsare followed over time; and sornedata setsemploy various
other hybrid designs.In using the terms "panel" and "repeated cross-
section," I am referring to the core feature of the sample design:Does the
study primarily reinterview the same individuals or selectnerv respondents
each time? The best empirical studies in the social sciencestoda,vverv of-
ten rely on panel data or repeatedsurvey data or data from hybrid designs,
so there is good reason to be familiar with these research designseven if
you are not interestedin the study of changeper se.
This chapter presents a number of regression models designedto ex-
ploit the panel and repeated cross-sectiondata sets that are increasingly
available in the social sciences.The models are simple and flexible. Most
of the models can be estimated using ordinary least squares regresslon.
The next section setsup the discussionof the models by formalizing a key
difference between panel and repeatedsurvey designs.
ANeryrrc Drr,ppRENCps
BETwEENPauEr AND REpEATED
CRoss-sEcrroNDATA
Supposewe want to estimatethe relationship between X and Y from data
on X and Y for a sample of N, individuals at time 1 and N, individuals at
time 2. ril/hat analytic leveragedo we gain by follorving the same individ-
uals over time, as in the panel design?
The panel design offers two important advantagesover the repeated
cross-sectiondesign.First, with panel data we can measuregross individual
change; that is, we can measure all the change that is taking place for
174 . Chapter6

individuals (for example, how many individuals marry or remarry, and other terms are pirra:r-i.:
how many divorce, over some period). With repeatedsurveyswe observe ei fi c to t ir t r e/ ( r . r lr t . : : -
only net individual change, for example, change in the overall percentage and not otheru'ise.1J. -: '
married or divorced (though of coursewe can ask people about their mar- Y , and p, is t he ef f r r :
ital histories). effcet o[ X on ) dt';. :. .
The second advantage is that panel designs provide two or more de- the subscriptr.)
grees of freedom for each individual in the study. As a result, panel de- The t er m 7F, ho, c. : : .
signs enable us to use fixed-effects models for individuals. Recall that for the it h indir idu. r . ' -
with fixed effects we are able to remove the constant effects of enduring individual. In other r.. :-
individual traits, as described in chapter 5. This is an important virtue of enoughdegr eesof ir . . - -
panel data becausewith fixed-effectsmethods (for example, differencing) for each individr r . r :l : : - -
we can very often remove the effects of significant confounding variables greesof freedonrt,. u : '
even when we don't know what those variables are. Becausewe cannot To i l l ust r at e,suf ln. : .
apply those methods when we have only one data point for each individ- 1.000 individuals c. r , : . ,
ual, in the case of repeated cross-sectiondata we must actually measure N -1or 999 dum m r ", . '. :
enduring individual traits in order to remove their confounding effects. w e ha ve 1, 000x. 3= i-
This point harks back to what was called, in chapter 5, the "first-difference donr t o est im at ccqLl, r L.: .
advantage": The first-difference model replaces the standard regression for repeatedcross-:r.i "
assumption that measured and unmeasured causesare uncorrelated with ual s. r equir ing J. e9- *.
the less restrictive assumption of stability in the unmeasured causesand sectiondata, then. to., : :
their effects. i t si ncewe cdhnot r r : : .
In short, panel data sets, but not repeatedcross-sectiondata sets' per- fixed-effectsmethods.
mit you to use fixed-effects methods to control for unmeasured traits of
individuals (Liker, Augustyniak,and Duncan 1985). Becausethis point is
central to what follows, it is useful to write out a model that demonstrates TH nrE G ENEReIQ r r -
the panel advantage more formally and provides a point of comparison
for repeatedcross-sectionmodels. C hapt er6 is or ganizc; , -
Consider the simple casewhere X one of the causesof Y, changesover ti me. O ne Q uest ionf r ', - .- :
time for individuals, while the other causesof Y are constant over time. ci al change( aggr eg. r t r.-
To simplify notation, imagine that these constant traits-the ones that and socialchangeh) . , . .
confound our estimation of the effect of X on Y-can be captured by a cial change-
single variable F ("fixed traits") and that the effect of F on Y is constant 1. Changing- elle, :, . :
over time. Then we have a simple two-regressor model where Y is a func- The quest ion aic: .
tion of a variable X that changes over time (or a set of variables that changed0\ er t 'li; :
change over time-the model readily generalizesto multiple Xs) and a 2. Conuergence/tlti, ":
variable F that does not change over time: The quesr ionh. r . -
YI t.f '= a+ a+ y F + B X +e. (6.1) it changed in the ..',:
t'tttIt

Y', is the value of Y for the lth individual at time /, F, is the value of F for I Wi rh more than one . .: , .
the ith individual (there is no subscript / since F is the same for individual yF, (so v ou don' r i i reral l r ' :.. -
I regardlessof the time of measurement),X,r is the value of X for the ith ampl e, y ou c oul d ure .r fi r.--- ".
individual at time t, and e,,is the error for the ith individual at time t. The of dummy rari ahl eshere hr - r .
 The SixthRule , 175

* - .-..irr\ or remafry, and other terms are parameters:a is an overall constantr a, is a constant spe-
:- : - , 'r: : : iLI f V C Y SW e O b S e f Ve
cific to time / (capturing change in Y that is constant across individuals
and not otherwise accounted for by the regressors),y is the effect of F on
- - i- -: : :|le o|erall Percentage
Y, and F, is the effect of X on Y at time /. Our aim is to estimate6,. (If the
: r - . . -:'rFl e aboutt heir m a r-
effect of X on Y does not vary over time, then F,- F, und we could drop
the subscript /.)
:. - -' :- '..:.l env o 0r m or e d e -
The term yF, holds the key. Because7 is a constant and F- is a constant
r- : . ' i ' \. a r es ult ,P aneld e -
for the ith individual, y F, can be captured by a dummy variable for the lth
-r : : - - .;.\i d uals . Rec all t h a t
individual. In other words, we don't need to measure F at all if we have
.-: - .,..,-.:eff ec t sof endur i n g
: - - . .',r-important virtue of enough degreesof freedom to capture yF,by the use of dummy variables
- ,i- - r\-1mple,differencing) for each individual in the studl'. With panel data we have the neededcle-
--. :- 'nfoundingv ar iab l e s greesof freedom to do that. With repeatedcross-sectiondata we do not.
.-
:' . : ' .,:'i . Be c aus ewe c ann o t To illustrate, supposewe have three rvavesof panel data consisting of
- : -,- . : :,:nt f or eac hindiv i d - 1 ,0 00 i ndi vi dual s each. B ecausethere are 1,000 i ndi vi dual s, we r r eed
: l -..i :Iac t uallYm eas u re N- 1 or 999 dummy variablesrto capture the effect of the yF,. Because
:::,
--
: . r ctxfounding effects. we have 1,000x3=3,000 data points, rve have plenty of degreesof free-
- - - ,rr:r i. the "first-difference dom to estimate equation 6.1 without measuring }. In the analogouscase
' : r . - . --:- .trndar d r egr es s i o n for repeatedcross-sectiondata, hou,ever,there are 3,000 separateindivid-
-:: - r, -:-,l re unc or r elat edwith uals, requiring 2,999 dummy variables.In the case of repeatedcross-
sectiondata, then, to control for the effect of F, rve must actually measure
: -:rlleilsufgd causesand
it since we cannot remove its effects through dummv variables or other
fixed-effects methods.
il : - -i ,i -:-ctl on dat a s et s ,pe r-
-:- : : tlnmeasuredtraits of
-
-.-: - -'i , Be c aus et his P oin ti s
THnEn GENp,nerQur,srroNs ABour CHeNcr.
:: , - . :' lel that demonstrates
:- - . - .: ;o1nsof c om par is o n
Chapter 6 is organized around three general questions about change over
: ' '.-. - -:-.:esof Y, changesover time. One questionfocuseson individual-levelchange,one focuseson so-
..:. - : ,-::L- ConstantOVertlme. cial change (aggregate-level
change),and one bridges individual change
,: - - . : ,::: i ri l it s - t he ones t h a t and social changeby asking how individual-levelchange"adds up" to so-
' -.en be captured by a cial change.
r :- : - '---:,rf F on Y is c ons ta n t 1. Changing-effect models for panel and repeated cross-sectionddta.
'--:-- - ':-. Jel n'here Yis a func- The question addressedhere is: Has the association of X and Y
. - : : ,i iet of variables that changed over time?
.-. ' , i -:(r m ult iple X s ) and a 2. Conuergence/diuergencemodels for repeated cross-section data.
The question here is: Has Y changed over time for societies,and has
( 6 .1 ) it changed in the samedirection and at the samepace for all groups?

,: : -:.: .. F is the value of F for I With more than one observation per individual, you have severaloptions for capturing
. --: - .- :l e sa m ef or indiv idu a l yF, (so you don't literally need to enter 999 dummies). For two waves of panel data, for ex-
'i ample, you could use a first-differenceequation, as described in chapter 5. I use the example
, - :-- ral u e of X f or t he l th
of dummy variableshere becauseI assumenrost readersare familiar with dummy variables.
- :- : : - :'.Ji i ' idualat t im e / . Th e
176 . Chapter6

3. Bridging indiuidual and social cbange. The question here is: How pl ane , so we assunr . : - '
much of the aggregatechange in Y can be accounted for by change (This ;rssunrprionr: : : .
in the population composition? si ons . )Tn equat ior r, - . '
y-i nt er ceptar r inr r : . :
The models differ in the way time is treated. Time is treated as cate-
gorical in changing-effectmodels and servesas the key subscript for F2 is the cffcct oi \'
w c a ssum gdoc: n', . . . . .
variablesin the models. In convergencemodels, time is treated as con-
scri pt ivepur pu5qs, - '. .... :
tinuous and servesas the primary X-axis. In models that bridge individ-
fectswe obsen'e. r r r . . , .
ual and social change, time servesas a subscript to obtain parameter
the change in r he c; '- - :
estimatesthat are used in a secondstep to estimateoverall change over
i ng X at r im e I in . l - : - : -
the time interval.
changing effecrs.
A sr at ist icall). 'iS: - , -
effecr of X on ) h. r . , .
CueNcrNc-EFFECT
MoDELS,PART1: Two PorNrs rN Trl.r
sider again the e,r.rr: .
chapt er 5 r ve nor eJ : : . , : -
Changing-effect (or changing-parameter)models ask whether the assocra-
fect of sout her nr c: Jr '-
tion of X and Y has changed over time. For example, has education be-
'What that ef f ect has c/ . '. r , : --.:
come a more important predictor of individuals' earnings? about strongestin t he So. : : : "
the association of race and earnings-are racial differences diminishing? ci ati o n of r egion i: - : :
Do Catholics still tend to have larger-than-averagefamilies, or has that decades.To deterrr:,:..
fertility difference disappeared?To fix basic concepts, we begin with the mi rri s hedwc ean. \ '. . : . :
caseof data collectedat just two points in time. regatjonist at r ir uJc.
stati sr icalllsignr t i; . : : : -
Case 7: Ttao-Vaue Panel Data witb Time-VatyingXs differencesin sesrc=-,.:-
gi on a s a dum m r \ '. - , r .-. .
Imagine you have two waves of panel data, and you are especially inter- ti al re gion t o br r r - - . . :
ested in the effect of a particular individual trait that changes over rime. hypot hesisr har r he . ". - -
\We denote that trait as X to distinguish it from the other observed van-
now than earlier.
ables,Vs (not to be confusedwith Vi/s,the unobseruedcausalvariablesdis- tV it h paneldlr , r . : : . - .
cussedin chapter 5). With two waves of panel data we can regresschange our m easur edver i. t b. . . :
in Y from time 1 to time 2 on changein X and change in V from time 1 to purge X and Y oi r r -
time 2, as in standard first-differencemodels. The aim in a changing-effect virtue of fi rst-di t'fcr.: , =
analysis(as the name suggests)is to determine whether the effect of X has purgingis a ber r er( . r '- ' -
changed from time 1 to time 2. To find out, we add X,, to the first-difference chapter5. hou er er .r : . . - .
model: on the indeperrderrt-,: ..
Y,r.- Yu= (oz- c-r)+ F2 (X i2- X iI)+ $z- F)X,, denceon r acialat r ir r : - : - .
fecr of sour her nr c\ r - : ' - -
+ y (Vi z -V ,r) + (e ,r- e ,r) (6 . 2 )
who moved inro or , :.:
To simplify we assume only one X and one conrrol variable V; the to estimatethe el-rs,: l
model easily generalizesto multiple Xs and Vs. X and V are time- ti me 2 in t he abser r - . '
varying regressors(later we consider the casewhere X is constant) that senceof m igr ar ior r... : : - -
can be either continuous or discrete; nonetheless,it is convenient to for ever yone.I nt er c. : . :-.
s peakof t he a p a ra me te rsi n e q u a ti o n6 .2 a s y-i nterceprsi n a regressi on first-difference mutic.: : -
 The Sixth Rule . I77

plane, so we assumethat at least one of the regressors- is continuous'

How
-i"::.. The question here is: only; it does not affect our conclu
i.3 .iccorlntedfor by change iThir'nrru*ption is for convenience
-.1. sior-rr.1 In equation 6.2,then, ar is the y-intercept at time 1 aud ar.is the
y-intercept tim.2. The parameter F1-is the effect of X on Yat time 1'
: :.:.::1. Time is treated as cate- "t Y at time 2, and is the effect of V on Y' which
p, i, th. .ff..t of X on 7
r :r ', ; : .1 5 the k eY s ubs c r iP tfo r over time' (These models may be for de-
nrrrr-. does not change
: :' :-1.. tine is treated as con- -'. is, we do not necessarily assume that t6e ef-
r.riprit. purposesonly, thJt
. l:' ::rLrdels that bridge individ- fecti we observeare causal.) Note that the coefficient for X,, rs Fz- Fp
: - -. r: ; i l pt to obt ain par ame te r the cbange in the effect of X on Y from time 1 to time 2. Hence includ-
r :::lnate overall change over ing X at ii-. t in a first-difference model provides a ready-rnadetest for
changing effects.
A"stalistically significant coefficient for the X,, term indicates that the
effect of X on Y has changed from time 1 to time 2. To illustrate! son-
: l: r rrs IN T I M E sider again the example of southern residence and racial attitudes. In
chapter"5 we noted that difference models can be used to estimate the ef-
: -- ir ls ask whether the associa- feci of southern residenceon racial attitudes. Here the issue is whether
F : has education be- that effect has cbanged. Historically, segregationist attitudes have been
-.: '. . : .-rample'
. rl s' e ar nings ?V hat ab o u t strongest in the south. It is frequently claimed, however, that the
asso-
segregationist attitudes has weakened in recent
.:: :--.-r.r1 differencesdiminishing? ciatio"n of region and
has di-
r- rr--.r'reragefamilies, or has that decades.To determine iflhe South-Nonsouth difference indeed
waves of panel data, regress change in seg-
-: - , 'o nce pts ,we begin wit h th e minished we can, with two
on change in region and on initial region' A
:f. regationist attitudes
coefficient initial region indicates that regional
stitistically significant for
re-
differencesin segregationistattitudes have in fact changed. If we code
nr'e-\-.tningXs gion as d.r-*! variable for the South, we expect the coefficient for ini-
"
:::.:. -rnd )'ou are esPeciallYinter- Iial region to be negative (that is' we expect Fr<F',), in line.with our
::.',r ;r.rit that changesover time. hypoti'esis that the Jfe.t of region on segregationistattitudes is weaker
- : ::(ITl the other observed vari- now than earlier.
causalvariablesdis- \ffith panel data, then. we can tesr wherhereffecrschangeover time for
.-z :!': )bserLted
- ;,....1 data we can regresschange ou. *.urrrr.d variables. By using change scoresras in equation 6.2, we
\ --.r.1changein V from tirne 1 to purge X and Y of the persistenteffects of unmeasured causes (the chief
c: >. The aim in a changing-effect ui.ti. of first-difference models, as noted earlier). The promise of that
in
:::-.:e s-hetherthe effect of X has purging is a better esrimateof the true causal effect of X. Also as noted
,-:. '.,.: .rddX,, to the first-difference .tt"it.iS, however,there is a price to pay for the purging: reduced variance
orr th. independentvariables. consider again the effect of southern resi-
ef-
denceon racial attitudes.In the first-differencemodel our estimateof the
" '-:r fect of southern residence depends on artitude change among individuals
rc.z) who moved into or out of the South.
'We cannot use first-difference models
to estimatethe effect of southern residence on racial attitudes at dme 1 or
, .-.:': one control variable V; the
time 2 in the absenceof migration into or out of the South since,in the ab-
!. -'-l.l Vs. X and V are time-
senceof migration, change in southern residence would be constant (zero)
::.; -,:... rvhereX is constant) that
for everyone.Interestingly, though, in the absence of migration we cAn vse
: : :iiheless, it is convenlent to
first-difierence models to determine whether the effect of southern residence
-.- 1 . r-i nre rc ept sin a r egr e s s i o n
178 . Chapter6

on racial attitudes cbanged from time 1to time 2.That is the topic of the ln short: Although '.',. -
next section. gender qr Cat hulr ! u; . ': . . . ,
ferencingmcthods ro :': ^
i I w e a r e int er est clr: : . : *.
Case 2: Ttao-:WauePanel Data uith Time-Inuariant Xs
use the differencing - -,
Equation 6.2,1ike equation 6.1, removesthe confounding effectsof the F,- Equation 6,2 also c,li. r; -
the constant effects of fixed individual traits-without measuring those that var y over Linr c.L. . '.
traits. This is the feature of fixed-effectsmethods that makes them attrac- w ork for r epeat ed5ur .- -
tive: rWecan use fixed-effectsmethods to remove the effects of fixed indi- l ow i n divr dualsor r r : .
vidual differences,evenwhen we do not measurethose differences.It turns appro pr iat cf ur ir r rc: , . - . . .
out. however.that in some instancesthose are the effects of interest-and
we cannot estimatethe effectsof individual differencesif we have removed
them. Cqse 3: Repealed Ct'o-.-.-
In other words, differencingwill not work if we are interestedin the
One way to think abi.-:::r
enduring effects of fixed traits becausedifferencing removes the variance
Y has changcdor cr : . : :
of interest. But if we are interested in estimating change in effects-
the effecrof X on ) '. I : - . - .
which ls the oblective in the case of changing-effect models-then fixed-
i ncl uder im e in r hc n: : .
effects methods can be used even when the traits of interest are fixed.
of interest.
To illusrate, suppose we hypothesized (1) that Americans who are
l ma ginewe hnr c r . ',
"born Catholic" (that is, Americans who grow up in a Catholic home) are
the oth er adm inisr cr - : , - -
more likely to disapproveof abortion and (2) that the effect has weakened
focal variable X irn,i -: . -:
over time, reflecting the declining influence of official church doctrine '.
i zesto m ult ipleX' . i: . * -
among American Catholics. Fixed-effectsmethods can be used to exam-
or di s cr et e;noner hcl. . . . -
ine the secondhypothesis but not the first. Let Y be a measure of attitude
inrerceptsin a reg,rq.. - :
about abortion, coded so that high values reflect disapproval of abortion
one of the regressor\.-,.-
and low values reflect approval. Let X be a dummy variable for Catholic-
W e could est im . lr r: : : :
born (X-, =Xi2= 1 for those reared Catholic, 0 otherwise).Becausethe
lows:
term (X,r-X,,)drops out in equation6.2, we cannot use equation 6.2 to
test the hypothesisthat Americans who grew up Catholic are more likely Y ,,=d, t Br X, . ', \ ' -
to disapprove of abortion. The equation does shed light, however, on
whether the effect of Catholic upbringing changed from time 1 to time 2. Y ,z =0: I p, X, _- ', \ '-
The coefficient for X,, estimates(82- 91),so a negativecoefficient for X,,
The su bscr ipti indcr . - '
is consistentwith our hypothesisthat the effect of Catholic upbringing on
spondentin t he / t h : . : : . .
anti-abortion attitudes is weaker now (assumingthe effect was positive in
that the ef f ect ot , \ or , :
the first place).2
on Y a t t im e 1 is den, : . :
2 B e c a u s eC a th o lic u p b r in g in g is a fixe d tr a it fo r indi vi dual s,the term (X,r-X,,) drops noted Br.
out in equation 6.2 and we have:

Y,.-Y,r=(ar- ( v1 l+ ( 8 2 - B) X jt+ y( Viz- V,r \+ (e,r- e,rl , the Catholic-upbringinr t::.-:

eter for X , then, i rrdi t.rr.. :
where a, is the y-intercept for non-Catholics at time 2 and a, is the y-intercept for non- orrp het w eenC athr' l i .' .::.-
Catholics at time 1. F1 is the difference in y-intercepts for Catholics and non-Catholics at trme rtti tude'. The s i z eof rh. - : -
1-the Catholic-upbringing effect at time l-and p, is the difference in y-intercepts for much othersc hange.l i .:..:--.
C a t h o l i c s a n d n on - Ca th o licsa ttim e 2 .He n cep .- p ,,th eparameterforX,i sthedi fferencei n C ethol i c s ,then the L.rrh. -
 The Sixth Rule . I79

: - - - - .r.ri is the topic of the In short: Although we cannot estimatethe effects of fixed traits such as
gender or Catholic upbringing with differencing methods, we can use dif-
ferencing methods to estimate the changing effect of such variables.Thus
if we are interested in estimating changesin the effect of gendcr, we can
r- I-. -- -r'+ Yr
use the differencing model of equation 6.2 when we haye panel dara.
r - ' .-:'.,.:.r ef fectsof the F,- Equation 6.2 also can be used to estimatethe changing effects of variables
:; :-- .::-.,,-! nlcasuringthose that vary over time. Fired-effects models such as equation 6.2 do not
'r:: ,:. :- -,:ilakes them attrac- work for repeatedsurvey data, however,sincerepeatedsurveysdo not fol-
-:- , : :ffects of fired indi- low individuals over time. As we now see, an interaction model is more
:i. - -- .: .ltfferences. lt turns appropriate for investigating changing effects with repeatedsurvey data.
!- - '- r ::::JIi of interest-and
I - -: t :"::. i i n ' e hav er em ov e d
Case 3: Repeated Cross-section Data
i r, : .'.: -'.ieinterestedin the
One way to think about the question of whether the associarionof X and
'l:l:-, : ; : anlO V e S t h e v a f l a n C e
Y has changed over time is to think of time as a variable that moderates
::r --r - in effects-
- -'l:tnge the effect of X on Y. Instead of using time as the basisfor differencing, we
: - r-:,i :l l odels - t hen f ix ed -
include time in the model as a r.ariableand interact it with the variable(s)
-- :
: :-., .' . illtcrest are fixed. of interest.
'-
: :-..: -\mericans who are Imagine we have two repeatedsur\-e\-s,one administered at time 1 and
:: .-r I .i Catholic home) are the other administeredat time 2. To simplifv the model we assumea single
: -: : :r: e ffec thasweak ene d focal variableX and a singlecontrol r-ariable\'; the model easilvgeneral-
-,!: : :i;ial church doctrine
izesto multiple Xs and Vs. The variablesX and \/ can be either continuous
- : i- -:.Jn b e us edt o ex am -
' or discrete;nonetheless,so that we can speak of the a parametersas )-
:: : r: .1measureof attitude intercepts in a regressionplane, it is often convenient to think of at ieast
' r : r : - r ;-sappr ov alof abor t io n
one of the regressorsas contlnuous.
: : - ':". .ariable for Catholic- 'We
could estimate the effects seDaratelvfor time 1 and time 2 as fol-
lows:
. , : - : -rtL )rus eequat ion6. 2 to
:: ,,. - : t :rholic are more likely Y i t= a t+ FtX ;t+ yV ,r+ e,, (6.3a)
' : :.
-rcd lrght, however' on
- : ' Y,r= ar+ BrXiz* 7V;:t €, (6.3b)
- -: : :ro m t im e 1 t o t im e 2 .
- : "r:-rtite coefficientfor X,, The subscript I indexesindividual, so {, is the value of Y for the lth re-
:-' : -- -r rrholicupbr ingingo n
spondent in the /th survey (t--1,2). Note that these equations assume
- - ' - - :-. e i fe c twas pos it iv ei n that the effect of X on Y might have changed over time : the effect of X
on Y at time 1 is denoted B. and the effect of X on Y at time 2 is de-
noted Br.
: .: .. :he term (X,, X,,) drops

the Catholic-upbringing effect on abortion attitudesat time 2 versusrime 1. A nonzero param-

a.
- r is the y-intercept for non-
: , , . rndn o n - C a t h o l i c ta r r i me
, ::fference in y-intercepts for
- . : rr iur X , i s r h e d i f f e r e n c ein
eter for X, then, indicates that the Catholic-upbringing effect has changed; rhat is, the differ-
ence berween Catholics and others (rvith V controlled) has changed with regard to abortion
attitudes. The size of the change in the effect depends on hou. much Catholics change and how
much others change.If changein attitudestoward abortion is the same{or Catholics and non-
Catholics, then the Catholic-upbringing effect does not change,and B,- B, is zero.
180 . Chapter6

One way to determine whether the effect has changed, then, is to esri- \7ith panel data se .;.: ...
mate separatemodels and compare coefficients.But there is a simpler way. a given set of inditic'-:: .
Consider again the issueof Catholic upbringing and abortion arrirudes.To whether the effecr ot -\
determine whether the associationof Catholic upbringing and attitude to- if we are interesre.l .: -
ward abortion has changed from one survey to the next, we can estrmate gencral populatior.r.r:-.. .
the parametersfor both equations in a single step by regressinga measure appro pr ir t e. This r r . : -
of abortion attitudeson a dummy variablefor time (7, coded 1 for time 2), fresh sarnplefclr errch.-.:,
a dummy variable for Catholic upbringing (X), a dummy variable repre- rcprescn t lt ive slt r r pl'
senting the interaction of Catholic upbringing and time (XxT,), and a chang ein t he cali- r . ll: : : , -
measureof the control variable V: metho d is likeiv r hr .h- : : . -
effectsof confor-rndinc', ..:
Yu= dr+ (cr2
- ar)T.+BrX,,+(82-B,)6 ,xTr) + yVo+ e, (6.4)
For r epeat edcr os: - : . . :
Equation 6.4 subsumesequations 6.3a and 6.3b (as you can see by efi ecrsr estBr + B, i< f : - : - .
substituting Tu=0 into equation 6.4 to reproduce 6.3a and Tz=7 rnto models as descr ihcJ. : .
equation 6.4 to reproduce6.3b). No important information is lost by es- 7997, chap.13 ). In sr:: * ,::
timating 5.4 in place of 5.3a and 6.3b-in facr, equarion6.4 is better be- to kno w. f or exar npl. . . , .- .
cause it gives you an easy way to determine whether or not observed for al l gr oups,r h. r rii. . , .'
change in the effect of X is statistically significant. Note that equarion 6.4 i s the s am ef or nr er r . r . ,
contains terms for the focal variable X, for time (Tr), and for the interac- and no nsout her ner . .- , : .
-
tion of X and time.3 not refer to gender -
-'-ir:-
The aim of equation 6.4 is to determine whether the effect of X on Y to time-there is .r :::--
has changed over time. The key is the interaction term, XxTr. The pa- know is whet hert lr c : i : - .
rameter for this interaction term is (82-81), change in the effect of X groups, in the s.rm. .'..:.. : -
from time 1 to time 2. Hence a statistically-significantcoefficient for the has rhe sameeffeer, :.
interaction term indicatesa change in the effect of X over the time period i cansas it doesior n: , : =
examined. In the language of interaction effects, rhe effect of X on y is S upposewe f inJ : : . . : : '
"moderated" by time. change?In the caseo: :.:. ,
In the case of Catholic upbringing and abortion, rhe parameter for i ng populat ionconr l. . :
the interaction term indicateswhether or nor the effect of Catholic up- ti me I t o t im e 2. ecl! ! :
bringing has changed over time. Recall thar equation 6.2 also esti- vesri g at ingchangeri: : - - -
mates Fz-F, except that equation 6.2 has the advantage of Statesin 1970 r.ersu:l
differencing out the effects of the confounder V without measuring it. ti al changein t he [ - . r -
Even if we measure V perfectly in equation 6.4, we do not expect this rul es out such ef ie. r : : - , .
equation and equation 6.2 to yield the same value for Br-B, because (thoughcom posir ior '. r ,- '
the meaning of Fz- Ft differs for the two methods, as rhe ,ri*t puru- enti al a t t r it ion) . I n r - . -
graph explains. changingpopular iun. _
The interaction model of equation 6.4 and the differencing model of observedchangein r h. , - - -
6.2both can be used to investigatechanging effects of X on Y. But the \fl e c onsiderr he . : : . - .
two models ask somewhat different questions about changing effects. detai l l a t er in t he chap: . . .
H ere th e im por r anr r , - . :
rA note on notation: a typically denotesy-intercept ri on i s t hat ir can ch. - .-' , :
and p typically denotesslope. Equa-
tions 6.3 and 6.4 use .r as the parameter for T and not X becauseT is always a categorical another .To det er m r n. - - .
variable in changing-effectmodels, whereas X could be either categorical or continuous. secti ondat a, r eest ir l. lt : I - :
 The Sirth Rul.' . 1\

\fith panel data we can ask whether the effect of X on Y has changed for
a given set of individuals. Nfith repeatedcross-sectiondata u.e can ask
whether the effect of X on Y has changed in the generalpopulation. Thr-rs
if we are interested in describing change in some association in rhc
general population, the repeated survey/interaction method is the more
appropriate. This result should not be surprising since, by selecung a
fresh sample for each survey, repeated surveys generally provide a nors
representative sample of changing populations. If we are interested in
change in the causal effect of X on Y, however, the panel/differencing
method is likely the better choice, since differencing removesthe enduring
effects of confounding variables.
For repeated cross-sectiondata where X is continuous, the changing-
effects test B2+ B1 is the tesr for nonparallel slopesin analysisof covanance
models as described in statistics texts (for example, Agresti and Finlay
1997, chap.13). In standardanalysisof covariancemodelswe might want
to know, for example, whether the income return to education is the same
for all groups, that is, whether the effect of years of education on income
is the samefor men and women, for whites and nonwhites, for southerners
and nonsoutherners,and so on. In the caseof equation 6.4, "group" does
not refer to gendercategoriesor race categoriesor regional categories,but
'$fhat
to time-there is a time 1 group and a time 2 group. we want to
know is whether the effect of X on Y is the same for the time 1 and time 2
groups, in the same way that we might want to know whether education
has the same effect on income for African Americans and Hispanic Amer-
icans as it does for white Americans.
Suppose we find that the effect of X on Y has changed. IThy did it
change? In the caseof repeatedcross-sectiondata, one possibility is chang-
ing population composition-the population membership changes from
time 1 to time 2, especiallyif the time interval is large. For example, in in-
vestigating changes in the determinants of voting behavior in the United
Statesin 1970 versus2000. we would need to bear in mind the subsran-
tial change in the U.S. electorate over rhe period. The use of panel data
rules out such effects since the same individuals are followed over rlme
(though compositional effects on a small scale might occur due to differ-
ential attrition). In the case of repeated cross-section data, however,
changing population composition might account for some or all of the
observedchange in the effects of the Xs.
\7e consider the effect of changing population composition in more
detail later in the chapter, when we examine cohort replacement effects.
Here the important point to nore about changing population composr-
tion is that it can change the observed association of one variable rvith
another. To determine if that has happened we can, with repeatedcross-
section data, reestimate the association using a sample restricted ro the
782 . Chapter6

birth-cohorts present in the population at the initial measurementpoint.a thirty coefficienrs.s.' :-' :
(The term "birth-cohort" refers to people born the same year.) Larer in w i th m acr o- leleI r ar r . ,
this chapter I describe a method that isolates the contribution of chang- -.
w ard mult ilevelln. r i, . : - .
ing population composition to societal trends. As I show there, the
method can be used to estimate how much changing population compo- 'Vbat'Ve'Want
sition contributes to any trend in societS from declining voting rates to to Knr't
rising rates of online shoppingto changein norms about sex (to cite just \/hat ar e t lt e r r r . t r r r, . . .
a few examples). w hat c ondit ionsh. r r . - : . . .
In short: We can do quite a bit with data for just two points in time. chartg covEr [ ir uei q. , . :
That holds for both panel data and repeatedcross-sectiondata. Certainly feren cesin r at ss ui , r : . .... .
two points in rime is better than one. But multiple data waves are better voti ng f or Dcr r r u. r . r r r . .
still.'tThe next sectiontakes up that issue. -
w hy the ef f ectoi eJLr ., -
Thesesorts oi qrr..:
varyi ngm acr o- lc\ cl. ': . - :
CneNcrNc-EFFECTMoDELS,PART2: Murrrrp.vEL MoDELS levelanalysiswas delt-, :".
WITH TIME ASTHE CoN:tIx:t i t has pr oved useiLr:l : : -
merhodherewe think
This section describeshow to use multiple waves of data to examine the machiner y.includir r ,::: . . .
correlates of changing parameters.The question of how to exploit multi- approachinspired1., :- - -
ple waves of data is becoming more relevant as we cumulate longer trme- tion (seeDiPrerei.lnJ ( ---
seriesin the social sciences. effectsituationu hqr. ..
The two-wave changing-effect models described above are limited in
what they can do. The models are designedto detect, not explain, change
in individual-level associationsor causal effects. The major advantageof Tbe General Mtrltilet e.
"
the two-wave models above is that they make few data demands: Two To i nt r oducet he g. ': - - . , -
surveyssuffice. Increasingly in the social sciences,however, we have mul- gregat ion,wher e inc: . -
tiple surveys to exploit. The changing-effect models above quickly be- he extendedto firorcr: ,' .
come unwieldy as we add waves of data. If we want to srudy how the rooms nest edin s'h. ,
effect of X varies over (say) thirty surveys,we would have thirty regres, pl i cationsof r he nr ,', = . '
sion coefficients to compare, one for each survey, To summarize those consist sof a "lelcl l'
To fix basicconcepti.j -.
aTo illustrate, suppose we found a steadily rising gender gap in voting for
Democratrc arrdfwo level2 r cg, r . . . - ,
versusRepublican candidatesfrom the 1984 national election to the 2004 national election.
Part or all of this growth in the gender gap could be due to greater gender differer"rcesin vot- , -. , -,
L rv L L
I
l
/ ^", n-n\
rvt L
\ rvllLnul

ing among the new cohorts ofvoters (those rvho became eligible to vote between 1984 and
2004, that is, citizens born from 1967 to 1986). To examine the effect of changing popula- The levelI equatiur.
..,
tion composition, then, we could eramine the gender gap rvith and rvithout the new birth
cohorts. If gender differences diminish significantly when lve remove the nerv cohorrs, we
Y rl =8..+B,X
' \tl ' rl ;
-t
rvould ir"rferthat the cohorts recently added to the U.S. electoratecontributed to the growth j -
Y ,,i s t he valueof ) :-
in the gender gap.
i With three or nlore rvavesofpanel data it is possible
to bring evidenceto bear on the is-
val ue of X f or r hc : :
sue of whether the treatment effect in an experiment is better estimated bi' using change in from the standardrc: :..
Y (that is, Yr-Y,) as the outcome variable or by using regressionto conrrol for the initial acrosscont e\ t s, A) : : - - - . .
value of Y(that is, by including Y, as a conrrol variable in a rnodel of Yr). SeeAllison (1990) (nor j ust t he var i. r h. . -
and Morgan and Winship (2007, chap. 9).
F,, i s rhe slopet t hc . '- . - -
 Thc S i rth R u ie . 1: r

- -- r.rsurement point,4
. . ::ilc \ ear. ) L ate r in
- -: : ih utio n of ch an g-
.. r.. i shorv there, the
' - - -: :r)pu latio n co mpo -
,: - -.:r]q \-otlng rates to
.ri ss-\ (to cite ius t
.;r.; - .--. :.r.o points in time.
:: . .,. j-: . -, n d ata . Ce rtain ly
t.::-:- \\'aves are better
l . l ,-,p r ls
--riJ to exarline the
- .'. t o er ploit m ulti -
- ..:':. r l. t tlonger
e r ime -
,[ -,- : r ,:rrrveare limited in
, -r. :,rt er plain, c hang e
. -: ::r.ljor advantageof
, '. :,ria demands:Two
\:-- - r -\ Pr ..
"
h.rr"
::.. -rfror-equickly be-
..::: to study how the
j 1.
..r,.
\ \ e rh
r ' r "ri rtrr
) re
, L6'oLJ
-
thirty coefficients,we might very well correlate the coefficientsthemscllcs
with macro-level variables for the thirty measurementpoints-a step to-
w a rd mul ti l evelanal ysi s.
WhatweVant to Know
I7hat are the macro changesthat matter for the X-Y relationship, that is,
what conditions have changed over time to causethe X-Y relationship to
change over time? Ve might want to know, for example, why racial dif
ferencesin rates of imprisonment have grown, or why the gender gilp in
voting for Democratic candidateshas grown. Or we might want to know
why the effect of education on earningshas increasedin the United States.
These sorts of questionslead naturally to a muldlevel model with time-
varying macro-levelconditions as the contextual variables.Although multi-
level analysiswas developedinitially to study the effectsof school contexts,
it has proved useful for the study of other contexts as well. To apply the
method herewe think of time as a context, so we can wheel in the multilevel
machinern including multilevel software,to apply to change.The multilevel
approach inspired by the school-effectsliterature can, with some modifica-
tion (seeDiPrete and Grusky 7990a, 1990b), be applied to the changing-
effect situation where it is time. not school,that forms the context.
The General Multileuel Model
To introduce the generalmultilevel model, let us focus on two levelsof ag-
mrtl-
gregation, where individuals are nestedin time. (The multilevel model can
be extendedto more than two levels-for example,studentsnestedin class-
rooms nestedin schoolsnestedin school districts, or four levels.Most ap-
rec
- plications of the model employ two levels,however.)The two-level model

T , summarizethose
- - . ,r r i n g f o r D e m o c r a tic
'-. I | ) + n a t i o n a l e l e c t i o n .
' - : . : : ie r d i f f e r e n c e si n v o t-
: 'le l.etlvecn 1984 and
-' : o i e h r n s i n s o o o u l a
: -: . ' . i t h o u tt h e n e w b i rth
- -' . . t h e n e w c o h o r t s , w e
: , :r::lbrlred to the growth
- . . i- r . ' et o h e r r o n r h e is-
. . : : -. : : : . 1 b V u s i n g c h a n g ein
. . : : ) io n t r o l f o r t h e i n i t i a l
i: : )' . i . S e eA l l i s o n ( 1 9 90 )
consistsof a "level 1" (micro) equation and a "level 2" (macro) equation.
To fix basic concepts,consider the simple caseof one level 1 regressor(X)
and two level2 regressors(2, and Z, refl,ectingmacro-levelconditions).
revrl 1 (urcno) EeuArIoN
The level 1 equation is:
Y ,i= Foi+F, jX,i+ (i =1, 2, . . . , J contexts) ( 6.s
)
",i
Y,, is the value of Y for the lth individual in the lth context and X,, is the
uJl,.teof X for the ith individual in the fth context. Equation 6.5'differs
from the standard regressionmodel in that effects are permitted to differ
across contexts, as reflected by the subscript i attached to the parameters
(not just the variables).Thus Bn, is the y-intercept for the ith surve-vand
fr, is the slope (the effect of X on Y) for the ith survey.
L84 . Chapter6

Becauseequation 5.5 permits effects to vary acrosscontexts, it is natu- for estimating multiler': -
ral to use the multilevel framework for changing-effect models where we R a udenbush199f r , t . . . - .
have measurementsat multiple points in time. The key is to think of time appropriate methods it,: .
'With in multilEvEl,ttoglsl5,fo: :
as the contefi. multiwave panel or repeatedcross-sectiondata, then,
we specify survey wave as the "group" and apply the multilevel modeling Although multilevel-:r :
approach to estimate the Bs in equation 6.5 that reflect the changing i ngl y pr act ical as nr . , . . .
effects over time. i mport ant t o em plr r t t r zr: '
To simplify matters, multilevel models generally use deviation scoresto tinue to lack porver it,r .
center X on zero. By centering X on zero, the y-interceprsin equation 6.5 is still limited in the nr(,-: .
are Ys (Y-means).6Thus Bo, is the mean of Y for the first survey,Bo, is the and a paucity of time ', .
mean of Y for the second survey, and so on. few variables.In the c:..
ampl e ,t her e ar e 46. i I
rBvsr 2 (r,recno)EeuArroN ti me cont ext s( sur ver '.. '
Becausewe are modeling both intercepts and slopes,we have an equation one everytwo years.In .::,
for each.For two Zs,the equationsare: time is the contert. -\t '-.-
still lack power.
(6.6a) 'Weturn our attenti(.,.-
Foi=Toot TorZri+ TozZzj+zn, (intercePts) :
(6.6b) chapt er .t he conver sc: l-
F r i= r r c + T rrZ ri + T ttZ z i + 2 ,, (s l o Pe s ) pol i ti c al indicar or sclt . '. -:
Note that the dependent variables in the level 2 equations are coeffi- in a society?
cients from the level 1 equations. The first equation models the level 1 in-
rerceptsas functions of the two Zs.The second equation models the level
1 slopesas functions of the two Zs. I follow standard multilevel notation C oN v t ncI NCE \ loD: -
(Bryk and Raudenbush 1992), where the first subscript for y identifiesthe
level 1 coefficient being used as the dependentvariable (0 for intercept, I S oci a l scient ist sh. t r . , -
for the slope for the first micro-level regressor,and so on), and the second and e t hnic dir isior r - . - . '
subscript for gamma is the conventional notation of 0 for intercept in that srates "ver sus"r cJ - - '. : --
equation,l for the slopefor Z, and so on. Substantively,then: and th e poor . r nJ - -
deepe ningdivisit 'n. " - - -
. 7e1is the effect of macro-level variable Z, onthe Ys across surveys,
i zati on") , m or al r r l: . =- . '
and y,r.is the effect of macro-level variable Z, on the K across sur- arrd g ender .Br ". 1. . : - '
veys. soci e t allevel.so r sf ; . '. ri -
. 7r, is the effect of variance in macrolevel variable Z, on variance in
by rule 6.
the regressionslopes,that is, on variance in the effect of X on Y over To f ind out if dt '. : . - :
time. Similarly, 7n is the effect of variance in Z, on variance in the w i th r epeat edsr uss- : :- .
effect of X on Y over time. In short, the model is designedto tell us uni que r cgr essionl, . : - .
whether cbange in the effect of X on Y over time is associatedwith groups. ( Alt er n. r r ir , . ,
change in macro conditions Z, and Z, group. A singler t r , ''.
Becausethe level 2 equations involve clustering effects that violate key ever.s incea singlr : : : *.
a b o u tth e e rro r te rm, o rd i n a ryleastsquaresi s not appropri ate
as s um pt ions encesin t he gr oup : - . - -
featu r eof t his m ( , J; --
6This follows from the fact that a regressionline goes through the point (X,Y), so when eachgr oup r ef lect .: t . - -
X i s c e n t e r e do n ze r o r h e y- in r e r ce p rit th c p o in r r 0 ,Yr. thought of as n LJr : - : . : - .
 The Sirth Rule . IB-5

:- -. - .! -' )ltte rts,itisna tu- for estimating multilevel models (DiPrete and Grusky 1990a; Bryk and
i- : ' * :l: : . : nlod els whe re w e Raudenbush 1992). Severalcomputer packagesare available that employ
-: - :r:r sto thin ko f tim e appropriate methods for estimating coefficients and their standard errors
rt r " . -' -- r3 atiOn d ata , the n, in multilevelmodels (for example,Raudenbush2004).
,: :" . . .: -:-,:rltilevelmodeling Although multilevel analysesof replicated surveyswill becomc increas-
' * ' ' r: :::lect the changing ingly practical as we continue to cumulate such surveys over time, it is
important to emphasizethat our macro-levelmodelsnonetheless will con-
t:- . ' " . -: Jella tion sco res tO tinue to lack power for some time to come. The number t-rftime contexts
": - l:1-: li> ln e qu atiOn 6.5 is still limited in the most popular replicated surveysin the social sciences,
- - : i-l ilirve\-,
F62 iS the and a paucity of time contexts in turn limits the macro equation to just a
few variables.In the caseof the cumulative General Social Survey,for ex-
ample, there are 46,570 respondents(as of the 2004 survey)but only 25
time contexts (surveys),and the surveysare being added at the rate of only
i-: :r.. .,.: h: t |e an equat io n one every two years.In effect, N=25 for GSSmacro-levelequationswhere
time is the context. At the macro level, then, our changing-effectanalyses
sdll lack power.
( 6. 6 a ) 'We turn our attention now to the secondgeneral question posed in the

( 6. 6b ) chapter, the convergencequestion. Are significant economic, social, and

political indicators changing in the same direction and pace for all groups
-L: - :J.rations are coeffi- in a society?
: -.- -- - : 'Jels the level 1 in-
:- -- : -,,:. )n n- r odels
t he lev e l
CoNvpncrNCE MoDELS
:-. - - . -.-: :: fo r y ident if iest h e
Ll: - - - .: --- t) for intercept, 1 Social scientistshave an abiding interest in divisions in societies-race
-- - . - --i ,,n. andt hes ec on d and ethnic divisions, gender gaps, differencesbased on religion, "blue
rr: - : :or interceptin thdt states" versus "red states," disparities in health outcomes for the rich
, ::'..1 . .t.hen: and the poor, and so on. For example, there is talk these days about
deepeningdivisions in the United Stateson the basisof age ("age polar-
- . -- ::--; )l acrosssurveys, ization"), moral values and religion ("culture wars"), and perhapsrace
r* r - : .-_ ,i th e Y s ac r os ss u r- and gender.By "deepeningdivisions" I am referring to divisions at the
societallevel, so repeatedcross-sectiondata are appropriate,as dictated
- '.'., Z t on v ar ianc ei n
b y r ul e 6.
.--: - .:.: .rtec t of X on Y ov e r To find out if divergence is indeed taking place in a society we can,
i: : - -: :' Z on v ar ianc ein t h e with repeatedcross-sectiondata, estimatea regressionmodel that permits
i - : - ' -: :s des ignedt o t ell u s unique regression coefficients to capture the time trends for each of the
r :: :.::tL'is associatedwith groups. (Alternatively, we could estimate separate regressions for each
group. A single model with interaction terms is more convenient, how-
'-: :::ects t hat v iolat ek e y ever, since a single model provides a ready-made test for whether differ-
--: : -
: j . : - - ...'.:i :i : not appr opr ia te ences in the group trends are statistically significant.) The distinctive
feature of this model is that time is the X-axis, so the regression line for
: ... ' : r : re p o i n r ( X , Y - ) ,s o w h en each group reflects the group's linear time trend. Becausetime itself is not
thought of as a cause, the model is descriptive. The model nonethelessis
186 . Chapter6

useful for addressing a wide range of issuesincluding, most notably, is- The qucst ion of c, , : ', . :
suesrelatedto fissuresin society. si gnsof t hc f s ; nd r lr r . '.
The convergencemodel provides a simple way to test hypothesesabour Y is diverging bctu'eer:::..:
the reduction or deepeningof social divisions in a society.For two groups are both positive,fo. e-.,
the model can be expressedas follows, with T (time) and G (group) as the basel i n egr oup on I inr : : ,
indeoendentvariables: ing over time-dir-erge :-.::
Y is convergingbetse;.. :
Y,,=u+ BT+ SrGr+QrtTxG.)+ e,, (6.7)
the trends have crossei ,
The first two terms in equation 6.7, a+BT, refer to the baselineor ref- Differenr group !(\ir .-..
erencegroup, where a and B are the y-intercept and slope,respectively. with different busclins
-
In contrast to the changing-effectmodel, the convergencemodel treats vergi ng,f or exam plc.r , . .
time as continuous. Because time is continuous in the convergence the baselinegroup. Br ..--
model, 7 has no subscript in equation 6.7. The groups, however, are vergenceor divergen.. : -
discrete, and numerical subscripts are used to distinguish them. Thus
ILLU ST R A TI O I \ : AI I IIL DI
G, is a dummy variable coded 1 for the secondgroup. The parameter B
SPEN D I N G O N E D L : C . \ T. ].
is the linear time trend for the baseline group, 6, is the difference in
y-intercepts between the second group and the baseline group, and $, S omeo bser ver sar su( : : . . :
is the difference in the slope for the second group and the baseline A meri cansociet r '- t ui: :
group. i n A mer ica,or gener r r ' -
T he c onv e rg e n c e s mul ti pl e groups (1,2,...,
mo d e l g e n e ra l i z e to G Marx has been suppl::-.:.
groups) in a straightforward manner: tions. The convergencs:r.
case of gener at ionr l\ '. . - - :
Y , , =a+ B T+ 6 rGr+ 6 ,G, + . . .+ 6 c c c + d r (TxGr)
young a nd r he old 'h, . . *
+ @, ( Tx G,)+ ...+ $ " (T x G" )+ e ,, (6.8) issuessuch as fundinc : :
In equation 6.8, as in equation 6.7, group 1 is the baselinegroup. Thus ti on. Considert his qu: : :
the parametersfor the other groups are in comparison to group 1. 'We are faced g'ith :-,.
be s olved easil' o: - ,
The Sign Test for Conuergence: CompaingYour Ss and 6s problem s,and f c, r . . , . , -
we're spcndingtoc, :'-.
To accommodateadditional groups, equation 6.8 adds dummy variables
ri ght am ounr , , . . . \ : ;
(G' etc.) to capture differences in intercepts, and interaction terms
ri ght ar r our r ron -.
(Tx C' etc.) to capture differencesin time trends. It is convenienrto code
cial Security?
T as zero for the first measurement,so a is the predicted value of Y for the
baselinegroup at the first measurementor wave of data. The 6s are dif- Thesei t em s invit e u> : : :
ferencesin y-intercepts between the baselinegroup and the other groups, di vi si o ns acr ossgenc: . : :
so 5, is the difference in y-intercepts between the first group and the sec- spending on educarro: ..
ond group, 5, is the difference in y-intercepts between the first group and S urveysince l97j. n: . . :
the third group, and so on. In other words, the 5s reflect initial group dif- (natsoc)has beenask- - ,
ferenceson Y (frtted by linear regression). Americansbelieveeirhr: .-
Similarly, the @sare differencesin slopes (linear time trends) berween money is spent on eJ. : , . . :
the first group and the other groups. Thus /, is the difference in linear cent be lievet hat t oo - . ,
trends for the first and secondgroups, @..is the difference in trends for the centagehas beendc. l: : . : -
first and third group. and so on. di vergr ngat t it ude: f , , r - - .

-, .-i.:1q.most notablY,is-
-:.:
:: : : :esth y pot hes es ab o u t
-- - : : ,aret\-.
For two groups
'. :nd G (grouP)as the
' . .
] . '. r , ' :,r th e bas elineor r e f-
:- -: rr ,. 1 :l ope, r es P ec t iv e l Y.
:- : - r-,:rgef lc em odel t r ea ts
: - ..: in the convergence
- -- ::Ol r ps , howev er ,ar e
-
.:':- . *.::,nguishrhem.Thus
,,:- -: :: -rp.The parameterB
in
:- . . ' r. the difference
: : -: -.l :i l i l le gr oup, and $ t
-: -: ',rp ar ndt he bas eli n e
- -' : : :roups(1',2,...,
- : : laselinegrouP.Thus
-_ - : _ i, rll frl gf olr n
":; 1 :, ' -',sand 6S
:' - . \ :Jds dummy variables
: - - : r: . . r:J i nt er ac t ion t er m s
s convenient to code
:::i e d r - alueof Y f or t h e
- . . : J.rta. The 6s are dif-
-,: - .-: ,..ndthe other groups,
t :- --:::.t g r ouP and t he s e c -
ri- -:i.,',:'-r1the first grouP and
i-, ::l,ect ittitial group dif-
- :-:
t':- :.:.:Iime trends) between
r . :re differencein linear
-:
. : -.i * :r-rencein trendsfor the
The Sixth Rule . 787
The question of convergenceand divergence of tends turns on the
'When
signs of the $s and the 6s. / and 6 have the same sign for a group,
Y is diverging between that group and the baselinegroup. When @rand 6,
are both positive, for example, rhen the secondgroup was higher than the
baseline group on Y initially, and that initial relative advantageis incrcas-
'When d and 5 have different signs for a group,
ing over time-divergence.
Y is converging between that group and the baseline group (or possibly
( 6. 7 ) the trends have crossed).
Different group comparisons can be made by reestimating the rnodel
with different baseline groups. To determine if groups 2 and 3 are cQn-
verging, for example, you can rerun the model with group 2 or group 3 as
the baselinegroup. By repeatingthis processit is possibleto test for con-
vergenceor divergencefor all pairs of groups.
AT T IT U D ES OF \I'OR KER S .{\D R ET IR EES T OW AR D
ILLUSTR AT ION :
SPEN D IN G ON ED U C AT ION AN D SOC I.\L SEC U R IT ]-
Some observersargue that, u'ith the rise of other sourcesof division in
American society-such as "culture \\'ars, racial/ethnic"balkanization"
in America, or generationaldivisions-the classtvar envisionedby Karl
G Marx has been supplanted bv a culture \\'ar or a n'ar between genera-
tions. The convergencemodel is designedto test iust such notions. In the
case of generationalwars, for e\ample, grou'ing dir-ision bet'uveen the
young and the old should shou'up in diverging attitudes on contentlous
(6.8) issuessuch as funding for Social Securitvand funding for public educa-
tion. Considerthis question in the GeneralSocialSurvey:
I
\7e are faced with manr- problems in this country, none of which can
be solved easily or inexpensivelr'.I'm going to name some of these
problems, and for each one I'd like you to tell me whether you think
we're spending too much mone,v on it, too little money, or about the
right amount. . . . Are \ve spending too much, too little, or about the
right amount on . . . Improving the nation'seducationsystem?On So-
cial Security?
These items invite us to bring data to bear on the question of deepening
divisions across generations in the United States. The question about
spending on education (nateduc) has been asked in the General Social
Surveysince 1973, while the question about spendingon Social Security
(natsoc)has been asked since 1984.k turns out that the vast maiority of
Americans believeeither that the right amount of money or that too little
money is spent on education and on Social Security; fewer than 10 per-
cent believe that too much is spent on either, and in both casesthis per-
centagehas been declining over time. But doesthis overall pattern obscure
diverging attitudes for the young and the old?
 188 . Chapter6

To find out, we apply the convergencemodel of equation 6.7 to nateduc and "natural iogarithr:-" .
and natsoc.Becausethose in the paid labor force are paying into Social Se- 10. To convert a lo-uirr....-
curity while retireesare withdrawing from it, a comparison of the attitudes logit coefficientof Ll.v1*.
of paid workers and retirees toward Social Security should be especially BE,;ause the depcni<..:
telling. A comparison of those two groups is strategicin the caseof spend- spendir r g, spendi'r
, ii i . -
s,
ing on public education as well, since both groups pay school taxes even rwo different wfl\ \ ,rrr..r-
though retireesare much lesslikely than paid workers are to have children pendentvariableis clicr, :
in public schools.If attitudes are determined by self-interest,then, we ex- right" or "too little." Ir :r
pect retireeswill be more likely than workers to favor increasedspending ing" versus"about rigir:"
for Social Securityand lesslikely than workers to favor increasedspending Retirees in fnct ilru -
for education.Moreover, if the deepening-divisionsargument has merit, we spendingt oo m uch ol. : . .
should find an exacerbation of these differences over time. of tabl e 6. 1) and less, . .
Table 6.1 reports results for spending on public education. I use logit umn 2) . The ohser r cJ ; - -
models. In a logit or "log-odds" model, the dependent variable is ex- ti sti callysignif canr
i . r n-
pressedas the natural logarithm of the odds of Y, where Y is a dichotomy above t he y- int er ccpt: :
l ogi ts below on "t t , o l: : : . .
1973, the first survel rr...:
TesrB6.1 ucati on.The f inding. ' -
ChangingAttitudestowardSpendingon U.S.PublicEducation: lievesthat we spend rc-tc, :-.
1,97
versus'Workers,
Retirees 3-2004
odds thar a wor ker b. ,
Too much spending Too little spending tion, and the odds that -.-
uersusabout right or uersusabout right or cati on is only ahour h. : -
too little spending too much spending rhat w e spendr oo lir r l,
(lolit) (losit) \il/ith regard to speni.: :
substantialgenerariona j
Convergencecoefficients -
- workers,initial year (5r)
Retirees +0.91.9 -0.613 The results in table 6.1 s
(.0ee) (.06e) suppor t f or spendinqr ': .
- workers,trends( frr)
Retirees -0.012 -0.001 trend (slope)for "roo nr.^-:
( .007) (.004) tle spending" (last rorr' ,-':::
group herew or ker s beclnr c lr . .
Baseline
Interceptfor workers(d) -) )9 +0.1.27 on educat ion, r nd ur ur . . . .
(.051) (.028) But retireesexhibit rhe .,::',
Trendfor workers(B) -0.028 +0.038 the trend differences c, -.
(.003) (.002) l n shor t : O ver r lr r p. . : .
for more spendingon F.::. -
N = 1 9 , 8 1 6 . S t a nd a r de r r o r s a r e in p a r e n th e se s.
Dataz 1973-2004 General Social Survey, adjusted for black oversamplestn 7982 and 1987. De- genera t ionaldif f er en, . . , - -
pendent variable is GSS variable nateduc. \l'orkers are respondents who are working full-time or portive than workers fc,: . -.
part-time in the paid labor force (codes 1 and 2 on the GSS variable wrkstat), and retirees are those ive today. Yet there :. :
coded 5 on wrkstat.
widening.
Model: logit(y,,\ - a + pT + SrGr+ Qr(T x G.)
The subscript "2" denotes group 2. Year is coded I973=0 so the intercepts pertain to 1973.
N ow let 'slook . r t r h: . : -
"Workers" is the baselinecategory and "retirees" is group 2, so 5, is the estimated difference be- ri ry l table 6. 2) . lf r h. r = .
tween retirees and workers in 1973 and @, is the estimated difference in the linear time trends for gested,then we shoul; ::
retireesversus workers. Trends are reported as averageannual change. w orkers over t he piiir : . , .
 The SixthRule . 189

- '-: - iJirrtion 6,7 to natedsc and "natural logarithm" refers to the use of the basee instead of the base
r - '-, . : p.r\-il1g into SocialSe- 10. To convert a logit back to an odds, you take the antilog of the logit. A
- :- : - ::'.:rrison of the attitudes logit coefficientof 0.919, for example,implies an odds of e)'ete-2.5L
..,: -":-,.r :, should be esPeciallY Becausethe dependentvariable consists of three categories(too much
- - -: ', ::.:, i n t he c as eof s P e n d - spending,spendingis about right, and too little spending),I dichotomize Y
r- a- .":: I'1. school taxes even two different ways and report results for both. In the first model the de-
l,; : ....::) .rreto havechildren pendent variable is dichotomized as "too much spending" versus "about
-r- .:.:-:n ter c s tt,hen, we e x - right" or "too little." In the secondmodel the contrast is "too little spcnd-
:-- - :.-'.,rrincreasedspending ing" versus"about right" or "too much."
-.
,:.:-i - -..-','(rr increasedspending Retirees in fact are more likely than workers to say that we are
j - - .,::ument has merit, we spendingtoo much on public education in the United States(column 1
':- , -:: .:: iifilf . of table 6.1) and less likely to say that we are spendingtoo little (col-
. - : . . - ., ;d uc at ion.I us e lo g i t umn 2). The observeddifferencesbetween retireesand workers are sta-
: r -, i :r:ri dent v ar iableis e x - tistically significantand large. The y-intercept for retireesis 0.919 logits
::-' : .'.rer eY is adic hot om y above the y-intercept for workers on "too much spending" and 0.513
logits below on "too little spending." The y-interceptsrefer to valuestn
l 9 7 J ,1hs fi rst surveythar asksthe questi onubout spendi ngon publiced-
ucation. The findings indicate that, in 1'973, the odds that a retiree be-
lievesthat we spendtoo much on public educationis eo'ele-2.5times the
odds that a worker believesthat rve spend too much on public educa-
:,;{ Tcto little spending tion, and the odds that a retiree believesthat rve spendtoo little on edu-
tr' ot' ;= 0.54)that a rvorker believes
- | uersusabout right or c a ti on i s onl y about hal f the odds (s
i: too much spending that we spendtoo little.
(logit) \7ith regard to spendingon education,then, there does appearto be a
substantialgenerationaldifference,as leastas of 1973. But is it growing?
-0.613 The results in table 6.1 sr.rggest not. Among workers there was growing
(.06e) support for spending for public education, as indicated by the negative
-0.001 trend (slope) for "too much spending" and the positive slope for "too lit-
(.004) tle spending" (last rou, of table 6.1). In other words, over the time period
here workers becameless likely to believethat we are spending too much
+0.t27 on education and more likely to believe that we are spending too little.
(.028) But retireesexhibit the sametrends, as indicated by the nonsignificanceof
+0.03
8 th e tr end di fferences(drs).
(.002) In short: Over the past three decadesthere has been growing support
for more spending on public education in the United States.And there are
: : . . rnrp l e si n 1 9 8 2 a n d 1 9 8 7 ' De -
generational differences:Thirty years ago retirees tended to be less sup-
r -: .: r: r: : \ \ 'ho a r e w o r k i n g f u l l - tim e o r portive than workers for such spending, and retireesare still lesssupport-
: - : ' : \\ rkstat), and retireesare those ive today. Yet there is no evidence that the generational division is
widening.
Now let's look at the trend in attitudes about spending on Social Secu-
-- : - . : le in t e r c e p t sp e r t a i n t o 1 9 7 3 .
.- ,-: - I : s t h e e s t i m a t e dd i f f e r e n ceb e -
rity (table 6.2).If there is generationalpolarization, as some have sug-
- - -: - : - '. : : : : ; c t n t h e l i n e a r t i m e t r en d s fo r gested, then we should observe an increasing gap between retirees and
: . .:..:.
' workers over the past two decades.As expected, we find that in 1984
 790 . Chapter6

T,tsrg 6.2
versus\)forkers,
ChangingAttitudestoward Spendingon SociaiSecurity:Retir:ees
1984-2004

Too much spending Too little spending

uersusabout rigbt or uersusabout right or
too little spending too mtrch spending
(losit) (losit)

Convergencecoefficients
Retirees-workers, initial year ( 6r) -0 .882 -0.345
( .171) (.06e)
Retirees- r.vorkers,trends ( $, ) + 0 .014 -0.003
( .014) (.006)
Baselinegroup
Intercept for workers (a) - z.-) r +0.062
(.053
) (.02e)
i

lrend tor worKers(p)

,:'
-0.02r +0.022
(.005
) (.002)
N=22,098. Standard errors are in parentheses.
Data: 1984-2004 General Social Suney, adjusted for black oversamplesrn 1987. Dependent
variable is GSS variable natsoc. Workers are respondentswho are working full-time or part-time in
the paid labor force (codes 1 and 2 on the GSS variable wrkstat), and retirees are those coded 5 on
wrkstat.
M o d e f : l o g i t tY , , \= a + P1 + ' r Cr + r b , tT xCr )
The subscript "2" denotes group 2. Year is coded 1984=0 so^the intercepts pertain to 1984.
"-Workers" is the baselinecategory and "refirees" is group 2, so 6, is the estimated difference be-
tween retirees and workers in7984 and @, is the estinrateddifference in the linear time trends for
retireesversusworkers. Trends are reported as averageannual change.

retirees were less likely than workers to believe that we spend too much
on SocialSecurity:According to the table 6.2 estimates,in 1.984the odds
that a retiree believesthat we spend too much on Social Security were less
than half the o{ds that a worker believesthat we spend too much on So-
cial Security (e 0 882=0.4).Contrary to the polarization thesis,however,this
gap did not grow over the next two decades.since @, is not statistically
significant. \7e conclude, then, that the time trends for retirees and work-
ers are parallel.
An interesting wrinkle appearsin our results when the dependentvari-
able is coded as spending too little on Social Security. Recall that retirees
rn 1984 were lesslikely to say that we spend too much on Social Security.
The wrinkle is that retirees were also less likely than workers to say that
we spend too little on Social Security. In other words, retirees were more
likely than workers to say that spending on Social Security is about right,
whereasworkers were more likely than retireesto say that the spending is
 The S i xthR ul e . l9l

off one way or the other, either too high or too low.7 Interestingly, work-
:::-ej lersus workers, ers over the past two decadeshave become more likely to say that we
s p e n dtoo l i ttl e on S oci alS ecuri tytB = 9.922. hi ghl y si gni fi cant) Ycr
. r e-
tirees have also becomemore likely to say we spend too little, and the gap
Too little spending
t,ersusabout right or between retirees and workers has remained stable from 1984 to 2004
too much spending (dz=-0.003, not significant).As we saw, the gap is constant also in the
(logit) caseof "spend too much." In other words, there is neither divergencenor
convergence;the trends are parallel for workers and retirees.
\Wefind no evidenceat all, then, for the notion that generational divi
-0.345 'We
(.06e) sions are growing in the United States. do find evidenceof growrng
-0.003 support for increasedspendingfor both public education and Social Secu-
(.006) rity in the United States, at least over the time periods examined here
(1973-2004 in the caseof public education and 1984-2004 in the caseof
Social Security). Critically, though, this support has increasedin tandem
+0.062
for retireesand workers. Even with a sample size close to 20,000 for both
(.02e)
issues,we found no statistically significant differences in the trends for
+0.022
(.002) workers and retirees.There appearsto be no generationalwar around the
corner.
Finally, it is important to stressthat convergencemodels should not
: : ). rm p l e s i n 1 9 8 7 . D e pe n d e n t
--: rr g .i u l l - t i r n eo r p a r t -tir n ein be estimated blindiy, rvithout first becorning familiar with one's data and
: ': : : t i r e e s a r e t h o s e c o d e d 5 o n building reality checksinto one'sresearch,as dictatedby rule 3. It is pos-
sible, for example, that change in Y is not monotonic over the time
interval-Y might exhibit a secular increaseover the first part of the in-
- r r e r r ' e n r \n e r t a i n t o 1 9 8 4 .
i :. rhe estinrateddifference be- terval and a secular decline over the second part of the interval. Or Y
--1. . r h e l i n e a r t i m e t r e n d s fo r
might change monotonicallv for some groups but not for others. It is a
', r good idea, then, to graph the mean of Y over time before estimating con-
::-.::.
vergencemodels. In some instancesthese graphs will show that the time
interval needs to be broken into smaller periods or "periodized" for
-,: : : li-][ $'e spendtOOmuch analysis.
: - - : -t.:tl .l tesin
, 1984 t he od d s

i : - . r: . '.: :pc'n dt oo m uc h on s o - Conuergence Model uersus Ch anging-Effect Model

: ;, -,- i-r::(rn. thesis,however,this I
It is useful to review what rve have learnedso far about the study of change.
.i i :- . i i ,-i d . is not s t at is t ic a l l y
Rule 6 dictatesthat you use panel data to study individual change and re-
-: ii: '* i fo i ret ir eesand work -
peatedcross-sectiondata to study social change.Paneldata can used to in-
vestigatewhether the association of X and Y has changed for a fixed cohort
r:-, i> '.,,:ler-rthe dependentvari-
of individuals. Repeated cross-section data can be used to investigate
.- - . : , .:::rr. R ec allt har r et ir e e s
,:-- : ::ruchon SocialSecurity.
TThe tendency of retirees to say that Social Security spending is "about right" might be
,: .. :.i ::.rn $.orkersto say that
an example of the psychological principle that people seek to reduce their cognitive disso-
:r.:: ','..rrds.retireeswere more nance. Becauseretirees are more likely than workers to be the recipients of Social Security
- . - -.-.Securitvis about right, benefits,retireesare more subject to the dissonancethat would result if they were not tobe-
:: :::: :,1sa| that the spendingis lieve that spending on Social Security is about right.
192 . Chapter6

whether the associationof X and Y has changedfor society in general.Re- public cducationquest:' : , -
peated cross-section data can be analyzed using either changing-effect (therewas no 2001 sLr : ', . ,
models or (if there are enough measurementpoints) convergencemodels. geststhat ir chungir r g- . - -.,-
Changing-effectmodels treat time as categoricaland use regressionmodels reportc dabuue,I r r r hr . . .
-
of the individual-levelrelationship of X and Y to ask whether the effect of cent of wor kcr s and . l s. . - .
X on Y has changed over time. Changing-effectmodels can be descriptive tle on public educarro:. -.
(has the association of X and Y changed over time?) or causal (has the worker:sand retirees.I:' :r.
causal cffcct of X on Y changed over time?). Convergencemodels are de- ures had jumped ro a l
scriptive models that focus on change in Y at the aggregatelevel. Conver- ti rees,a dif f er enceof 1- : - '
gencemodelstreat time as continuoustnot categorical,and use regression Although there is Dlliur .-:
to model the trend in Y over time. As the name suggests,convergencemod- publ i c educat ionr h. r u: . . : -
els ask whether differenr groups are converging (or diverging) on Y. port has incr easedcq, r . - .
When applied to social divisions, the convergencequestion (are group Thes e im pr essit r nr , . '. -
trends converging or diverging?) and the changing-effectsquestion (have S oci alSur veyconsi'r -
group differences changed?) are similar but not identical. A changing- of the changing- ef f cir- ' -
effect test is often a natural supplement to a convergencetest. For the ed-
Y ,,= o r +( a. - o, J- - . - '
ucation and Social Security examples above it would be useful to know
whether we would have reached the same conclusions about the stability Recall that the suf'.,: ::-
of generationaldifferenceshad we used a two-wave changing-effectmodel model. The indepenclci-:.':
instead of a convergencemodel based on all the relevant waves of GSS ous or d iscr et e.I n r hr , . . '
data. On the basis of data fbr the two endpoints, a changing-effect test w e code X as a . lLr r r r : : . . . .
tells us whether the difference in attitudes between workers and retirees is this model:
larger or smaller in the initial and final years. Although the two-wave '.
Y i ,= dt +\ az- a. , T - :
changing-effectmodel discards data for intervening years, it can still be a
powerful method becauseit is basedon individual-level data. Tabl e 6. 3 r epor r . ': . r -
The convergenceand changing-effecttests are not redundant. The con- GSSdata on attirud.> : ,,,,:
vergencemodel usesdata for intermediate years, as well as the endpolnrs, S tares.As bef or e.r hr : : : . - '
to determine if there is linear convergenceor divergence.If there is linear spendi n gver sr r slf ', ) Llr- - - -
convergence,the difference between retirees and workers will be smaller or too m uch. Resr r lr ..
in the final year than in the initial year. If there is linear divergence,rhe The changing-efiee r ::. -.
difference will be larger. The converse is not true. A larger difference at on publ ic educat ion\ \ . : . . -
the final year (ch?nging-effectmodel) does not necessarilymean that rhe i n 1973- 7S- wor kcr . : -
trends are still divdrging, since the trends might be nonlinear, with the di- too much on educ. r r r . , -. . ' .
vergenceoccurring earlier in the period. For the same reasons, a smaller (dr- du colum ns 1 an; I -
difference at the final year does not necessarilymean that the trends are the conver gence nr oJ. . . - . , -
still converging. In other words, while the results of a two-wave changrng- ti on 6.10 also squar e. . '.: :
effect model can tell you whether the effect of X on Y has changed over i n the m id- 1970sr r cr . . - '
time, it cannot tell you when the change occurred, and whether it is con- we spendtoo much on :-.-
tinuing. w orkers t o say t hr r \ \ q . - . . -
Consider again the issueof generationaldifferencesin support for public The key f inding in r : - . -
education in the United States,this time investigatedusing a changing- teraction term rel,et'-. ' .'
effect model. To enlargethe samplefor the changing-effectanalysis,I com- cati ng that r he dif f er . n. . l.
bine the data for 1973-7 5 (the first three surveysin which the spendingon sameas it was thirtr' \'e.rr: :,,
 The Sixth Rule . 193

^ -i-:, r : : io.iet,vin general.Re- public educationquestionappeared)as well as the 2002 and 2004 surveys
-;- .- .: ;irher changing-effect (there was no 2003 survey). A quick inspection of the percentagessug-
-,:-. - ..:: aonvergence models' geststhat a changing-effectanalysiswill yield the samestory as the results
': : -." ::'.j rse regressionmodels reported above.In the combined 1973-75 surveys,for example,53.8 per-
,: r : '..< n'hether the effect of cent of workers and 38.8 percent of retireesthought that we spendtoo lit-
i-: --: -- ': .iel: can be descriPtive tle on public education, a difference of 15 percentage points between
j :' r rr-.ri or causal (has rhe workers and retirees. In the combined 2002 and 2004 surveys,those fig-
--' , r:qencemodels are de- ures had jumped to 76.0 percent for workers and 62.0 percent for re-
: - . ::srcgate level. Conver- tirees, a difference of 14 percentagepoints between workers and retirees.
'' - . : - -.-.i 1.. r r r Jus er egr es s i o n Although there is much greater support today for increasedspending on
-j:-' :' . -:-rts. ConvefgenCe mod- public education than there was three decadesago, the change in that sup-
::: -- :.rl \'!'r ging)O n Y . port has increasedequally among retirees and workers.
- :: (are group These impressions can be tested more formally. Becausethe General
- -::-..i qllestion
: - - r--', :-rffccts ques t ion( h a v e Social Survey consists of repeated surveys,I employ the interaction form
| . --. . ientical. A changing- of the changing-effectmodel:
: - - ' ::ience test. For the ed-
Yi ,= dt+ l az,-ar)T,+ B rX ,,+ @ 2- B r)i ' X i /71)+ e,t 6. 9)
r, : : , '-:iJ be useful to know
-: Recall that the subscripts L and 2 refer to time in the changing-effect
- , ..:.(rnsa bout t he s t abi l i ty
-, i, :.h .1nging- ef f ec t m o d e l model. The independentvariableX in equation 6.9 can be either contlnu-
- : : -.: :elevantwaves of GSS ous or discrete.In the erample under considerationhere, X is discrete.If
:-ir ::. .t changing-effecttest we code X as a dummv variable for retirees(X=1 for retirees)rve have
': -:r , : r:-.\\-orkersand retireesis this model:
: ,-- \lrhough the two-wave Yi ,= dt+ (az-arl T,+ B rreti rees+ l h- P t)(reti reesxT)\+si t ( 6. 10)
- i: - : -.r--ive ar s ,it c an s t ill b e a
:: : , .....-l e leldat a. Table 6.3 reports results for the application of equation 6.10 to the
::-:- , J -.(rtredundant.The con- GSS data on attitudes torvard spendir-rgon public education in the United
:: : , : -. ,...rl ell as t he endpoi n ts , States.As before, the dependent variables is dichotomized as too much
-: : - , I f t her e is lin e a r spending versus about right or too little, and too little versus about right
-::. .. : -:se nc e. will be smaller
.,..orkers or too much. Results closely resemble those for the convergencemodel.
- : : - . -. :. L i rr eardiv er genc eth , e The changing-effect model indicates that workers' support for spending
- - : ::..:. .\ larger differenceat on public educationwas substantiallyhigher in 2002-04 than it had been
:.cssaril-v mean that the in 1.973-75-workers in 2002-04 were less likely to say that we spend
' : lt ()nlin ea r, with rhe di- too much on education and more likely to say that we spend too little
: : ::'.. a,1mereaSOnS, a Smallef (a2- a p columns 1 and 2) . This result squareswith the trends observedin
::.,i. : :li.lfl that the trends are the convergencemodel. Results for the retirees dummy variable in equa-
r:--- r- ,)i a f\\'o-wavechanging- tion 6.10 also squarewith results from the convergencemodel. Retirees
:-. r .\ (rn )'has changedover in the mid-1970s were significantly more likely than workers to say that
'- - -- J..rnd uhet her it is c o n - we spend too much on public education and significantly less likely than
workers to say that we spend too little.
: ::: ::rics ir-rsupportfor public The key finding in table 6.3 is that Gl r- b ,1, rhe coefficient for the in-
: -.rt.J us ing a c hang i n g - teraction term retireesx T, falls well short of statistical significance,indi-
ysls,I com-
::!r-efiectanalvsis. cating that the difference between workers and retirees in 2002-04 is the
n'hich the spendingon sameas it was thirty yearsearlier. On the basisof this finding we conclude,
 794 . Chapter6

Tasls 6.3
ChangingAttitudestoward Spendingon U.S.PublicEducation:Resultsfor
Model, 1973-7 5 versus2002-04.
Changing-Effect

Too much spending Too little spending

uersusdbout right or uersusabout right or
too little spending too much spending
(logit) (logit)

Convergence
coefficients
Retirees-workers, initialyear(Br) +0.767 -0,610
(.140) (.106)
- workers,trends(Br* Br)
Retirees -0 .013 -0.002
(.00e) (.006)
group
Baseline
Interceptfor workers(&,) I )1
+0.154
(.0711 (.043)
Trendfor workers(42-Ar) -0.022 +0.034
(.004) (.002)
N=4,761. Standard errors are in parentheses.
Data:7973-75 and 2002-04 General Social Surveys. Dependent variable is GSS variable nate-
duc. Workers are respondentswho are working fulltime or part-time in the paid labor force (codes
1 and 2 on the GSS variable wrkstat), and retireesare those coded 5 on wrkstat.
M o d e l : l o g i t ( Y,,) - - a r + ( a .- a r ) T r + Br r e tir e e s+ ( 8 ,- Br)reti reesxT,, rvhere teti reesj s a dummy
v a r i a h l ec o d e d I fo r r e tir e e s.
In the changing-effect model, the subscripts I and 2 denote time. Year is coded 1973*75=0 so
the intercepts pertain to time 1. Note that all the variables,including time, are dummy variables,so
"linear trend" here is reflected by differences in intercepts. To facilitate comparison with results
from the convergencemodel (table 6.1), coefficients for workers'trend (a,-a,) and trend differ-
ence (Fr-Fr) were divided by 29 to convert from total to averageannual change over the period
1973-75 to 2002-04.

as before. that rhe rrends for workers and retireesare neither converging
nor divcrging. Viith the changing-effect model, however, the evidenceis
less direct since we do not know what the trend was between 1975 and
2O02-something that we can readily examine under the convergence
model. The evidencefrom the changing-effect model is also based on a
smaller sample.
In this instance, then, the changing-effect model and the convergence
model tell the samestory,with effectsizesthat are roughly similar (table6.4).
This will not always be the case.Becauseit usesall the waves of data, the
convergencemodel provides a more direct test of whether trends are con-
verging or diverging. On the other hand, changing-effectmodels are more
widely applicable in the social sciencessince they require only two waves
of data. They ask whether effects are the same at two different points in
time-a more delimited issuethan the issue of convergence.Convergence
 The Sirth Rule . 195

Trnrp 6.4
: . --: - r: ,r: Resultsfo r ChangingAttitudestoward Spendingon U.S.PublicEducation:ComparingResults
for Convergenceand Changing-Effect
Models

r, - .. . .:. :. Too little sPending Too much spending Tto little spending
: " -' ': ' L'ersusubout right or uersusdbout right or uersu.sdbout right or
too little spending too little spending
i (logit) (logit) (logit)
Convergencecoefficients (both models)
- 0. 610 Retirees- workers, initial year
( . 106) Convergencemodel + 0.9I9 -0.613
- 0. 002 Changing-effect model + 0.767 -0.610
( . 006) Retirees- workers, trend difference
Convergencemodel
+ 0. 154 Changing-effect model
( . 043)
+ 0. 034 Baselinegroup (both models)
( . 002) Intercept for workers
Convergencemodel -2.29 +0.I27
Changing-effect model I tl
+0.154
-. -: . : ri; rb l e i s G S S v a r i a b len a te - Tr end f o r w o r k e r s ( a n n u a lc h a n g e r
: :r. in the paid labor force (codes Convergencemodel -0.028 +0.038
.: ' ,,n *'rkstat. Changing-effect model -0.022 +0.034
: , 1.. rlhere retirees is a dummY
No /e :Resul tsfromtabl es6.l and5..l .\=l 9.S l 5forconvergencernodel and4,76 l forc hangi ng-
': . . \ 'eir r i s c o d e d 1 9 7 3 - 7 5 = 0 so effect model.
-: . : : : : inre ,a r e d u m m y v a r i ab le s,so " Not statistically significant.
c o m p a r i s o nw i t h r e su lts
: --. . : , it a r e
- : : rnd ( a , - 4 1 ) a r - r dt r e n d d iffe r -
r: : -rinu a l c h a n g eo v e r t h e Pe r io d
models will becomeincreasingiyuseful in the social sciencesas we accu-
mulate more and more measurementpoints in our data sets.In the mean-
time, with changing-effectmodeis we can often learn quite a bit from just
r::i Jre neither converging two waves of data, particularly if the measurementinterval is appropriare
.:'.rr\\'c v eff ,he ev idenc ei s fo r th e changei n questi on.
-.J \\..rsbet."veen1975 and
:'.: Lrnderthe convergence
: :r'.odelis aiso based on a BnrncrNc INorvrouer AND SocrAL CneucE: EstruetrNc Cononr
Rp,plecrlrpNT EFFECTS
::. ,Jel irnd the convergence
'Where
: :, ,r.r9hlr-similar(table6.4). does social change come from? By social change I am referring to
. ,..11the waves of data, the change on some aggregateattribute for a societS such as change in per-
': $'hethertrends are con- centage of peopie who vote or percentageof people who attend church
:rl-effecrmodels are more regularly. There are two major proximate sources: Individuals can
:', require only two waves change, and the composition of a population can change (Ryder 1965).ln
,:-:r\\'o different Points ln other words, change in public attitudes could be due to changing public
-,:i\ ergence.convergence opinion or to changing publics.
196 . Chapter6

The effect of changing publics-changing population composition due placernent,The ne-rr5i;:. ,.

to population turnover-is often ignored in discussionsof social change. partitioning.
For example, popular discussionsof the rise of political conservatismand
the increased popularity of the Republican party in the United Statesin
An Accounting Scltentc '' '
the 1980s were often framed in terms of questionssuch as "Why are peo-
ple converting to the Republican party?" Becauseof population turnover, S upposewc Ar e inr r : 1. : . . .
however, percentageRepublican could rise during a period when no one soci alc hangef r om . r , - -
switched to the Republican party. The composition of the U.S. electorate Y . Our aim is t o dcr cr : '
changes over time as older birth-cohorts gradually die off and are re- percen t Republic. lr r : \ , . -
placed by younger birth-cohorts. This type of population turnover, called much i s due t o r lr ecr i. . .
cohort replacement or cohort succession,leads to societal change when subsum esper cent . r sc.. : "- :
the attitudes and beliefs of the cohorts entering the electoratediffer from a bi nary var iahlec. Jc-
the attitudes and beliefs of the birth-cohorts that are dying off.8 able coded 0, 100. r
It turns out that cohort replacement in fact accounted for a large parr With repeatedsurr'.', :.::
of the 1980s'rise in Republicanism,as cohorts entering the electoratein w e cannor f ollon r h. , -
the 1980s tendedto be more Republicanthan the older cohorts they were as they age. We e,rn. .. :
replacing (Norpoth 1987). Similarly, by decomposing the change in seg- fol l ow t he changcin : - . -
regationistattitudesamong U.S. whites in the 1970s and early 1980s, we w i thi n-cohor tor it t r '. ; .
find that growing acceptanceof interracial marriage in the United States ci etal ch angeb\ ' \ cf . i- . . : ' -
in that period n'as driven primarily by cohort replacement-reflecting tercoho r tchange.uh. - , - -
generationaldifferences-whereas changein other racial attitudes resulted on the mean oi ) . L
largely from individual change acrossa broad swath of society (Firebaugh ment eff ect s:I f all . , , : ' - .
and Dav is 19 8 8 ). cohorr f or anot hcr\ \ . . . - .
Separatingthe effect of population turnover from the effect of aggregated C ross- cohor teh. r : : - , .
individual change is strategicin the study of social change. By identifying pl acementt o soei, r i . - , '- - .
the proximate sourcesof social change,we know better where to look for usedfor est im at ine. , - - l -
underlying causes.ril/hen we know that changing attitudes toward interra- cohortsis not cr pr r r r i- l
cial marriage are due to cohort replacementrather than to aggregatedindi- W i th in- cohor r. h. r : . - - - :
vidual changgx(re question "Why are the younger cohorts more tolerant?" ual changet o socir r i. ' . -
replacesthe question "'Why are people's attitudes changing?" Those rwo ti me w i t h r epear e,-.rl. - . .
questionsmay point in very different directions with respectto underlying served) t, he ner ef ic. :
causes. Supposewe find rhar : :
Becausepure panel designs do not have a mechanism for adding new D emocr at ,and I ndup; : '''.- -
cohorts as they enter a population, panel data alone cannot be used to in- W i th re peat edst r r r . r - . . .
vestigatecohort replacement effects. Repeated surveys, by contrasr, are curred o ver t he per i, '. . I . -
well suited for investigatingcohort replacementeffects.With repeatedsur- parry swit chilr g.t lr c . r . . : - : .
vey data it is possible to estimatehow much of the observedsocial change party swit chingr et ulr .* "
is due to aggregatedindividual change and how much is due to cohort re- vidual change(if anr' :: :.
W i rh r epeat edcr , , . . - - .- -
EPopul:rtionturnoyer effects consist of more than just cohort replacement i ndi vi du alchangeir , ': : -
effects, smce
populations can also change becauseof migratior-r.Typicallv, horvever,the cohort succession time. This strateg\i: l, . .
process dominates the population turnover effect, so studies generallv speak of separating might reflecrmore rh.r:' : .
the effect of cohort replacementfrom the effect of aggregatedindividual change. reflect migrationefie.r.
 The Sixth Rule . 797

placement.The next sectionsdescribea straightforward method for such a

:i :.:-.qpopulationcomposition due
-.: ,'i iitiutsions of social change' partitioning.
-: 1ic of political conservatismand
r..--..r lrarty in the United Statesin An Accounting Schemefor Social Change
: : -trsticlnssuch as "WhY are Peo-
Suppose we are investigating social change on some variable Y. Overall
i " F --a urc o f populat iont ur n o v e r'
social change from survey 1 to survey T is Y.- Y' where Y is the mean of
: ::i- Jltring a Period when no one
-: : ::rpositionof the U'S' electorate Y. Our aim is to determine how much of Yr-Y, {for example, change rn
percent Republican) is due to aggregated individual change and how
^ '.. s.,,d.,ully die off and are re-
much is due to the effect of cohort replacement. (Note that the mean of Y
, :'. :r irf population turnover, called
subsumespercentagesand proportions, since a proportion is the mean of
,, ''. le.,Jsto societalchangewhen
a binary variable coded 0, 1 and a percentageis the mean of a binary vari-
:i .:.:cring the electoratediffer from
able coded 0. 100.)
. ,. ,:r\ that are dYing off.8
\fith repeated survey data we cannot follow individuals over time, so
:-.: .:- flct accounted for a large part
... .,,horts entering the electoratein we cannot follow the change in racial attitudes for particular individuals
'We
as they age. can, however, follow birth-cohorts over [ime, so we can
.=., :1.r.rn the older cohorts they were
follow the change in racial attitudes for cohorts as they age; we call this
:'. JecomPosingthe changein seg- 'We
within-cohort or intracohort change. begin the decomposition of so-
:. . .: '.t he 1 970sand ear lY198 0 s ,v i ' e
States cietal change bv separating intracohort change from cross-cohort or in-
-:-.::,rl marriagein the United
replacement-reflecting tercohort change, where cross-cohort change refers to cohort differences
, r'. ;ohort
on the mean of Y. Cohort differences are necessaryfor cohort replace-
:r.:r it.tother racial attitudes resulted
ment effects:If all cohorts had the samemean on Y, then substituting one
, ,'-1:i,ad swath of society (Firebaugh
cohort for another would have no effect on the overall Y-mean.
Cross-cohort change, then, bears on the contribution of cohort re-
'*::'-.\\'cr from the effect of aggregated
,;-":. ,rf social change.By identifying placement to social change. As noted earlier, panel data alone cannot be
used for estimating cohort replacementeffects,since the effect of entering
::. '.'.i know better where to look for
-.:: ;1-rr.rgingattitudes toward interra- cohorts is not captured in panel data.
to aggregatedindi- Within-cohort change bears on the contribution of aggregatedindivid-
-:::-.;::l ,"ther than ual change to social change. Although individuals are not followed over
: :i: \'()unger cohorts more tolerant?"
changing?" Those nvo time with repeatedsurvey data (so change for specificindividuals is unob-
r
-'i -ittitudes respectto underlying served),the net effect of individual change on social change ls observed.
: -.:;.llons with
Suppose we find that political party preference (percentageRepublican,
Democrat, and Independent)remained the same over time within cohorts.
: :.-,.''i a mechanism for adding new
Nfith repeated surveys, we do not know how much party switching oc-
r---:'.:lJrii alone cannot be used to in-
by contrast, are curred over the period. But we do know that, to the extent that there was
-. R.:cilted surveys,
effects. With repeatedsur- party switching, the changescanceledeach other out within the cohorts, so
:.,..-< r.r.rel-rt
-.,. *-.,,;l-rof the observedsocial change party switching resulted in no net changewithin cohorts. As a result, indi-
l.rorvmuch is due to cohort re- vidual change (if any) produced no sociai change in this instance.
ti; -'-:--ul
\fith repeatedcross-sectiondata, we can glean the effect of aggregated
individual change from change in the Y-mean within birth cohorts over
- -: --.'::'.lLtstcohort replacement effects' since time. This strategy is not foolproof, since change in Y-mean within a cohort
,- ' . , ficallv, however' the cohort succession might reflect more than the effect of individuals changing on Y: It can also
. - '.- :. . , rrr'rdiesg e n e r a l l ' vs p e a k o f se p a r a tin g
reflect migration effects or the effect of differential cohort mortality. Imagine,
- :- : : . ', : iregat edi n d i v i d u a l c h a n g e '
198 . Chapter6

for example, that party preferenceis fixed for all individuals after age eight- S up posewe ha\ € iiir , .
een: Once a Republican always a Republican, and similarly for Democrats trni tsl individuals,r n : : . -
and Independents. Percent Republican could nonetheless increase within mon i n t he socialSur r - - .- .
birth-cohorts if Republicanstend to outlive others.Similarln the averageed- anal ysisas cum ular . J - . : :
ucation of a cohort could rise over time simply becausethe less-educated Lineardecomposirir)n
: ..
tend to die younger than the better-educated. Sfe could detect this sort of rnerge ddat a. and t lr c - . . .
effect with panel data but not with repeatedcross-sectiondata. Thus in in- threev ar iables{agc.\ : . : :
stanceswhere there is reason to believethat differential mortality or migra- The 6r st st ep ir r {', : - . .
tion is relatedto Y, it is a good idea if possibleto comparethe within-cohort year an d r espondent '.- , - , .
trend in repeatedsurveyswith the comparabletrend in panel data. the sake of dar a dr . p. . . .
cohort int o br oaderLr . : - :
Line ar D ecomp osition Metb o d l arl y i f t he sur verr . r r r j .
itself, however, one-\'--:
The cohort replacement accounring scheme divides total social change shorthandI r r ill r . f . , , lt , l
over some time period into (i) the part due to cohort replacementand (ii) as rhe appr opr iat cn'. : - -
the part due to aggregatedindividual change. The magnitude of the co- bi rth-y earand sur r . r . '- '
hort replacementeffect is determined by the size of cohorr differencesfor ti on of t he y- ir r t er cr F:..: .
the cohorts rotating in versus those dying off and the rate of cohort re- surveyand bir t h- re. r ". : .
placement. The contribution of aggregated individual change to social The next st epi< r o: - - - -
change is determined by the intracohort trend, that is, by the average vey (yeart hat r he r nJ ' *
change over time within cohorts.
Partitioning of this type makes senseonly when the pace of cohort re- Yu= Bor +B r , St t r t 'i^.
\..:'
placement is commensuratewith the pace of social change, so that cohort
E quation 6. || r ' r r c: : : :
replacementis a plausible sourceof the change.It would be pointlessto in-
val ue o f Y f or an in; - -
vestigatethe contribution of cohort replacementto the weekly or monrhly
the i nd ividualuJ:
swings in the popularity of a sitting U.S. president, for example, since the '. : : . -
From t he r esulr . , '- : -
ups and downs of presidential popularity happen at a much faster pace
changeon Y. To ASSI >: : ': -
than the pace o{aohort succession.It would also be pointless, ar the other
I to su r vey f is d<n: . . ,
extreme, to investigatethe contribution of cohort replacementto change
those meansb1 t akir - - -
in voting patterns from, say, 1910 to today, since there is no overlap of
and l ast SLlr V€VSnr - ,r '
voting cohorts-in a vacuoussenseall change since 1910 is attributable to
cohort replacementbecausethe replacementis complete. In the discussron E(Y,,)=Y,=F,,r-i )r.
that follows, we assumethat the pace of the social change of interest is in
ElY,r\=lr= B, r- r
fact commensuratewith the pace of cohort replacement,so that cohort re-
placement could sensibly account for some or all of the observed social SurueyYear, is ther-e;:::.
change. \7e also assumethat measurementintervals for the data are ap- theaverage yearol 1..::
propriate for the pace of social change under investigation, so that the for the Tth surver-.
data indeed capture the change. Socialchangein ) :: -
Linear decomposition provides a way to separate out the cohort re- Y from time 1 to rim. :.
placement component of overall social change. It is called linear decom-
position becauseit assumeslinear within- and cross-cohort change; other nY=Yr_Yr=Ft\;,
methods are possible when linearity assumptions are nor met (Firebaugh whereA denotes chrn!.. -
1989. 1997 \. time 7 (for example.; =

: tr :--:rtidualsafter age eight-
:.: .rmilarlv for Democrats
-:-. -
'- - :-..ihelessincreasewithin
: r-:i-. >:nrilarly,the averageed-
: - - .':i.ruse the less-educated
'.,.:
r:i Jtiuld detectthis sort of
r;- . r -:-Si C tiondat a.T hus in i n -
,i- - .. -:. rtial mortality or migra-
i - : :, 1 'tFz lr et hewit hin- c o h o rt
irr : - -:r-j i n paneldat a.
-:- : : . . .j e s to t al s oc ial c ha n g e
: r - :'f)rt replacementand (ii)
-.; - : ; -:rc n-ragnit ude of t he c o -
i - : .,. .-'r' rf cohort differencesfor
-: :: .:-.J the rqte of cohort re-
ir: * : ::r i d ual c hanget o s o c i a l
-. :::: i. rh.rt is, bY the average
:. .,.:.:n the pace of cohort re-
.: r : -.,11change,so that cohort
-, -.. ,: ri'ould be pointlessto in-
i-: - -: . -r :tr tl -rew eek ly or m ont h l y
tr:i i:r'-t. for example,sincethe
: -.:-:::n at a much faster pace
,- j .',.: he pointless,at the other
: - :- ,r. replacementto change
,.r:.. :.r-.Jctl.rereis no overlap oT-
-:-- : ! :rr!' 1 910 is at t r ibut ab l eto
-::.: - -,itllplete.In the discussion
: - - - - .rl eh angeof int er es ti s i n
:i ::t -iJifi]eflt, so that cohort re-
' ': - ...11 of the observedsocial
:.-.: -::i|els for the data are ap-
: --j:r i1\'estigation,so that the
: :::-lrate out the cohort re-
r r :--: l r i s ca lledlinear dec o m -
- -r-.- -rrrSS-cohort change;other
:-:: ::) Jre not met (Firebaugh
The Sirth Rule . 199
Supposewe have repeatedsurvey data consisting of T surveyswith N,
units (individuals) in the rth survey. Such data sets are increasingly com-
mon in the social sciences,and generally the surveyscome ready-madefor
analysisas cumulated data files; that is, the T surveysare already merged.
Linear decomposition is attractive in part becauseit is so easy to do n ith
merged data, and the data requirements are so minimal: It requires only
three variables (age,year of survey, and an outcome variable Y).
The first step is to create a birth-cohort variable. \With data on survcy
year and respondent'sage, we calculate birth-year as year minus age. For
the sake of data display, it might be useful sometimesto collapse birth-
cohort into broader intervals (e.g.,five-year or ten-year intervals), particu-
larly if the surveysare five years or ten years apart. For the decomposition
itself, however, one-year intervals generally are most appropriate, and for
shorthand I will refer to the cohort variable as birth-year. 'We assumeyear
as the appropriate metric for surveys as well, so cohort is measured as
birth-year and survey is measured as survey-year.To facilitate interpreta-
tion of the y-intercept, it is useful to code survey-yearas zero for the first
survey and birth-year as zero for the oldest birth-cohort.
The next step is to regressY on individual's birth-year and date of sur-
vey (year that the individual was surveyed):
Y, = B or+BrrSurueyYearit+ F rrBir th Year,,+ e,, (6.11)
Equation 6.11 is meant to be descriptive,not causal. It statesthat the
value of Y for an individual is a linear and additive function of the year
the individual was surveyedand the year the individual was born.
From the results of this very simple model we can decompose social
change on Y. To assistintuition, note that social change on Y from survey
1 to survey T is defined as Yr-Y' and that we can determine both of
those means by taking the expectedvalue of equation 6.11 for the first
and last surveysas follows:
E(Y,r)=Y, = Fot+ F rr surueyYe
ar, + B, BirthYear, (6.12)
E(Y,rl=Yr= Fut+F, rSurueyYearr-
Brr-BirthYror-, (6.13)
i s the yearthe fi rst surveyw as admi ni steredand N rthYr *, it
S u ru eyY ear,
the average year of birth for respondentsin the first survey, and similarly
for the Tth survey.
Social change in Y from the first survey to the Tth survey is change in
Y from time 1 to time T, or equation 6.13 minus equation 6.12:
tY = Yr - Yr= F, A SurueyYear
+ Br A.BirtbYear, (6.14)
where A denoteschange, and we assumethe Bs are the same at time 1 and
time T (for example,Fs= Fs).
200 . Chapter6

Equation 6.14 is the key equation in linear decomposition.On the ba- Finally, let me strc:: -.- ' :
sis of equation 6.14 we can partition social changeinro two components: nratesuur ces :. . .
ul clr lr r r gc.
using linear decompo:r::-
. The first component, BrLSurueyYeal, is the contribution of aggre-
look for rhe underlriui .....-:
gated individual change to overall social change from survey 1 to i ndi vi dualconver sion: - . : '
survey T. Recall that B, is the linear within-cohort slope (a weighted tion composition(asoi;.: .
averageof all the within-cohort slopes),or averageannual change is an irnportant first sr.: .'
in Y within cohorts. To get the total contribution of aggregated
individual changeto social change over the entire time inrerval, then,
we weight the average change per year (Fr) by A,SurueyYear,the
SuuuaRv
number of years from the first survey to the last survey.
r The second component, BrABirthYear, is the contribution of cohort
R ul e 6 dicr at est he r r >: - :
replacement to overall social change from the first survey to the last
use of r epeat edur osr - - ; - -
survey. B, is the linear cross-cohort slope, or averagedifference in Y
useful f or bot h descr : : : . . -
from one cohort to the next. To get the total contribution of cohort
over time, panel data .. :-i i
replacement to social change over the entire time interval, then, w-e
to rel no vet he enJt r r r n- : - - :
weight the averagecross-cohort change l!z) by ABirthYear, the dif- causalmodels.
ferencebetween the averageyear of birth for respondentsin the first P ane ldat a and r cp- '. : '.-
survey and the last survey. To illustrate, supposethe averageage for gate the f ir st gener . r .l '. . .-
respondentsis forty for a survey taken in 1980 and forty-two for a questj o n:Hcs t he . f r - , :
survey taken in 2005. The 1980 respondentson averagewere born ask w het hert he clic- :
i"J279rgd 2005 respondents on average were born in 1963, so al s.\X /it hr epeat edCf , : : - - : -
LBirtbYear=1963-1940=23 years.To esrimatethe contribution of has changedin t hc - - - . -
cohort replacementto social changein Y from 1980 to 2005, then, poi nts a r e r equir eJ : . .
we would multiply Frby 23.
tw o w a vesor t \ \ 'o ai, - - - - - -
The linear decomposition method has been used in a number of recent in- associ a t ion changeJ: : '-
vestigationsof social change (for example, Alwin 1996; Brewster and Pa- causeso r cor r ellt er . : : - .
davic 2000). The key assumption is that the effectsare linear and additive. mentpo int s.Wit h cr : , : -
In particular, equation 6.14 assumesthe same intracohort and intercohort models with time l' ::'.- - -
slopesat times 1 and T(compare equations6.12 and 6.13). We can check B y c ollecr inguf J. : : : - . .
the plausibility of our assumptionsby summing the trvo components.Gen- poi nt, repeat edcr o. r - . =- -
erally the two componentsdo not sum exactly to the differencebetween Y, ti gati ngsocialchanq. . l- , '-
andYn but the discrepancyshould not be large. If the components do not for addressingthe st; :-: .
sum approximately to Yr-Y' the problem could be survey-to-survey convergencequestion H,,-
bouncinessin the Y-trend. In that caseperhaps more reliable estimatesof al l groups?)and t hc s, '. . .
the overallchangein Y might be obrained by using regressionor moving i n Y ca n be accour r r . : -
averagesto smooth the trend in the Y. If a large discrepancyremains after popul at ion?)The
. enr - '.- - -
smoothing, the underlying assumptionsof the linear decomposition model interaction terms to i::::'-'
are dubious, and another decompositionmethod should be used. i ng on Y. The conr - : l'. -
The linear decomposition merhod is simple-making very few data be estim at edusing t r : . - , - '
demands-and elegant. It applies to the study of social change for binary regress ionof Y or r hi: : - - -
variables(changein proporrions or percentages) as well as to the study of The issuesin t hi. . : '. : : -
chanse for continuous variables. the quant it at ivest uJr

.:-'-r Jecomposition'On the ba-
.l-i:ngeinto two conponents:
'....
,.:-. .: the contribution of aggre-
. -':-rl change from surveY 1 to
: : , , . . , hrn-cohorstlope( a weig h te d
: l:i or averageannual change
: : . , . ; ontri but ion of aggr eg a te d
: ';: Ihe entirc time interval,then,
-.:- ,;.1r (F,) bf LSurueYYear,the
last Survey.
_-: :Lrrhe
:r:'.
. i. the contriburion of cohorr
-
.:. ::itu.tthe first surveyto the last
: : . . , , F e. o r ave r agedif f er enc ei n Y
:-': ::l- rotal contribution of cohort
l: ::-.. errtire time interval, then, we
,:-:.:'.i. LE,) by LBiirYeal, the dif-
: ,'',rh f,r. respondentsin the first
averageage for
-:tr.:.i3.slrpposethe
::\::', in 1980 and fortY-two for a
::r:\rlldents on avefagewere born
:-..:.\eragewere born in 1963, so
::.. Trr .rti-nt. the contribution of
.-: - : ir Y from 1980 t o 2005, th e n ,
recent in-
--,::,--Lrsedin a number of
.::... -\lrvin 1996; Brewster and Pa-
,,: ::. effects are linear and additive'
..-.::.:.nlcintracohort and intercohort
:: : -. 6 .12 and 6' 13) . ' W ec an c h e c k
.-:::-:ringthe two comPonents'Gen-
. ::\...-t1\'to the differencebetween Y,
: :: l.rrge.If the comPonentsdo not
:: :-.:rl could be survey-to-survey
": :::r.lps more reliableestimatesof
or m o v i n g
' - : * rr\ usi ngr egr es s ion
discrepancy remains after
-: . irrge
-: : :he linear decomPositionmodel
r ':r:thod shouldbe used.
: - .rrrple-making verY few data
::,: ::.idv of social changefor binarY
.::-::li.1geS)as well as to the study of
Thc Sixth Rule . 201
Finally, let me stressagain that the accounting schemehere locatesproxt-
mate sourcesof change,not the underlying causes.Partitioning social change
using linear decomposition does, however, provide insight about where to
look for the underlying causesof change.Knowing whether the changelies rn
individual conversion(individualsactually changing)or in changingpopula-
tion composition (asolder cohorts die off and are replacedby newer cohorts)
is an important first step in locating the root causesof the change.
Suuntenv
Rule 6 dictates the use of panel data to study individual change and the
use of repeatedcross-sectiondata to study social change.Paneldata are
useful for both descriptive and causal studies. By tracking individuals
over time, panel data enableresearchersto describeindividual change and
to remove the enduring effects of unmeasuredindividual heterogeneiw in
causalmodels.
Panel data and repeatedcross-sectiondata both can be used to investi-
gate the first general question posed in this chapter, the changing-effect
'With
question, Has the effect of X on Y changed over time? panel data we
ask whether the effect of X on Y has changed for a given set of individu-
'$fith repeatedcross-sectiondata we ask whether the effect of X on Y
als.
has changed in the general population. Becauseonly two measurement
points are required to estimate a changing-effect modei, we need only
two waves or two cross-sectionsof data to determine whether the X-Y
associationchanged from time 1 to time 2. To identify the aggregate-level
causes or correlates of the change, however, we need multiple measure-
ment points. With enough measurementpoints we can estimatemultilevel
models with time as the context.
By collecting updated samplesof the population at each measurement
point, repeatedcross-sectiondata setsare especiallywell-suited for inves-
tigating social change.Hence repeatedcross-sectiondata are recommended
for addressingthe second and third questions posed in this chapter: the
convergencequestion (Has Y changed in the same direction and pace for
all groups?)and the cohort replacementquestion (How much of the change
in Y can be accounted for by change in the cohort composition of the
population?). The convergencemodel usestime as the X-axis and includes
i n reracti onterms to derermi new hethergroupsare convergi ngo r diver g-
ing on Y. The contribution of cohort replacement to social change can
be estimated using the regression coefficient for cohort obtained in the
regressionof Y on birth-cohort and year of survey.
The issuesin this chapter are timely for social research.Prospectsfor
the quantitative study of social change have never been better' and thev
202 . Chapter6

should continue to improve in the future as ongoing panel and repeated A P P E, NDI X
survey data sets steadily add new waves of data. It is likely, then, that the
issuesof this chapter-the investigationof changeor stability in individual-
leveleffects,the investigationof convergenceand divergenceof groups, and StudentExercise
s
the investigation of cohort replacement'scontribution to social change-
will be central to social researchin the twenty-first century.

ThE exErsises agaln ..!: ;

vey,Co t u lhe r r c[ 's; : '. . .
bel o w.By f ollo* r r r r : : . :
statisticsyou needru.--:.-
j
N ot e: Recallr harr l=
most recentGSSdrr: ., ..
working with more r-l-
rreed to us e thc "rc l c .'
2004. Also r ecallr h. r : . .
somewhatsincethesei.:.
tions below might be , :::
the logic belorr ro ol-: . -

A ssrcNvr Nr l: Cor i, . -
IN A TTITU D E S A B .I ..- .

[n most of t he sur r , . - -
labeled racmar:

Do you t hink r L. - ,
groes/Blacks/-\ir.-.. .

B eginningin l9Et r . . . . : -
as to whit e Am cr ir ', : ', :
cans sey that ther f.: . -
exercisewe f ocus o" ''
A s we will .:: -
'ee
w hi te adult s u hu . . . :
l astt hr eedecade. r. : - ' -
est trendsobser ve. : ' - -
declinewas dr ir '. n : - -
chan gingr heir vicr i . '
to answert hat qr . r . . : .
Th e exer ciseior - - : - : '.
trend f r om 19- l r , -
 i- : - :-.spanel and repeated AP PE N D IX
: :; . . , .: .s l i k elr ' ,t hen, t hat th e
-- :-: : :.tabilit v in indiv idu a l -
,J: ',,- I - r',a:genceof groups, and StudentExerciseson Rule 6
, - - :. ,n to social change-
.: - - - ':-.i n tur ) - .

The exercisesagain use the 7972-2004 data from the General Social Sur-
vey. Go to the web site http://sda.berkeley.eduand follow rhe instrucrrons
below. By following the instrucions you should be able to generateall the
statisticsyou needto answer the questions.Variable namesare in boldface.
Note: Recall that the figures given below arc tor the 1972-2004 GSS,the
most recent GSSdata avaiiablewhen this book was written. Thus if you are
working with more recent data, to replicate the figures eractly you will
need to use the "seiection filter" in the SDA program to remove years after
2004. Also recall that the format in the SDA web site might have changed
somewhatsincethesedirections were written, so some of the specificdirec
tions below might be out of date. You nonethelessshould be able to follow
the logic below to obtain the tablesyou need to answer the questions.

AssrcNur,Nr 1: CoHonr REpracEllENT'sCoNrnreurroN To CHANGE

IN ATTITUDES ABour INIERRRcIRTMeRnrecn

In most of the surveys from 1972 to 2002 the GSS asked this question,
labeled racmar:
Do you think there should be laws against marriages between (Ne-
groes/Blacks/African
Americans)and whites? (yes/no)
Beginningin 1980, the questionwas posed to African Americansas well
as to white Americans. Horvever, only about 6 percent of African Ameri-
cans say that they favor laws against black-whire intermarriage, so in this
exercisewe focus on the trend for whites.
As we will see shortl,v,the GSS results indicate that the percentageof
white adults who say they favor such laws has declined sharply over the
last three decades,from 4 in 10 in 1972 to 1 in 10 today, one of the steep-
est trends observed in the GSS. It is important to determine whether the
decline was driven primanly by cohort replacement or by individuals
changing their views on interracial marriage. Our aim in this exerciseis
to answer that question.
The exerciseconsists of five steps: Recode racmar; look at the racmar
trend from 1972to 2002 (the question was not asked in 2004): creare a
 Chapter6

variable called birthcohort and calculate change in the mean of birthco- shou ldobt ain ir r c{r r : :
hort from 1.972 to 2002; regress racmar (recoded) on birthcohort and for birthcohort.
year (year of survey); finally, estimatethe contributions of cohort replace-
ment and aggregatedindividual change by inserting the appropriate val- Question1. How do r',.-:
Which one reflectsrriti..:.
ues into equation 6.14, above. Here is some help on how to do that:
cohortchange?Erplairr,"
o Use the recode command (found in the pull-down menu under "Cre-
ate Variables") to create a new variable racmar2 that is coded 100 -
Question 2. (a) Use equ.,.:.
for those who think there should be laws against interracial marriage contributions of cohorr i i. . .,- ; -
and 0 for those who think there should not be such laws. (We use the 1972 - 2002de'lr r r c. . . . . . . :
100 instead of l becausewe want to use percentagesinstead of marriage.Round off r c".r::-: :-.
proportions-percentages are more convenient for the decomposi- ':
\flhat do you findr (HI\l: -
tion later.) For the 1972-2004 cumulative data set, you should find counts for about nvo-th^:-, - .
22,155 respondentswho say they opposesuch laws, and 6,629 who changeaccountsfor a1.,,-:: -: r-
say they favor such laws. (b) D o the t wo com pon. : ': - - - '
r The second step is to examine the trend over time by crosstabulating the percentagesfor racnrrrl ''
racmar2 (row variable) with year (column variable). Because we i q i mnorrnn r
want to restrict the analysisto whites, enter "race(1)" in the box be- i c) S umm ar izeyour r cr L: . : - . .
side "selection filter(s)." Select "No weight," and run the table. replacementis the larger r ":: -
Note the decline in the percentageof those who think there should
be laws against black-white intermarriage.
r Now use the "Compute a new variable" command under "Create A ssrcrl ' , t ENr2: CH r r e t r :
Variables" to create a variable called birthcohort, as follows: birth- P uerrc Epucer r ox: R. . - , , '
c ohor t -y e a r-a g e - 1 8 8 3 . T h e v a l u e " 1883" i s addedto setbi rthco-
hort at zero for the first birth-cohort in the sample. (The oldest The challengein this .r!) - - :-
people in the GSS cumulativedata were born in 1883.) We do this chaptera nd in t he nolr - : ' -
for conveniencein interpreting the y-intercept. table. You can use the E.:. . ,
o Next we need to find the mean of birth-cohort for 1972 and 2002,
l ogi t analysisis one ( r : : : : :-
the first and last years we are examining. Go to "Comparison of " A nal ysis") . M ake sur . : :
means" under "Analysis." Enter birthcohort as the dependent code the durnmy r-ari.ti,.. : .
variable and year as the row variable. Again use the filter "race(1)"
to restrict your analysis to whites. Also enter racmar2(0-100),
birthcohort(0-103), and year(1972-2002) as fiiters. These filters Hints
are necessaryto make the sample here consistentwith the sample . You will needto ::- *:
you will use in the next step,the regression.Your samplesizeshould
create two dumnr" ..:.: r - :
be 24, 6 7 7 . -
i s coded 1 f or "sp. : : : :
e Now go to "Multiple regression"under "Analysis." Estimateequa-
tion 6.11 by regressingracmar2 (dependentvariable) on year and
roo lit t le. " Tlt c . . . -. : .
rl e" and 0 f or ". r i'' ..: - - '
birthcohort.e Rememberto use the filter "race(1)" to restrict your e R ecodewr kst at t ( r a: : . . . : -
analysisto whites. Select"No weight," then "Run regression."You
for wr kst at =5 ani r . : : : . ' =
eln this casewe arrive at the same conclusion rvhether we use logistic regressionor Ol-S r C re at ea neu vr r i. : i'. : : - -
regression so, to simplify matters, the assignment asks for OLS even though the outcome to th e f ir st ) ear . I - - :
variable is dichotomous. o C re at et he int er r ; . . .'

t":ril
 ChapterTitle . 205

:i: -r',.rre in the mean of birthco- should obtain a regressioncoefficientof -0.318 for year and -0.122
:r.rr :;;oded) on birthcohort and for birthcohort.
.: - -riihutions of cohort replace-
Question 1. How do you interpret the coefficients -0.318 and -0,722?
: :', ::lrcrting the appropriateval-
Which one reflects within-cohort change and which one reflects cross-
:---:.:clp on how to do that:
- cohort change?Explain what these coefficients mean, in plain English.
:. :-: :.ril-down menu under "Cre-
:: ..:.;- tacmar} that is coded 100 Question 2. (a) Use equation 6.14 and a hand calculator to estimate the
',: ,r',',i against interracial marriage contributions of cohort replacementand aggregatedindividual change to
,..; not be such laws. ('Weuse the 7972-2002 decline in support for laws prohibiting black-white inter-
--.
:' - :" use percentagesinstead of marriage. Round off your estimatesto the nearestone-tenth of a percent.
:: - ,irlenient for the decomPosi- V/hat do you find? (HINT: You should find that cohort replacemenrac-
-:-.,-.,r:i'edata set, you should find counts for about two-thirds of the decline and that aggregatedindividual
'
l:^, )\e such laws, and 6,629 who change accounts for about one-third of the decline.)
(b) Do the two componenrs sum approximately ro rhe observeddecline in
-:::-.i orer time by crosstabulating the percentagesfor racmar2 from 1972 to 2002? Explain why this result
.ir ;,rlLrmn variable). Becausewe is important.
:. : : i. . itte r "ra c e( 1) " in t he bo x b e - (c) Summarize your results in a paragraph. Are you surprised that cohort
-\ n'eight," and run the table. replacementis the larger contributor to the decline?Why or why not?
:: : :hose who think there should

, :...-,rie'' command under "Create AssrcNl,rENt2: CueNcrNGArrrruDES TowARDSprNorNc oN U.S.

,.,:: birthcohort, as follows: birth-
'
-. I S83" is addedto setbirthco-
.:i ir.r the samPle. (The oldest
- ..
:- : .,, , : : e b orn i n 1883. ) W e do th i s
:: ' .-l: 1 tercept .
: : : : h -co hortf or 1972 and2 0 0 2 ,
', - - -: : rl nl n g.
Go t o " Com par is o n o f
:: : bir th co hor t as t he deP e n d e n t
.: :. . \g .ri nus e t he f ilt er " r ac e (1 )"
- : :. - \l so e nt er r ac m ar 2( 0-1 0 0 )'
- -l- 1, 1 02 ) as f ilt er s . T hes e f i l te rs
. .-.::econsistentwith the samPle
:: *:;-:iott, Your samplesizeshould
- . . : : . r " \n al y s is . " E s r im at ee q u a -
I :::rndent variable) on Year and
: , : i: r e r "race ( 1) " t o r es t r ic t Y o u r
: . : . : . " tl -re n"R un r egr es s ion ."Yo u
':::1 : r \ \ 'e us e l o g i s t i cr e g r e s s i o nor o L S
: - : r: \ : lor O LS e v e n t h o u g h t h e o u tco m e
Puerrc ErucetroN: RnrrREEsvERSUS'Wonr<nns,I97
3-2004
The challenge in this assignment is to use the information given in this
chapter and in the notes to table 6.1 to reproduce the coefficients in that
table. You can use the Berkeley SDA web site to do the assignment since
logit analysis is one of the options (it's on the pull-down menu under
"Analysis"). Make sure that your sample size is 19,816, and that you
code the dummy variablesthe way I did.
Hints
. You will need to recode the dependent variable, nateduc, twice, to
create two dummy variables.The first-let's call it toomuchspend-
is coded 1 for "spend too much" and 0 for "about right" and "spend
too little." The second, toolittlespend, is coded l for "spend too lit-
tle" and 0 for "about right" or "spend too much."
. Recode wrkstat to create a dummy variable retiree, where retiree= 1
for wrkstat = 5 and retiree= 0 for wrkstat = 1-2.
. Createa new variabletrend=year-1973 (so the y-interceptperrains
to the first year,1973).
o Create the interaction term retireextrend=retiree'rtrend.
206 . Chapter6

This givesyou the variablesyou need to reproduce the results in table 6.1. C H A PTE, R 7

Question 3. (a) What are your estimatesfor retiree? Is either coefficient

statistically significant?In plain English, describewhat your result means. The Seventh
Rr:..
(b) What are your estimatesfor trend? Is either coefficient statistically sig-
nificant? In plain English, describer,vhatyour result means. LET MTTHOD BL ...-
(c) What are your estimatesfor the interaction term? Is either coefficient
statisticallysignificant?In plain English,describewhat your resuit means.
(d) What are your estimatesfor the 1'-intercepts? Interpret.

Statisticalestimatii,: :- .'-
and resear ch. lct il: . . : .
t() the researchqr.c-:. , .
-i.
designand qucstr,r;--. .
ti ce mer hodear rlr c-
parti c ularm er hoJ : : : , :
than thc method dr's:-,
l n this bt - r okI lr : .
ods bu t have lo. L: : :- -
phasison det iqr r, . : - - -
Ii ttl e value in lr s, - j : -
si gn right - \ o \ \ ': : . - . - . '
refining our estinr-r:
ti cs and econor "- : - - - '.
resear ch. Fin. r ll'. : '.
uni maginat ir e r c. ; . - - - -
searcht hesedr r r : . r \ . . - - -
resear chdesign. .T'- - - .
searchr odal li. . : : : : - : : . -
conj un ct ion r r ir h . : : . : . -
methods per se.l
In t har light . r r r . -
j
i s i mplicit in m u. l:
[O Sefv e a s a L O L I l l t i : .r :
-
wright2004,p.lt': ..-',
better r esear . h.) , , . : . , :
w ho wor ship at r : . '- : . -
thi nking t hat r ceh: ' - ,

' lt i. imnnrr rr' , ._-

nrcthod. Tl rc fi rqJ :--..-.

rearch des i gnor .r. : :r:.
imnnrrrnce nl d-.i..' .
.1".i".. r..."1 ,l ,', .
. :-: rc5ult sin t able 6. 1 . C H AP TE R 7

:;ri Ir either coefficient

,.- -:.:\'ollr result means. The SeventhRule
s ig -
:. :;te nr s t at is t ic allY
LET METHOD BE THE SHRVANT.NOT THE,MASTER
-.1 ltf.
rl I. either coefficient
- -,-:\ our result means.
| ::lffl.

Statistical estimation methods should serve as the handmaiden to theory

and researchdesign, not the other way around. You fit the researchdesign
to the researchquestion, and rhen fit the estimation method in turn to the
design and question. \X/hile u'e all agree to this rule in principle, in prac-
tice method can becomemaster becausewe becomeso enamoredwith a
particular method that our researchis designedaround the method rather
than the method designedto fit the research.
In this book I have not s;rid rnuch about statisticalestimation meth-
ods but have focused more l.rearilr' on researchdesign issues.The en-r-
phasis on design over estimtltion is deliberate.In the first place, there is
little value in preciseestimatesof the rvrong thing. Until we get the de-
sign right-so \r'e are estimaringthe right thing-there is little point in
refining our estimation. Second,rhere are already many excellentstatis-
tics and econometrics te\ts covering estimation methods for social
research. Finally, the emphasis on design reflects my belief that
unimaginative research design is the greatest bottleneck in social re-
searchthesedays. Our statisticalmethodsvery often are better than our
researchdesigns.The grearestpotential for breakthroughsin social re-
searchtoday lies in berter researchdesign,or in better researchdesignin
conjunction with appropri:lte esrimation, rather than in better estimarion
methods per se.1
In that light, rule 7 is a fiming conclusion for the book. Although rule 7
is implicit in much of r'vhathas come before, I include it as a separarerule
to serveas a counterweight to the "technification" (Collier, Brady, and Sea-
wright 2004, p.266) assumptionthat more complex methodologyequals
better research. Sometimescomplex is better, and sometimesnot. Those
who worship at the altar of complex methods are prone to the error of
thinking that technical sophistication can substitute for knowledge of the

I It is important to stressthar there is r.rosharp boundarv

befiveendesign and estimation
method. The fixed-effects rnodel, for exarnple, could be thought of either as a type of re-
search design or as a tlrpe of statistical estimation method. Either wav, we underscore the
importance of design, since fixed-effectsmethods dictate the use of certarn types of research
designs(panel data, sibling data, etc.).
208 o Chapter7

subject matter, careful theorizing, and appropriate research design.2As to end on a positile :: :.
social researchersour aim should not be researchwith the most bedaz- scribe the domin.rni. I
zling estimation method-in effect making method the master-but re- pline, sociologv.
search with the most appropriate researchdesign and estimation method
for the question at hand. Often simplestis best. If percentageswill do, then
use percentages. Onsrs sr oN u't r H R: . , '
There are two ways that statisticalestimation can becomethe master rn
social research. The first is through the hegemony of a single statistical To gauge t he pr edr . . - :
technique or set of closely related techniques.Although social scienrists I coun t edt he u's \ , : - : : - -
give lip serviceto the principle that the tool should fit the researchdesign SociologicalRetic:; .--
and objective,in practicethere is a remarkablesameness in the tools used quantit at ivear t i'lcr : '- -
in much nonexperimental research.To determine the causesof some out- or regression-iiken.:: -,
come variable Y, researchersroutinely regressY on some set of other vari- study.
ables.The problem is not regressionitself, but the rourine use of regression, In c hapt er 5 I r r , ': - . ' '-
as if rcgressionwere the only tool available.When there is only a hammer pl ression as t hr pr r r t . '
in the tooibox, then everythingbeginsto look like a nail. The danger is that menti o lr edt her c.lr ; ; *. . .-
social researcherswill adjust their questions to fit the method, instead of (1985) havem adc iir r : ', : :
the other way around. " del i ver t he goo, 1i." : - -
A secondway that statistical method can becomethe master is through that, evet tif alu. t . . * . ,
unrealistic expectations about what statistics can do. Statistics becomes soci alr esear che. r r :: . - '
the master when we begin to believethat fancy statisticscan substitute for of soc ial r ese, r r ; h: ''. - - - -
hard thinking about data collection and researchdesign. The danger here ti oner s m r ghr l'c r : ', - - '- -
is that false hope in the remedial power of statisticscan undercut careful the " rc gr ess) on \ " ^
thinking about design. Consulting statisticians often complain about be- the hegem onrof , t . . : '- . -
ing contacted for statistical advice too late-afrer the data are collected. Y on X, socialr . - . . '. '- - -
Sfe cannot rely on statistical wizardry to overcome faulty data and re- thi nki ng about , , : : ', - :
search design. anal ys is. decision- t : . . . -
Unfortunately, social sciencemethods too often have become impedi- i ncapa cit yt o t hink . . - - -
ments to social researchtoday, along the lines just described.The knee- ci ous ef f ect sof oLr -
jerk use of regression methods and false hope in the remedial power of cial sciences.
statistical estimation methods both limit our imagination in research. A gu [ r . I st r ess : : . . . : : .
Quantitative research in the soC,ialsciencestoday tends to be longer on unthi nking useo1': . j'
statistical sophistication than on imagination. di fferentappr o. r eh -
My aim in this chapter is to provide examples to serve as a counter- cal w or k in r he : --
'oi
weight to the privileged position of the standard stand-alone regression to do the helr r ll. ' - .
sffategy in social research.The examples are meant to be suggestive,nor
exhaustive.They barely scratch the surface with regard to the options we
should be considering in social research. I intend the examples to be N e.ruRelr Y O cr L. . '.
positive-examples of what to do, not of what not to do. I want the book
N atural exper ini- : . : - - -
2In addition, there is the danger that reliance on statistical soci alr esear chcr:.. : : -
fixes can lead to a detelora-
tion of research qualit,v as researchersare beholden to techniques that they can easily use seemt hat nar ur . r l: i I - -
(due to user-friendly computer softr.vare)but don't fully understand. facr ra r e. But inr . r : . : : .
 The SerenthRule . 209

:. - : - r:i r.j ear c h des ign. 2A s to end on a positive note. Before discussingthose examples,though' I cle-
-: - : i- ,',i rht he m os t beda z - scribe the dominance of regression analysis in one social sciencedisci-
- - .- :.' * lhe master-but re- pline, sociology.
- ::. - '-::Jestimationmethod
-'.'-- - lr:'llltlt$€S rvill do, then
Oe s sssroNw ITH R E GR E S S IoN
1:-lL'corTlcthe master in
To gauge the predilection for regression methods among sociologists,
'' .r . ingle \ r r lt is r i c a l
:^, Jughsocial scientists I counted the use of regression in the articles published in lJteAmertcqn
r :i rhe researchdesign Sociological Reuiew in 1999, the last year I was editor. Of the thirty-eight
':::ess rn the tools used quantitative articles published that year, all but four employed regression
r aJusesof some out- or regression-like methods (for example, event-history analysis) in their
. Jrlteset of other vari- study.
In chapter 5 I noted the problem with relying on standard linear re-
-.:.narlseof regression,
::lrrc is only a hammer gression as the primary, or onlv, tool for comparing like with like. As I
::-:r1.The dangeris that mentionedthere,Freedman1I991,1997), Leamer (1983), and Lieberson
::-.rmethod, insteadof (1985) have made similar points about the limited ability of regressionto
"deliver the goods." Here I want to make a somewhat different point:
: - :.: :hc masteris through that, even if always done well, the dominance of the regressionmodel in
social researchcan be stultif,ving;it limits our imagination.In some lines
- " -: ,. Statisticsbecomes
- ...::tJsc an s ubs t it ut ef o r of social researchthe regressionapproach is so domineeringthat practl-
'- - -:.r{r1. T he dangerhe re tioners might be hard-pressedeven to conceptualizeresearchoutside of
' -i ,: careful the "regressY on X" box. Clevernessand originality are the casualtiesof
-.1Ilundercut the hegemony of a single method.'$(hen the automatic routine is to regress
. ::.:- con.rplainabout be-
, :- :ir. data are collected. Y on X, social researchersquit looking for natural experiments and quit
:-- .--: i.tulty data and re- rh i nki ng about other opti ons-computer si mul ati on, decomposit ion
analysis,decision-treemethods, network analysis,and so on. That trained
: ::'- ::l re bec om eim ped i - incapacity to think outside the regression box is among the most perni-
' ...: Jcsc r ibed.T he k ne e - cious effects of our over-relianceon standard linear regression in the so-
.: ::--: remedial Power of cial sciences.
:r',.i l n. lliO nin r es earc h . Again, I stressthat the fault lies not with regression itself, but in the
. - , . .,-,,d, to be longer on unthinking use of regression,as if it were the only game in town. Often a
different approach would be better. Some of the most promising empiri-
:- :.r Sefve aS a COUnter-
cal work in the social sciencesusessomething other than regressionper se
: .:.rnd-aloneregression to do the heavy lifting, as we now see.
- ': .rr-ri () be s ugges t iv e,
not
i-- ::..irtl oPtionswe
::i.irtl to the options
- : ::; rh e er am P lest o b e Nerunerrv OccuRRrNc RaNoou AssrcNru.rNt,AcAIN
' : :,' Jo. I want the book
Natural experiments are potential gold mines for social research. Yet
social researchersrarely mine natural experiments. At first blush it might
.' .' -. - i.\ . i i n l e a d to a d e te ri o r a-
- '
seem that natural experiments are seldom exploited becausethey are in
- - . . - : : I n at tne v ca n ea sl ry USe
. - -.'.: fact rare. But interruptions of various sorts are commonplace in the social
2I0 . Chapter7

realm-new laws or policies, natural or human-made disasters,and so marri agesin dir or ; : , '- . ''
on-and many of these interruptions, with a little imagination, might be transmissionof n. . r- : . ,
exploited as natural experiments or quasi-experiments.More than a half on divorce itself ai :-:' :r -
century ago, Trygr.e Haavelmo (1,944)noted the "stream of experiments generat ion.hor r r . . . . . -
that nature is steadily turning out from her own enormous laboratory." l essi n vest m entin : : '. . :
Arguably, the relativescarcityof natural experimentsin social researchis tow ar d r nar r ilgc u . : : :
due to the lack of imagination of researchersmore than it is due to the w E ca n exper l . h. . - . : . .
lack of raw material in the social world. servi r t gt heir par cr . : . . .
\Wefirst encounterednaturall,voccurring random assignmentin the dis-
havi o r st hat r cdu, ; : . . : .
cussion of rule 5, r.vhererve noted that natural experiments are strategic Oth er expl. r n. n. , - - - -
for creating instrumental variables.Recall that the term "naturally occur- as thc pr incipirl t r . : -
ring" does not necessarilymean that nature made the assignments,but learning and morc - :i
that the assignmentswere not done for the sake of randomization, as one generalr ule singlcf-:': .:::" -.
would do in a controlled experimenr. In this secrion I describe more ex- dren. Less t im c or : - :
amples of naturally occurring random assignmentthat social researchers often rneansloir cr t - .. : r .
have been able to exploit. From theseexamples I hope to convince you of cation, financial pri:: -.i:
the value of thinking hard about rvhat a relevant natural expenment i ncreasingt he r isk ' ': ; .
would look like, and of doing this thinking early in your researchproject. The dccim at ior ,r . r : '
I do not mean to suggestthat you will always find an appropriate natural a natur alexper ir r r cr :. : . .
experiment (though of course you are more likely to find a natural exper- ent thar is t he ke. r : : ". ^
iment if you are looking for one). But the exercise itself is worthwhile: men ageseighrern '. '. :'
Even "dry searches" are useful becausethe discipline of thinking about morralit y r at e f or
natural experiments helps to sharpen your thinking about threats to the treate das a r elr r i\ . . - -
r eliabilit yof y o u r re s e a rc hfi n d i n g s . the me n who sr r r t : . i- - -
To stimulate thinking along these lines, I describe four investigations matter here, such .i: i-.: -
that have used natural experiments of some sort to gain leverageon a sig- dren to t he t r ear nr c: :- . . -. .
nificant issue in the social sciences.The first investigation addressesthe to w i dowhood- s. 1r : r :
question of how marital instability is transmitted across generations,the Ger m an childr cn -
second addressesthe question of birth-cohort effects, the third estimates and for r nedf ar nilr c, : : .
peer effects in schools, and the fourth addressesthe question of whether gel har dt{ |999r uir '- ' : - -
schooling in America is an equalizing force or merely a mechanism for re- Germln populr r t i, )n r-
producing classprivilege acr/ss generations.The issuesare diversein sub- chi l dr enhavc f ar . . l r , :
stance; what the examples have in common is their use of some type of and I 945. 77 per ecr . .*: .
natural experiment (along with regressionmethods) to gain analytic trac- in single-parentfam: ::
tion on a question that has proved difficult to answer using more conven- single-parentfamiL:. : -:.
tional methods. paren t s) Dieknr
. . r nr..: . . * :
vari atem odel usin: , . . - -
Random Assignment Based on National Tragedy: basel ine.O t her t h r r . . . . - .
Post-Woild War II Germany for respondent\lror: :
paren tf am ily.l/ r ir . . :: - . " -
The first example usesfamily structure in post-\7orld War II Germany to w ho gr ew up ir r . r . : . - ,
gain leverageon the causesof divorce. Divorce feeds divorce, that is, chil- ri sk of divor ce \ \ . 1: - : '. - : : -
dren whose parents are divorced are more likely themselvesto end their on the r eason ior . : : . , - .

-
-- - -r--ridedisasters,and so
- - . . : .:rrg inat ion.m ight b e
- : 1 . : : . -J:,!s\.. { or e t han a ha l f
r i :i i- - r:re .r m of ex per im e n ts
-,:. . ::rorlrQus laboratory."
:', t,r I -' ::'Ii itl social researchis
,-, :'- rl :: fhan it is due t o t h e
:., -r-.signment in the dis-
:'.:i :i rnent s ar e s t r at e g i c
: :rrm "naturally OCCur-
,:: :he assignments,but
: :.rndomization, as one
-:r i I describemore ex-
' : ::r.lt social researchers
- .-.,:e to convinceyou of
. .1.: natural experiment
- . our researchproject.
.: ,:r1appropriate natural
: ' trnd a natural exper-
- .. itself is worthwhile:
- :.rne of thinking about
. :-: ebout threats to the
- :.-:rr. four investigations
- :.rin leverageon a sig-
-:i : : - ::ri qation addr es s es
the
';- ::: j ,:-JTOSS
generations, the
- - '-.
-r.. rh e r hir d es r im ar e s
*- - :- -: - :.l - q ues t ionof whet h e r
: : :::-\ e mechanismfor re-
-. - -: i r,re sar ediv er s eins u b -
- : : -.:'::Llseof some type of
:.r.n s ei n e n e l
o....._.._ .yflc trac -
: : :' -',,. uslng more conven-
-r
r ::-' t' . baseline.Other things equal, thev find no difference in the risk of divorce
r' ::-'1,, :,d \\ar II Germany to
: - : : : :-l i di to r c e, t hat is , c hil -
: , : '. :rcmselvesto end their
The SeventhRule . 277
marriages in divorce. Observershave offered various explanations for the
transmission of marital instability acrossgenerations.Some of thesefocus
on divorce itself as the factor that increasesthe risk of divorce in the next
generation. For example, respondentsfrom divorced families might placc
less investment in their own marriages becauseof more negativeattitudes
toward marriage that they acquired while growing up. In addition, smce
we can expect children to learn conflict-resolution behaviors from ob-
serving their parents, children of divorce are more likely to acquire be-
haviors that reduce the chancesof successin their own marriages.
Other explanations identify the absenceof a parent, not divorce itself,
as the principal factor. These explanationsplace less emphasison social
learning and more emphasison the obstaclesfaced by single parents. As a
generalrule single parents have lesstime and money to invest in their chil-
dren. Less time often means less supervision and control; less rloncy
often means lolver investment in a child's education. Along with lessedu-
cation, financial pressure in the home can lead to earlier marriage, thus
increasingthe risk of divorce.
The decimation of the German'World War II male population provides
a natural experiment to test whether it is divorce or the absenceof a par-
ent that is the key transmitting mechanism. About one-fourth of German
men ages eighteen to forty were killed in the war. I7ith such a stunning
mortality rate for young men, it is plausible to assumethat death can be
treated as a relativeli' random event,or at least that the men who died and
the men who survived did not differ greatly with respect to things that
matter here, such as their parenting skills. Thus the assignment of chil-
dren to the treatment caregory-growing up in a single-parentfamily due
to widowhood-can be vieu.ed as random, or nearly so.
German children of the World \War II generation have now grown up
and formed families of their orvn. Andreas Diekmann and Henriette En-
gelhardt 11999)use marital data from a 1988 random sample of the \fest
German population ages eighteen to fifty-five to investigare how those
children have fared in their own marriages.For those born between 1933
and 1945,77 percentgrew up in two-parent families, 16 percentgrew up
in single-parent families due to war deaths, and 2 percent grew up in
single-parentfamilies due to parental divorce (5 percent grew up with no
parents).Diekmann and Engelhardt(table2) report the resultsof a multi-
variate model using children who grew up in a two-parent family as the
for respondentsfrom a two-parent family and respondentsfrom a single-
parent family, i/the absenceof the parent is due to death. For respondents
who grew up in a single-parent household due to divorce, however, the
risk of divorce was increasedby 180 percent.Thus divorce risk depends
on the reason for a parent's absencel "risks are higher for children of
zIz . LnaDtef /

divorce than for respondents who grow up with a widowed parent" Constitution, rvhich :-.:. '
(p.792). This evidencesuggests,then, that it rs divorce that is transmitted. women were eligib'1.:
i nto rwo gr olr p\ - r - '
voti ng agc.ar r elJ ir ' . . .
Random Assignment Based on National Laws: Voting in
P ost-Ninete enth Amendment America A mendm enrchungc- . .
fect of hist or ic. r-l . ': - '
A venerablebody of work in the social sciencesassertsthat historical con- voti ng r at esof u, ': '
ditions presentwhen one "comes of age" have lasting effectson one's atti- vote wit h t he vot in: - . : : . .
tudes and behaviors (Mannheim 1927; Ryder 7965). Becausedifferent '
amendm entin lq- ^ : '
birth-cohorts are subjectedto different "slices of history" as they come of voting ratesof nrq:'.','.-
age, cohort differencesarise. As Norman Ryder (p. 844) explains, "Each of age on vot inc. I : \ ' -
new cohort makes fresh contact with the contemporary social heritage and of age bef or e\ \ 'onr : : ' -
carries the impress of the encounter through life." Or as Ron Lesthaeghe age male count er f . : . - : - .
and Johan Surkyn (1988, p. 40) put it, "Cohorts develop distincrive w e have st r ong r r '. : . - - - .
meaning-giving universesearly in life and seemto maintain thern through- tioning.
or-rtadulthood." \Wecall this embedding of history in cohorts a cohort ef- Fi rsrir is uscf Lr:l . .
fect: "Cohort effects occur wheneverthe past history of individuals exerts to vote t han m cn. \ : . ': : , '
an influence on their current behavior in a way that is not fully captured ment wom en in i. i. : . ' - - .
by an age variable" (Hobcraft, Menken, and Preston 1982, p. 10). to do so. I n t hq ": . : . : - '
Yet it is very difficult to determine how much cohort effects matter, or pol i ti cal scienuc" | : ' ":
evenwhether they exist at all. The problem turns on separatingcohort ef- ri am a nd Har olJ F
fects from age effects. Peoplewho come of age at the same time are the the 19 23 m a\ '() r . t - . - -
same age, and age together. So if we find, for example, that people who data for the * h, . '
were born in the 1940s differ from those who were born in the 1960s, it 1982, t able2: I ll; '-
is hard to know whether the difference is due to their age differencesor to for men and \\'onri::
their cohort differences(arising from their experienceof different histori- The ke1 que( ! . ' '- - -
cal conditions during their formative years). Adding data for other rime ti nuedt oer hil. it . '- -
periods only complicates the problem, since that introduces the possibil- and t2 ) whet herr : . : * . :
ity of what is called period effects, and cohort (year of birth) equals pe- K evi n Chcn and i. '-
riod (year of measurement)minus age (Glenn 2005). teenth Am en. lnr . : - : . - . '
Note that the opeqativemechanism for cohort effects is historical con- w ornell t hc ir t t pr , ': ; . : .
ditioning: Different/cohorts have different tastesor values or worldviews tel l i ng f acr or .lor i. '. : - . -
becausethey were exposed to different historical conditions during the i n vot e t ur nout . "
formative period of their lives. To separatecohort effects from age effects, Mor eover .\ \ 'c r \ : . , -
t hen, we need to i ma g i n ec i rc u m s ta n c e isn w hi ch i ndi vi dual si n a gi ven horts t hat is kerc- : :
society are randomly assignedto different historical conditions as they l angu ageof er p. r , : ' - - : '
come of age. By framing the issuesthis way, we see immediately that the j. : - :
S omewom en in
effects of historical conditioning-cohort effects-could be isolated from cenrs dur ing di. r : '- - '- -
age effects through a controlled experiment in which people of the same no mem or \ of Ji. : : . : - . ' -
age were randomly assignedto different historical conditions during their duri ng ef f ector r ' - - - . -
formative years. greate st exposur c: - '.
Can we find something resembling such an experiment in the real have "t he gr eat c. : . : - : "-
world? Consider the passageof the Nineteenth Amendment to the U.S. vote, wit h a 1c. . . - : - - . - -

.,. -:: \\'ith a widowed Parent"
.: : ;-.dir orce that is transmitted'
:. L-ttt's:Voting in
ir:i-c: ilSserts that historicalcon-
-
-: , r Lrsting effects on one's atti-
; R . :cr 195-5).Becausedifferent
':".-.s of history" as they come of
. R '. ier (p .844) ex plains ," E a c h
social heritageand
- :-.:-nrporary
:::' lLfe." Or as Ron Lesthaeghe
,. "Coho.tt develoP distinctive
to maintain them through-
- ::::n
: : :lstorv in cohorts a cohort ef-
: :-rr: history of individualsexerts
:. -:-'.,..r\that is not fully captured
. :. : '. iP re sto n1 982, P . 10) .
',.":-.ruchcohort effects matter, or
.:t :urns on separatingcohort ef-
-: i Jge at the sametime are the
::. :or example,that PeoPlewho
.. '...ho u'ere born in the 1960s' it
.; :-.. ro their age differencesor to
-:r: c\perienceof different histori-
:,'-:. . .\dding data for other time
..r:: rhat introducesthe Possibil-
. ;, ;v1rrt(-vearof birth) equals Pe-
t ' .: : lrl )0 05).
:': :,rhort effectsis historical con-
':.: :.lstesor valuesor worldviews
-: :..iorical conditions during the
:=:. -,rhort effectsfrom age effects,
:: : \ \ l ri ch i ndiv idualsin a g i v e n
::::': historical conditions as they
.,,..:.'..\\'e see immediately that the
-
.: ::iccts-could be isolatedfrom
:',:..: in rvhich PeoPleof the same
': : i:llrlcal conditions during their
: .,--h an exPeriment in the real
Ame ndm entt o t he U .S'
' , .. : . en th
. - . "" , .1
The Seventh Rule . 213
Constitution, which extended voting rights to women in all states-Before
women were eligible to vote, sex randomly separatedchildren at birth
into two groups-a group that would be eligible to vote upon reaching
voting age, and a group that would not be eligible to vote. The Nineteenth
Amendment changed the historical conditions. To compare the lasting ef-
fect of historical conditioning on birth cohorts, then, we can compare the
voting rates of women who came of age before women were eligible to
vote with the voting rates of women who came of age after passageof the
amendment in 1920. By comparing the voting rates of women with the
voting rates of men who are the same age, we can separateout the effect
of age on voting. If Nineteenth Amendment women-women who came
of age before women could vote-are less likely to vote than their same-
age male counterparts, and this gender gap disappearsfor later cohorts,
we have strong evidence for the enduring effects of a cohort's condi-
tioning.
First it is usefulto know whether,in the 1920s,women were lesslikely
to vote than men. Scattered evidence suggeststhat Nineteenth Amend-
ment women in fact were less likely to vote (than men) when first eligible
to do so. In the "first sample survey recorded in the annals of American
political science" (Bennettand Bennett 1987, p. 158)' Charles E' Mer-
riam and Harold F. Gosnell (1924) discovereda substantial gender gap in
the 1923 mayoral election in Chicago-a result consistent with turnout
data for the whole of Illinois for the 1920 national election (Kleppner
1982, table 2; Illinois was the only state to report vote turnout separarely
for men and women).
The key questionsare (1) whether women of the amendmentera con-
tinued to exhibit lower voting rates than men thirty to forty years later,
and (2) whether the disparity disappearedamong later birth cohorts. As
Kevin Chen and I explain (1995, p.978): "After passageof the Nine
teenth Amendment, historical conditions no longer 'taught' young
women the impropriety of voting. So if historical conditioning is the
telling factor, postamendmentcohorts should exhibit no sex differences
in vote turnout."
Moreover, we expect a pattern of disappearinggender gaps across co-
horts that is keyed to the passageof the Nineteenth Amendment. In the
languageof experiments,there are severallevelsof the "treatment effect":
Some women in fact were denied the right to vote, others were adoles-
cents during disenfranchisement,others were children, and others have
no memory of disenfranchisement.If historical conditioning has an en-
during effect on cohorts, the effect should be greatestfor cohorts with the
greatest exposure to the conditioning; hence disenfranchisementshould
have "the greatest lasting effect on women actually denied the right to
vote, with a lesser effect on women who were adolescentsduring the
2I4 . Chapter7

disenfranchisementera, followed by those who were children during that C l assr oom soao. , a. - _- ,
er a" ( F ir ebau g ha n d C h e n 1 9 9 5 , p .9 7 9 ). classmates speud.r !:;-. -:
Data from the 1952-1988 American National Election Study strongly truc pe eref f ect rir , ', . . . -
confirm thesepredictions. Even though the right to vote had been extended ful to concepnr , r ll/ .'-
to women more than thirty years-and in some instances,more than fifty ti on i s f he t r eat r nt L: -
years-before the electionsexamined here, women who came of age before uutculllewouhl h. t r r . : .
or during the Nineteenth Amendment era continued to vote at lower rates peerswhu $gr c Ju; : : .
than their male counterparts.And the stronger the historical "treatment," one' si nt er estis r n : : - : : : ,
the greater the difference.For men and women born before 1905, the odds The nt assirc r c. l- . - :
that a man voted is twice the odds that a woman did, controlling for age ol i n.r) SchoolDir i; . .
and age-squared,education, family income, region, party partisanship, how studentaulr ir rr . :
marital status, and election year (Firebaugh and Chen 1995, table 2). For l n an ef f or : tt o h, r l. i- . , - .
those born from 1,906to 1915, the odds were 43 percenrgreater for menl (as measur edl'r i- ; - - - ;
for those born from 1916 to 1,925,the odds were 28 percent greater for reduc ed- pr icelun. : - .: - - -
men.eVoting ratesare the samefor men and women born after 1925. popul a t ion.t he di. : : - : - .
In short, the ratification of the Nineteenth Amendment provided a nat- to 5 pe r centof ir : : : . .- : .
ural experiment to test the concept that the values of birth-cohorts are of the const Anrr c. . - : : : ^ -
shaped by their common youthful experiences-a central social science student sin t he dr . : : - : '
concept that is often assumedbut hard to demonstrate empirically. ti on du e eir hert . ' : . = - , - .
(H oxby and \ \ cir : : : - : ' -
For r esear . hcr . :. : . r . .
School Reassignments in Nofth Carolina ti gate p eer ef ici: . ^- - '
Our third example of the use of natural experiments in social research mentsin which . : . . - - : . -
comes from an econometric investigation of peer effects on student and Weingar r h.. r . - - : - - -
achievementby Caroline Hoxby and Gretchen Ifeingarth (2005). Peeref- dents in t he t hir - : - -
fects, like cohort effects, are often surmised but seldom proved. In addi- rhe 1994-95 s.h, : :
tion to the usual concernsabout measurementerror and omitted-variables ent var iableis : r r r ; : : '. :
bias, the empirical search for peer effects is complicated by the possibility ol i na' s end- of - qr . r ; =. . . . .
of self-selectionon the dependenrvariable. For example, adolescentswho permi t t he er ar r r r r : . . :
have an unmeasurgd propensity for delinquency might befriend other encech angesin p. . - . '- -
delinquency-prondadolescents,resulting in a positive associationbetrveen datasetincludesnr : - . -- . - .
individual's delinquency and peers' delinquency. The unwary researcher reducedpr ice lr r ni; : - - -
might interpret this associationas indicating a peer effect when in fact the i ncl ud esinf or r nlt r t ': .: . - ,
delinquency-prone individual would have committed the same acts of B ec ause t he pLr r p .
delinquencywithout the peers.There is also the possibility of a "reflection" prevent( or lt le, r . t '. .'
(Manski 1993) or "social multiplier" (Glaeser,Sacerdote,and Scheinkman the rea ssignm enr. \i: : : :
2003) effect. BecauseI am a peer to my peers,ifthere is a peer effect, then gradesin crou'ded s;:.
my outcomeswill affect their outcomes,which in turn will affect my our- i n l ess- cr or , r 'ded'r r , : - - . :
come, and so on in a continuing loop. This loop further complicates the rThe rel rri grrrnc T:.:- -.
analysis of peer effects. -
rel sri gnc d s tudent. :' - .'
pl e. or to honre-..l -,' , : -
eMoreover, this pattern of gender differences occurred di ffi cul t, horv erc r' ... F
only for voting, not for other
measuresof political involvement. with the reassignneni.

::: ;hildrenduringthat
:.:-:ior-rStudystrongly
:: h,rdbeenertended
. -. . , : . , <5.nro re th an fift y
' .,.i(r .ame of age before
: -r t,l \'ot€ at lowef rates
f .tl "tre clme nl "
. t r, r iu
':. :.itore 190.5,the odls
: j. controlling for age
- :. Pilrrv partisanship,
.-:r 1 995,t able2) . Fo r
:::-ent greaterfor men;
:: l\ percent greater for
: :.(rrnalter 1,925.
;:-.*:rentprovided a nat-
.--. ,rf birth-cohorts are
- : ---. Jintra l so cia l sclen c e
'..
,.-
.ri! - - lPr r^i
Lrl .;^^
r Lar r).
l l .,
: r :: : r-:l l rs i n s oc ial r es ear c h
- :--r effects on student
- . - '. ' .:r:ri .rrt h
- ( 2005) .P eere f-
.: - - . i ..,Jo m pr ov ed.I n add i -
*:- :: : : rn d om it t ed- v ar iab l e s
: - -'r .:-rted by the possibility
: I - :r ..:rple,adolescents who
-.rrphrhef r iend ot h e r
- - : - :..e a ss oc iat ion bet we e n
r -: - , lh.' unrvary researcher
- i r r::r .iiecr rvhen in fact the
. - -- :-::::d the same acts of
: - : : --:rrl i rv of a " r ef lec t io n "
:':.::. '., -:r[]ote, and Scheinkman
:-:-.. : : -::i is a peereffect,then
- - ' :-.rn u-ill affect my out-
- ' :-rherc om nlic at est h e
rr voting, not for other
Rule .
The Seventh 215
Classroomsprovide a strategic place to search for peer effects because
classmatesspenda great deal of time together.The challengeis to s€parate
true peer effects from selectioneffects and other types of biases.It is help-
ful to conceptualize this in terms of an experiment where peer composi-
tion is the treatment effect. Wl-ratwe want to know is whether a stlrdent's
outcome would have been different had she been embeddedin a group of
peerswho were different on the characteristicsof interest (such as race, if
one's interest is in the effects of race segregationon some outcclme).
The massivereassignmentof students in the Sfake County (North Car-
olina) School District provides a grand narural experiment for studying
how student achievementis affected by the peer composition of the class.
In an effort to balance its schools on the basis of race and family income
(as measured by percentage of srudents participating in the free or
reduced-pricelunch program) in the face of a rapidly growing srudent
population,the district has,eachvear for more than a decade,assignedup
to 5 percent of its studentsto different schools.rFor students,the result
of the constant reshuffling is changing peer composition: 62 percent of
studentsin the district have experienceda changein their peer composi-
tion due either to the reassigr.rrr.rent
of others or to their own reassignment
(H o x by and W ei ngarth2005, p. 15).
For researchers, the reshufflingpresentsa unique opportunity to lnves-
tigate peer effects becauseof the "very large number of r-raturalexperi-
ments in which studentsexperiencenew peersin the classroom" (Hoxby
and Weingarth, abstract).Hoxb.v and \Teingarth use panel data for stu-
dents in the third through eighth grade in the \fake County district for
the 1.994-95 school 1'earthrough the 2002-03 school year. The depend-
ent variable is student achievement,as measuredby scoreson North Car-
olina's end-of-grade tests in reading and mathematics. The panel data
permit the examination of students' scores before and after they expen-
encechangesin peer composition. In addition to students' test scores,the
dataset includes measures of race, gender, participation in the free or
reduced-pricelunch program, and parents' education. The datasetalso
includesinformation to identify eachchild's classroom.
Becausethe purpose of the reassignmentis to relieve overcrowding and
prevent (or at least alleviate) segregationof schools by race and income,
the reassignmentsare not random. Specifically,students in more-crowded
grades in crowded schools are more likely to be reassignedthan students
in less-crowded situations. And, given the aim of racial and income
3The reassignmentstrategv for balancing schools
could be sabotagedif parents removed
reassignedstudents frorn the public schools (they might srvitch to private schools, for eram-
ple, or to home-schooling).The procedures used by the school district rnake noncompliance
difficult, however (seeHoxby and Weingarth 2005, p. 1,1),and most fan.riliesin fact comply
with the reassignment.
216 . Chapter7

balance, in any given year students of a particular race or a particular in- Cognitive inequ.llr;. -.;
come category are more likely to be reassigned. i s w ell docum cnt . J : : . ,
The reassignmentsare, however, conditionally random: Conditional children frorn fam'l:.. ',' :
on a student's race, ethnicity, lunch participation (the income measure), rai n m or e r t lu. ilt t r '' .
school, and grade level-variables included in the dataset-students are educa t ion,t t t t t r r r '
equally likely to be reassigned.In other words, for two students of the thi s ger r er ulSLS : . ,
same race with the same level of family income, those in the same school ti on -+ Childr en's: t :
and grade are equally likely to be reassigned.In particular, as Hoxby and l n a ddit ior rt o r . ': , . .
'Weingarth (table
2) show, once the reassignmentvariables (crowdedness frorrrh ighcr SE5I . i: . . . : - -
of school and so on) are controlled for, the probability of reassignmentis student sgencr . r ll'. r . . . . . -
independentof students' initial test scores.This independenceis critical to to hi g her abilr r i '*: - . . .
their investigation,sincetheir aim is to determinewhat effect peers'achieve- favora bleint er acr r :' . .
ment has on one's own achievement, and how that effect might differ age i n equalit ie'" r r : : - .-
acrossgroups (perhaps,for example, the presenceof high-achieving peers some obsen'er: hrr
benefitsonly other high achieversand has no effect or even a negativeef- from s er vingAs. r n: , . l
fect on low-achieving students). exace r bat einequr t l; : : . . .- '
Hoxby and \fleingarth conclude that there dre peer effects on students' sentsone of the mos: :--:
achievements,but these effects hinge on peers' achievementlevels,not on S E S advant age. or
-.-:-
their income level or their race. Quoting Hoxby and rWeingarth (p. 28; serve t he int er cst : : : . .
bracketed clausewas added for clarification): (B ow lesand G inr r - - - -
N o one disput - . : , - '
The vast majority of the apparent impact of a concentrationof racial
tO recelVem or c . l: - ^ - - '
minorities, ethnic minorities, or poor students[that is, studentsin the
tion to transntir a. -,r- -' ",
free or reduced lunch program] is really the effect of their achieve-
cl ass dif f er ences.D. . : : .
ment. Put another way, if we seetwo schools with the samedistribu-
nonethelessf un. t r ' ' .
tion of achievement(not merely the same mean), we should expect
l ow er SESst udent . .. : . - .
their students' achievement to evolve similarly in the future, even if
14) explain:
the schools have quite different racial, ethnic, and income composi-
t ions . E ve n if scht loi.. r : - . . '
i n s chooler t r ir t 'r . : . . . . .
In short, on ,he basis of the natural experiments provided by the Wake
vironments. Son',. - -
County rea(signments,Hoxby and lfeingarth find that what matters for
enc es,but t he di. . i; . - -
school achievement is the achieuement composition of one's peers, nor
even more Selerc. -..
r,vhetherthey are black or white or poor or not poor. It remains to be seen
qualit y schoolr r r - : "
whether other researchersare able to replicate their findings with other
child am endir r .gr l: - - - .
data, but the important point here is Hoxby and Ifeingarth's imaginative
vant agedst udenr - .. - '
use of a school district's reassignmentplan to provide analytic leverageon
the difficult issue of peer effects in schools. In short, to determin. '.'.-.
considert wo lear ni'r - . : '
ideal experiment, \\'e \,..
Using Seasonal Comparisons to Assess Scbooling's
and nonschoolcondir , : -
Impact on Inequality
schoolsfail to ler-elrh. :
The fourth example, a study by Douglas Downey, Beckett Broh, and Paul inequality in learnins :.:
von Hippel (2004\, also involves schools:"Are schoolsthe great equalizer? tions as it is under ti:: '

' -.-,.. or a particular in-
' :.-.nJolr-r:Conditional
. -: incornemeasure),
-..r:aser-studentsare
' : :r\ o students of the
- i: in the sameschool
.::.;u l a r . as Hox by and
, ..:r-rhles (crowdedness
- :, of reassignmentis
-,:::ndence is critical to
-.,: ;fiect peers'achieve-
...: eiiect might differ
: ::ieh-achievingpeers
-- ': cvena negaf iv eef -
r, .: efiectson students'
' :-..ment levels,not on
, ' - I \ \ ' e i n o a rrh
- No one disputesthe fact that children from higher SESfamilies tend
- :r.entration of racial
:: - : - ::r.1 its , s t udent sin t he
--. ,::r'.t o f t heir ac hiev e -
::.rthe same distribu-
- : -.:.:.:'. \\-e should expect
.: ::r rhe future, even if
.::J rnc om ec om pos i -
::,,r ided by the
.. ::rlt n-hat matters{d
' ,: ,rf one's peers, not
:. Ii remainsto be seen
: : hndings with other
.,,.. :rS.rrth'simaginative
:: ,rnalvticleverageon
: ': a -i
. . :'..:ketr Broh, and Paul
Rule .
The Seventh 277
Cognitive inequality during the summer months and the school year." It
is well documented that, in the United Statesand other'Western nations,
children from families with higher SES(socioeconomicstatus) tend to ob-
tain more education than children from lower SES families. And one's
education,in turn, is a major determinantof one's own SES.So we have
this general SES transmission model: Parents' SES--; Children's edusa-
tion-+Children's SES.
In addition to obtaining more years of schoolingon average,children
from higher SESfamilies also ter-rdto receivebetter education.Higher SES
students generally attend schools rvith better resources,tl-reyare assigned
to higher ability grolrps, and the1.and their parents typically have more
favorable interactions rvith teachers.Due to what one author calls "sav-
age inequalities" in public education in the United States(Kozol 1991),
some observershave concluded that education in the United States.far
from serving as an equalizing force. servesinstead to reproduce or even
exacerbateinequalities.Indeed,some scholarsarguethat schoolingrepre-
sentsone of the most important channelsbv u.hich parents transmit their
SES advantage(or disadr,antage)to their children. In that way schools
tn
'r'
)g
- ui serve the interests primarilv of those at the top of the SES hierarchy
(B o wl esand Gi nti s 1976).
to receivemore and better schooling.The issueis rvhetherschoolsfunc-
tion to transmit classdifferencesfrom parentsto children or reduce such
class differences.Despite large disparities in resources,schools might
nonethelessfunction to reduce disparities in skills between higher and
iower SESstudents.As Dorvner',Broh, and von Hippel (2004, pp. 613--
14) explain:
Even if schoois are unfair, they may serve as equalizersif the variation
'Sfake in school environrnentsis smallerthan the variation in non-schoolen-
vironments. Some children may have relatively poor school experi-
ences,but the disadvantages in their non-schoolenvironmentsmay be
even more severe.As a result, a disadvantagedchild attending a low-
quality school can enjov a larger "school boost" than an advantaged
child attendinga high-qualitl school.In this way schoolscan favor ad-
vantagedstudents,i'et still serveas equalizers.
In short, to determine r'vhetheror not schoolsserveas equalizers,we must
consider two learning environments: in school and out of school. In an
ideal experiment, we would randomly assign children to various school
and nonschool conditions to try to separatetheir effects. The claim that
schools fail to level the playing field would be supported if we found that
inequality in learning tended to be just as high under the school condi-
tions as it is under the nonschoolconditions.
218 . Chapter 7

Although such an experiment is not feasible,we can use summer vaca- On the basisof r hc. . :
tion for students as a natural experiment to separatethe effects of school vari ab leY will , . lr . lnl-
and nonschool environments on learning. During the school year, children able X.
are exposed both to school and nonschool environments. During the sum- D ecom posit ion ir : ', . .
mer, the large majority of children are exposed only to nonschool envi- change or v ai l J rrrc l r. : :
ronments. To separate out the school effect on learning, then, we can ti ons of subpopulit r i( ) r r:
compare rates of learning over the summer with rates of learning when qoul rtiugpr ocedr r r . . , , ,
school is in session.As Downey, Broh, and von Hippel (p. 616) observe, vari anceunder inr r r : - .
this design is "analogous to a crossoverdesign in health research,where the proximate sourar. : :-
the samepatients are exposed to different treatmentsin different periods." not con st it ut et he c- . . - : - . .:
To compare students'learning in the summer and during the schooryear, can point us t o r he l: : J: :
Downey, Broh, and von Hippel use test score data from a nationally repre- ways that overall ch:::. -
sentativesample of about 20,000 children who enteredkindergartenin the the subpopular ions: - . *
United Statesin 1998. By beginning when formal education first begins, tive, or policy inreres:.I:..
Dor.vney',Broh and von Hippel are able ro examine differences across chil- populations:The nrc,r: .::.
dren that are relativelyuncontaminatedby schoolprocesses.Consistentwith decomposition.
prior research,they find significant differencesin the reading and mathemat- To givc r sim plec\ r - . -
ics skills of higher and lower SESchildren as they begin their schoolingca- rate ri s es over sonr L : . '
reers. On the first day of kindergarten a child whose family SES is one cause o f t hat r ise. \ \ r '. ,...
standard deviation higher than that of another child will, on average,have poverty primarili' ant,,:.:
nearly a t\,vo-monthschool head start on that other child lp. 62a). young or t he oldi I . : : . .
The question is whether schools tend to exacerbatethose SESdispari- country? And so on.
ties. By comparing learning rates during kindergarten and first grade with D ecom posit ioni: l: : . - . - -
learning rates during the intervening summer, Downey, Broh, and von seenas m er ell der . r r p: -
Hippel find that schools in fact reduce the inequality among students of j ust to descr ibe.Bu! : : . :
differing SESbackgrounds. Children learn ar much more equal rares dur- w hat th er e is t o er pl. r . : . .
ing the school year than they do during summer vacation. As a result, the factsar e. [ r al50f , . . : .
even though the learning gap between SES groups grew during kinder- Decompositionhas : ,.
garten and first grade, the gap grew much more slowly when school was or rates,or what \\ c :-.._.-.
in sessionthan when it was not. In the language of an experiment, expo- anal ysisof var ianee. ". '. -
sure-rd.the"treatment" (school) reduced the disparity in learning rates for si s of i n equalir l. l : -
higher and lower SESchildren. In that sense,schools are "great equaliz- 'p. .
ti on. The well- knor .r . ,
ers" in America (Downey,Broh, and von Hippel, p. 633). Apparentlv the group medns(Blinder 1-- -.
disadvantagesthat lower SES children face in school, as savageas they decomposit ion.The l : : =. , -
might be, are less consequential for their learning than are the disadvan- di vi di ng socialchanc. . -
tages they face in their homes and neighborhoods. replacementand the p.r:: *,.
exampl elt he "gr oup ': ': . : .
poi nts i n t im e, sinc. i: : . , .
DEcouposrrroN Wonr rN THESocrer ScrrNcns ods, such as the Blintle:-, ,. ,
(as describedin chapri: : :
Regressionanalysisproducescoefficients(slopes)that summarizethe as- posi ti onof inequalr r r.'. : . - . .
sociations among variables.Under certain conditions these slopes are B ecausechapt er h *: . - -
estimatesof the averagecausaleffectsacrossindividuals in somepopulation. i n th i s chapr er I f , , . :.
 The SeventhRule . 219

i: I :. ' : a. ln U S e S U m m e f V a Ca - On the basis of these slopes,then, w.ecan predicr how much an olltcolne
r -:. :i.iii the effectsof school variable Y will change on averagefor a one-unit change in a causal vari-
-. :-3 5choolyear,children able X.
-"--
: ...:r-nts.During the sum- Decomposition analysis, bv contrastJ accounts for the observed
rr. -:- :-.1\to nonschool envi- change or variance in some ourcome variable Y by isolating the contribu-
.w e
- = . - , . ii l-1- i l H ,
-.il
,L--
tl l g l l , C an tions of subpopulations ro rhe overall change or variance in y. The ac-
',:- :
r-1.t1's
of learning when counting proceduresrvill varv dependingon the narure of the changeor
',: - tr:pcl ( p. 616) obs er v e , variance under investigation, bur the objective is the same: to determine
::'i :- - ;ealth research,where the proximate sourcesof the in Y. The proximate sourcesmight
-:ri--:r- r> rn differentperiods." not constitute the causes per'ariance
se of the variance, but proximate soufces
-- - - ' -i Ju ri ng t he s c hooly ea r. can point us to rhe underlr'ing causes.There are an endlessnumber of
r: :.,:.:.::()nt a natronallyrepre- ways that overall change and variance can be partitioned, of course, so
- : r:::J kindergartenin rhe the subpopulationsshould be groups of particular theoretical, substan-
- ' --:.: cducation first begins, tive, or policy interesr. The firsr srep, then, is to identify the strategic sub-
t,: : -riicrencesacrosschil- populations:The more strategicthe subpopulations,the more relling the
*., . . : '-essesc.ons is t ent wi th d e c o mposi ti on.
To give a simple erample of decomposition,supposethe U.S. poverry
: -: .'.gin their schoolingca- rate rises over some period. As a first step toward understanding the
-)
- ,.. ,rse family SES is one cause of that rise, \\.e \\'ant to determine its source: Is the growth in
- r
r-j ' - : riill. o n avera ge . h av e poverty primarily among blacks or whites or Latinos? Is it among the
:: :-: ;:.iild@.624). young or the o1d?Is the increaseconcentratedin certain regions of the
: i , , :: r,tre those SESdispari- country? And so on.
I Icr I )-t ^' ^- r de with Decompositionis underutilizedin socialresearch,perhapsbecauseit is
-: --: . -:.1 (1lrU 5I .

seen as merely descriptive. As social scientistswe are apt to explain, nor

just to describe. Bur firsr \\'e must get the facts right, so that we know
^^" ^l- rat es dur-
LYuar what there is to explain. Good decomposition work helps us know what
r - - '-- r:'.rcation. A s a r es ul t, the facts are. It also points us toward likely explanations.
: : :: ,.:. qre\\' during kinder- Decomposition has pro'ed its mettle in the analysisof changein means
- '. . .,r'll u,hen school was or rates, or what we mighr call "decomposition of first-moments',; in the
i - - -: : -ln experlment, expo- analysisof variances("decompositionof second-moments")1 in the analy-
- .' . rrr le rrnin s rate s fo r sis of inequalitn a special q'pe of variancel and in the analysisof segrega-
- i ,: . '- - ,]s a r e " gr eat equali z - tion. The well-known Blinder-oaxaca decomposition of differences in
:-:
- : ' _:: . . l_. ,ri -l ). 'A
^r r ppar
* _: 4^_^J ent lyt- _l:
he group means (Blinder I973; Oaxaca 1973) is an example of firsr-moments
-: - . , - , irl. as sa va ge a s th ey decomposition. The linear decomposition model describedin chapter 6-
::: - :::r-.tt.ta re th e disad va n- dividing social change o'er some time period into the part due to cohort
replacementand the part due to aggregatedindividual change-is another
example; the "group means" in this case are the means on y at different
points in time, sinceit is change over time that is decomposed.some meth-
ods, such as the Blinder-oaxaca decomposition and linear decomposition
(as describedin chapter 6), employ regression;orhers, such as the decom-
. ::.rt sLlmmarizethe as- position of inequality and segregationindexes (below), do not.
-: :t,)n s thes e s lopesar e Becausechapter 6 describesa method to decomposemeans or rares,
.-.-..:..in somepopulation. in this chapter I focus on the decomposition of variances and the
220 . Chapter7

decomposition of segregation.In keeping with the theme of this chaprer, the rate of change is the .:
the methods for the most part are not fancy, but they are appropriate for w ay of conccpt ualizingi: ': : r ', : '
the task at hand. I rely on examples from prior research, beginning with tive here becausetherr .- .,
the decomposition of variance and inequality. (ANOVA), for deccrrl;.,,.,,
-
mctter, th en, t o use of t : . . .
Decomposition of Variance and Inequali4t varianceof log(earninss .:..
nents.And, sincevarianct :
Someof the most compelling examplesof decomposition in social research betw een- class and u ir hir , - - . .,
involve the partitioning of variance or inequality into successivelysmaller
cl assi neq ualit y.r espe. r ir .
subpopulations, each contained or "nested" within a larger population.
Weeden and colleasue. ...,
The citizens of the world, for example, can be partitioned geographically:
The Theil index is approp: .-::
Individuals are nestedin local communiries;local communities are nestedin
usi ng al gebr a( Jenkins1"- . .
larger jurisdictions or regions; regions are nested in countries; and coun-
decompositions(of the r;- ..-
tries are nested in the world. Likewise, students are nested in classrooms
dicate that inequalitv s'i:r : :-
that are nestedin schoolsthat are nestedin school districts, and so on.
the wage inequalitv in ::'.
The decomposition of variance or inequality within and across nested grown more rapidlr'. hor..:'.::
groups is often very telling. Some inequality measures,for example, have
Iumpier as well as more i..-;
the property of additiue decomposabilily (Jenkins 1991), meaning that As the exampie of c:::-. -
total inequality can be partitioned exactly into its within- and between- w ork i n the social scicn. . . - . .
group components (for nested groups). This property has been used, for changesby apportionins ::-.
example, to partition global income inequality into the parr due to w e can uset he logic in r sr ; : - :
within-nation inequality and the part due to berween-nation inequality Consider,for example.h .'.
(recallthe discussionof global income inequalityin box 1.1 of chapter 1).
al l i ndi vidualsin r hc r i - -
As another example, \Teeden et aL. (2007) additively decompose the (with a fast computer)\\'. - -
growth of earnings inequality in the Unired States over recent decades,
culating somemeasllre(): -:-:-
7973-2004. Earlier analysesin economicsand sociologyhave focusedon
do not have that datl..r.r...: .,..
the causesof growing inequality. Weeden and her colleagues,in conrrasr,
al l the po pulousnet i, , n. - -
investigate the consequencesof growing inequality for the class structure '
i ty), and est im at esoi
in the United States. Nested-group decomposition is appropriate here, "'. . : '
strategyi n r his in: r . r r r.e.
since individual workers are nested in occupations that are nested in '
p o s r i l o n eq u a tl o n s to i !r ::i '1 i :
larger categoriesthat \Teeden and colleaguescall "big classes."The ques-
Goesl i ng2001; Schulr
tion is whether growing inequality is strengthening or weakening the big-
Fi nal l y consider
, inr . r - . . .:
class structure in the United States. More rapid growth in between
fundame nt alinequalr n : - . , .
big-classas opposed to within big-classinequality would indicate an in-
and Tul l apur kar200i. : . - - "
clining importance of class for earnings in the United States, that is, a
spanfor all individuLr l. -
lumpier earnings distribution with relatively stronger class distinctions.
tries over the secontlh.r,:
More rapid growth of the within-class component would indicate a de-
uariance in adult life sr,-, . ,,
clining relative importance of classfor earnings in the United States.
'Weeden among those who live i, :,-:
and colleagues use two standard measures of inequality to j apurkar2 005) .Var ianc; . - - ; - :
study the trend in earnings inequality. The first measure is simply the
than it is in other high-in; :--: - ,-
variance of logged earnings. The variance of the log is often used as a
France,Japan,and SueJc:- : : ,
measure of inequality becauseit meets the essentialrequirements for ine-
W hat is t he sour er ' : - -
quality measures.In particular, rhe variance of the log is constant when
classic study of differc:: ., -'

' :Irrr neol t his c hapte r,
. ::t;\ Jre appropriatefor
re;rch, beginningwith
,:;',
-::- -r'r .lrion in sclcialresearch
c i - : - :t:()su c c es s iv ely
smaller
r:- -. :r .l larger population.
:- -- - -:: o neJ geogr aphic a l l y :
.- . - ::rmunitiesare nestedin
-: - : -': - :-iaount r iesand; CO u n -
:-i:.:- ...:. nestedin classrooms
- - : '. .',::hi n and ac r os snes te d
r -- : ::,rilS. for example,have
: -: ..s 1991) , m eaning t h a t
-: :. ii'ithin- and between-
.- - -: ::rtv h as been us ed,fo r
:: : - r: '. rn ro t he par t due to
-. -. ::i-.r cefl-Ildtion inequality
:*, : : .:: L .or 1. 1 of c hapt er1 ).
- ::;irively decomposethe
- (with a fast computer) rve could estimateglobal income inequality by cal-
- :: - a:..-::iover recent decades,
.: : --, ::ol o gv hav ef oc us edo n
- :-: -.:: ;Lrlleagues, in contrast,
-: - ,..: :-. ior the classstructure
-: i :. ,it is appropriate here,
"- --: . :..,)r15that ar e nes t ed i n
' " The ques-
- -: - - .. rrg classes.
-
- :: :1- :'.i (-lr\\:eaken-ngthe big-
''. : : ..:.J growth in between
:' ri o uld indic at ean i n -
. * i-: t ri re d S t at es ,t hat is , a
: . -- :.!(r c las sdis t inc t io n s .
.: .::-.: ri'ould indicate a de-
- uariance in adult life span, as measured by variance in the age at death
.r- - -: . :he United States.
-..:: :--:-:.:Lires of inequality to
re.lsure is simply the
:-: ltrg rs often used as a
" : .r rrquir em ent sf or in e -
: :^.r-log is constant when
The SeventhRule , 221
the rate of change is the same for everyone-consistent with the standard
way of conceptualizinginequality (Allison 1978). The measureis attrac
tive here because there is a ready-made method, analysis-of-variance
(ANOVA), for decomposing variance. It is a relatively straigl,tforward
matter) then, to use off-the-shelf computer packages to decompose the
variance of log(earnings) into its between-classand within-class compo-
nents.And, sincevarianceof log(earnings)is a measureof inequality,the
between-classand within-class components reflect between- and within-
classinequality, respectivell'.
'Weeden
and colleaguesalso use the Theil index to mcasure inequality.
The Theil index is appropriate here becauseit is additively decomposable
using algebra (Jenkins1995; Firebaugh2003, pp.127-28). Both setsof
decompositions(of the varianceof logged earnings,and of the Theil) in-
dicate that inequality within occupationsremains the source of most of
the wage inequality in the Unired States.Between-classinequality has
grown more rapidly, however; as a result, the labor market is becoming
lumpier as well as more unequal.
As the example of earnings inequalit,vdemonstrates,decomposition
work in the social sciencescan be used to shed ncw light on observed
changesby apportioning the u'hole irrto its constituentparts. Sometimes
we can usethe logic in reverse,to trv to estimatethe whole from the parts.
Consider, for example, hon' one might estimate income inequality across
all individuals in the rvorld. If rve had income data for everyone, then
culating some measureof inequality such as the Theil index. Of coursewe
do not have that data, but rve do have estimatesof the averageincome for
all the populous nations in the world (to estimatebetween-nation inequal-
ity), and estimates of rvithin-nation inequality for many of them. The
strategy in this instance, then, is to sum up to the total by using decom-
position equations to estimate the whole from the parts (for example,
Goesling2001; Schultz 1998).
Finally, considerinequalitv in individuals'life spans-"arpluably the most
fundamental inequalitv that exists among human populations" (Edwards
and Tuljapurkar 2005, p. 665). Life expectancyat birth (that is, aueragelife
span for all individuals) convergedamong twenty-one industrialized coun-
tries over the second half of the twentieth century (White 2002). However,
among those who live to age ten, has not converged (Edwards and Tul-
japurkar 2005). Variancein the adult life span is higher in the United States
than it is in other high-incomecountries,such as Britain, Canada,Denmark,
France,Japan,and Sweden(Edwardsand Tuljapurkar 2005, fig. 5).
Vhat is the source of this U.S. exceptionalism? 'We know, from the
classic study of differential mortality by Evelyn Kitagawa and Philip
222 . Chapter7

Hauser (1973), that there are substantialdifferencesin mortality rates (Reardon and FircL'.rr:...-
acfoss social classesand racial categoriesin the United States.As Kita- ro w hi ch t he lo. al . r r , '
gawa and Hauser (1973, p. 4) note, "The importance of socioeconomic or someot her popui: t : , - - '
differentials in mortality is that they point to the possibilities of reducing l f ou r aim is f o t s- . . . ;
morrality through the betterment of socioeconomicconditions in the pop- mtrst 6 r st det er nt 'ni ', ',- '
ulation." In addition, socioeconomic and racelethnic differencesin mor- schoul sunr . lsubur l. , i. ,r - . ,
tality might help explain the puzzle of the greater heterogeneityin adult suburb s?Wit hir r >*1, .
life spans in the United States. Greater heterogeneity in adult life spans ousl l th e r educt ion , : . . -
might be tied to greater heterogeneity in the U.S' population, since the i ted eff ect if nr or r - .
United States tends be more racially heterogeneousthan other rich na- Li kew i s er eduer t t gr ; - : ; - - .
tions. To test this coniecture, r'vecould decomposethe variance in adult l i trl e i m pact on or r r . : - -
life span by race. !(hat we want to know is whether the variance in adult of the s egr egat ionli. - ,
life span within race/ethnicgroups in the United Statesis similar in size to and the suburbs.
'Sfhatever In fashioningpoii; - - -
the variance for racially homogeneousgroups in other nations.
we discover regarding variance in age at death within racial groups, we first to decomposes.i:::..:
could probe further by decomposing the within-nation, within-race vari- of the s egr egat ion.: . . . '
ancesalong other key dimensionssuch as sex, social class,and causeof S eanR e ar don.John \ ': - .
death. The point is that successivepartitioning of the variance is likely to Thei l ' se nt r opvin. 1c. , ' - : -
tell us quite a bit about heterogeneityin adult life spans and reveal where overal l segr egar ion, . ,
between-aggr€gdte :ci:: i ::
we should look-and not look-for explanations of the greater life span
uncertainty in the United States. ti onal unir , su. lr . t . . -
raci al g r oup. \ er r : : - - - -
posedfur t her :t hr r - .
Decomp osition of Segregation Indexes ther pa r t it ioned ir : : - :
Segregationrefers to the extent to rvhich individuals' local environments di stri cts in t he sub, : : : - . ,
differ on the composition of some population trait (such as race or so- schoolswithin sch,-.,,- : .:-
cioeconomic status). School racial segregation'for example, refers to the C on siderf igur c - . . . . -
extent to which racial composition varies acrossschools.The more that i n U .S . m et r opolir . r r- , - - -
racial composition varies across schools, the more that a student's local
racial environment depends on the school that he or she attends, so
Total metrtri,
school racial segregationcan also be thought of as the association be-
tween race and school. From the raw data we can tell immediately when
school racial segregationis zero-that. would occur when racial composi-
tion is the same for all schools. so race and school are not related. We
could also tell immediately when there is maximum or complete Within-citr
segregation-all schoolswould be monoracial, with whites attending all-
white schools, blacks attending all-black schools, Latinos attending all-
Latino schools, and so on. Across schooi ;
In virtually all cases,of course,the racial segregationof schools is nei-
Within schooi ;
ther zero nor complete but falls somewherebetween the two extremes. In
that caseit may not be obvious, from the raw data, whether school segre-
gation is greater in city A or in city B. A number of segregationindexes Figure 7.1. Decon-,:
have been developedto solve this problem, including multigroup indexes Areas.Source:Re-,.::
 Ruie .
The Sever.rth 223

, ": 1:- ,r: LII lTlortolity rates (Reardon and Firebaugh 2002). Segregationindexes measlrre the extent
- ' : to which the local environments of individuals differ, on averageJon racc
- ...:<.1 states. As I(ita-
iJ ot socloeconomlc or some other population characteristic.
- : . .. i .i l it iesof r educ in g If our aim is to reduce the overall racial segregationof schools,we
' , ,,rditionsir-rthe pop- must first determine where the segregation is located-between city
-
'' : Ji rt r r enc c sin m or- schoolsand suburban schoolsJBetweenschool districts within qities or
i i - 'i: i -.:ero geneit Yin adul t suburbs?'sfithin school districts in central cities or in suburbs? Obvi-
:: : - -: '::: i n adult lif e s pan s ously the reduction of segregationwithin school districts will have lim-
:-: - :rtPu lat ion,s inc et he ited effect if most school segregationlies between school districts.
- . . r : -: --- :h .r n ot her r ic h na - Likewise reducing segregationacrosssuburban school districts will have
:- -: i: :ite t':rriancein adult little impact on overall school segregationin a metropolitan area if most
. , - -..-: :he lirriance in adult of the segregationlies within the centrai city, or betweenthe central city
- - . - .-.:-. i . and the suburbs.
' im ilar in s iz et o
t-:. - :-..: nations. $Thatever In fashioning policies to alleviate school segregation,then, it is helpful
:,: : : - : ::1:l lacial grOuPS,we first to decompose segregationindexes to pinpoint the principal sources
: - .-: ,n . rr ir hin- r ac ev ar i - of the segregation,so that vou knou- u'here to concentrate your effort.
.; , . : ,1-1cl - as s ,and c aus eo f Sean Reardon, John Yun, and Tamela \{cNultv Eitle (2000) show that
- - : : :.-.:\'.1riance is likely to Theil's entropy index of segregation(H) can be additivelydecomposed,so
: * : l: :l-]flS and reveal where overall segregationcan be expressedas a sllm of within-aggregateand
-:: : rhe greater life span berween-aggregate segregation.rvhere"aggregate"refersto an organiza-
tional unit, such as school district, or to a demographicgroup, such as
racial group. Very often the n-ithin-aggregate component can be decom-
posed further; thus, for example,segregationr,vithina suburb can be fur-
ther partitioned into the part that is due to segregationacross school
.: :. .'..' local environments districts in the suburb, and the part that is due to segregationacross
:: - :-.: : such as race or so- schoolswithin school districts.
.: - . : .\Jmple, refersto the Consider figure 7.1, a fir.e-partdecomposition of school segregation
:r - -- :-a()ols .T he m or e t ha t in U.S. metropolitan areas.In this schemeoverall school segregationfor
:-: -' :: :i-r;rta student'slocal
* : -: . : :- o r s he at t ends ,s o
Total metropolitan school segregation,as measured by H
,* :-- : -r: the as s oc iat ionbe -
i : -.,," :.11immediatelywhen J
- - . -, . - 'r hcrtr ac ialc om pos i -
IWe Citiesvs. suburbs(I) e+
:- j -, '. rre not related.
': - ::'-'.rlmum or C0mplete (II) <+ Within-suburbsegregation
lfithin-city segregation (III)
,- : :: ri'hitesattendingall-
\i - ,. l,-rtinosattending all-
Across school districts (IIa) Across school districts (IIIa)
: -: : : : :-t:i on o f s c hoolsis ne i - \Within school districts (IIb) \X/ithin school districts (IIIb)
' : r .r-1 ::-:1i he tr v o ex t r em esIn
.
:: ' :..:-:-.'.rhetherschool segre-
- -:---:: ,l tegregationindexes Figure 7.1. Decomposition of School Segregationin U.S. Metropolitan
-. -- -,-...: multigroup indexes Areas.Source: Reardon, Yun, and Eitle (2000).
224 . Chapter7

metropolitan areasis first partitioned into three components: the part that TrYrv 7,l
is due to differences in racial composition of the central city as a whole C ompo nentof
s Sr 'hoo.
::::
yersus suburbs as a whole, the part that is due to differences in racial Metropoli tan A re.rs

composition across schools within central cities, and the part that is due
to differences in racial composition across schools within suburbs. The
second and third parts-within-city and within-suburb segregation-are
further apportioned into segregationdue to differing racial compositions
I. Segregationof citr r.. - -
across school districts and segregationdue to differing racial composi-
tions across scbools within school districts. So there are five components II. Within-city segres.i:.-
in all (I, IIa, IIb, IIIa, IIIb), and they sum to total segregationfor the I I a . A cr o sssch o o l .:: -
-
metropolitan area. I I h . W i th i n sch o .,li .: -
Note the differing policy implications for the components. Compo-
I I I . Wi t h i n - su b u r l .>q :- r- .:
nent I can be reduced only by convergencein racial compositions of I I I a . Acr o sssch o o l - .:'
cities and suburbs as a whole. Component II is reduced by convergence IIIb. \Tithin school c .::
in the racial composition of schoolswithin central cities. By decompos-
ing II further, we can determine whether significant reductions in school
segregationare possible by changing the racial composition of schools Sozrca:
Reardon.
\'Lrr:.
.::.: : :
within school districts (IIb), or whether changesacross school districts
will be required (IIa). The samereasoningappliesof coursefor reducing segreg at iont o t hs t , r . . . . : -
school segregation within suburban areas, where components IIIa and lessthan one-third rr: ::-:::
IIIb are strategic. S econd,r he lr r r sr - : . "- .
Simultaneousdecomposition along two dimensions is also possible nori ty c om posit i, r n, r - : - -
j .:
with H. For example, in their analysis of average school segregation in D i ffere nces
in r hc i. i, .
2L7 metropolitan areasin the United Statesin 1995, Reardon,Yun, and suburbscontributc. -',:-- -:, -
Eitle (2000) partition simultaneously by the five-part scheme above and overall white/blackH:.:..-
by race. In the case of race, they u'ant to know how much of the total school s would r edu,
school segregationis due to the white-minority divide and how much is (36.9%*3. 9oot in r h: . r . -
due to segregationamong minorities themselves(blacks, Hispanics, and Thi rd , segr egr t r r , '.
, r,... - -
Asians). There are two race components (segregationof whites from mi- quarter of all m er r ,':
norities, and segregation across minoritiesa) in their scheme, so the central cit ies, m ost : ; i: : jl
Reardon-Yun-Eitletwo-way decompositionyields 2x5= 10 components thern.The opposirerr ::.-
tf school segregationin U.S. metropolitan areas.Becauserheiruesrlt, ,o posi ti on acr ossscho -
nicely illustrate the value of decomposition work in the social sciences,I encesi n r acial cur r r p,. . :
reproduce them in table 7.1. reduce school segres-:.:
From this two-way decomposition we see,first, that about four-fifths of mated 1 8. 6 per cenr .
the overall school segregationin the 217 metropolitan areas arises from ln shorr, decomp,. :
the segregationof whites and minorities. This means that eliminating all in pinpointingu'her. :,
segregation among black, Hispanic, and Asian students would reduce pol i cy i n t er vent ionii. r '..
overall segregationon averageonly by about one-fifth. Reardon, Yun, and Y un-E idedecom poiir, -
Eitle (2000, p. 358) note that the contribution of black/Hispanic/Asian segregation within ';l'.' -
l essthan l0 per cenr , : :
aBecausewhites are considered a single race, there is a within-group segregationcom Srates,whereassegrcl.'::
ponent only for minorities (blacks, Hispanics, and Asians), and not for r'"'hites. for nearly25 per eer r,r : : -

- :rf onents: the part that
.: .cntral city as a whole
..: : , differencesin racial
.. ,-.:rdthe part that is due
-- .. i i i th in s ubur bs .T h e
: ,". -,rurb se€yegation-are
' " - "j r,reinlc om pos it io n s
. '-.' i rr q r ueiel c om po s i -
. :-::a- .rre five components
\ t so r F o l ] r i o n
' r-
-r)ITlpofleflts.
,.rci.ll compositions of
:. j Lrcedby convergence
::.. ;i ti es . B y dec om p o s -
-.,:-.:reductionsin school
- lposition of schools
:- .r,ross school districts
,. ,i coufsefor reducing
j:: .()Ilponents IIIa and
': rrro flSis als o pos s i b l e
-- ..h ool s egr egat io ni n
: : - " -* -i . R ear don,Y un, an d
-- - - j':-rrt schemeabove and
, :()\\' much of the total
- ' : '. ide and how much is
:.hcks, Hispanics,and
*: - -: - ,ri ,,n of r v hit esf r om m i -
i : .' : :hei r s c hem e,s o t h e
- : -: l x5 = 10 c om pone n ts
- .t-: ,. :i:c.1lib their results so
r:r the social sciences,I
'r: : ::rrr aboutfour-fifthsof
.:renareasarisesfrom
:,r:15that eliminating all
.:udents would reduce
-::h. Reardon,Yun, and
: l.lack,/Hispanic/Asian
:i :r -:roup segreg:rtioncom-
- r ': lor \ \ ' h i t e s .
The SeventhRuie . 225
Tevrn 7.'1.
Componentsof SchoolSegregatior-r
in the Unitcd States:Averagesfor 21-7
MetropolitanAreas
White/Minority Black/Hispanic/Asian
Segregation Segregation
(percent) (percent)
L Segregationof city vs. suburbs 36.9 3. 9
II. !Tithin-city segregation
f O f th e lla. A c r o s ss c h o o ld i s r r i u r s 6.5 1.8
I I b. Wi t h i n s c h o c rdl i s t r i c t s 17.l 7.4
COmpO-
I ll. Wi t h i n - s u b u r b 'e g r e q a r i , , n
I I I a . A c r o s ss c h o o ld i : r r i c t s 14.2 4. 4
I I lb . Wi t h i n s c h o o ld i . r n c t s 5.5 2. 2
> = 80.2 t- tq 7
Sor;rre: Reardon, Yun. and Eitle 12000). table 3.
segregationto the total is small in part becausethese groups consritute
lessthan one-third of metropolitan public school enrollments.
Second,the largest single componenr is the difference in the white/mi,
nority composition of central cities versus suburbs (36.9% of the total).
Differencesin the blacklHispanic/Asiancomposition of central cities versus
suburbscontributes an additional 3.9 percentto the total, so equalizing the
overall white/blacUHispanrc/Asian composition of city versus suburban
schools would reduce overall school segregation just over 40 percent
(3 6 .9 % + 3.9' hl i n the arerageci ty.
Third, segregationrvithin city school districts accounts for nearly one-
quarter of all metropolitan school segregation(17.7%+7.4%). Within
central cities, most segregarionlies within school districts, not across
them. The opposite is true in the suburbs, where differencesin racial com-
position across school districts are more important. Eliminating differ-
encesin racial composition acrossschool districts in the suburbs would
reduce school segregation in the average metropolitan area by an esti-
mated 18.6 percent.
In short, decomposition r,vork in the social sciencescan be invaluable
in pinpointing where to look for underlying causes,and in indicating where
policy interventions have the greatest potential for change. The Reardon-
Yun-Eitle decomposition of segregation, for example, indicates that racial
segregationwithin schooldistrictsin suburbanareason averageaccountsfor
lessthan 10 percent of total metropolitan school segregationin the United
states,whereassegregationwithin school districts in central cities accounts
for nearly 25 percentof the total. On the basisof that decomposition,then,
226 . Chapter7

we know that policies to reduce segregationwithin school districts have context ef f ect s.t he: . ''. - .
a potentially greater payoff (for the reduction of total segregation)when nation. This inter..:
applied to city school districts rather than to suburban school districts. w l tcrrwc f ushr ur r , . . : : .
few i ss t r es
of t t t , , , .
ior is affc,,tcelb, -. ',.... .
THr Errscrs oF SocIAL CoNrpxr

One of the central notions of social scienceis that social context matters. Context Effects .7-sQf7'. --;-.
Growing up in a crime-infestedpoor neighborhood, for example, surely is
Let' s begin br Jct , : : " - -
worse than growing up in a safe middle-classneighborhood. Note, criti-
key soc ial and c, : ' - -
cally, that this sort of effect is due to the child's neighborhood context,
nei ghbor hoodsh. r '- - -
not to characteristicsof the child or of the child's family. To determine
school sar e bet t eri. . : : - . - - -
the effects of neighborhoods, the telling comparison is not between poor
el s of ra cial \ eqr c! . : :
children and middle-classchildren, but betrveenpoor children who grow
therei s f l cont e\ t t r i: - . .
up in poor unsafe neighborhoods and equally poor children who grew up
a c t e r i s ti cs o f L o l i tL \:.
in more affluent, safer neighborhoods.
i n d e p e nd e r r r o l 'tl i . - ... - - .
Although it might seem virtually self-evidentthat children who grorv
a m p l c . th e r e i s ,r .. ^
up in bad neighborhoods will tend to fare worse in life than they would
d e n t s i n i n n cr cr tv r ."
have had they grown up in better surroundings, that claim has proved de-
s t u d e r i rs i n su L ,r r r I.'.::.
-,
vilishly difficult to demonstrateempirically. The major roadblock is selec- Moreform.rlll.- '- -
tion bias, a problem we encountered before, in our discussion of causal
/ contexts
{forrr.r"': ..'
inference (chapter 5). Becausepeople self-selectinto neighborhoods, it is
val ue on s ofl te i l Lr:- " .
hard to determine whether differenceswe observe across neighborhoods
andX denot er h. - . .
are due to the neighborhoods themselvesor to the way that people sort ,t
(tor eram ple. . r \ t - r - :
themselvesinto those neighborhoods. Returning to the example in the pffprt p v i c t< u' l t,' rt \'
previous paragraph, suppose we find that poor children in unsafe neigh-
Ieuclcauscs of \. :':. . '
borhoods fare worse than poor children in safe neighborhoods: That
put it. "The indiri-l:.r .
difference might reflect, not a neighborhood effect, but the effect of
the questionis u'h.:.:.--'.
unmeasured family characteristics (such as differences in parental com-
al so exe r t sso. iel . ', : . . . '
mitment to their children) that would induce some poor parents to sacri-
dent of r he inf lucnl- i - . - - -
fice more than other poor parents do to live in a safer neighborhood for
In a classicst uJ, : . '.
their children.
l eagues ( St ouf ier . t l
Context effects,then, are both important and contentious in the social " prevalenceof vr l. r c. .
sciences.There is, for example, the question of whether racial segregarion
havi orsof Anr er ie. r r r)
(a contextual property) has negativeconsequencesfor minority groups-
that i nexper i5'q; sl. r , . : : - -
and, if so, what kinds of segregationlead to what kinds of harm. With tho. .r!au\
.".1, i,,. ,, .
LrrL/ vvl...
'* rr ivl !'.--
respect to residential segregation,for example, Douglas Massey (2004)
recently has claimed that the segregationof races leads to poorer health
among African American adults. \fith respect to school segregation, ' l t i s important r(' r' :.
there is longstanding interest in the effect of racial segregationon the ac- rrf popul lti on (utnp,' ;tt: '
ow q ro di f fererrc es i rr .rr r-
ademic achievement of minority students (Card and Rothstein 2006;
are hi ghe r i n c ountrr . l rl : -
Coleman et aL.1966;Charles,Dinwiddie, and Massey2004; Hoxby and ferencei n ferri l i rr i ' J L,. :
Weingarth2005). Becausesome of the burning issuesof our day turn on rhe w onten i n c uurrtrr'- \ :: -

:-:r school districts have
- :,,tal segregation)when
- -.:ian schooldistricts.
'i: : :-.:-:.ocial context mattefs,
-- :, - - :. i o r e x am ple,s ur elyi s
-t ,.r -: ti'rhorhood.Note, criti-
: - - -: :ei g hbor hoodc ont e x t,
:-: -- : : family. To determine
- - I : i: , rt] i S nO t bet W eenpo o r
' , :: . i: : - :',or childr enwho gr o w
-i . :;hi l dr en who gr ew u p
., ::rrt children who grow
:.: :n life than they would
. :'l.rt claim has proved de-
r :r.1jorroadblock is selec-
:'. ,,rrdiscussionof causal
: ::r., n eighbor hoodsit, i s
:r-.: .lCtoSS neighborhoods
:ne \\-a,vthat people sort
.::: ro the example in the
- ,:: l dre r rin uns af eneig h -
..:.:: neighborhoods:That
.:iecr, but the effect of
::::3nces in parental com-
'l :c
l oor par ent st O S a C ri -
,.eferneighborhood for
: .(rntentiousin the social
,..::rherr ac ials egr ega ti o n
,.. for minority groups-
, .:rr krnds of har m . W i th
Massey(2004)
,. D.,uglas
-:. ieadsto poorer health
-: ro school segregation,
,:,ri :egregationon the ac-
,,:j .rnd Rothstein2006;
\1.:sser'
2004;Hoxby and
: .:uesof our day turn on
Rule .
The Severrth 2)7
context effects,there has been a reslrrgenceof interestlately in their estr-
mation. This interest is rvelcome. 'Wc must "let method be the servant"
when we fashion our methods to addressissuesof import, and there are
few issuesof more irnport in the social sciencestl'ranhow hunran behav-
ior is affectedby social conrexr.
Context Effects as Objects of Study
Let's begin by defining the term context effect. Social contexrs vary on
key social and economic characteristics,just like individuals do: Some
neighborhoodshave higher crime rates than other neighborhoods,some
schools are better funded than other schools, some cities have higher lev-
els of racial segregationthar-rother cities, and so on. \il/hen we say that
there is a context effect, u'hat we mean is that there are one or more char-
acteristicsof contexts that matter for some individual-leveloutcome y,
independent of tl-recharacteristicsof the individuals thernselves.For ex-
ample, there is a school context effect on high school complction if stu-
dentsin inner-cin-schoolsare more likely to quit high school thar-rsimilar
studentsin suburbanschools.
More formal l v,consi deri = 7,2,. . . ,N i ndi vi dual snestedi n j = 1 , 2, . . .
(for erarnple,neighborhoods,schools,cities).Let Y,,denotethe
/ contexts
value on some outcome variable for the ith individual in the 7th conrexr,
and X, denote the r-alueon some explanatory variable for the ith context
(for example, averagehouseholdincome in a neighborhood).A context
effect exists when X, hds an effect on Y,,independent of the indiuidual-
leuel cattsesof Y, includingX,,(if defined).As PeterBlau (1960, p.179)
put it, "The individual's orientation undoubtedlyinfluenceshis behavior;
the question is whether tlre prevalenceof social values in a communrty
a l s o e xerfssoci alcon:i rri i ntsupon patternsof conduct that are i n depcn-
dent of the influencesererted by the internalizedorientations."5
In a classicstudy of the U.S. military, Samuel Stouffer and his col-
leagues (Stouffer et al. 7949) found numerous instances where the
"prevalenceof values in a community" influencedthe attitudes and be-
haviors of American soldiersin \forld'$(ar II. They found, for example,
that inexperienced soldiers in vereran units were less likely to say that
they were ready for combat than inexperiencedsoldiersin inexperienced
't lt is important to note the diiference betu.eenthe effect of social context and the effect
of population compositirtn. ln the c:rseof con-rpositioneffects, differences betu,eengrgups
ow-eto differencesin charircteristicsof the individuals making up the groups. If fertility rarcs
are l-righerin country A than countrv B, for example, we would 6rst want ro knorv if the dif-
ference in fertilitv is due to difference in age composition: Perhapsa higher proportion of
the women in country A are of child-bearing age.
228 . Chapter7

units-an indication of the influence of the vererans (who believed the most other sar e poor ir J--
inexperienced soldiers were not ready for combat) on the inexperienced w here ut her sir r e lr ul l.
soldiers. el assi rr a r r r iJcllc-1.
. n.
Interest in context effects among social scientiststoday centers on the classneighborhood.
effect of peers, schools,and neighborhoods on life chances,especiallyfor I usea st u. l1 f lc, Lt.,-
the yonng (Borlas1995; Cutler and Glaeser1997;Hoxby and Weingarth and n study f r r t m . o. 'i" '
2005; Massey and Denton 1993; Mouw and Entwisle 2006). The intel-
F XAC T R BA TI N C C O \ TI \i' .
lectual forebear for much of this research is the so-called Coleman Re-
port on the stateof American educationin the 1960s.In one of the largest In addressingthe quesr:,::
and most influential social sciencestudies ever (the sample consisted of E dw ard G laeserllvu-
more than 150,000 students),JamesColeman and colleagues(1966) found raci alor et hnicgr or r p\- -
that school effects on student achievementare relatively small compared groups in question. "C':-..:
to the effects of family background, and that disadvantagedblack stu- w ri te (p. 827) , "especL- r - .
dents profit from schooling in racially integrated classrooms.The surpri- within a segregatedsr(r-.: :
sing finding that school effects are relatively unimportant prompted a S o the q uest ionof r r h. : : ', - -
school-effectsresearch literature (searchingfor such effects) that contin- ring to a raciailv sepirr.r;* -
ues today. The finding that black students may benefit from school rnte- i s an ernpir icalqt r . . r i,'r '. , ''
gration served as a catalyst for the implementation of busing to achieve U.S. cities,Cutler :rnd C'.,-.
racial integration. That conclusion also served to boost interest in re- sinceblacks in more Ssgi;i,
search on the social consequencesof residential and school segregarion of schooling,emplovnre;::..
(researchthat continues to flourish today). regatedcities.Becauseb....-
In thinkingrabout the effects of social context, it is important ro note these three outcomes C..: ,
that some ch/racteristics of contexts have a counterpart at the individual context (racial segreq.ru,:'.
level and some do not. A city, for example, can be characterizedby aver- l eveleff ectof heing l. l. r . .
age household income, or by the percentageof residentswho are white or
who are renters.'Wecan call theseaggregatiueproperties. Other character- .\IIT IGAT INC C O N TE . \ 1 ' L I : :

istics of contexts-integral or global properties, such as the jurisdictional JamesDavis's( 1966, st - . : .

boundaries in a city-are not formed by aggregatingindividual character- example of a mitie.rtin: , .
istics and have no obvious counterpart at the individual level (Selvin and encesthe sayingt lt r r tr r - ,
Hagstrom 1963). frog i n a big pond. \ \ c ; 1. r :
Context effects arising from aggregativeproperties can be classifiedas by rel at ivest andinqir r : . - ,
either exacerbating or mitigating. In the case of exacerbating effects, the l ari on" ( Davis | 966. F. - '
context effect and the individual-level effect have the same sign. In the such context, since ii-.*:--
caseof mitigating effects,the context effect and the individual-level effect school s,r egar dles of - - - - -
have different signs. curveha s r eal- lif ccor 'r-.: .
Some of the most intriguing issuesin social scienceinvolve the possi- to obtain highergr r lJt - . :
bility of exacerbating versusmitigating context effects.We know, for ex- reer decisiclns.
ample, that being rich confers certain advantageson individuals; but does D avis'scent r alar) Tr,.: : -
associatingwith others who are rich confer additional advantage?And, at {for exam ple.one s. r l': : - .
the other end of the spectrum, does disadvantagework in the same way, pl i ed w idely. f r or r . i; . . -
so that living in a poor neighborhood is harmful independentof the effect rel ati ve,not ab: olt r t . . : - : -
of one's own poverty? Or do the context effects instead mitigate the (w hat m at r er sis onc'. . ,
individual-level effects? Perhaps being poor in a neighborhood where our purposeshere.the;: : ,

: -: ,: -:,1ns(rvho believedthe
: - ^'".: on thc inexperienced
r :. ": ...:is roday centerson the
,rr - rr ihances,especiallyfor
- . :. . ---. H oxb . vand' W einga rth
.- - " .,rsl e2006) . T he inte l -
- : - : . '-^ rs.In one of t he lar g e s t
:-: : :r :1e SampleconSiStedof
-i- . , -: ( 1966) f ou n d
-i )l l eagues
- .rri rell s m all c om pare d
- - -'. -.i .l J\ent aged blac k s ru -
:: :- - : : :;l a ssro om s . T he s ur p ri -
: - . :'. rpor t ant pr om pt ed a
. . ! , ,L !^l:f^^r.
, ' ! ! rr/ r +l^
r,,at con tl n-
r: .-: :.nefit from school inte-
.. o f hr r s inot o ac hie v e
: . : : . : : :r) boos t int er es tin r e -
. j :- i. , . :nd sc hool s egr ega ti o n
-::'.:. lt is important to note
-
- : -::rnart
' ' _ r *_ _ at t he indiv idu a l
:. -.:.:-. :.- characterizedby aver-
i: : -..idents who are white or
:: , --.' :erties. Other character-
rr .: -, . ..r.h as t he jur is dic t io n a l
::- - - r ::n { i ndiv idualc har ac te r-
: : . -: - : i vi dual lev el( S elv inan d
: :: :crties can be classifiedas
-::: : rr:rcerbatingeffects,the
' :- - - ':.c the s am es ign. I n t h e
-. ::: :le individual-leveleffect
:. ...'. >rienceinvolve the possi-
-::'.-: \fle know, for ex-
-ifects.
: :-r::: on individuals;but does
:: :.j.::::, trttlladvantage? And, at
:r::.:-:-i!-$'ork in the sameway,
-r- :-. rndependentof the effect
:i.: :::;ctS instead mitigate the
',., r :
-r neighborhood where
Rule .
The Seventh 229
most others are poor is less alienating than being poor in a neighborhood
where others are not poor. Similarly, one might be better off to be n-riddle
classin a middle-classneighborhoodthan to be middlc classin an upper-
classneighborhood.
I use a study from economicsto illustrateexacerbatingcolltcxt cffccts
and a study from sociology to illustrate mitigating contcxt effects.
EXACERBATING CONTEXT EFFECT
In addressingthe question "Are ghettos good or bad?" David Cutler and
Edward Glaeser (1997) begin bv noting that the spatial separation of
racial or ethnic groups could have either positive or negativeeffects on the
groups in question. "Ghettos mav have benefits as well as costs," they
write (p. 827), "especiallyif thev allow for mixing acrossincome classes
within a segregatedgroup and for positive spilloverswithin that group."
So the questionofwhether ghettos(usedin the nonpejorativesense,refer-
ring to a racially separatedcommunin,) are good or bad for their residents
is an empirical question. On the basis of their analysisof censusdata for
U.S. cities,Cutler and Glaeserconcludethat segregationis bad for blacks,
since blacks in more segregatedcities are significantly worse off (in terms
of schooling, emplovment, and single parenthood) than blacks in lessseg-
regated cities. Becauseblacks are also worse off at the individual levei on
these three outcomes(Cutler and Glaeser,table IV), the effect of social
context (raciai segregation)in this instance exacerbatesthe individual-
leveleffect of being black.
M ITIGAT IN G C ON T EXT EF F EC T
JamesDavis's(1966) studv "The Campusas a Frog Pond" providesa nice
example of a mitigating context effect. The term "frog pond" here refer-
encesthe saying that it is better to be a big frog in a little pond than a big
frog in a big pond. We expect frog pond effects where "successis judged
by relativestandingin the social group, not by standingin the total popu-
lation" (Davis 7966, p.25). The collegecampus, Davis suggests,is one
such contert, since student grades tend to be about the same across
schools, regardlessof selectivity.This tendency of collegesto grade on a
curve has real-lifeconsequences becauseit implies that a studentwill tend
to obtain higher gradesin a lesscompetitive college, and gradesaffect ca-
reer decisions.
Davis's central concept-that what matters is one's reldtiue rank on X
(for example, one's ability relative to others on the campus)-has been ap-
plied widely, from accounting for revolution (that revolution is fueled by
relative, not absolute, deprivation) to explaining satisfactionwith income
(what matters is one's income relative to the income of one's peers). For
our purposeshere, the critical point is that X,,(the individual value) and X
 230 . Chapter7

(the value for the conrexr) have opposing effects. In Davis,s example, ca- \\-illlace.(In both surve\':. i - -
reer aspirationsare boosted by high academicability at the individual level .rad voted for'Wallacc.r
but lowered by high academicability among an individual's peer group. The naiveexplanationr: .
1968 election,rvhites\\ ci ' -
Context Effects as Nuisance heavi l ybl a ck dist r ict s.I n , ':
j or rol e i n t hat elect ion.. \ . : :
In the last section we observedthat, iust as individuals can be distin- lace,their presenceaffectr.: : :'
guished on the basisof their values on some attribute X, so too neighbor- .' -
the proper multilevelmocl.
hoods and other social conrexrs can be distinguished on the basis of the .Y,,(race of respondentli -- -:
prevalenceof that attribute among those in each context. Becauseit could rhe odds of voting for \\: ,-
be said that thesediffering propertiesof social conrextsput the "social" The problem is thitt \\'- ..-.--
in social research,determining the effect of social context is fundamental rhe individual-leveleffect :r-,
to social research.Increasingly,social researchis multilevel, neasuring both l eve ls.I n t he cir se( ) : i. ; .
properties of conrexrs as well as properties of individuals. In many in- ..'
that we cannot obserle t
stancesthe goal of the researchis to estimarecontext effects. amine data aggregate d br :-..
Unfortunately, data in the social sciencesdo not always permit us to postelectionsurvevs.For :-.'
separatecontext effects from individual-level effects. In those insrances sinceavailabledata vel\ L\:.:
context effects can be a nuisance. consider again rhe case of Governor time to poll individual'.
George'Wallace's bid for the presidencyin the 1968 U.S. election(chaprer Context effectsArc a n'.. :.'- - i
1). Becauseof Wallace'sreputation as a segregationist, we would nor ex- to estimatethem (irnd th-.. : . '.
pect ril/allaceto garner many votes among blacks. Yet, when we look at the most comfiton occur:.. ':
the vote in the 1968 election, we find that George Wallace received a inferencesabout indrtidl:,'. ^: - -
greater share of the vote in regions with higher percenragesof blacks. ti mesresear cherignors . : . '. . "
Across congressional districts in the South, for example, the correlation sions about individutl. :: :- .-
was r=0.55 betr,veenpercenr black and percent who voted for Wallace W i l l i am S. Robinson 1- i -- :
(Firebaugh 1978).'we see the same pattern acrosslarger regions in the
calleda fallacy because.i rr't, :' -
United States.\7allace's srrongestshowing by far was in the south, where shipsat the aggregatelir.. :: ::
blacks constitute nearly 25 percent of the population. In the East, where ci al conte xthasan el'lc, : .
blacks make up only about 10 percent of the population, \Tallace fared w e should avc, iddr . t , ': : - -
much worse, capturing a much smaller share of the vote (lessthan 6 per- aggregaterelationships.i:'i. , --
cent of the vote in the East versusmore than 20 percent in the South).6 show, relationship'.tr rlt= " - - -
The naive erplanation is that blacks must have been more inclined than posite directions.
other racial groups to vote for \fallace. This explanation is quickly ruled Aggregationbias is ,rr-.:-. :
out by postelection survey data. In one postelection survey, the 196g , t he Pr oblr : '
sci encesbut ::-
American National Election Study (ANES), we find that, of the g7 black and epidemiologl'-)u'h.':. - ,:
respondentswho said they voted, not a single respondent had voted for growing technical liter.r:,... -
\X/ailace.Among the other respondenrswho
vored, fully 1 in 8 said they solutionsto the probler:.
had voted for rwallace.The GSS resulrs for 1972 and 1973 confirm the
ANES findings: of the 237 black vorers, only 2 said they had voted for
C nrrrcer TEsr s r N Socl. . -
6These figures are based
on the 1972-73 General Social Sur'e1'. To complete the ac-
count, \(allace recervedless than 10 percent of the vote in the west and Midwest, u.here In this book I haverepe;:.:
blacks constitute roughly 8-9 percent of the total N in the 1972-73 CSS.
ods and m ult ipledat . r . . - '

-'' I-t.rvis'serample, ca-
. rt i h c indiv idunllev e l
,,
- rdual'speer group.
-. . :Juals can be distin-
--' -,.:- X, so too neighbor-
-. ,.'.;.1on the basisof the
- - - :-.ic\t.Becauseit could
-r\rS nr r rt he " s oc ia l "
,- - ,ntext is fundamental
. :l rrrlt ilev el.
m eas ur i n g
- - r rJuals .I n m any in -
. :\teffec t s .
- amine data aggregatedby precincts,or we can (in someinstances)rely on
- - . : .rin'ays permit us to
: : ' :.i 5 . l n t hos e ins t anc e s
- . - :'. rhe case of Governor
- . - : : :ri st.w e would not e x -
: , - - -. \et, when we look a t
: r:L- Wallace receiveda
-,, :ercentagesof blacks.
' i- rr:,inrple,the correlation
- - - - .,.ho voted for \fallace
: - : .. l .rrgerr egionsin t h e
'. - '.,..rs in the South,where
: . :.::(rnIn . t he E as t ,whe re
: - : . :'.l l .rtion, ' W allacfear e d
-: : ::'.- r'ote (lessthan 6 per-
- - :-:.ent in the South).6
- , : .-.cn rr.roreinclined than
- : . : ,.:rition is quickly ruled
' . . : :-i i ol l S ur v ey ,t he 196 8
- " I rl r:t. of r he 87 blac k
:: :..ttl d ent had v ot ed f o r
: ,:. fu l l r . 1 in 8 s aid t he y
- -- - -:nd 1973 confirm the
: ' - :,.-rdthey had voted for
: '. : r. T o c o m p l e t e t h e ac-
I rsr and Midrvest, where
- T C;S S .
Rule .
The Seventh 23I
Wallace. (In both surveys,11.3 percent of respondentsoverall said they
had voted for Wallace.)
The naive erplanation is off the mark becauseof context effects. In the
1968 election,whites were more likely to vote foI'Wallaceif t|ey livcd irr
heavily black districts.In other words, social context effectsplayed a ma-
jor role in that election.Although blacksthemselvesdid not vote for'Wal-
lace, their presenceaffected the propensity of wbites to do so. Thus, with
the proper multilevel model we would find opposing signs for the effect of
X,, (race of respondent is black) and X, (percent black for the region) on
the odds of voting for \(allace.
The problem is that we cannot always separatethe context effect from
the individual-level effect becausewe do not always have accessto data at
both levels.In the caseof elections,for example,the secretballot means
that we cannot observehow individuals actually vote; at best we can ex-
postelection survevs. For historical data we often are even more limited,
since available data very often are aggregated,and we cannot go back in
time to poll individuals.
Context effectsare a nuisancewhen we do not have the appropriatedata
to estimatethem (and thus must simply assumethat they do not exist). In
the most common occurrenceof this problem, a researcherwants to draw
inferencesabout individual behavior on the basis of aggregatedata. Some-
times researchersignore the possibility of context effectsand drarv conclu-
sions about individuals from aggregatedata anyway' committing what
rWilliam S. Robinson (1950) called the ecologicalfalldq'. That practiceis
called a fallacy becauseit makes the problematic assumption that relation-
ships at the aggregatelevelmirror relationshipsat the individual level.If so-
cial context has an effect, however,there is said to be aggregationbias, and
we should avoid drarving conclusions about individual-level effects from
aggregaterelationships. Indeed, as the 1968 presidential election results
show, relationshipsat the individual and aggregatelevelmay evenbe in op-
posite directions.
Aggregation bias is an issuein virtually all the social and behavioral
sciences,but the problem is particularly acute in fields (such as history
and epidemiology) q'here data often are not available for individuals. A
growing technical literature (for example, King 7997) continues to seek
solutionsto the problem.
CRrrrcer Trsrs rN Socrer Rnsr,eRcn
In this book I have repeatedlyemphasizedthe importance of multiple meth-
ods and multiple data sets in social research.In chapter 3, for example,
232 . Chapter7

I cited a number of examplesof multiple methods used as validity checks- scores rand ior r 'r r"
testing the plausibility of the claim that Bush lost 10,000 votes in the was Steepesttor n',r- ',',
Florida panhandle, using ethnographic data to interpret survey responses that i n 1993 black , : . .
on race, using foilow-up in-depth interviews to investigateabuse wirhin Qrral i 6 cat ior r Test t i .
farnilies.hr chapter4, I extolled the virtues of identical analysesof paral- scoredin t hc t t - , pi: , , . :
lel data sets. thirty yearsearlier.\,.^: - -
In this section I want to return to the reality check rule and the repli- the A F'Q T ear ned, . -
cation rule to make the point that progress in social research is often a the nrtionll r \ o , l- i
matter of progressivelyeliminating rival explanations by a seriesof criti- seoreslor hlrt.kr i: -
cal observations or tests. In some instances a single critical test might earni n gsf or hln. \ - : '-
suffice, as in the earlier example (chapter 5) of John Snow's classicinves- The quest ionoi, ' : .
tigation of how cholera is spread. More often, multiple causes are in- question of r','hat is ;,.
volved, and progress in sorting our those causesis often slow and halting. subj ectof a som er r : : : . .- ,
To illustrate the latter, consider the longstanding line of research on the first demonstrated::- :
black-white test score gap in U.S. schools.I begin with three observa- and sinceth.n .r.,.rp,,.
' -.
tions: (1) The gap is real. (2) The gap is importanr. (3) The causeof the starts.a s Jencks. 1r 'J
gap is the subject of a vigorous debate today in the social sciences.
First, the test score gap is real. Although the black-rvhite gap has nar- B y now r hiser '. - '. , -
rowed considerably in recent decades-the black-white reading gap was bl ac k- whit eg, . r p - .
cut by almost one-half, and the math gap by one-third, from 197I to The debat e
'r . r r r .
1996-the average black student in the United Statesstill scores below al l y pr ecipit et c.. : '
r,,i l ., -^-..t..^
ruil)/ rg)(r\fu.
r L,.-
r'1.. -.. :
three-fourths of white students in the United Stateson most srandardized
tests (Jencksand Phillips 1,998a,p. 1). On some tests the gap is even end t-rfrhe deb-rt. : .
greater, with more than half of black Americans scoring lower than 85 It ha pper r ed, ls. r : . . '
percent of white Americans. The black-white test score gap is no longer much can 11-gfqr, :i
disputed, since it is found in study after study (for example, Bali and Al- E du calionalR. : ; - . .
varez 2003; Hedges and Nowell 1998; National Center for Education
Most l ikelv t her c . r : . -
Statistics2007). The gap appearsearly, even before children enter kinder-
senseth e t esrgJp f i: 1. '-
garten (Fryer and Levitt 2004; Wesr, Denton, and Reaney 2001), and it
as note d above.BLr :: : .
persists into adulthood. There are similar, though typically smaller, resr
cases:Tr y t o f ind ir ': - '
score gaps between white students and Latino students.For our purposes
That i s r he appr or r eh: . - .
here, it sufficesto focus on the black-white gap.
possrb leexplanlt ion- '
The black-white test score gap is enormously important because "re-
ducing the black-white test score gap would do more to move America to- o On e hypot hesi.r . :
ward racial equality than any politically plausible alrernative" (Jencksand gap. If so, then r.r.
Phillips 1998b, p. 51). Scholarssuch as ChristopherJencksand Meredith tively constairr , .
Phillips believe that is the case becauseof pretty strong evidencethat, in ".
scoresfor bl,rck s:-.
today's world, reducing cognitive inequality would go a long way toward raisedin u,hite f;::'
reducing economic inequality. On the basis of recent data on test scores
and earnings from the National Longitudinal Survey of Youth (NLSY), -A mo np
\r l rdel r. \\ -
for example, we find that the black-white earnings ratio for men (black
than rvl t i te \tudent\ t' , r-:-
earnings as a fraction of white earnings for men in their thirties) increased poi nr our . equ:rl i z i ng. rt.t .- -
for men with lower-than-averagetest scores, for men with average test ment as well.

.: Lrsed as validity checks-
.(.)st10,000 votes in the
-rerpretsurveyreSponses
investigateabusewithin
. .:rrri ci lanaly s esoI par a l -
;heck rule and the repli-
: ,cial researchis often a
,::''rrsb v a s er iesof c r it i -
-.:rqle critical test might
,rn Snow'sclassicinves,
. nrLrltiplecausesare in-
. ,rtfen slow and halting.
- ilne of researchon the
r:,n with three observa-
. ...:.r3 ) T he c aus eof t h e
- .r;k-rvhite gap has nar-
. -..'.l.rrtereading gap was
:-:-thi rd, f r om 1, 971t o
::-rtc-sstill scoresbelow
::: ()n most standardized
:-i tests the gap is even
. .coring lower than 85
-. - .!ore g ap is n o lo ng er
: : cxample,Bali and Al-
,.C.enterfor Education
::children enter kinder-
"-
RcJnev200 | t , and i t
::l rr pically smaller,test
,.:.nts. For our purposes
)rt ant be cau se " re-
: : ' . t a{
-: t ' ) ntove AmeriCa tO-
.: :ernative" (Jencks and
:: lcn' ks a nd Mere dirh
i::ong evidence that, in
.- ;r1 .r long way toward
r,.-:rr data on test scores
- '.:r' of Youth (NLSY),
'-) r.rrio for men (black
::err rhirties)increased
:::rn $'ith average test
The SeventhRule . 233
scores) and for men with higher-than-averagetest scores, The ir-rcrease
was steepestfor men with higher-than-averagetest scoresrhowev.r, ..,
that in 1993 black men who scoredin the top half of the Armed Forces
Qualification Test (AFQT) earned 96 percent as much as white men who
scoredin the top half of the test. This is in sharp conrrast to the results
thirty years earlier, when black men scoring abovethe national averagcor1
the AFQT earned only 65 percent as much as white men scoring above
the national averageon the AFQT. In today's world, then, equalizing test
scores for blacks and rvhites should go a long way toward equalizing
earningsfor blacks and whites in the United States.T
The question of hou to reduce the test score gap of courseturns on the
question of what is c:tusing the gap. The cause of the gap, in turn, is the
subjectof a sometimesheateddebatein the social sciences.The gap was
first demonstrated in the testing of U.S. Army recruits for \World 'S7arI,
and since then empirical researchon the subject has proceededin fits and
starts,as Jencksand Phillips (1998a, p. vii) explain:
By now this cycle has becomepredictable.First someoneclaims that the
black-white gap is largely innate. Then skeptics challenge the claim.
The debate starts off largely unencumberedby evidence,bur it eventu-
ally precipitatesa number of useful empirical studies.The issuers not
fully resolved,but the number of plausible explanationsis smaller at the
end of the debatethan at the beginning.This happenedin the 1920s.
It happened again in the 1970s, after Arthur Jensenpublished "How
much can we boost IQ and scholastic achievement?" in the Haruard
EdwcationalReuieu,in 1969.It seemsto be happeningagain.
Most likely there are multiple causes of the test score gap, and in that
sensethe test gap puzzle differs from the puzzle of how cholera is spread,
as noted above. But the methodological principles are the same in both
cases:Try to find critical testsof the predictionsof various erplanations.
That is the approachthar has beenused with partial successin evaluating
possibleexplanationsior the test scoregap:
e One hypothesis is that genetic differences account for the test score
gap. If so, then racial differencesin rest scoresshould remain rela-
tively constant over time, and there should be no difference in test
scoresfor black studentsraisedin black families and black studenrs
raisedin white families.But neither is true. As noted earlier.the test
: Among students with the same t\\'elfrh-gradetesr
scores,black students are mctre likely
th a n r vhi te studentsto gradr:atefrom col l ege.Thus, asJencks and Phi l l i ps (1998 b, p.46)
point out, equalizing test scoresshould rcclr-rce black-white disparities in educational attail-
m e n t a s w el l .
l3+ . L - h a n te r /

score gap has declined significantly in the United Statesin recenr de-
cades.Moreover, the preadolescenttest scoresof blacks are much
higher for blacksraisedin white families (Jencksand Phillips 1998a,
p.3), which is scarcelyconsistentwith the genericargumenr.Vir:tu-
ally all scholarsnow agreethat the causeslie elsewhere.
. Another hypothesislocatesthe causein the testsrhemselves: perhaps
the tests are unreliable and biased against some groups. If so, then
test scoresshould have little or no predictive ability of educational
and occupational attainment. But test scores do predict later life
chancesfor all racial groups, so for the most part researchershave
turned elsewherefor explanations of the gap.
e Another explanation focuseson differencesin school resources,as
rneasured,for example, by spendingper pupil. That explanation was
more compelling during the "separatebut equal" days of racial seg-
regation,where schooisfor whites and blacks may have been sepa-
rate but certainly were not equal.The explanationis lesscompelling
today becauseper pupil spending(someglaring exceptionsnotwith-
standing) on averageis about the same for whites and blacks (Jencks
and Phillips 1998a, p. 9). Equal spendingdoesnot necessarilyrrans-
late into equal educational quality, however, so the test score gap
may be due in large part to differencesin the way money is spent-
an important topic for further research.
. Other explanations focus on cultural differences. If cultural differ-
encesplay a significant role, then black children raised in white fam-
ilies should score higher than black children raised in black families
(as they do). Critical observationssuch as this strengthenthe casefor
the role of culture, but they do not tell us what cultural differences
matter. A particulariy influential cultural explanation, based largely
on ethnographic studies of minority students in schools (Fordham
and Ogbu 1986; Ogbu 2003), argues that black adolescentsare
characterizedby an "oppositional culture" that hinders their aca-
demic achievement.For example, in his classroom observations of
studentsin ShakerHeights, Ohio (an affluent suburb of Cleveland),
John Ogbu (2003) found that black students were more likely that
white students to arrive late to class, come without materials, and
disrupt lessons. Whatever the merit of the controversial claim that
black adolescentsexhibit an oppositional culture that undermrnes
their academic achievement(Ainsworth-Darnell and Downey 1998;
P. L. Carter 2005; Farkas, Lleras, and Maczuga 2002; Fryer and
Torelli 2005; Mickelson 1990; Tyson, Darity, and Castellino2005),
differences in adolescentculture alone cannot account for the resr
score gap since, as noted earlier, the gap appearswell before adoles-
cence.

..' :..; L'r.rited Statesin recentde-
-:: - . : . ; t tres of blac k s ar e mu c h
-- -. Jcncksand PhilliPs 1998a,
, .:.-.: -.- !],eneticargtlment. Virtu-
: - : . - r: l i e e l sewher e'
, -: :.r. teststhemselves: PerhaPs
: .-,-, ::>t some groups. If so, then
:-: : rrf i ve a bilit y of educ at i o n a l
r r : : . i i o r€s do pr edic t lat e r l i fe
' : :..: :ttost part fesearchershave
_ _ --
.,-.r ''1n
F!!r,
-. ., j: : . : c e s i n s c hool f es our ce sa, s
'- ^ ' :rrpi l . Thar ex planlt ion w a s
.. ,-.:: r-rt cqual" days of racial seg-
- ::.- :.lircksmay have been sepa-
-. :\Fli.tflxtionis lesscompelling
- tutes for careful theorizing and conceptualization.An irnportant theme of
, .:'.. !lllring exceptionsnotwith-
--::r; :,,r s'hites and blacks (Jencks
:: -,l..-.:doesnot necessarilytrans-
,' . .--\\\eler, so the test score gap
:--:: itLthe way money ls spent-
:"1:- --.
-:r ;ifierences.If cultural differ-
r -:,. ,hildren raisedin white fam-
-. ,.- l.lren raisedin black families
:,-. -..ls this strengthenthe casefor
: :;., us n'hat cultural differences
,.. :-::.rl erplanation, basedlargely
: :'. )iurlents in schools (Fordham
.:..:..;! rhat black adolescentsare
.. , , . , r u re " th at hinder st heir a c a -
. :- :ris classroomobservationsof
,::- -rftluentsuburb of Cleveland),
:-.\ ):'.ldentswere more likelY that
-.-:::. Cortl€without materials,and
.: : ,l the controversialclaim that
r ::-rrnol culture that undermines
:.'. :lir-Darnell and Downey 1998;
,.. ...:rd-\,{aczuga2002; Fryer and
. . :-..Dirrity, and Castellino2005)'
,r , ,:rc carnnotaccount for the test
::i i.lp .lppearswell before adoles-
.. "- . - . , ". t . * t t
The SeventhRule . 23,5
This brief summary scarcelydoes iustice to the richnessof the literature
on the test scoregap. My purpose' however,is 1ot to provide an overview
of researchon this issue,but to illustratehow social researchcrslook for
critical observations or tests ttl evaluate rival explanations, peeling off
various explanationsone by one as they fail to predict accuratelyacrossa
varietl'of settings.
C o NcrusIoN
Even a cursory comparison of empirical work in the social scienceiotlr
nals today with published work fifty vears ago will show how far we have
come in terms of the sophistication of our estimation methods. For the
most part, the greater sophisticationis good. There is the danger, how
ever,that researchers will come to rely on estimationmethods as substi-
this chapter is that we cannot collnt on technicaltricks to remedy defec-
tive theory or researchdesign.
Another danger posed by increasingstatistical sophistication is that the
tools of researchwill overshadowthe substanceof research."Let method
be the servant, not the master" is meant to be a poilted remilder that the
researchproblem should dictate the choice of the tool, not the other rvar-
around. Statisticalmetl-rodsshould not be the motivators of our research.
Nor should statistical methods be an afterthought: Data design and es-
timation method go hand in hand. Sometimesresearchersdo not think
much about estimation until the data have been collected. That is r iris
mistake because,generailyspeaking,estimationtechniquesare ill-suitecl
for overcominginadequaciesin researchdesign.Researchdesignis rtL'out
collectingthe sort of data that enablesyou to estimatethe right rl-t1t1q. \t-r-
tistical estimationis about providing reliableand preciseestimates. To erl-
sure reliableand preciseestimatesof the right thing, we must not tlir i'r.e
researchdesignand statisticalestimation.
The secretsof the social world are not easily revealed,so \\'e ll.-d to
find all the leversand apply all the leveragewe can. To advanc.-in ihe stl-
cial sciences, we need to hone our skills in the use of natur,rl .\perllllcllts
and decompositionsand critical observationsto enticethe sclcL.rl g 61lJ 1e
revealits secrets.Regressionanalysis-the workhorse l.nethodin :i.l; st,ci.rl
sciencesfor well more than a quarter century-should not L'ed:.;.rr.i"d.
but used more thoughtfully,and appliedstrategicalli'inr,,r.rtLrn.i:,,n u'ith
other approaches.Above all, what we need in sociirl res.r:;i.t tt.,.l.rvis
more imaeination.
A P P E NDI X v106 is cocle. l1= , ,
uhaugcLlt ssc. ' '. . ,t
\'arra[)-tc v ,/ U + - . . '
StudentExerciseson Rule 7 dr r t e, 2=r ot cJ .-.
nrr tv r.rtrdi ,l .rr,'
var iablecalled t hr r ; r . :
,, , l -: , -,g-V*l
1,..,,,,
d lllll l\ !.'..Y
-.

D em oer ; t , \ . ',
The lessonto be learned in this exerciseis that, due to contexrual effects, vot cd f ur . r t lr r : ; - : . '
relationships observed at the aggregatelevel do not necessarilyreflecr re- el ect ionslr or l I - : '
lationships at the individual level. In fact, as this exercise demonsrrares, for Georgc \\ .r -,
individual-level and aggregate-levelrelationship can even be in opposite fi lt er " t o i: t r l. r : :: - -
directions. e The ner t , r cl .
The more general point I wish to make, in line with rule 7. is that bl ack. Ent er r { 1or .
rnethod without substancecan be dangerous. In the 1968 election, re- ti o n. O b: cr r r : . ".
gions with a higher percenrageof blacks tended to vore at higher rates for no w \ O t e 1, , ' t '.
George Wallace. I7ithout an undersrandingof the issuesof that day, yt_ru l ^ . ^, - . , . ; . , 1, . . .
might be tempted to interpret that positive associarion to indicate that ri onsas \ oLr\ : - . - -
'Wallace.
blacks voted disproportionately for Yer, as you will see, black vo t er sin t lt . - . : " : . .
votes for'$Tallacewere extremely rare.
'
The routine should be familiar by now. Go to the web site http://sda Question l. \ \ cr '- -- .
.berkeley.eduand follow the instructions below. The instructions should GeorgeWall. r c. l. i :-. ..
enable you to generateall the staristicsyou need to complete the assign- o Th e t hir d . r - r - -
ments. Variable names are in boldface.
gion ot - , , . : "- - - ' .
Note: If you did the prior exercisesyou should be able to do the as-
Again selc. r '\
signment here. To avoid redundancy, the instructions below include only
toisolat elhc: : - -
the key commands. Recall that the formar in the SDA web site mieht have
pe r cent egehl. : . . '
changed somewhat since these directions were written, ,o ,o--. of the
Cent ages t o 'r ; . - , 1-
specific directions below might be out of date. You nonethelessshould be r Recodevlll r
able to follow the logic of the instructions to obtain the tables you need to
oh r ainedin r hr : '
complete the assignments.
i s coded "1" ir r 111
since bhck: . , i: . . . .
Northeast ir.rthc - .--'
AssrcNueNT 1: Groncs \X/errecp AND THE Brecr Vorn rN mu 1968 o Th
"'_
e f innl
"
sr'_teni.'
' -
Pnssrolxrrar ErpcrroN-Rpsurrs FRoM THEANES regionblack,1. , , 1. , : . '
1) as "Select iori-. r: . .
The aim is to compare the individual-level versus regional-level associa- rvei.htino n t -rrr R -. .

tion between race and vote for George rwallace in the 1968 presidential 1, 027,t he sAm c: -. : ''' -
election. From the SDA Berkeley web site, choose the 1948-2000 cumu- black and thirdpartr i
lative ANES data set. There are five basic steps:
Quest ion 2. ( a) \ or c : : ': -
r First recode the ANES race variable v106 into a variable called si gni ficant .O n r he l. . r - -
black, coded 1 for blacks and 0 for whites (everyoneelse).Currently, compa r e t he aggr es. r r ., '
 Rule ,
The Sever.rth 237

v 106 i s coded 1= w hi te,2= bl ack, and 3= other, and so you n eedt o

change these codes to 0 and 1. You also need to recode the voturg
variablev704, currently coded 1=voted for Democraticparty candi-
date, 2=voted for Republicanparty candidate,3=voted for third-
party candidate. Wallace was a third-party candidate,so create a
variable called thirdpartyvote by recoding 1 for those who voted for
a third-party candidate and 0 for those who voted Republican or
Democrat. You u'ill find that only 3.6 percent of voters say they
- - : . - , ::,) aont ex t ualef f ec t s , voted for a third-partv candidate. This figure, however, includes all
: j : l..essarily reflect re- electionsfrom 1948 through 2000. The percentageis much higher
:: :- - :r\;raise demonstrates, for George Y/allace (as vou will see later, when you use "Selecrion
- . - . -..' .\cn be in oppos ir e filter" to isolatethe 1968 election).
o The next step is to cross-tabulatethirdpartyvote (row variable) with
,:. -.: u'ith rule 7, is that black. Enter v4(1968) as "Selectionfilter" to isolatethe 1968 elec-
.
: :. l 9o8 elec t ion.r e - tion. Obsen'e that, bv isolating the 1968 election, thirdpartyvote is
,rr .rr higherr ar esf or now vote for George'Wallace.Choose "No weight" from the pul[-
r r r-: .)i ri es of t hat day , y ou down weighting menu, and otherwise choose the same default op-
: :-- -.-...ionto indicate that tions as you did in prior assignments.You should find, of the 87 black
- : : j: .,:i vo u will s ee,blac k voters in the sample,nctt a single one voted for Georgetil/allace.

- r :r. \\'eb site http://sda Question 1. 'ff/ereblacks more likelv or less likely than others to vote for
George'S7allace? statisticallysignificant?Is it ver,vstrong?
Is the associ:rtior.r
:tr - --: instructions should
. . , -^ t - , -
- -: _ ! ( 'lllPl s r s .Uht,c a s s l g n - . The third step is to cross-tabulateblack (rorv variable)br.v112 (re-
gion of countr\'-Northeast, North Central, South, and \West).
, : ,- i rc able to do the as- Again select"No u"eight" and insert v4(1968) in the selecrionfilrer,
-.: : - -: :s b elow inc ludeonly
to isolate the vear 1968. This cross-tabulationis neededto obtain
- : -: : --r\ u'eb site might have
percentageblack in each of the four regions.We will use those per-
.i ::: .' : : if en, SO SOme Of the
centagesto createa ne$.variableregionblack.
.::: : -. lonethelessshould be o Recode v'1.12to create regionblack by inserting the percentagesyou
-:.-..:':he tablesyou needto
obtainedin the prior step.To illustrate:The Northeast region, which
i s coded " 1" i n v112, shoul d be recodedas " 6.8" i n regi onblack,
since blacks constituted 6.8 percent of the respondentsfrom the
Northeast in the 1968 ANES.
. : - ,. \ \'olr. Ix l:g't 1,968 . The final step is to cross-tabulatethirdpartyvote (row variable) with
' .1 . - . r\ES regionblack.Isolatethe 1968 electionby enteringv4(1968\,black(0-
1) as "Selectionfilter," and choose"No weight" from the pull-down
: -- . .. :: j i onal- lev elas s oc ia - weighting menu. Run the table. The number of casesshould be
r r - : - :h e 1 968 pr es ident ia l 1.027. the samenumber as in the individual-levelcross-tabulationof
-- :: ::r 1948-2000 cumu- black and thirdpartyvote.

Question 2. (a) Note that the aggregate-levelassociation is statistically

' r1 c r!o a v ar iable c alled significant.On the basis of the values for R (a measureof association),
::- :'. ::','Oneelse).Currently, compare the aggregate-ievelassociation of race (blacks/nonblacks) and
238 . Chaprer7

vote for George Nilallace with the individual-level association you ob- . Do the cross-talrul-..:l
tained previously. and aggregate-lercl . ,.
(b) The context effect here is mitigating-expiain what that means, sub- kl.-L
ur4rN/
\ vv rrrr
'.-ith
t-^ra
v vL! t,
t ,-
-1 '

stantively,in this case.

(c) Give one or two reasonswhy the vore of whites would be affected by Question 3. (a) Comp.:1r. : l
'Whar
the racial context of their region. That is, why would whites in a more from the ANES. .r:.
heavily black region be more likely to vore for a segregarionistcandidate of R for the GSSdata? F, : :
j
than would whites in a lessheavily black region? {b) C om par er he valuc- ' :
discrepancyroughli' thr ..'.'-'.
norahl y lar gerin 0r r oi '

AssrcNuENr 2: GroncE n7erlecs ANDTHEBrecx Vorr rN rne 196g

Pnr,sropNlrar-ELECTToN-REsurrsrnou rHs GSS A ssrcxllr , Nr 3. Exr r r lL. - - :

The second assignmentis to examine George \il/allaceand the black vote Tabl e 1 in Angr ist r r ndK. , . . -
again, this time using GSS data insteadof data from the ANES. Because make use of natr,rralrrtiY:
you can use the instructions for the ANES as your guide, the instructions ies, describe (in onc or" :',,
below will be abbreviated. used.
o The 1958 election variableis called pres68 in the GSS,coded 1 for
vote for Hubert Humphrey (the Democratic candidate), 2 for vote
for Richard Nixon (the Republican), 3 for vote for George rJfallace,
and 4 for vote for someoneelse. You should recode this variable, as
before, as 1 for vote for \il/allaceand 0 otherwise. You can call this
new variable wallacevote.You should find that 1 1.5 percent voted for
Vallace, and 88.5 percent voted for Humphrey, Nixon, or some
fourth-party candidate.
. Similarly, recode the GSS race variable, cailed race, ro creare a new
variable black that is coded 1 for black and 0 for others. (you can see
the codesfor race by entering race in the "selected" box and clicking
on "View.") Of the 46,510 respondentsin the 1972-2004 cumula-
tive GSS,you will find that 6,399, or 13.8 percenr,are black. (Note
that this percentageis likely to be somewhat different in your subse-
quent tables that include pres68, since the use of pres68 restricts the
analysisto the 1968 election.)
. The GSS region has nine categories:1-2=Northeast; 3-4=North
Central; 5-7=South; 8-9='West. Use the recodecommand to creare
a new variable region4_gss that parallels the four-category region
variablein the ANES.
r Cross-tabulate black (row variable) with region4_gss (the GSS re-
gion variable)to obtain the percentageblack in each region. Insert
year (1972-1973) as "Selecrion filter."
. Create regionblack from the percentagesyou obtained in the cross-
tabulation of black with region4_gss.
 The SeventhRuie . 239

r Do the cross-tabulationsnecessaryto examine the individual-level

and aggregate-level(four regions) associationof racc (black/non-
black) with vote for Wallace.

Question 3. (a) Compare the GSS results with the results you obtained
from the ANES. N7hat are the individual-level and aggregatelevel values
of R for the GSSdata? For the ANES data?
(b) Compare the valuesof R: Is the individual-levelversusaggregate-level
discrepancyroughly the same size in both data sets, or is the discrepancy
notably larger in one of the data sets?

AssrcNusNr 3. ExelrprEs oF NATURALExprRrltENrs

Table 1 in Angrist and Krueger (2001)lists fifteen economic studiesthat

make use of natural experiments. From your reading of one of the stud-
ies, describe (in one or two paragraphs)the natural experiment that is
u se d.
References

Abbott, Andrew. 2003. Methocls ctf Discc,n,ery:Heuristics for the Social Scient:es.
New York: W.'$7. Norton.
Aghion, Philippe,Eve Caroli, and Cecilia Garcia-Peiralosa. 1999. "lnequality and
economic growth: The perspectiveof the new growth theories." .lournal of Eco
nomic Literdture 37 :7615-50.
Agresti, Alan, and Barbara Finla1,.1,997. Statistical Methods for tlte Social Sci-
ences.Third edition. Upper SaddleRiver, NJ: PrenticeHall.
Ainsworth-Darnell, James \7., and Douglas B. Downey. 1998. "Assessing rhe
oppositional culture explanation for racial/ethnic differencesin school perfor-
mance." American SociologicalReuiew 63:536-53.
Allison, Paul. 1978. "Measures of inequality." Anterican SociologicalReuiew
43 : 8 6 5 - 8 0 .
1990. "Change scoresas dependentvariablesin regressionanalysis."Pp.
93-174 in SctciologicalMethodology 1990, ed. Clifford C. Clogg. Orford:
Basil Blackwell.
2002. Missing Ddtd. SageUniversity Paper Serieson Quantitative Appli-
cationsin the SocialSciences,07-136. ThousandOaks, CA: Sage.
2005. Fixed Effects Regression Methods for Longitudinal Datd Using
SAS. Car.v,NC: SAS Press.
Alwin, Duane F. 1996. "Coresidence beliefs in American society-1973 to
1997." Journal of Marriage and tbe Family 58:393-403.
Alwin, Duane F., and Robert M. Hauser. 1975. "The decompositionof effectsin
path analysis." American Sociological Reuiew 40:37-47.
Amato, Paul R., and Joar.rG. Gilbreth. 1999. "Nonresident fathersand children's
well-being: A meta-analvsis."Journal of Marriage and the Family 61:557-73.
Amsden, Alice H. 2003. "Good-by dependencytheory, hello dependencytheory."
Studies in Comparati t'e I tttentational D euelopment 38 :32-38.
Angrist,JoshuaD., and \\'illiam N. Evans.1998. "Children and their parents'la-
bor supply: Evidencefrom exogenousvariation in family size." Americdn Eco-
nomic Reuiew 88:450-77.
Angrist, Joshua D., and Alan B. Krueger. 2001. "lnstrumental variablesand the
search for identification: Frorn supply and demand to natural erperiments."
Journal of Economic Perspectit,es 15:69-85.
Arrighi, Giovanni, Bevellv J. Silver, and Benjamin D. Brewer. 2003. "Industrial
convergence,globalization, and the persistenceof the North-South divide."
Studies in Compdratit'e Irtterrtational Deuelopment 38:3-31.
Bali. Valentina A.. and R. \.lichael Alvarez.2003. "Schoolsand educationalout-
comes: What causesthe 'race gap' in student test scores?"Social ScienceQuar-
terly 84:485-507.
Becker, S. O., and A. Ichino. 2002. "Estimation of averagetreatment effects
basedon propensityscores."STATA Journal 2:358-77.
242 . References

Benabou, Roland. 1996. "Inequality and growth." NBER Macrrtecottotnics C a r d , D n vi d . ,r n d T.- - - -

Ann ua l 11 :1I - 74. t e \ t \L o |e g a l '. (
Bennett,Stephen,:rnd Linda Bennett.1987. "Political participation." Annnal Re- N B t R Wu r [i r ,,- l ' ,
uiew of Political Science2:157-204. Curter. .1, Sl c,tr. l " l
Benson,Rodney, and Abigail C. Saguy.200-5."Constructing social problems in c o mp u r i so nu l tl i t:;:: ..
an age of gi<lbalization:A Frcnch-American comparison." Anterican Sociologi- j o u ma l ) .
cal Reuietu70:233-59. C a r t e r , Pn r d o r c.'l .l
tslake,Judith. 1989. Family Sizeand Achieuement.Los Angeles:University of Wl i /t'. N cu \, 'r . ,
California Press. C h r r l e s, C n m i l l . ,/ .
Blau, Peter M. 1960. "Structural effects." American SociologicalReuiew 25: c ( ) n l i n u i n g !o r l ..- i - .
1 7B-9 3. p e r fo r r n ,r r r u c.' ) ....
Blinder, Alan S. 1973. "\7age discrimir.ration:Reduced form and structural vari- C h e r l in . Attd r cu . I .
ables."Journal of Human Resources8:436-55. " T h e i n fl u cn i ( - :'
Bollen, Kenneth, and Robert \(. Jackman. 199-5."Income inequalit,vand demo- A m e r i ca n So .r ' . - .
cratization revisited: Comment on Muller." American Sociological Reuieu C l o g g . C l i ffo r J t .. ' .
6 0:9 83 -89 . ing socialstarirlr.:: :-
Borjas, Ceorge. 1995. "Ethnicity, neighborhoods,ancl human capital externali- C t 'l e m a n . f.tr r r c.i . :'
ties." American Economic Reuiew 85:365-90. A l e xa n Jr r \1 . \1
Bourguignon,Francois,and Christian Morrisson. 2002. "Inequalitv among world i t 1 'oI E,l u ,,ttt 't- .
citizens: 1820-1992." Anterican Economic Reuietu 92:727-44. Eclucation.
Bowles, Sanruei,and Herbert Gintis. 1976. Schoctlingin Capitdlist Americd: Edu- C c , l l i cr , D a r i J. H . - :
ccttional Refonn and the Contradictions of Economic Llle. Ner.v York: Basic l e v e r a g ei tt.ttt..t '.
Books. C h ap te r l i i n R .:, :':-
Bradatan, Cristina, and Glenn Firebaugh. 2007. "History., population policies, e d . H e r tr r E. Br .r - . .. .
and fertility decline in Eastern Europe: A case study." Journal ctf Family His- field.
to ry 3 2:1 79 - 92. C o n l e y.D a l to r r .l l 'l - r .
Brady, Henry E. 2004. "Data-set versus causal-processobservations: The 2000 L o n l e t. D a l to u . a n J R - . ...
U.S. Presidentialelection." Pp.267-77 in Rethinking Social Inquiry: Diuerse a n d ch i l d r e n ': Jr .t.i :
Tools, Shared Standards,ed. Henry E. Brady and David Collier. Lanham, MD: I r i r th o r d e r i r o n r .r - .
Rowman and Littlefield. 11,302.
Brady, Henry E., and David Collier, eds. 2004. Rethinking Social Inquiry: Di- C o o p er . H a r r i s. i r n d L ::-
uerseTools, Shared Standards. Lanham, MD: Rowman and Littlefield. S y nth csi s. N e u \i 'r k: r
Brewster, Karin L., and Irene Padavic. 2000. "Change in gender-ideology, C u t l e r , D a vi d \1 ., .r n i : -
1977-1996: The contributions of inrracohorr change and population turnover." -
Q u a r l e r l l ' Jo u r r ttl , : .
Journal of Marriage and Family 62:477-87. l ) a v r '. Ja n r e sA. l q b r - '. ' ;
Brown, B. W., Jr. 1972. "Statistics,scientificmerhod, and smoking." Pp.40-51 o f r e l a ti ved e p r i va r i . r :
tn Statistics: A Guide to the Unknotun, ed. judith M. Tanur, Frederick Sociology72:1,7-3L
Mosteller, \7illiam H. Kruskal, Richard F. Link, Richard S. Pieters,and Gerald D e a t o n , An g u s. 2 0 0 - 5 . "\1 .
R. Rising. San Francisco:Holden-Day. growth in a poor ri orl.: .
Bryk, Anthony, and Stepher"rRaudenbush. 1992. Hierarchical Linear Models: Diekmann, Andreas, ar-r.if i:
Applications and Data Anall,sis Methods. Thousand Oaks, CA: Sage. d i v or ce :Effe ctso f p .r r .: .
Butterfield,sherri-Ann P.2004. " ''we're lust black': The racial and ethnic idenrities lugical Reuiew 64:-5 i-- :
of second-generation\WestIndians in New York." Pp. 288-312 in Becoming D i P r e te ,Th o m a s A.. .r r ,i .
New Yorkers:Ethnographiesof the New SecondGeneration, ed. philip Kasinirz, t r e nd sw i th r e p e r te di r - -
John H. Mollenkopf, and Mary C. \(/aters.New York: RussellSageFoundarion. o l o gy 1 9 9 0 ,e d . C l i r i ,'r ;
Campbell, Donald T., and Julian C. Stanley. 1.966. Experimental and Quasi- 1 9 9 0 b . "Str u ctu r ..:
Experimental Designsfctr Research.Bosron: Houghton-Mifflin. men and women." Ant;' : -
 References . 243

Card, David, and JesseRothstein.2006. "Racial segregationand thc black-white

test score gap." Can-rbridge,MA: National Bureau of Economic Research.
NBER \ilorking Paper no. 12078.
Carter, J. Scott. 2005. "Reassessingthe effect of urbanism and regionalism:A
comparison of different indicators of racial tolerance." Sociatkn Toda1,J (wgl
journal).
Carter, PrudenceL. 2005. Keepin' It Rettl: School SttccessBeyond Black dnd
W/:i/e. Neu' \brk: Orford Universitl' Press.
Charles, Camille 2., Gniesha Dir.rrviddie,and Douglas S. Massey. 2004. "The
continuing consequencesof segregation:Family stressand college academic
performance." Soci,tlScienceQuarterly 85:1353-73.
Cherlin, Andreu', Linda M. Burton, Tera R. Hurt, and Diane M. Purvin. 2004.
"The influenceof phvsical and serual abuse on marriage and cohabitation."
American SociologicalReuieu,69:768-89.
Clogg, Clifford C., and Aref N. Dajani. -|991. "sources of uncerraintyin model-
ing social statistics:An inventory." Journal of Official Statistics7:7-24.
Coleman, James S.. Ernest Q. Campbell, Carol J. Hobson, JarnesMcPartland,
Alexander NI. l.lood, FredericD. Weinfeld, and Robert L. York, 1,966.Equal,
ity of Educational Opportu zitl'. \Washington,DC: United StatesDepartrnent of
Education.
Collier, David, Henrv L,. Brady, ar-rdJason Seawright. 2004. "Sources of
leveragein causal inference: Toward an alternative view of methodology."
Chapter 1,3in Rethinking Social lnqr.tiry: Diuerse Tools, Shared Standards,
, -r Herr.'F. Brad v a nd Dav id Collier . Lanham . NI D : R o u , m a n a n d L i t t l c

:\ Dalton. 2004. The Pecl<ins(Jrder. New York: Pantheon Books.

: ;r. Da lton , a nd Reb ec c aG lauber . 2005. "Par ent al e d u c a t i o n a li n v e s t m e n t
.::.: children's academic risk: Estimates of the impact of sibship size and
- -:rr .rder from exogenousvariation in fertility." NBER Working Paper no.
. . t r) 1.
:-. H.rrris, and Larry V. Hedges, eds. 1994. The Handbr.tok of Research
'' ,;1/.'3-sl-s.
Nerv York: RussellSageFoundation.
::. David -VI.,and Edward L. Glaeser.1997. "Are ghertos good or bad?"
t )rrtrtcrly .lournal ot' Econontics | |2:827--2.
-).rvrs,JamesA. 1966. "The campusas a frog pond: An applicationof the theory
,I re'lativedeprivation to career decisionsof college men." American lournal of
' ;tolog! 72:1'7-31'.
:.:.ioD. Angus. 2005. "Measuring poverty in a growing world (or measuring
:.()\\'th in a poor world)." Reuiew of Economics and Statistics 87:1-79.
- :.r1rrflfl, Andreas, and Henriette Engelhardt. 1999. "The social inheritance of
.iivorce: Effects of parent's famiiy type in postwar Germany." American Socio-
I ogi cal Reuiew 64 :7 8 3-9 3.
I)iPrete, Thomas A., and David B. Grusky. 1990a. "The multilevel analysis of
trends with repeatedcross-sectionaldata." Pp. 337-68 in Sociological Method-
ology 1990, ed. Clifford C. Clogg. Oxford: Basil Blackwell.
1990b. "Structure and trend in the process of stratification for American
men and women." American Jr,turnalof Sociology 96:107-43.
214 . References

Downey, Douglas. 1995. "\fhen bigger is not berter: Family size, parental Firebaugh,Glenrr. rr: i.-
resources,and children's educational performance." American Sociological Re- 1 9 7 2 - 1 q 8 4 : R e l r ,': . -
uieru60:746-61.. 25r-72.
Downey, Douglas, Beckett Broh, and Paul von Hippel. 2004. "Are schools the F i r e b a u g h .C l e r ttt..t: -
great equalizer?Cognitive inequality during the surnmer months and the school A m e r i ca n 5 o r l ./, r :. ..
year." American Sociological Reuiew 69:61,3-35. F i r e h a u g h .Gl e n r r ..'.:- .
Downey, Douglas, Brian Powell, Lala Carr Steelman, and Shana Pribesh. 1999. Anrerican: orr .r lr,J -
"Much ado about siblings: Change models, sibship size, and intellectual devel- i r r go, f r l r e Ar r r r r i ..r .
opment (comment on Guo and VanWey)." American Sociological Reuiew F i s h e r ,R o r r a l d.\. I 'r -
6 4:1 .93 -98 . and Boyd.
Duncan, Greg J., and Graham Kalton. 1987. "lssuesof designand analysisof sur- F nr dham \ i o n i rl rrr -'
veys across time." International Statistical Reuieu 55:97-1,1,7. a^^i --
vvPU L 6
., i.L .l,- .: -
Duncan, Greg J., \7. Jean Yeung, Jeanne Brooks-Gunn, and Judith R. Smith. F r a n k, R o b e r t H . I ""-
1998. "How much doeschildhood poverty affect the life chancesof children?"
.lournal 107:I S-ll-r-
American Sociological Reuiew 63:406-23. 1 9 9 9 .Ittr t:':
Easterlin, Richard A. 1974. "Does economic growth improve the human lot?" In t o n : Pr i n ce to n['::.., - ,
Nations and Households in Economic Growth: Essaysin Honour of Moses F r e e d ma n ,D a l i d . I"" .
Abramouitz, ed. P. A. David and M. \7. Reder. New York: Academic Press. 353-58 (reloinder : '
1996. Growth Triumphant: The Twenty-lirst Century, in Historical Per- O r f o r d : R a si lBl ,r ci ..''.
spectiue.Ann Arbor: University of Michigan Press. Ie 9 - . "f- ,, .- .,
2001. "Income and happiness:Towards a unified theory." Economic C atr s a l rt f i / i ( , i -i j . i : t'.

lournal 11I:465-84. i n t h e Su ci .tl\. /, ,: .. . .

Edwards,Ryan D., and ShripadTuljapurkar.2005. "lnequaliry'in life spansand a Bend,IN: Univer:i:'. -
new perspective on mortality convergence across industrialized countries." F r y e r ,R o l a n d C .. .u t;':.
Population and Deuelopment Reuiew 31:645-74. scoregap in the hr.: :.'.
Farkas, George, Christy Lleras, and Steve Maczuga. 2002. "Does oppositional 86447-64.
culture exist in minority and poverty peer groups?" American Sociological Re- F r y e r . R o l a n d C .. l r .. r ".'
uie w 67 :74 8- 55. white.' " Cambridct. \I -,
Felson, Richard B. 2002. Violence and Gender Reexamined. rWasl.rington,DC:
C a l r o n , Si r Fr a n i i .. l r - i .
American PsychologicalAssociation Books.
Macmillan.
Firebaugh, Glenn. 1978. "A rule for inferring individual-level relationships from
C e r r e tt, Ge o ffr e ,. I' r '':. '
aggregatedata." Americdn Sociological Reuieu 43,557-72. vember/December: \l-- -
1989. "Methods for estimatingcohort replacementeffects." pp.243-62 G h o s h, S.. a n d C . R . R
in Sociological Methodology 1989, ed. Clifford C. Clogg. Oxford: Basil Black- A n a l ysi so f ExP.,;"' '
well. G l a e ser , Fd r va r d I .. h '. . .
1992. "Growth effects of foreign and domestic investmenr." American s o c i a l m u l ti p l i r r ." / :' :
Journal ctf Sctciology 98:105-30. G l a s s,C e n e . | 9 - 0 . "l :.:' .'
1997. Analyzing RepeatedSurueys.SageUniversity PaperSerieson Quan- tional Researc| -i:-i-: .
titative Applications in the Social Sciences,07-115. Thousand Oaks, CA: Sage. ..Tn tc:- :, .._
1972.
2003. The New Geography of Gbbal lncome Inequality. Cambridge: R e se a r chi n .l tt,- t:: :'
L
Harvard University Press.
G l e n n,N o r va l D . l ,) , r r .
Firebaugh, Glenn. In press. "Replication data sers and favored-hypothesis bias
o n Qu a n ti ta ti r e.\p p - .-
(Comment on Jeremy Freeseand Gary King)." Sociological Methods and Re-
C o e s l i n g . Br i a n . t,t l
search.
t i o n :: N e r v e i i Jcr r ... ,
Firebaugh, Glenn, and Kevin Chen. 1995. "Vote turnour of Ninereenth Amend-
G r o v es,R o b e r t \1 .. | -
ment women: The enduring effect of disenfranchisement."American Journal of
E l e a n o rSi n g e r ..r n - r - . '
Sociology 100:972-96. N J : Jo h n Wi l cr .r r r - .

. - . ... srze , p are nta l
' ": \;tti616ti'o1 P"-
- ".\re schoolsrhe
- Firebaugh, Glenn, and Jack P. Gibbs. 1985. "User's guide to ratio variables."
:.:is a nd th e scho ol
: . - :.:r.rPribesl-r.1999.
. .: J lntcllectualdevel-
. : : ok'tg ica l Reu iew
.
-:: .rndanalysisof sur-
-- _ i - .
' -:' : Ju dith R. Smith.
' . , r: of children ?"
-,.nces
' .: : hc hu man lo t?" In
: ..: Hottour of Moses
-,.: ^\;ademicPress.
::::',, in Historical Per-
--: : heon'." Eco no mic
- . . . , . . : .in lfe sp an san d a
' - ...::rr[ze d cou ntrie s."
"Does oppositional
- 86:447-64.
':. :-.i11.SociologicalRe-
' ':...r.\\hshington, DC:
. .l relationshipsfrom
-- -l
r' : . f lec ts." Pp.2 43 -62
--. tlrford: Basil Black-
. ,ist m en t." American
-'-rter Serieson Quan-
. '-:r.l(Jaks, CA: Sage.
,::.t ttt I ity. Cambridge:
. . , , red-h r.po the sis
b ias
,::.il \Iethods and Re-
- : \ineteenth Amend-
:. ' .\rterican lournal of
References . 245
Firebaugh, Glenn, and Ker.rnethE. Davis. 1988. "Trends in ar.rtiblackprejr,rclice,
1972-1984: Region and cohort effects." American Journal of Sociology 94t
25 1 - 7 2 .
Ameri can Socio logicaI Reuiew 5 0 :7 73-22.
Firebaugh, Gienn, and Laura Tach. 2005. "Relative income ancl happiness:Are
Americans on a hedonic treadmill?" Paper presentedat thc 100'h annual meet
ing of the American SociologicalAssociation,Pl.riladelphia (August).
Fisher,Ronald A. 19-59.Smoking: The Cancer Controuersy.Edinburgh: Oliver
and Boyd.
Fordham, Signithia, and john U. Ogbu. 1986. "Black students'school success:
Coping with the 'burden of acting white."' IJrban Reuiew 78:176-206.
Frank, Robert H. 1997. "The frame of referenceas a public good." Ecotrcntic
Journal 107:1832-47.
1999. Luxurl' Feuer:Money and Happiness in an Age of F.xcess.Prince-
ton: PrincetonUniversityPress.
Freedman,David. 1991. "Statisticalmodels and shoe leather." Pp.291-313 and
353-58 (rejoinder)in SctciologicalMethodobgy 19c)1,ed. l'eter V. Marsden.
Oxford: Basil Blackrvell.
1997. "From assocrationfo causation via regression." Pp. 113-61 in
Causality in Crisisi Statistical Methods and the Search for Causal Knowledge
in the Social Sciences,ed. Vaughn R. McKim and Stephen P. Turner. Sourh
Bend,IN: Universin'of Notre Dame Press.
Fryer, Roland G., and StevenD. Levitt. 2004. "Understanding the black-white test
score gap in the first two vears of school." Reuiew of Economics and Statistics
Fryer, Roland G., Jt., and Paul Torelli. 2005. "An empirical analysisof 'acting
wl.rite."' Cambridge,-\IA: NBER rWorkingPaperno. \( 11,334 (May 1),
Galton, Sir Francis. I874. English Men of Science:Nahtre dnd Nurture, London:
Macmillan.
Garrett, Geoffre,v.2004. "Globalization's missing middle." Foreign Affairs (No-
vember/December): 84-9 6 .
Ghosh, S., and C. R. Rao, eds. 1996. Handbook of Statistics13: Design and
Analysis of Experimenf-s.Amsterdam: Elsevier Scrence.
Glaeser, Edward L., Bruce L. Sacerdote,and .fose A. Scl-reinkman.2003. "The
social multiplier." .lournal of the European Economic Association 1:345-53.
Glass,Gene. 1.976."Prinan', secondary,and meta-analysisof research."Educa-
tional Researcb5:3-8.
1977. "lntegrating lindings: The meta-analysisof research." Reuiew of
Researchin Edrtcatic.trt 5:ii l--9.
Glenn, Norval D. 2005. Cohort Anall,srs.Secondedition. SageUniversityPaperSeries
on QuantitativeApplications ir.rtl'reSocialSciences,
5. ThousandOaks, CA: Sage.
Goesling, Brian. 2001. "Changing income inequalitieswithin and berween na-
tions: New evidence." American Sociological Reuiew 66:745-61,.
Groves, Robert M., Floyd J. Fou'ler Jr., Mick P. Couper, James M. Lepkowski,
Eleanor Singer, and Roger Tour:rngeau.2004. SttrueyMethodology. Hoboken,
NJ: John'Wiley and Sons.
216 . References

Cuo, Guang, and Leah K. Van\7ey. 1999. "Sibship size and intellectual develop- J e n c k s.C h r i 'tu p l r cr . ..: -
ment: Is the relationship causal?" American Sociological Reuieut 64:169-87. S c or eGa p . \\r i l ::- :: -
Haavelmo, Trygve. 1944. "The probability approach in econometrics."Econo- .fencks,Christophcr..r : '.
metrica 12 (Supplement): 1-1 15. t e s t:C l u r i r r g tl '. l ...
Halaby, Charles N. 2004. "Panel models in sociologicalresearch:Theory inro ber/October)J 1:-4:-'-:
practice." Annual Reuiew of Sociology30 507-44. Jenkins,StephenP. 1"-
Harding, David. 2003. "Counterfactual rnodels of neighborhood effects: The ef- E c o n o tn i c l r t,.1 tt.:..:'.'
fect of neighborhood poverty on dropping out and teenagepregnancy." Ameri- A r m o n k, N \ : \1 . i '
too\- .'l
cdn Journal of Sociology 109:676-719. -

Harris, David R., and JeremiahJosephSim. 2002. "Who is multiracial?Assessing U . K., l e - l - S". - .
the complexity of lived race." American Sockiogical Reuietu 67:61.4-26. K a l d or , N i ch ,,l .r - . .' '- -
Hedges, I-arry V., and I. Olkin. 1985. Statistical Methods for Meta-Analysls. San n o m i c Stu ( l i .sI ::' :-
Die go : Acad em iePr es s . K a l t o n,Gr r tl r .r r r .I''
Hedges,Larry V., and Am,v Nowell. 1998. "Black-white rest score convergcnee lications.
since 1965." Pp. 149-81 in The Black-Wbite Test ScoreGap, ed. Christopher K i n g . C a n . l ( ) "- . :
'Washington, '.
Jencks and Meredith Phillips. DC: Brookings Institution. t tr ..- - -
P r i n ce to r ['r
Ho, Daniel E., Kosuke Imai, Gary King, and ElizabethA. Stuart. 2007. "Match- K i n g . Gr r r . R ,,l ';- - , : .
ing as nonparametric preprocessingfor reducing model dependencein paramet- ScientificlrtfL,..::. . : ..
ric causal inference." PctliticalAnalysis 15:199-236. K i s h . Le sl i e . l .') ^i . \
Hobcraft, John, Jane Menken, and Samuel Preston. 1982. "Age, period, rnd 1 9 8 ,. - \i .;::.:
cohort effects in demography: A review." Population Index 48:4-43. K i r r g ,r w l . tr c ' \1 .
Holland, Paul W. 1986. "Statisticsand causalinference."lournal of the Amcrican United Stttts: -. ,. :
StatisticaI Association 81,:94 5-7 0. UniversitvPre...
Hoxby, Caroline M., and Gretchen \Weingarth. 2005. "Taking race out of the K l e p pn e r , Pa u l . i ': -
equation: School reassignmentand the structure of peer effects." Manuscript, t U r no U I. l , ,tt-.:.;.
Department of Economics, Harvard University. Retrieved March 2006. K o n o w , Ja n t.'. .: - .
Hsiao, Chcng. 2003. Analysis of Panel Dara. Secondedition. Cambridge:Cam- n o m i cu s Ir .r p r :.- i '
h ridg e Univer s it yPr es s . m o un t U n l \cr ..: : ,
Hummer, Robert A., Richard G. Rogers, Charles B. Nam, and Christopher G. Kozt-,1. Jonrth.lr'r,.
Ellison. 1999. "Religious involvemenr and U.S. adult mortality." Demographl, York: Cron'n.
36 :27 3-8 5. K u h , Ed r vi n . .r r '; -
ln.rai, Kosuke, and David A. van Dyk. 2004. "Causal inferences with general when the d.rr.r.i-. -
treatment regimes:Generalizingthe propensity score." Jourttal of the American L a y a rd .R i ch .r r .l .
Statistical Association 99 :854-66. gu1n.
Jaccard,James,and Robert Turrisi. 2003. Interaction Effects in Multiple Regres- Leamer, F.du'ar.l L. - - -
slon. Secondedition. SageUniversity Paper Serieson Quantitative Applicarions E c on o m i c R r '/;, i .
in the Social Sciences, 07-72.Thousand Oaks, CA: Sage. Lesthaeghe,Ron. .-.:- -
Jackman, Robert !(. 1980. "Keynesian governnent intervention and income ine- t h c or i e so ffe r - ' .- -
quality (Cornment on Stack, ASR, December 1978)." American Sociological L i e b er so n ,St:r n l re. - ' '
Reuiew 45:1.31.-37. a n d Th co r t. Bc:. -
Jacobs,Jerry A. 2005. "Multiple methodsin ASR." Footnotes(December).Wash- l 9 9 l . ":" ..
ington, DC: American Sociological Association. i n co m p l r l r tr q .:..- - .
Jencks, Christopher, r,vith Marshall Smith, Henri' Acland, Mary Jo Bane, David 307-20.
Cohen, Herbert Gintis, Barbara Heyns, and Stephan Michelson. 1972. Ine- L i k e r . Je ffr e r .5 L r r.\..- .-
quality: A Reassessmentof the Effect of Family and Schooling in America. o f ch a n g e :A. ", -
New York: BasicBooks. S c ie n ceR csc,r ,., : -

, .-t t-,-,,,"t,.t-...1.,^_
:,.: i eu'64: 169- 87.
:, :-r{)nle triCS."
Eco no -
:\..rrch: Theory intcr
' -:'rL)cleffects:The ef-
, *. ar.gnancy-" Anteri-
- : .:irir:rcial?Assessing
.:-:r 6=:6 14 -26 .
' .\Iet,t-Analysrs.San
i:ri SCOfe COnVergence
'r-;p. ed. Christopher
-. I1; tilu rion .
i: -: -rr t. 10 07 . "Match-
,r,lrrr,-a
^ in
. i. ' . \ge , pe riod , an d
' : . . , 4S:-i-4 -3.
:, ':-tl oi the Anterican
- . -. : lq ra ce o ut o f th e
- : rrccts," Ma nu script,
- \ l.rrch 20 06 .
- - a rmhrirloe.
-:. -rnd Christopher G,
::.:lrrr'." D emography
_ i - -n,- ,-< rr:ith
:, ':.il ol the American
-.: :,: ,\htltiple Regres-
..- :::.]liYeApplications
: :.,)n .rnd income ine-
:,. :., i L',urSociological
;. -. L)rcember).Wash-
. \. 1...:r. .lo Bane, David
' .1 , rrlso n. 19 72 . Ine -
: -, . , litt.qin Ame rica .
References . 247
Jencks, Christopher, and Meredith Phillips, eds. 1998a. The Black-White 'lest
'Washington,
Score Cap. DC: Brookings Institurion,
Jencks,Christopher,and Meredith Phillips. 1998b. "America's next achievemenr
test: Closing the black-white test score gap." The American Prospect (Scptem-
ber/October)41.:44-5 3.
Jenkins,StephenP.1991. "The measurementof income ineqr-rality-" Pp. .3-.38in
Economic Inequdlity tmd Pouerty: International I'erspectiues,ed. Lars Osberg.
Armonk, NY: M. E. Sharpe.
1995. "Accounting for inequality rrends: Decomposition analysesfor the
U.K . , 1 9 77 - 8 6 . " E c o n o m i c a 6 2 : 2 9 - 6 3 .
Kaldor, Nicholas. 1956. "Alternative theories of distribution." Reuiew of Eco-
nomic Studies23:83-100.
Kalton, Graham. 1,983.Introductiott to SurueySampllrug.BeverlyHills: SagePub-
licatiorrs.
King, Gary. 1997. A Solutiott tct the Ecological Inference Problem. Princeron:
Princeton Universitv Press.
King, Gary, Robert O. Keohane,and SidneyYerba, 1994. Designing SocialInquiry;
norampt- Scientific Inference in Qualitatiue Researcb,Princeton: Princeton Universitv Press.
Kish, Leslie. 1965. Suruey Sampling. New York: John Wiley and Sons.
1987. Stotistical Desigtt for Research.New York: .|ohn \filey and Sons.
Kitagawa, Evelyn M., and Philip M. Hauser. 1973. Differential Mortality in tbe
United States:A Study in SctcioeconomicEpidemiology. Cambridge: Harvard
University Press.
Kleppner, Paul. 1982. "'Were r,vomen to blame? Female suffrage and voter
turnout." lournal of lnterdisciplinary Historl, 12:621-43.
Konow, James, and Joseph Earle1'.2003. "The hedonic paradox: Is Homo Eco-
Crm- nomicus happier?" Manuscript, Department of Economics, Lovola Mary
mount Universit_v.Retrieved December 2004.
Kozol, Jonathan. 1991. Sat,ageInequalities: Children in America's Sc/.,t,r.,/s,New
York: Crown.
Kuh, Edwin, and Jol.rn R. Meyer. 1955. "Correlarion and regression esrimares
oe..r" I when the data are ratios." Econometrica23:400-16.
Layard, Richard. 2005. Happiness..Lessonsfrom a New Science.Nerv York: Pen-
guin.
Leamer, Edward E. 1983. "Let's take the cor-rour of economerrics."Amerrcan
Economic Reuiew 73:3 1-43.
Lesthaeghe,Ron, and Johan Surkyn. 1988. "Cultural dynamics and cconomic
theoriesof fertilit,vchange." Populationand DeuelopmentReuiew I4:I-45.
Lieberson, Stanley. 7985. Making It Count: The Improuement of Social Resedrch
and Theory. Berkeley: University of California Press.
1991. "Small N's and big conclusions:An examination of the reasoning
in comparative studies based on a small number of cases." Social Forces 70:
307-20.
Liker, Jeffrey, SueAugustyniak, and Greg Duncan. 1985. "Panel data and models
of change: A comparison of first difference and conventional models." Soclal
ScienceResearch14:80-101.
218 . Refelences

Little, R.J.A. 1992. "Regression with missing Xs: A review." Journal of tbe M u l l e r, Ed w a r d \. 1 - - r
ia I ion 8- : | 227- 37.
A tneri catt St a t i st ical Assoc SociologicalR.,rr, :, -
Lombard, Herbert L., and Carl R. Doering. 1928. "Cancer studiesin Massacl.ru- M y e r s . Sco tr l \[. l q - ^ ''
setts:2. Habits, characteristics,and environments of individuals rvith and with- p o r ta n ceo f fr m r l r J :': -
out cancer." New England lournal of Medicine 198:481-87. N a t i o n a l C e n r e r f.r r L - ...
Lott, John R., Jr. 2000. "Gore might lose a secondround: Media suppressedthe 2 0 0 /. R e p o r r 2 0 tl 1 - - .:
Bush vote." Philadelphia Inquirer, November 14, p. 23A. N o r p oth , H e l m u r . l q \-
Luhrr.ran,Reid. 2002. Race and Ethnicity in the United States: Our Differences I 9 8 0 s? " Pu h l i c u S':,::
and Our Roo/s. Fort Worth: Harcourt. N u x o ll , D a n i e l . l e e +. - ".
Mannheim, Karl. 1927 [1952]. "The problem of generations."Pp. 276-322 in in ornrrrth rcr". " 7..

Essayson the Sociologyof Knowledge. London: Rouledge and Kegan Paul. Oaxaca,Ronald. I 9--i. '1.I ,
McCloskey, Deirdre, and Stephen Zthak. 1996. "The standard error of rcgres- I n t er n a ti o n ,r lL c .':.,':.
sions." .lournal of Economic Literature 34:97-1,14. O g b u , Jo h n U . 2 0 ( l - 1 8. ..;, ,
Manski, Charles F.1993. "ldentification and endogenoussocial effects:The re- A c a d e m i cD i - sc,r {.r -,-, ;, '
flection problem." Reuiew of Economic Studies60:531-42. P a g e r .D e r a h . l r ) {) 1 . I
Massey, Douglas. 2004. "Segregationand stratification: A biosocial perspective." I n c ar ce r a ti o riro r r r r . I ' .
Du Bois Reu iew l: 7- 2- S. e r s ." Ph .D . d i \>( r r .l r :..
Massey, Douglas, and Nancy Denton. 1,993. American Apartheid: Segregation P a r r k er t,Fe l i r . l q - i . " I"-
and the Making of the Underclass.Cambridge: Harvard University Press. . ^ ., Vr
- . , - YCy
5Ur ^ r EV
^ ,,:l ^ ,..^ " ,... -
rUglllr.
Matthews, Robert. 2000. "Storks deliver babies (p=.008)." Teaching Stdttstlcs Pearl,Judea,2000, C,;r;-,-
22:36-38. Cambridge Unilcrsirr'f'-:,
.What '
Mayer, Susan. 1997. Money Can't Btry: The Effect of Parental lncome on P eerl,Raym ond.I ' 1. .
Children'sOutcomes.Cambridge:Harvard UniversityPress. M er edit h.199". .
P hi l l ips,
N4elchior,Arne, and Kjetil Telle. 2001. "Global income distribution 1965-98: k n o w a r td r l h l t r i c .: :
Convergenceand marginalisation."Forum for DeuelopmentStudies1:75-98. ican SociologicalRci :.,:. :
Merriam, Charles E., and Harold F. Gosnell. 1924. Non-uoting: Causesand R a u d e n b u sh Sr. e p h cr rI. - :
Methods of Control. Chicago: Universit.vof Chicago Press. l r g . L i n co l n u 'o .J. II , .-
Mickelson, Roslyn Arlin. 1990. "The attitude-achievement paradox among black Ravallion, Martin. I1.11-r :. \,
adolescents."Sociologl, of Education 63:44-61.. H o w w e l l d o n a r i o n .:.
Middleton, Russell.1976. "Regional differencesin prejudice." American Socio- Statistics.85:64-5--il.
logical Reuieu 41:94-117 . R e a r d o n .Se a nF.. .l n .t ( . . -
Milanovic, Branko. 2005. Vlorlds Apart: Measuring International and Global In-
B a t io n ." Pp . j j - ^ - .:
equdlity. Princeton: Princeton University Press. b e r g . Bo sto n :Bl r ,\r ,.,
Mirowsky, John, and CatherineE. Ross. 1999. "Economic hardship acrossrhe R e a r d o n .Se r n F...l u h :. .
life course."American Sociolctgical Reuieu 64:548-69. s l r u ctu r e o f sch o . . ..- .
Moffitt, Robert. 2005. "Remarks on the anal1,sisof causal relationships in popu- metropolitln-.1rc.l., -
lation research." Demography 42:9 l-108. R o b i n so n ,Vi l l i .r n r '. . - '
Morgan, StephenL. 2001. "Counterfactuals,causaleffect heterogeneitnand the trals." Americ,rrt 5, ::
Catholic schooleffect on learning." Socbktgy of Education 74:341-74. R o s e rr b a u m ,Pr u l R ,. .'*
N{organ, StephenL., and David J. Harding. 2006. "Matching estimatorsof causal p r o p e n si r vsco r e i :. .:'
effects: Prospects and pitfalls in theory and practice." Sociological Methods 41-55.
and Research35:3-60. |9 8 4 . "R c* - -
Morgan, Stephen 1.., and Christopher \finship. 2007. Counterfactuals and o n fh e p r o p e n \r t\ .- - -
Causal Inference: Methods and Principles for Social Research. Cambridge: 79:516-24.
Cambridge University Press. 19 8 5 . "( n r ::- -
Mouw, Ted, and Barbara Entwisle. 2006. "Residential segregationand interracial nl i no me t h n , l s rh : - -
r ""'b '-
friendship in schools." American Journal of Sociology 112:394-441. 39:33-38.
 References . 249

- r'., " .lountaLof the Muller, Edward N. 199-5. "Economic determinantsof democrac1,."Amerrcan
SociologicalReuiew 60:805-2 1.
- . - -...; r c sin M as s ac h u - Myers, Scott M. 1996. "An interacrivemodel of religiosity inheritance:The im-
.: :.rL-;
s'ith andwith- portanceof farnily conrexr." American SociologicalReuiew 6l:8-i8 66.
National Center for Education Statistics. 2001, The Condition of Educatiott
' '' l :rr J s Llppr es sted
he 2001. Report 2001072. 'Washington,DC: NCES.
Norpoth, Helmut. 1987. "Under way and here to stay: Party realignment in the
'. .. :: Ottr Differences 1980s?" Public Opinion Q.uarterly 51:376-91.
Nuxoll, Daniel. 1994. "Differences in relative prices and internatiolal difterences
: .." I'p. 276-322 in in growth rates." The American Economic Reuiew 84:1423-36.
-: -.:rtlKegan Paul. oaxaca, Ronald. 1973, "Male-female wage differentials i' urban labor markets."
- "', .,.:clerror of regres- International Economic Reuiew1.4:693-709.
Ogbu, John U. 2003. Black American Students in an Afllttent Suburb: A Study of
- . :rrl effects:The re- Academic Disengagemenl.Mahway, NJ: Erlbaum.
__-. Pager, Devah. 2002. "The Mark of a Criminal Record: The Conseouencesof
- . oci a l p e r s p e c t i v e . " Incarceration for the Ernployment opportunities of Black a'd whire Job seek-
ers." Ph.D. dissertation, University of \(/isconsin, Maclison.
- . ; 't l: e t d : S e g r e g a t i o n Paukert, Felix. 1973. "Income distribution at differe't levels of development:A
. : :',er s i$' P r es s . survey of evidence." International Labour Reuiew 108:97-125.
- Tt:ching Statistics Pearl, Judea. 2000. Causality: Models, Reasoning, and Inferences. Cambridge:
Cambridge University Press.
' l'.irerttttlIncome on Pearl,Raymond. 1938. "Tobacco smoking and longevity." Science87:2L6-17.
Phillips, Meredith. 1999. "Sibship size and academic achievemenr:'What we now
- -::rf.utio n 1 96 5-9 8: know and u,'hatwe still need to knou. (comment on Guo and Van\7ey)." Amer-
i - ) f//.//r's Ii / J-Y 6. i can Sociologi cal Reuiew 64 : 188-92.
:-: ,tutg: Causesand Raudenbush, Stephen.2004. HLM6: Hierarchical Linear and Nonlinear Model-
lng. Lincolnu'ood, Il-: Scientific Software International.
' :.r:.1Lloxamong black Ravallion, Martin. 2003. "Measuring aggregatewelfare in developing counrnes:
How well do national accounts and surveys agree?" Reuiew of Economics and
- .:." .\nterican Socio- Statistics. 8 5 :645-52.
Reardon, SeanF., and Glenn Firebaugh.2002. "Measures of multi-group segre-
,.:.: ,,ttl .trtd Glob al In- gation." Pp. 33-67 in Sociological Methctdology,2002, ed. Ross M. Stolzen-
berg. Boston:Blackwell.
'" , rrrdship acrossthe Reardon, SeanF., John T. Yun, and Tamela McNulty Eitle. 2000. .,The changrng
structure of school segregation: Measurement and evidence of multiracial
: --:.rrionshipsin popu- metropolitan-area school segregarion,7989-199 5." D emogrdphy 37 :3 5 1-64.
Robinson, \Tilliam s. 1950. "Ecologicalcorrelationsand rhe behavior of individ-
- - _.-*..,,-,.^ir-. vr ^-,.1 +t -
,<rrrLrt uals." American Sociological Reuiew 15:351-57.
. .: ': -1 : 347- 74.
Rosenbaum, Paul R., and Donald B, Rubin. 1983. "The central role of the
: - r-ifimatorsof causal propensity score in observational studies for causal effects." Biometrika 76:-
' :rrtlctgical Methods 4i- 5 5 .
1984. "Reducing bias in obser:vationalstudies using subclassificatron
-
' ,rptterfactuals and on the propensity score." Journal of the American statistical Association
:.. -.i,r'cD. Cambridge: 79:516-24.
1985. "Construcring a control group using multivariate matched sam-
- :: :.r:ion :md interracial pling methods rhat incorporate the propensity score." American statistician
l: 19-1-'{'1
1. 39:33-38.
250 . References

Rosenthal,Robert. 1979. "The'file drawer problenr' and tolerancefor null re- S mi th,TomW. 1990."Tr r : -
srrlts." Psyc/:ologicalBulletin 86:638-41 . 1972- 1. ) 89C55. " t , \ r
1.994."Ptrametric measuresof effect size." Pp. 231-44 in The Hand- C )pini on R es e:rrc h C enr.:.
book of ResearchSt'nthesis,ed. Harris Cooper and Larry V. Hedges. New S n o w,Jo h n . ( 1 8 5 - t) 1 9 6 - i ,
York: RussellSageFoundation. New- York: Hafner.
Rosenzu'eig,\lark R., and Kenneth I. Wolpin. 2000. "Natr-rral'natural expen- Stack, Steven.1978. "Thc
-
ments' in econornics."Journal of Economic Literature 38:827-74. om,v on the degree of i:..
Rubirr, Donald B. 1973. "The use of matchedsamplesand regressionadjustmenr SociologicalReuieu,1) :''
to removebias in observationalstudies."Biometrics29:185-203. Stor-rffer,Samuel A., Eds.i:-
1979. "Using rnultivariatematched sampling and regressionadjustment and Robin M. '!(illirrr:.
to control bias in observational studies." Journal of the American Statisticdl Army Life. Volume 1. P: :
Association74:3 1.8-28. Sudman, Seymour, Norn.i.r:
1987. Mubiple Intputation for Nonresponse in Swrueys.New york: .|ohn about Answers: The .\;:
\7iley and Sons. ogy. Josse,v-Bass Sociel .'
Rubin, Donald B., and Neal Thomas. 1996. "Matching using estin'rared propen- B a ss.
sity scores:Relating theory to pracrice." Biometrics 52:249-64. Summers, Robert, :rnd -\j...:
2000. "Combining propensiry score matching with additional adjust- expanded set of interr-Li:
ments for prognostic covariates." Journal of the American Statistical Associa- Econonics 1 06:-12 -i. :.
tio n 9 5:5 73 -85. Tourengeau.Roger. L.rrr..
Russett,Bruce M., Ha1'wardR. Alker, Jr., Karl'W. Deutsch,and Harold D. Lass- of huuey Rcspon..c.C.i.:.
rvell. 1964. Vrorld Handbook of Political and Social Indicators. Ner,v Haven: Triplett, Jack E. 199-. ' \1.
Yale University Press. t h e r e se r r ch i ssu e r ." l .
Ryder,Norman B. 1965. "The cohort as a conceprin the study of socialchange." 9-42.
American SociologicalReuiew 30:843-61. T u c h , Ste r e r r .l q 8 - . "L .
Saia-i-Martin, Xavier. 2006. "The rvorld distribution of income: Falling ciologital Retiett il: i - -
poverty and. . . Convergence, period." Quarterly Journal of Econr,tmics T u f r e , Ed w a r d R . l 9 5 l . /
1.21.:351.-97. CT: GraphicsPress.
Schafer, Joe L. 1997. Analysis of lncomplete Multiuaridte Data. London: Chap- T y s o n. Ka r o l l n . \\ i l l :.r "'
man an d Ha ll. b l e ck th i n g ': U r .l e r '.r. -
hi " h " l:
schultz, T. Paul. 1998. "lnequality in the distriburion of personali'come in the " , ' h i " . , " ' . rc rr,
world: How it is changing and why." Journal of Population Ecctnomics lI: U g g e n . C h r i sto p h e r '.r ", '
307-44. C O l l se q tl e n C e t r\' l (
Schwartz, Barry, and Howard Schuman. 2005. "History, commemoration, and SuciulogicalRc'z/. i, ..- - -
belief: Abraham Lincoln in American memory, 1945-2001." Arnerican Socio- U n i r e d N cti o n '. l o o l . r ' .:
logical Reuiew 7 0 :l 8 3-203. U z z i . B r i a n . a n d R r .: . .'-
Seawright,Jason,and David Collier. 2004. "Glossary." Pp. 273-313 in Rethink- the corporarel.ru " :-
'$ 7 a i t e,
ing Social Inquiry: Diuerse Tools, Shared Standards, ed. Henry E. Brady and L i n d a J...r r r .1r .
David Collier. Lanham, MD: Row.man and Littlefield. l i oi nn i rr t l re I " t, --
m c n t R e u i e t(l q :l i t -'
Selvin,Hanan C., and \farren O. Hagstrom. 1963. "The empirical classificarron
'Waters,Mary
of formal groups." American Socblogical Reuiew 28:399-41I. C. 1q'':.
Sherman, Lawrence W. f992. Policing Domestic Violence: Experiments and Pluralism: Tltt t .
Dilemmas. New York: Free Press. We nd y F. Kr r ki r . \ - .
sherman, Lawrence'w., and Richard A. Berk. 1984. "The specificdeterrentef- nois Press.
fects of arrest for domestic assault." American Sociological Reuiew 49:26I-72. I9 9 9 . 3 7 ',- . ' ' ' '
Smith, Herbert L. 1997. "Matching with multiple conrrols ro estimate rrearment R e a l i ti cs.N o r ' \. :.
effects in observarionalstudies." Pp. 325-53 in Sociologicdl Methodology, ed. 1 0 0 1 . " fl .. .
Adrian E. Raftery. Boston: Blackwell. f r o m d e m o g r .r fr r ..' : - '

::.lnce for null re-
++ h The Hand-
-:
' -' \'. Hed ge s.Ne w
. : . . : . - .,'n atu ral expe ri-
: - .l--- 4 .
I -: -:. !iio n a dju strn en t
. . __r ,.).
- : --iiiiorl a diu stme nt
-...;r'1i c,u1 Statistical
'-.. \e'n, York: John
-
_
-.rinrrrrerl nrnnen-
r_.-r.-''
- - --b+.
:r .r.lditional adjust-
' : ;t-tt istical Associr.-
. ,:rrlFiarold D. Lass-
' ..: :.;:|)t s, Neu' Haven:
. :'. ,rf socialchange."
" ,jin co me:Fallin g
:,' ,:tl o l Econ omics
London: Chap-
--r,;:.1.
:::i()r1ill inCClme
in the
: -;:: ,11LcotrcmiCS11l.
. , ,iltlremoration, and
i." .\tnerican Socio-
- _ ) - )l Jl n l ae tn l nR -
:. H;urv E. Brady and
r ::::irical classification
_- I I I Mary C. 1998. "Multiple ethnic identity choices," Pp,28-46 in Beyond
': '. : E x p e r i m e n t s
. : >lec i f i c d e t e r r e n t e f -
-. ; . Rc t' i e u t 4 9 : . 2 6 7 - 7 2 .
i :, cslrnlate treatment
, : : -;i \Iethodology,
Referenccs . 2-51
Smith, Tom \7. 1990. "Timely artifacts: A review of measurernentvariation in thc
1972-1989 GSS.' GSS Methodological Report no, 56. Chicago: National
Opinion ResearchCenter.
Snow,John. (18.5.5)1965. On the Mode of Commtmication of Cholerd.Reprint.
New York: Hafner.
Stack, Steven. 1978. "The effect of direct government involvement in the econ-
()n1yon the degree of income inequality: A cross national study." Americatt
SociologicalReuiew 43:8 80-8 8.
Stouffer,SamuelA., Edward A. Suchman,Leland C. DeVinney, Shirle,vA. Star,
and Robin M. \Villiams. 7949. Tbe Antericdrt Soldier: Adjusttnent During
Army Life. Volun're1. Princeton: Princeton University Prcss.
Sudman, Seyrnour,Norman M. Bradburn, and Norbert Schwarz. 1995, Thinking
about Answers: The Applicatbn of Cognitiue Processesto Suruey Methodol-
ogy. Jossev-Bass Social and Behavioral ScienceSeries.San Francisco:Jossey-
Ba s s .
Surnmers,Robert, and Alan Heston. 1991. "The Penn\WorldTable (Mark.5): An
expanded set of international comparisons, I950-1988." Qtnrterly Journal of
E conomics 1.06:327-68.
Tourangeau, Roger, Lance J. Rips, and Kenneth Rasinski. 2000. The Psychology
of SttrueyResponse.Cambridge: Cambridge University Press.
Triplett, Jack E. 1997. "Nleasuring consumptior.r:The posr.-1973slowdown ar.rd
the researchissues." Lederal ReserueBdnk of St. Louis Reuiew (May{une):
9-42.
Tuch, Steven.1987. "Urbanism, region, and tolerancerevisited." Americdn So-
ciological Reui ew 52:.504-10.
Tufte, Edward R. 1983. The Visual Display of Q,uantitatiueInformation Cheshire,
CT: GraphicsPress.
Tysor.r,Karolyn, !(/illiam Darity Jr., and Domini R. Castellino.2005. "lt's not 'a
black thing': Understanding the br-rrder.r of acting white and other dilemmas of
high achievement." Am er i cdn Socio I ctgi cal Reuiew 7 0 :5 82-60 6.
Uggen, Christopher, and Jeff N4anza.2002. "Democraric contraction? Political
conseqLrences of felon disenfranchisementin the Llnited States." American
So ciological Reuiew 67 :77 7 -803.
United Nations. 1993. Systetnof National Accounts.Brussels:United Nations.
Uzzi, Brian, and Ryon l-ancaster.2004. "Embeddednessand price formation in
the corporate law market." American Sociological Reuiew 69:31.9-44.
Waite, Linda J., and Evelyn L. Lehrer. 2003. "The benefitsfrom marriage and re-
ligion in the United States:A comparative analysis." Population and Deuelop-
ment Reuiew 29 255-75.
'\X/aters,
and Plttralism: The Conception of Groups and Group Identities in America, ed.
Wendy F. Katkin, Ned Landsman, and Andrea Tyree. Urbana: Universitv of Illi-
nois Press.
'Vlest
1999. Black ldentities: Indian Immigrant Dreams and Amerrcan
Realities. Nerv York: Russell SageFoundation.
ed. 2002. "The social construction of race and ethnicity: Some examples
from demography." Pp. 25-49 in American Diuersity: A Demographic Challenge
252 . References

for the Twenty-first Century, ed. Nancy A. Denton and Stewart E. Tolnay. Index
Albany: State University of New York Press.
2004. "Race, ethnicity and immigration in the Unired States."Pp.20-38
in Social Inequalities in Comparatiue Perspectiue,ed. Fiona Devine and Mary
C.'Waters.Malden, MA: Blackwell.
'Weeden,Kim, Kim Young-Mi,
Matthew Di Carlo, and David B. Grusky. 2007.
"Social class and earnings inequality." American Behauioral Scientist 50:
702-36. Abbon, Andrew, 4--i
'West, abortion: attitudes tos'.tr.l. : -:
Jerry, Kristin Denton, and Lizabeth Reaney. 2007. The Kindergarten Year:
Findings from the Early Childhood Longitudinal Study, Kindergarten Class of pol i cy i n R omani :r.i \-tr
1998-99 (NCES 2001-03). U.S. Deparrmentof Education,Narional Centerfor add-r-regrerrorapfr,' .r-:.. . -
Education Statistics.I(/ashington, DC: U.S. Governmenr Printing Office. addi rrv edec orrpo' .rl ' i l rr,.- -
advocac,vresearch,1
\fhite, Kevin M. 2002. "Longevity advances in high-income countries, 195-5-
age effe c ts ,212
96." Population and Deuelopment Reuiew 28:59-76.
'Winship, age pol a ri z ati on,18-i
Christopher, and StephenL. Morgan. 1999. "The esrimarion of causal
eggregr tedi nJ rr i .l rr.rl-:r.:: -.
effects from observational data." Annual Reuiew of Sociology 25:659-706. aggregcre-l ev el rn.rl r .i .. -
Zajonc, Robert B.7975. "Birth order and ir.rtelligence: Dumber b1' the dozen.', eggrege ri onbi as .I I l . r.. .. .
Psychology Todatt 8 :37-43. fal lacy
Ziliak, Stephen,and Deirdre McCloskey. 2004. "5r2e matters: The standard error .d o r p o .fi r Tp nr,rnerrie< " .

of regressionsin the American Economic Reuiew." Economic lournal V,ldtch A l l i son, P aul , 6-5-66. l 6---
1:3 31 -58 . Amato, Paul, 107-9
American Natior.ralElr;:: : .
111
A msden, A l i c e, l .{ --6
analytic porver. 2..3.lt'
Arrighi, Ciovirnni. --1--:
assuci a ti on:l rrd .-ru..L.
spuri ous ,14, 1l l . l r:
avai l ab l ec as eanah' s i s .::

balance of treatn.)cnr-r:
147. Seea/s o prop.:.
B erkel eyS D A { S urrc r
Anal ,v s i s ),3l
B i as.S eefav ored hr:,
vari a bl c sbi as rs el e;:
bi rth cohort, 181. \. .
black-white
tests.()i-
Brady,Henr,v,7---r
Bush,George\i',. --

cancer:and l l uori dl i r,
127n ; and,s mok i n:
box 5 .1 )
case'"visedeletion. {.i
causala nal y s i s ,l ogrc ,
causa l i ty
causal chains. 1.1-I -1
causald i rec ti on. 1-{
causal -proc esobs s .-r,..
 Sren'art E. Tolnay. Index
Sriites."Pp. 20-38
r Devine and Marv

. :.:J B. Crusky. 2 00 7.
t '.;'.t,tnl Scientist 50:
.. . Kindergarten Year:
' :l:tt,lergarten Class of
: . \ational Centerfor
. ' : ; nring Office .
-- ::. co un tries,19 55 -
': ::rim:rtion of causal
, qr'2 -5:6 .59 -70 6.
.::rl.erbr.'the dozen."
-.:-:: The stanclarderror
'' ,itic lournal Vlatch
Abbott, Andrerv, 4--5
abortiol.r:attituclestoward, 3 1-33, 180;
p o li cy i n R ornani a,38-40
add-a-regressorapproach, 133-34
addrtive decompos:rbilit.v,220
advocacy research, I
a g e e ffects,2l 2
a g e p ol ari zati on, 1B 5
aggregatedindividual change, 197
a g g r egate-l evel anal vsi s,7 1, 17),, 231
aggregation bias, 23 l. Seealso ecological
fallacy
aggregativeproperties, 228
Allison, Pavl, 6 5-66, | 69-70
Amato, Paul, 107-9
American National Election Stud.v(ANES),
111
Amsden, Ahce,74-76
analytic porver, 23, 26
Arrighi, Giovanni, 74-76
association:and causality, 12 1-29 ;
spurior.rs,14, 722, 143
available case anal,vsis,66
b a la nceof treatmentand control groups,
747. Seealso propensity scores
Berkeley SDA (Survey Documentation and
An al vsi s),31
Bias. Sea favored-hypothesisbias; omitted-
variables bias; selection bias
birth ccrhort, 182. Seealso cohort effects
black-n'hite test score gap, 232-3.5
Brady, Henry, 77-78
Bu sh , GeorgeW.,77-78
cancer: and fluoridation of u'ater,
l2 7n; and smoki ng, 122-29 (and i n
b o r.5.1)
casewisedeletion, 65
causal analysis,logic of, 127-29. Seealso
ca usal i ty
ca u salchai ns, 13-14
causal direction, 1,1
causal-processobservations,77-78
cansality, l3-14, 120; and correlation,
12 1-29 ; counterfactual approach to.
122, 127-29; deterministic versus
probabi l i sti c,25n, 120; effect of a c aus e
versuscausesof an effect, 39, 130-31;
lau's of, 121
Central Limit Theorem, 105
change: aggregatedindividual, 197; cross-
cohort, 197; gross and net individual,
173-74 intracohort, 197. Sepalso soci:rl
change
changing-effectmodel, 176; versus conver
gencemoclcl, 191-9-5; antl rnultilevel
mcthods, 182-85; w i th pancl data,
1,76-7 9; with repeatedcross-sectron
data,179-82
chol era,l 58-59
cl assi cexperi mentaldesi gn,129
Clogg, Clifford, 92
cohort effects, 212-14
cohort replacement, 195-96; estimation of
effecrs,| 9--20 l ; \rudent e\erc i s eson,
203-s
cohort succession.Saecohort replacement
C ol enun R eport,228
collected sample,93-94; and unit nonre-
sponse error, 94-9.5
common metric, 42; converting variables
to,44-46
comparing like with like. Seematching
complete caseanalysis,6-5
composite measures,72-74
confidence intervals, 109-10
confoundi ngvari abl es,130, 132, 140-45,
159-62 (andi z tabl e 5.1)
C onl ey,D al ton, 140, 156
consumption surveys, 102-3
conrerr effecr'. 22o-2- : exrcerberi ng,
versurmi ri gati ng.2291as nui sr nc e.
230-3r
contextual ization, 27 -28
control variables, 14-15. Seealso
residualization
convergence,1 85; sign test for, 186-87
254 r Index

convergencemoclel, I 85-86; versus expenditure surveys, 102-3

changing-effectmodel, 191-95; examples e xp e r i mentaldesi gn,129, 152
o f , 1 8 7 - 9 1 ( a nd in ta b le s6 .1 - 6 .1 ) explanatorv variables:an-tountof variance
correlation: and causality, 121-29 I in, 37; maximizing variance in, -19-40
s p u r i o u s ,1 4 , 1 2 2 \a n d in fig u r e 5 .7 ) e r te r n al repl i cati on,107
counterfactual approach to causal
i n f e r e n c e ,1 2 2,1 2 7 - 2 9 farnily incon-reand children's schooling,
coYcrageerror, 91, 98-99, 117 1 4 3 -45
c r i t i c a l t e s t s ,2 3 1 - 3 5 fan-rilysize and children's schooling, 140,
cross-cohort change, 197 r 55-57
cross-sectiondata. Seerepeatedcross- fa vo r ed-h1'pothesibis as, 151
sectlon data fe lo n y convi cti on: and empl ovment,49,
curse of dirnensionalitv, 147 5 6 - 57; and voti ng, 50
Cutler,David, 229 fertility rates in Rorn:rnia, 37-38 (and in
fig u re 2.1 )
Davis, James,229 file drarver problem, 108
t l e c o t n p o . i r i , r r t2. | 8 - lq l o f \cg r e g n r io n . finite population correction, 19n
222-26 (and in figure 7.1 ); of social fi rst-differenceadvantage, 13 6, 174
clrangc (change in means), 197-207; fir st- d i fferencemodel s, 134-38, 16 1,
of variance and inequalitv, 220-22 7 7 6 -77
design. Saeresearchdesign F ish e r ,R onal d A ., 126-27
Diekmanr.r,Andreas, 21 1 fixed-effect method, 136, 167-62, 178-79,
disenfranchisement:of felons, .50;of 207n; exarnplesof, 13.{-45; studcnt
\\ro[ren, 212-74 exerciseson, 165-70
divergence.Saeconvergence Florida vote in 2000 presidential election,
divorce in Cermany, 210-12 77-78
Dorvnen Douglas, 216 flvin g machi nes,26-27
d u r n m y v a r i a b l es,4 6 , 1 75 fo r e ig n i nvestnrcnti n l our countri er,
Duncan, Cireg, 143 7 0 - 73 l and l n box 3.1)
formal equivalenceof models, 71-76
Easterlin, Richard, 54 frame population, 93-91i and coverage
ecological f allacv, 7 4n3, 23 | e r r o r,98-99
effect of a cause, 39; versuscausesof an Frank, Robert, 54
effect,130-31 F r e e d man,D avi d, 133
effect size: examples of verbal comparisons, frog pond eifects,229
19-50 (in box 2.1);methods for gauging,
42--50; relative versus absolute, 42; and Ga llu p, George,21
statistical significance,41 GallLrp polls, 20-21, 172
elasticities,44 Ccn e r al S oci al Survey,31, 172-73
c l c e t i o n .S c c p r e tid e n r ia le le ctio n g lo b a l income i nequal i ty,9-70 (i n bor 1.1)
errpirical nihilism, 27-28 g lo l.a l porertl . methodr for e'ti meti ng.
er.npiricalregularities, 17 i0 1 - 5
endogeneit,vproblcm, 76. Seealso selection Gore, Al, 77-78
bias g r a p h s ,43-44, i |, 52 l l i r hox 2.21
E q u a l R i g h t s A m e n d m e n t ( ERA) , 1 1 1 - 1 5 g r o wfh rate n'rodel s,138-39, 161
errors in measurement..Seemeasurement
estillation nethods. Seestatistical estimarion happiness:and health, 59-60, 166-69; and
ethics in research,3 in co me,54-.55;and marri age, 8-9,16,
ethnographic studies, 79-81 6 7 ,1 66-69; studentexerci seson, 58-62;
exclusion error, 93; sourcesof, 94-99 (and of wor:king women versushouservives,
in frgure 4.1J 4 9 - 51..60
 . l--l hedonic treadmill, .5.5 matching to reduce nrodel dependence,
- . I il Ho lme., Sherl ,,ck,4 | 5l -i 2
. :::r()unt of variance Hoxby, Caroline. 214 mean substitution (mean imputation), 66
- i '1nce
in'3e-'10 rruman
rlight'
26-27
"':.:T::::::::;llil'ii**ll;i:-.]$"-
in co me:and happi ness,54-.55;and errors i n, 88-89
- . : : . . - n 'ssc h o o l i n g , ch il dren'sschool i ng,143-45 measurementerror, 88-89, 100-1 01, 118
irccr.neinequalit,v:global, 9-10; national mediating variables, 13-14
- : . s c h o o l i n g ,1 4 0 , d ifferencesrn,87 l i n tabl e 3.1); student meta-anal ,vsi s, 106-9
exerciseson, 84-89 metric. Saecommon metrrc
. :.. l.r 1 industrialization in poor countries, effects missing-at-random (MAR) assunrption, 58
,: .:rplovnrcnt,49, o f,74-76 mi ssrngdara, 05, 94; methods for al l e-
- i i inequalitv: in earnings, 220-21; in income,
-. viating, 65-68: as a source of bias,
: : : : - r .. l - - 3 8 { a r r l l r u 9 - 10, 84-89; i n studentl eami ng, 96-98 (i t box 4.1)
2 1 7-18; i n i ndi vi dual s'l i fe spans, model depender"rce, 15I
' , \ 2 2 1-22 moderati ngvari abl es,15-16,179
' --: i: r o n . 1 9 n In stituti onal revi ervboards,.3n moti on, probl em of, .{, 28
: : : :. l - 1 6 , 1 7 4 in stn tmentalvari abl es(l V), 153-.58, mul ti l evelmodel s, 182-8-5
: , -. 1 l - l - . l l J , 1 6 1 , 1 6 l -62 (andi r fi gures.5.2-5.4) mr-rl ti pl e
i mputati on, 6T-68
in te r acti oneffects,l 5-16, i 79 rnul ti pl emethod studi es,8l ; erampl esof,
; ---l- in te r nal repl i cati on.106 82
.:. :-. i61-62,178-79, in te r raci almarri age.atti tudestorvard,203-.5
- l-l-J.j; student Inter-university Consortium for Political national producticxr data, 103-4
- a n d S oci alR esearch(IC PS R ),22 nei ghborhoodeffects,149,226
: : : .i J e n t i a l e k c r i o n , in tr a cohort change,197 ni hi l i snr,27-28
investment, effects of foreign and domestic, Nineteenth Amendment, 212-14
- _- _- 7 0 -73 r.roneffects,2
' : ,,r'countrier, ite tn nonresponse,94 nonrnedi ati ngvari abl es,14
- : I nonresponseerror, 95-98
. : : rr , ) J e i s7, 1 - 7 6 l( e lvin, Lord,27 no-surpri sesobj ecti on to researc h,6
- : -- 1 : , r n d c o v e r a g e l( e ynesi antheory, 85

:I
"T:::j"li:119
:;i:i,i:::1,1ffi
:'J
Leamer, Edlvard, 133 r,ariables
. :- l-ieberson, Stanley, I .5, 133 overs:rmplingof minority groups, 22,
--
life sp:rns,clifferencesin. Sec inequalin' 2"4-25,40
orprincipre
ofsa'rpri'g'
18
. : ixllS::lnl'J!l;1"',0t
- . I 1. 1-2-73 L ite r ary,D i gest,21 P ager,D evah,49, 56
'. : : .. . q - 1 0 { l z b o x 1 . 1 ) L o n d on \vafercomp:.l ni es
i n the l 8-i 0s, pai nvi secl el eti on.66
- :. :or estimating, 1 5 8-59 panel data, 172-73,176-79; vers us
repeatedcross-sectionclata, 173-75
market share principle, 53 path analysis, 1-5
'l :' r L . o r 2 . 2 ) m a r r iage:and hi rppi ness,8-9.1'+,16,61, path di agram, 15, 153
. : r-.19.
16 1 166-69; and ser, 34 peer effects: on student achievement,
rnatching, 130; comparison of stratcqies 214-16 on happiness,-5.1-55
, i' t - { . i 1 . 1 6 6 - 6 9 ; a n d fo r ,159-62; exact, 1,16-47;usi ng fi red- peri od effects,212
-
- :r.'.:rjrtse,8-9,16,
: e ffectsmodel s, 134-461r:si ngpropensi t,v pol i ti cal pol l s, 19-21
' : J{rr.ises on, 58-62; scores,147-51; usi ng natural l l ' occurri ng popul ati oncomposi ti on,227n; effec rsof
, : :i : r ! h o u s e w r v e s , r a ndom assi gnment,1.52-59;usi ng changesi n, 181. S eea/so cohort
standard regression, I 3 1-32 replacement
256 . Index

povert,v.Seeglobal poverty reverseuniverse, 129 stati sti cals i gni fi c l nc r ' ..:.. .
p r e s i d e n t i ae
l l e ctio n :o f 1 9 3 6 , 2 1 , o l 7 9 6 8 , r e vie w arti cl es,6n significance,36. { 1
1 9 l .o f 2 0 0 0 , 7 7 ,7 9 Ro m a ni a, 37-39 Stouffer, Samr-iel.ll -
prior research,7; challengesto, 8-l l; srrati 6c J random ..r' r : .. - -
clarifications of, 1 1; ertensions of, sanrple, 1B; advantageof large samples, substan ti r.e profi l i ns . ;"-: -
11-r7 2 3 , 26, 129. represenfati veness
of, 21 ; synthes i z i ngs tudi es1r . -
problenr
of motion,
6,39-40 sirnple random, 18; stratified random,
p r o p e n s i t ys c o r es,1 4 7 - 5 1 , t6 0 - 6 1 1 ))-)i 40 target population. f -i---
preprocessingwith, 1.51--52 sa m p l esi ze:pri nci pl esof, 19-21; i n test score gap, 2.ll--r l
p s e u d o c o n t r o l s,l5 q u " ri i t;rri ve
\rudi e.. 2J-2{- Thei l i n der: r' i i nc ".., . --
sanrpling: principles of, 18-23. Seealsct segregation,2l.j
qualitative research,23-26, 7 9-82 sa mpl e rheory.irnportrl t.- ' . ' - ' -
sir n r p l i ngerror,94-97,105, I 10, 129 total survev error n--::.r--:- .
race/ethnicity, 80; and ernplovmenr, 49, sampling frame, 115. Seealso {rame triangr-rlation,6.{
5 6 - 5 7 ; s e l f - i d e n tifica tio n o f, 8 0 - 8 1 ; a n d population truncJted v .l rttnc a. , -
v o t e f o r G e o r g eWa lla ce ,2 3 0 - 3 1 , school reassignmentsin North Carolina, Tuite, Edu'ard, ,i I
236-39; and vote in 2000 presidential 274-16
election, 79 school segregation,components o1,225 lin uncertai ntyi n s oal f,.:::::-,
random assignment,128-30; eramples ta b le 7.1 ) data c ol l ec ti on.el --. -
o f n a t u r a l l r .o ccu r r in g , 1 5 2 - 5 9 , SDA (Survev Documentation and Analvsis), ment error, 100- I
2 0 9 - 1 8 ; a n d t he p o we r o f p r o b a b ilit.v, 31
t52 secondar,yanalysis,22
random san.rple.See sarnple segregation,components of, 225 (lr table
ratio variables.Seecomposite measures 7 .1 )
reality checks, 64; external, 76-82; selectedsanrple,93-94; ilnd sampling error,
internal, 65-76 9 5-97
R e a r d o n ,S e a n , 2 2 3 - 2 5 selection bias, 14, 128-29, 226
r e g r e s s i o nm e t h od , 1 3 1 - 3 2 , 1 6 1 ; cr itiq u e selection effect. Seaselection bias
o f , I t 2 - 3 4 : o hse ' sio nr vir h . 2 0 9 : p :r th se x: a rti rrrdc'to$ard,62.: hurrrrrrgcnJ
model of, 160 (}r figure .5.-5).Seea/so frequency of, 34-35; marital status and
residualization frequency of, 34
r e i i g i o s i t y ,1 5 sib lin g rnodel s,140-.+.5,161
repeatedcross-sectiondata, 172-7 3, smoking and cancer, 122-29 land inbox
179-80; r'ersuspanel data,173-75 5 .1 )
replication, 90; internal and external, Sn o r v,.fohn,1.58-59
106-7; versusnultiple--methodsresearch, social change, 172; deconrposition of,
8l 197-201- key questior.rsabout,
research:as an ongoing conversation!4i t7 5 -7 6
establisl'redfindings of, 7; ethics of, 3; social contert effects. Saacontext effects
o b j e c t i v e so 1 , 7 2 0 - 2 7 social multiplier effects, 214
researchdesign, 129, 1-52,23.5; as ke.yto "so-rvh:rt" objection to research,6
b e t t e r s o c i a lr e se a r ch ,9 1 ,1 6 4 ,2 0 7 ; n e e d spending on Social Security and education,
for morc imagination in, 207, 235; and attitudes about, 187-91 (and in tables
o m i t t e d - v a r i a b le b
s ia s, 1 3 3 - 3 4 6 .1 - 6.1)
researchquestions, 2; interest criterion, 4- spurious association, 14, 1,22, 113
I 8; researchabilitycriterion, 2-4 stability assumption (first-difference
residualization, 74n2; as matching m o del s),136
m e t h o d u s e d bv r e g r e ssio n7, 4 ,1 3 3 , standardization of variables,44-46
150 statistical estimation, 207; as servant to
responserates, 95-97 theorv and researchdesign, 207-8,
reversecausation, 16 235
 Indcx . 2) /

-: : i l r g e s a r n p l e s ,
- -: : : : - ' l t l r e n e SOSf , 2 l ;
' . . . : . r t r f i e dr a n d o r n ,
. .. lq-21;in
. t r_16
.129
ll.;. 110,
t i. i . l / - . oi r a m e
r : : L r ( ) n e n ros f , 2 2 5 ( i n
- data collection, 92-99; due to rneasure
: : : : . r t i o na n d A n a l y s i s) ,
statisticalsignificance
versussubstantive
signi fi cance,36,41
Stouffer, Samuel, 227
stratified randorn sample, 22, 40
substantiveprofiling, 46-49
synthesizingstudies, 16-1 7
target population, 93-94
tcst score gap, 232-35
T h e il i ndex: of i nequal i ty,221; of
segregation,223
theory',importance of, 54-.56, 235
total survel' error paradign.r,91
triangulation, 64
trlrncated variance, 137, 177
T u fte, E drvard,5l
uncertainty in social research,91; due to
m e nt error, 100-101
unit nonresporrse,
94-98 land in l:tox4.1),
1 17; versusi tem nonresponse 94
,
tttrmearured!crrse5.5ce confound i ns
variables
variance on explanatorv variables,37-39;
qualitl,-quantity tradeoff, 138, 177 ;
stratified sampling to increasesize of,
22-25,40-41
voting rates: for George Wallace, bv race,
230-31, 236-39; surveyesti m aresof,
1 15- I 8; for rvomen versus rlen, 21,2-14
\Wallace,George C., 19-20, 2"30-31,,
236-39
Waters, N{ary, 80-81
'Weedcn,
Kim,22t)
rX/right, Orville and Wilbur, 28
Zeno's paradoxes,28

- l- , . n f , 2 2 5 ( l n
table

: -l: ; n d s a r n p l i n ge r r o r ,

.-._r.) tlA
_

- : . Jri o n bi as
- - : . r l : h u n ti ng an d
..- I i: rr.irital st:ttus and

- :i. 161
,. -. l l-19 l a rd i n bo x

--
: : : o t l l P 6 1 5 i 1 i 961f ,
---
-. 1 , ,r ' r sr r b o u t ,

-: . . \ i . ' c o n f e x t e f f e c t s
- -: _ 111

- : , r ase i trch ,5
- r , : t f i r\ :rnd ed u ca ti on ,
9 7 t ,tntl iru tables

. . . j. ltt. 1 43
::. r-r1
ifierence

:', .:-.:rLe s.,14 -46

. - -: .rsservanfto
. - --: . .:::ten .2 07 -8,