Supplementary materials for this article are available online. Please go to www.tandfonline.
com/r/TAS
General
Exact Binomial Confidence Intervals for Randomized Response
Jesse FREY and Andrés PÉREZ
We consider the problem of finding an exact confidence inter-
val for a proportion that is estimated using randomized response.
For many randomized response schemes, this is equivalent to
finding an exact confidence interval for a bounded binomial pro-
portion. Such intervals can be obtained by truncating standard
exact binomial confidence intervals, but the truncated intervals
may be empty or misleadingly short. We address this prob-
lem by using exact confidence intervals obtained by inverting a
likelihood ratio test that takes into account that the proportion
is bounded. A simple adjustment is made to keep the intervals
from being excessively conservative. An R function for comput-
ing the intervals is available as online supplementary material.
KEY WORDS: Binomial distribution; Bounded binomial pro-
portion; Likelihood ratio; Simple random sampling
1. INTRODUCTION
In surveys that ask sensitive questions, there is a danger that
respondents will refuse to respond or will answer untruthfully.
Randomized response techniques encourage truthful answers
by ensuring that the respondent’s answers do not provide any
definite sensitive information. In the original formulation of
randomized response, in Warner (1965), one is interested in
estimating the proportion of a population that belongs to a sen-
sitive group A. Since respondents who belong to group A are
unlikely to admit membership under direct questioning, a ran-
domizing device is brought in. Out of sight of the interviewer,
the respondent uses a spinner, a die, or some other device to
generate a binary outcome X that is 1 with known probability
p ̸= 1/2 and 0 otherwise. If X = 1, then the respondent truth-
fully answers the question “Do you belong to group A?”, and
if X = 0, then the respondent truthfully answers the opposite
question “Do you belong to group Ā?” A routine calculation
shows that if πA is the chance of belonging to group A and
Jesse Frey is Associate Professor, Department of Mathematics and Statistics,
Villanova University, 800 East Lancaster Avenue, Villanova, PA 19085 (E-
mail: jesse.frey@villanova.edu). Andrés Pérez is PhD student, Department of
Statistics, Temple University, 1801 N. Broad Street, Philadelphia, PA 19122
(E-mail: andres.perez@temple.edu). This work was done while A. Pérez was
a Master’s student at Villanova University. The authors thank the editor, an
associate editor, and the two referees for their comments and suggestions.
8 © 2012 American Statistical Association DOI: 10.1080/00031305.2012.663680
πyes is the chance of responding “yes” to the question, then
πyes = pπA + (1 − p)(1 − πA ).
If π̂yes is the sample proportion of “yes” answers obtained
from a simple random sample of n respondents, then one ob-
tains an unbiased estimator for πA by setting π̂A = (π̂yes − (1 −
p))/(2p − 1). However, it is possible for this unbiased estimator
to be less than 0 or bigger than 1. When we take into account that
the true πA must satisfy πA ∈ [0, 1], we find that the maximum
likelihood estimator (MLE) for πA is π̃A = median{0, π̂A , 1},
which always lies in the interval [0, 1].
Since the original work of Warner (1965), much additional
work has been done on randomized response. Techniques for
handling multichotomous and continuous variables have been
developed in articles such as Greenberg, Kuebler, Abernathy,
and Horvitz (1971), and improved methods have been pro-
posed for estimating proportions πA . Greenberg, Abul-Ela,
Simmons, and Horvitz (1969) explored the unrelated ques-
tion technique, in which Warner’s question “Do you belong
to group Ā?” is replaced by an innocuous question with a
known “yes” probability. Kuk (1990) proposed a method in
which, out of sight of the interviewer, the respondent gener-
ates two independent binary outcomes X1 and X2 with known
success probabilities p1 ̸= p2 . If the respondent belongs to
group A, then he/she reports X1 . Otherwise, he/she reports X2 .
Mangat (1994) developed a method in which the Warner ran-
domized response method is used only for members of group
Ā. Those in group A respond “yes” regardless of the value of
X. Campbell and Joiner (1973) gave an interesting introduction
to randomized response, and more extensive reviews of the ran-
domized response literature appear in van den Hout and van
der Heijden (2002) and in Chaudhuri (2011). Recent contribu-
tions to the randomized response literature include Quatember
(2009), Tan, Tian, and Tang (2009), and Nayak and Adeshiyan
(2009).
The randomized response literature includes techniques for
point estimation, techniques for estimating the variability of
a point estimate, and techniques for comparing randomized re-
sponse schemes in terms of how well they handle the tradeoff be-
tween privacy protection and efficiency of estimation. However,
it seems that relatively little work has been done on confidence
intervals. In the binomial case, the only confidence intervals that
seem to have been used are the standard Wald intervals, which
perform poorly (see Brown, Cai, and DasGupta 2001). In this
article, we develop exact confidence intervals for proportions
that are estimated using randomized response. These intervals
The American Statistician, February 2012, Vol. 66, No. 1
 P-Value Precision and Reproducibility

Dennis D. B OOS and Leonard A. S TEFANSKI

< 0.05. In this article we systematically study p-value variabil-
ity with an eye toward developing a better appreciation of its
P-values are useful statistical measures of evidence against a magnitude and potential impacts, especially those related to the
null hypothesis. In contrast to other statistical estimates, how- profusion of scientific results that fail to reproduce upon repli-
ever, their sample-to-sample variability is usually not consid- cation.
ered or estimated, and therefore not fully appreciated. Via a Under alternative hypotheses, p-values are less variable than
systematic study of log-scale p-value standard errors, bootstrap they are under null hypotheses, where for continuous cases the
prediction bounds, and reproducibility probabilities for future Uniform(0,1) standard deviation 12−1/2 = 0.29 applies. How-
replicate p-values, we show that p-values exhibit surprisingly ever, under alternatives their standard deviations are typically
large variability in typical data situations. In addition to provid- large fractions of their mean values, and thus p-values are in-
ing context to discussions about the failure of statistical results herently imprecise. Papers such as Goodman (1992) and Gel-
to replicate, our findings shed light on the relative value of ex- man and Stern (2006) address this variability, yet the subject
act p-values vis-a-vis approximate p-values, and indicate that deserves more attention because p-values play such an impor-
the use of *, **, and *** to denote levels 0.05, 0.01, and 0.001 tant role in practice. Ignoring variability in p-values is poten-
of statistical significance in subject-matter journals is about the tially misleading just as ignoring the standard error of a mean
right level of precision for reporting p-values when judged by can be misleading.
widely accepted rules for rounding statistical estimates. The science of statistics has recently come under attack for
its purported role in the failure of statistical findings to hold up
KEY WORDS: Log p-value; Measure of evidence; Prediction under the litmus test of replication. In Odds Are, It’s Wrong:
interval; Reproducibility probability. Science Fails to Face the Shortcomings of Statistics, Siegfried
(2010) highlighted Science’s love affair with p-values and ar-
gued that their shortcomings, and the shortcomings of statis-
tics more generally, are responsible for the profusion of faulty
1. INTRODUCTION
scientific claims. Pantula et al. (2010) responded on behalf of
Good statistical practice demands reporting some measure of the American Statistical Association and the International Sta-
variability or reliability for important statistical estimates. For tistical Institute, pointing out that Siegfried’s failure to distin-
example, if a population mean µ is estimated by a sample mean guish between the limitations of statistical science and the mis-
Y using an independent and identically distributed (iid) sample use of statistical methods results in erroneous conclusions about
√
Y1 , . . . ,Yn , common practice is to report a standard error s/ n the role of statistics in the profusion of false scientific claims.
or a confidence interval for µ , or their Bayesian equivalents. Lack of replication in studies and related statistical issues have
However, the variability of a p-value is typically not assessed been highlighted recently in interesting presentations by Young
or reported in routine data analysis, even when it is the primary (2008), who pointed to multiple testing, multiple methodologies
statistical estimate. (trying many different methods of analysis), and bias as partic-
More generally, statisticians are quick to emphasize that ular problems. In fact, results fail to replicate for a number of
statistics have sampling distributions and that it is important reasons. In order to correctly assess their impacts, it is necessary
to interpret those statistics in view of their sampling variation. to understand the role of p-value variability itself.
However, the caveat about interpretation is often overlooked We study three approaches to quantifying p-value variabil-
when the statistic is a p-value. With any statistic, ignoring vari- ity. First, we consider the statistician’s general purpose mea-
ability can have undesirable consequences. In the case of the p- sure of variability, the standard deviation and its estimate, the
value, the main problem is that too much stock may be placed standard error. We show that − log10 (p-value) standard devia-
in a finding that is deemed statistically significant by a p-value tions are such that for a wide range of observed significance
levels, only the magnitude of − log10 (p-value) is reliably deter-
−k
Dennis D. Boos is Professor and Associate Department Head, and mined. That is, writing the p-value as x · 10 , where 1 ≤ x < 10
Leonard A. Stefanski is Professor, Department of Statistics, North Car- and k = 1, 2, 3, . . . is the magnitude so that − log10 (p-value)
olina State University, Raleigh, NC 27695-8203 (E-mail for correspondence: = − log (x) + k, the standard deviation of − log (p-value) is
10 10
boos@stat.ncsu.edu). This work was supported by NSF grant DMS-0906421
so large relative to its value that only the magnitude k is reli-
and NIH grant P01 CA142538-01. We thank the Editor, Associate Editor, and
two referees for thoughtful comments that resulted in substantial improvements ably determined as a measure of evidence. This phenomenon is
to the article’s content and its exposition. manifest in standard errors derived from both the bootstrap and

c
⃝2011 American Statistical Association DOI: 10.1198/tas.2011.10129 The American Statistician, November 2011, Vol. 65, No. 4 213

Nicholas CHAMANDY, Omkar MURALIDHARAN, and Stefan WAGER
Modern data and applications pose very different challenges
from those of the 1950s or even the 1980s. Students contemplat-
ing a career in statistics or data science need to have the tools
to tackle problems involving massive, heavy-tailed data, often
interacting with live, complex systems. However, despite the
deepening connections between engineering and modern data
science, we argue that training in classical statistical concepts
plays a central role in preparing students to solve Google-scale
problems. To this end, we present three industrial applications
where significant modern data challenges were overcome by
statistical thinking.
KEY WORDS: Big data; Data science.
1. INTRODUCTION
Technology companies like Google generate and consume
data on a staggering scale. Massive, distributed data present
novel and interesting challenges for the statistician, and have
spurred much excitement among students, and even a new dis-
cipline: Data Science. Hal Varian famously quipped in 2009
that Statistician would be “the sexy job in the next 10 years”
(Lohr 2009), a claim seemingly backed up by the proliferation
of job postings for data scientists in high-tech. The McKinsey
Global Institute took a more urgent tone in their 2011 report
examining the explosion of big data in industry (Manyika et al.
2011). While extolling the huge productivity gains that untap-
ping such data would bring, they predicted a shortage of hun-
dreds of thousand of “people with deep analytical skills,” and
millions of data-savvy managers, over the next few years.
Massive data present great opportunities for a statistician.
Estimating tiny experimental effect sizes becomes routine, and
practical significance is often more elusive than mere statistical
significance. Moreover, the power of approaches that pool data
Nicholas Chamandy is Data Scientist, Lyft, San Francisco, CA; formerly with
Google (E-mail: ). Omkar Muralidharan is Statistician in Ads Quality, Google,
1600 Amphitheater Pkwy, Mountain View, CA 94043 (E-mail: ). Stefan Wager
is Ph.D. Candidate in Statistics, Stanford University, Sequoia Hall, 390 Serra
Mall, Palo Alto, CA 94305-4065 (E-mail: ). This article is derived from a talk
presented at the Joint Statistical Meetings on August 5, 2013, in Montreal,
during the session entitled Toward Big Data in Teaching Statistics. The authors
are grateful to Amir Najmi, Hal Varian, and three anonymous reviewers for
their many helpful suggestions. They would also like to thank Nick Horton, Jo
Hardin, and Tim Hesterberg for encouraging them to write the article. (E-mail:
nickchamandy@gmail.com).
Color versions of one or more of the figures in the article can be found online
at www.tandfonline.com/r/tas.
© 2015 American Statistical Association DOI: 10.1080/00031305.2015.1089790
Teaching Statistics at Google-Scale
across observations can be fully realized. But these exciting op-
portunities come with strings attached. The data sources and
structures that a graduating statistician (or data scientist) faces
are unlike those of the 1950s, or even the 1980s. Statistical mod-
els we take for granted are sometimes out of reach: constructing
a matrix of dimensions n by p can be pure fantasy, and outliers
are the rule, not the exception. Moreover, powerful comput-
ing tools have become a prerequisite to even reading the data.
Modern data are in general unwieldy and raw, often contami-
nated by “spammy” or machine-generated observations. More
than ever, data checking and sanitization are the domain of the
statistician.
In this context, some might think that the future of data science
education lies in engineering departments, with a focus on build-
ing ever more sophisticated data-analysis systems. Indeed, the
American Statistical Association Guidelines Workgroup noted
the “increased importance of data science” as the leading Key
Point of its 2014 Curriculum Guidelines (ASA 2014). As Di-
ane Lambert and others have commented, it is vital that today’s
statisticians have the ability to “think with data” (Hardin et al.
2014). We agree wholeheartedly with the notion that students
must be fluent in modern computational paradigms and data
manipulation techniques. We present a counterbalance to this
narrative, however, in the form of three data analysis challenges
inspired by an industrial “big data” problem: click cost estima-
tion. We illustrate how each example can be tackled not with
fancy computation, but with new twists on standard statistical
methods, yielding solutions that are not only principled, but also
practical.
Our message is not at odds with the ASA’s recent recom-
mendations; indeed, the guidelines highlight statistical theory,
“flexible problem solving skills,” and “problems with a sub-
stantive context” as core components of the curriculum (ASA
2014). The methodological tweaks presented in this article are
not particularly advanced, touching on well-known results in the
domains of resampling, shrinkage, randomization, and causal
inference. Their contributions are more conceptual than the-
oretical. As such, we believe that each example and solution
would be accessible to an undergraduate student in a statistics
or data science program. In relaying this message to such a stu-
dent, it is not the specific examples or solutions presented here
that should be stressed. Rather, we wish to emphasize the value
of a solid understanding of classical statistical ideas as they ap-
ply to modern problems in preparing tomorrow’s students for
large-scale data science challenges.
In many cases, the modern statistician acts as an interface
between the raw data and the consumer of those data. The term
“consumer” is used rather broadly here. Traditionally, it would
include the key decision-makers of a business, and perhaps the
The American Statistician, November 2015, Vol. 69, No. 4 283
 Three Examples of Accurate Likelihood Inference
C. L OZADA -C AN and A. C. DAVISON
The modern theory of likelihood inference provides im-
proved inferences in many parametric models, with little more
effort than is required for application of standard first-order the-
ory. We outline the relevant computations, and illustrate the cal-
culations using a dilution assay, a zero-inflated Poisson regres-
sion model, and a short time series. In each case the effect of
the higher order correction can be appreciable.
KEY WORDS: Autoregression; Bias reduction; Dilution as-
say; Higher order asymptotics; Likelihood; Zero-inflated Pois-
son distribution.
1. INTRODUCTION
Likelihood is a mainstay of statistical modeling. Inference
in many applications is based on familiar large sample the-
ory for the maximum likelihood estimator and the likelihood
ratio statistic (Cox and Hinkley 1974, chap. 9). This theory
involves approximations that may be justified when the sam-
ple size is large, but simulation is often needed to establish
whether a given sample is sufficiently large in a new applica-
tion. Thus the basic theory is often applied in cases where its
performance is unclear, either because the sample is small but
no better approach is readily available, or because although the
sample appears large, the information content per observation is
much smaller than is thought. The best-known situation where
this second issue arises is in logistic regression, for which spe-
cial approximations have been widely studied (Davison 1988;
Strawderman and Wells 1998). It is not widely appreciated that
standard likelihood theory can be readily improved and that the
corresponding computations are relatively straightforward. The
purpose of this article is to illustrate these ideas in three sit-
uations of increasing complexity: a single-parameter dilution
assay; a zero-inflated Poisson regression model; and an autore-
gressive time series.
2. MODERN LIKELIHOOD THEORY
2.1 Basic Notions
We assume a parametric model with probability density func-
tion f (y; θ ), with a d-dimensional parameter θ and a vector
Claudia Lozada-Can is a Postdoctoral Researcher (E-mail: Claudia_
Lozada@live.com) and Anthony Davison is Professor of Statistics (E-mail:
Anthony.Davison@epfl.ch), Institute of Mathematics, Ecole Polytechnique
Fédérale de Lausanne, IMA-FSB-EPFL, Station 8, 1015 Lausanne, Switzer-
land. The work was supported by the Swiss National Science Foundation
through the National Centre for Competence in Research on Plant Survival
(http://www2.unine.ch/nccr). We are grateful to the referee, associate editor and
editor, and to Alessandra Brazzale and Nancy Reid for their helpful comments
on the work.
© 2010 American Statistical Association DOI: 10.1198/tast.2010.09004
of continuous responses, y = (y1 , . . . , yn ). The log-likelihood,
ℓ(θ ) = log f (y; θ ), is maximized by the maximum likelihood
estimator ! θ . Under standard regularity conditions ! θ has an ap-
proximate normal distribution with mean θ and variance ma-
θ )−1 , where ȷ (θ ) = −∂ 2 ℓ(θ)/∂θ ∂θ T is the observed in-
trix ȷ (!
formation matrix, and the Wald pivot {ȷ (! θ )}1/2 (!
θ − θ ) has a
standard normal distribution. The approximate normal distrib-
ution of ! θ provides the most common basis for inference about
elements of θ , but the implicit assumption that the estimator
is symmetrically distributed around its target means that the
resulting confidence intervals for components of θ tend to be
centered wrongly, or equivalently that true significance levels
for one-sided tests on such components may differ substan-
tially from their nominal values, because asymmetry of the log-
likelihood about its maximum is not accommodated. Moreover,
these inferences are not invariant to transformations of the pa-
rameters.
Write θ = (ψ, λ), where ψ is a scalar component of θ for
which inference is required and λ represents the remaining
components, and let ! θ = (ψ!,!
λ) and ! θψ = (ψ,! λψ ) respectively
denote the overall maximum likelihood estimator and the maxi-
mum likelihood estimator when ψ is held fixed. In this notation
a preferable basis for inference on ψ is the likelihood root
" #1/2
r(ψ) = sign(ψ ! − ψ) 2{ℓ(! θ ) − ℓ(!θψ )} , (1)
which takes potential asymmetry of the log-likelihood into ac-
count, and may be treated as an N (0, 1) variable. The quantity
r(ψ)2 is the familiar likelihood ratio statistic. When testing the
null hypothesis that ψ = ψ0 against the one-sided hypothesis
that ψ > ψ0 , we regard small values of the p-value '{r(ψ0 )}
as casting doubt on the null hypothesis; here and below we use
' to denote the cumulative probability function of the standard
normal distribution. When θ = ψ is scalar, λ disappears from
the expressions above and ȷ (θ ) is scalar. It is easy to verify that
apart from a possible sign change, r(ψ) is invariant to repara-
meterizations of the form (ψ, λ) "→ {g(ψ), h(ψ, λ)}, in which
g and h are bijective, so-called interest-respecting reparameter-
izations.
2.2 Higher Order Approximations
Improved likelihood inferences may be obtained through
higher order asymptotics, on which there is a large literature
summarized in books by Barndorff-Nielsen and Cox (1994),
Pace and Salvan (1997), Severini (2000), and Brazzale, Davi-
son, and Reid (2007), and in review articles such as Reid
(2003). One basic formula is the so-called p ∗ approximation to
the density of the maximum likelihood estimator conditioned
on an ancillary statistic (Barndorff-Nielsen 1980, 1983, 1986),
manipulation of which yields the modified likelihood root
$ %
1 q(ψ)
r ∗ (ψ) = r(ψ) + log , (2)
r(ψ) r(ψ)
The American Statistician, May 2010, Vol. 64, No. 2 131
 Use of R as a Toolbox for Mathematical Statistics Exploration
Nicholas J. HORTON, Elizabeth R. BROWN, and Linjuan QIAN
The R language, a freely available environment for statistical
computing and graphics is widely used in many fields. This
“expert-friendly” system has a powerful command language
and programming environment, combined with an active user
community. We discuss how R is ideal as a platform to sup-
port experimentation in mathematical statistics, both at the un-
dergraduate and graduate levels. Using a series of case studies
and activities, we describe how R can be used in a mathemati-
cal statistics course as a toolbox for experimentation. Examples
include the calculation of a running average, maximization of
a nonlinear function, resampling of a statistic, simple Bayesian
modeling, sampling from multivariate normal, and estimation of
power. These activities, often requiring only a few dozen lines
of code, offer students the opportunity to explore statistical con-
cepts and experiment. In addition, they provide an introduction
to the framework and idioms available in this rich environment.
KEY WORDS: Mathematical statistics education; Statistical
computing.
1. INTRODUCTION
The R language (Ihaka and Gentleman 1996; Hornik 2004)
is a freely available environment for statistical computing and
graphics. It allows the handling and storage of data, supports
many operators for calculations, provides a number of tools for
data analysis, and features excellent graphical capabilities and
a straightforward programming language. R is widely used for
statistical analysis in a variety of fields, and boasts a large number
of add-on packages that extend the system.
This article considers the use of R not for analysis, but as a
toolbox for exploration of mathematical statistics. Following the
approach of Baglivo (1995), we introduce examples to illustrate
how R can be used to experiment with concepts in mathematical
statistics. In addition to providing insight into the underlying
mathematical statistics, these example topics provide an intro-
duction to R syntax, capabilities and idioms, and the power of
this environment.
We begin by providing some motivation for the importance
of statistical computing environments as a component of math-
ematical statistics education. Section 2 reviews the history of R
(and its connection to S/S-Plus), details resources and documen-
Nicholas J. Horton is Assistant Professor, Department of Mathematics, Smith
College, Northampton, MA 01063 (E-mail: nhorton@email.smith.edu). Eliz-
abeth R. Brown is Research Assistant Professor, Department of Biostatistics,
University of Washington, Seattle, WA. Linjuan Qian is Undergraduate Re-
search Assistant, Department of Mathematics, Smith College, Northampton,
MA 01063. We are grateful to Ken Kleinman and Paul Kienzle for comments
on an earlier draft of the article, and for the support provided by NIMH grant
R01-MH54693 and a Career Development Fund Award from the Department of
Biostatistics at the University of Washington.
© 2004 American Statistical Association DOI: 10.1198/000313004X5572
tation available for the system, and describes an introductory R
session. Each of the activities in Section 3 include a listing of the
R commands and output, and a description to help introduce new
syntax, structures, and idioms. Finally, we conclude with some
overall comments about R as an environment for mathematical
statistics education.
Over the past decade, there has been increasing consensus re-
garding the importance of computing skills as a component of
statistics education. Numerous authors have described the im-
portance of computing support for statistics, in the context of
numerous curricular initiatives. As an example, Moore (2000)
featured papers on technology that fall within the mantra of
“more data, less lecturing” (Cobb 1991). Curriculum guidelines
for undergraduate programs in statistical science from the Amer-
ican Statistical Association require familiarity with a standard
software package and should encourage study of data manage-
ment and algorithmic problem-solving (American Statistical As-
sociation 2004). While we are fully in agreement with the need
for software packages to teach introductory statistics courses, in
this article, we focus on aspects relating to algorithmic problem-
solving, at the level of a more advanced course.
We also make a distinction between statistical computing and
numerical analysis from the toolbox (or sandbox or testbed) for
statistical education that we describe. Although there is a cru-
cial role for statisticians with appropriate training in numerical
analysis, computer science, graphical interfaces, and database
design, in our experience relatively few students emerge from
undergraduate or graduate statistics programs with this train-
ing. While somewhat dated, Eddy, Jones, Kass, and Schervish
(1987) considered the role of computational statistics in grad-
uate statistical education in the late 1980s. A series of articles
in the February 2004 issue of The American Statistician (Gen-
tle 2004; Monahan 2004; Lange 2004) readdress these issues.
Lange (2004) described computational statistics and optimiza-
tion courses that “would have been unrecognizable 20 years ago”
[though considerable overlap is noted with topics given by Eddy
et al. (1987)]. Gentle (2004) listed six areas of expertise valuable
for statisticians:
1. data manipulation and analysis
2. symbolic computations
3. computer arithmetic
4. programming
5. algorithms and methods
6. data structures
Most of these topics, while quite valuable, often require addi-
tional training beyond the standard prerequisites for a mathemat-
ical statistics class (Monahan 2004; Gentle 2004; Lange 2004).
The use of “rapid prototyping” and “quick-and-dirty” Monte
Carlo studies (Gentle 2004) to better understand a given setting
is particularly relevant for mathematics statistics classes (not
just statistical computing). For many areas of modern statistics
(e.g., resampling based tests, Bayesian inference, smoothing,
The American Statistician, November 2004, Vol. 58, No. 4 343
 General
“Not Only Defended But Also Applied”: The Perceived Absurdity
of Bayesian Inference
Andrew GELMAN and Christian P. ROBERT
The missionary zeal of many Bayesians of old has been
matched, in the other direction, by an attitude among some
theoreticians that Bayesian methods were absurd—not merely
misguided but obviously wrong in principle. We consider sev-
eral examples, beginning with Feller’s classic text on probability
theory and continuing with more recent cases such as the per-
ceived Bayesian nature of the so-called doomsday argument. We
analyze in this note the intellectual background behind various
misconceptions about Bayesian statistics, without aiming at a
complete historical coverage of the reasons for this dismissal.
KEY WORDS: Bogosity; Doomsdsay argument; Foundations;
Frequentist; Laplace law of succession.
1. A VIEW FROM 1950
Younger readers of this journal may not be fully aware of
the passionate battles over Bayesian inference among statisti-
cians in the last half of the twentieth century. During this pe-
riod, the missionary zeal of many Bayesians was matched, in
the other direction, by a view among some theoreticians that
Bayesian methods are absurd—not merely misguided but obvi-
ously wrong in principle. Such anti-Bayesianism could hardly
be maintained in the present era, given the many recent practical
successes of Bayesian methods. But by examining the historical
background of these beliefs, we may gain some insight into the
statistical debates of today.
Andrew Gelman, Departments of Statistics and Political Science, Columbia Uni-
versity, New York, NY 10027 (E-mail: gelman@stat.columbia.edu). Christian
P. Robert, Université Paris-Dauphine, CEREMADE, IUF, and CREST, Paris,
France (E-mail: xian@ceremade.dauphine.fr). We thank David Aldous, Ronald
Christensen, the Associate Editor, and two reviewers for helpful comments. In
addition, the first author (AG) thanks the Institute of Education Sciences, Depart-
ment of Energy, National Science Foundation, and National Security Agency
for partial support of this work. He remembers reading with pleasure much of
Feller’s first volume in college, after taking probability but before taking any
statistics courses. The second author’s (CPR) research is partly supported by
the Agence Nationale de la Recherche (ANR, 212, rue de Bercy 75012 Paris)
through the 2007–2010 grant ANR-07-BLAN-0237 “SPBayes.” He remembers
buying Feller’s first volume in a bookstore in Ann Arbor during a Bayesian
econometrics conference where he was kindly supported by Jim Berger.
© 2013 American Statistical Association DOI: 10.1080/00031305.2013.760987
We begin with a Note on Bayes’ rule that appeared in William
Feller’s classic probability text:
Unfortunately Bayes’ rule has been somewhat discredited
by metaphysical applications of the type described above. In
routine practice, this kind of argument can be dangerous. A
quality control engineer is concerned with one particular ma-
chine and not with an infinite population of machines from
which one was chosen at random. He has been advised to
use Bayes’ rule on the grounds that it is logically acceptable
and corresponds to our way of thinking. Plato used this type
of argument to prove the existence of Atlantis, and philoso-
phers used it to prove the absurdity of Newton’s mechanics.
In our case it overlooks the circumstance that the engineer
desires success and that he will do better by estimating and
minimizing the sources of various types of errors in predict-
ing and guessing. The modern method of statistical tests and
estimation is less intuitive but more realistic. It may be not
only defended but also applied. W. Feller, 1950 (pp. 124–125
of the 1970 edition)
Feller believed that Bayesian inference could be defended
(i.e., supported via theoretical argument) but not applied to
give reliable answers to problems in science or engineering,
a claim that seems quaint in the modern context of Bayesian
methods being used in problems from genetics, toxicology, and
astronomy to economic forecasting and political science. As we
discuss below, what struck us about Feller’s statement was not
so much his stance as his apparent certainty.
One might argue that, whatever the merits of Feller’s state-
ment today, it might have been true back in 1950. Such a claim,
however, would have to ignore, for example, the success of
Bayesian methods by Turing and others in code breaking dur-
ing the Second World War, followed up by expositions such as
Good (1950), as well as Jeffreys’s Theory of Probability, which
came out in 1939. Consider this recollection from physicist and
Bayesian E. T. Jaynes:
When, as a student in 1946, I decided that I ought to learn
some probability theory, it was pure chance which led me to
take the book Theory of Probability by Jeffreys, from the li-
brary shelf. In reading it, I was puzzled by something which,
I am afraid, will also puzzle many who read the present book.
Why was he so much on the defensive? It seemed to me that
Jeffreys’ viewpoint and most of his statements were the most
obvious common sense, I could not imagine any sane person
disputing them. Why, then, did he feel it necessary to insert
The American Statistician, February 2013, Vol. 67, No. 1 1
 Supplementary materials for this article are available online. Please go to www.tandfonline.com/r/TAS
Statistical Computing and Graphics
Visualizing Longitudinal Data With Dropouts
Mithat GÖNEN
This article proposes a triangle plot to display longitudinal
data with dropouts. The triangle plot is a tool of data visual-
ization that can also serve as a graphical check for informative-
ness of the dropout process. There are similarities between the
lasagna plot and the triangle plot, but the explicit use of dropout
time as an axis is an advantage of the triangle plot over the more
commonly used graphical strategies for longitudinal data. It is
possible to interpret the triangle plot as a trellis plot, which gives
rise to several extensions such as the triangle histogram and the
triangle boxplot. R code is available to streamline the use of the
triangle plot in practice. Supplementary materials for this article
are available online.
KEY WORDS: Data visualization; Graph; Informative
dropout; Trellis plots; Triangle plot.
1. INTRODUCTION
Data visualization is an essential component of data analysis.
As stated in Cleveland (1993), “It provides a front line of attack,
revealing intricate structure in data that cannot be absorbed in
any other way. We discover unimagined effects and we challenge
imagined ones.” It is widely argued that graphical displays of
information are more efficient in capturing the readers’ attention
and result in a higher retention rate of the messages delivered
in an article or presentation. In fact the American Statistical
Association Style Guide (2011) states that “When feasible, put
important conclusions into graphical form. Not everyone reads
an entire article from beginning to end. When readers skim an
article, they are drawn to graphs. Try to make the graphs and
their captions tell the story of your article.”
This article is concerned with the longitudinal setting where
subjects contribute data over multiple time points. Dropout in
longitudinal data is common. It is rare that subjects drop out
at random. If the dropout process is informative, data analysis
needs to reflect this accordingly. The literature on the various
methods proposed to take into account informative dropouts is
very rich. Verbeke and Molenberghs (2000) and Little (2008)
are good overviews of this literature.
Mithat Gönen, Memorial Sloan-Kettering Cancer Center, 1275 York Avenue
Box 44, New York, NY 10065 (E-mail: gonenm@mskcc.org)
© 2013 American Statistical Association DOI: 10.1080/00031305.2013.785980
There are several graphical methods for displaying various
features of longitudinally collected data such as plotting the
means over time, event charts, follow-up PLOTs (FU-PLOTs),
spaghetti plots, and lasagna plots. Plotting the means, arguably
the most commonly used method in practice, is simple to un-
derstand and communicate but can hide the salient features of
the dropout process.
Event charts (Goldman 1992) and extensions (Lee, Hess, and
Dubin 2000; Atherton et al. 2003) display the timing of multiple
events of clinical interest. Related to the event chart is the FU-
PLOT of Lesser et al. (1995), which shows the timing of visits
or data collection (such as a blood draw). Event charts and the
FU-PLOT can be helpful in visualizing the dropout process, but
they are disconnected from the measured values and hence they
cannot guide the analyst as to how informative dropouts are.
The spaghetti plot is simply a longitudinal profile of each
patient, usually with a linear interpolation between the time
points. It is useful for small datasets where the patients can be
categorized in a few groups that can be coded either by color or
by a plotting symbol. With large datasets or a large number of
groups, the plot quickly becomes uninterpretable. The lasagna
plot (Swihart et al. 2010), a cousin of the spaghetti plot, displays
the longitudinal information in color-coded layers much like a
heatmap, with subjects as rows and time as columns. In the last
section, we will mention a connection between the lasagna plot
and the triangle plot.
All of these plots have features that have made them useful
in displaying various aspects of longitudinal data but none deals
with the issue of informative dropout. The goal of this article is to
close this gap using the triangle plot that retains the longitudinal
aspect of the data while uncovering the amount of information in
the dropout process. The triangle plot is most useful in situations
where subjects dropping out do not return, leading to monotone
missingness. This is further discussed in Section 5.
The rest of the article is organized as follows: Section 2 intro-
duces the triangle plot and Section 3 presents an interpretation
as a Trellis plot. Triangle plots from a study of quality of life
(QOL) in patients with gastric cancer are shown in Section 4.
The article is concluded with the discussion in Section 5.
2. TRIANGLE PLOT
The observed data will be denoted by Xit , where i =
1, . . . , n indexes the subject and t = 1, . . . , T denotes all the
possible time points at which measurements may be taken. Also
The American Statistician, May 2013, Vol. 67, No. 2 97