The American Statistician

ISSN: 0003-1305 (Print) 1537-2731 (Online) Journal homepage: http://www.tandfonline.com/loi/utas20

The Second Course in Statistics: Design and

Analysis of Experiments?

Natalie J. Blades, G. Bruce Schaalje & William F. Christensen

To cite this article: Natalie J. Blades, G. Bruce Schaalje & William F. Christensen (2015) The
Second Course in Statistics: Design and Analysis of Experiments?, The American Statistician,
69:4, 326-333, DOI: 10.1080/00031305.2015.1086437

To link to this article: http://dx.doi.org/10.1080/00031305.2015.1086437

Accepted author version posted online: 11

Sep 2015.
Published online: 29 Dec 2015.

Submit your article to this journal

Article views: 1471

View related articles

View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=utas20

Download by: [CAPES] Date: 15 October 2016, At: 20:07

 The Second Course in Statistics: Design and Analysis of Experiments?
Natalie J. BLADES, G. Bruce SCHAALJE, and William F. CHRISTENSEN
Statistics departments are facing rapid growth in enrollments
and increases in demand for courses. This article discusses the
use of design and analysis of experiments (DAE) as a nontermi-
nal second course in statistics for undergraduate statistics ma-
jors, minors, and other students seeking exposure to the practice
of statistics beyond the introductory course. DAE is a gateway
to approaching statistical thinking as data-based problem solv-
ing by exposing students to statistical, computational, data, and
communication skills in the second course. Given the somewhat
antiquated view of design and deemphasis of classical design of
experiments topics in the new ASA curriculum guidelines, DAE
may seem an odd choice for the second course; however, it ex-
poses students to the breadth of the statistical problem-solving
process, explores foundational issues of the discipline, and is
accessible to students who have not yet finished their advanced
mathematical training. These skills remain essential in the data
science era as students must be equipped to understand the po-
tential and peril of found data using the principles of design.
While DAE may not be the appropriate second course for all
statistics programs, it provides a strong foundation for causal
inference and experimental design for students pursuing a B.S.
in Statistics in a program housed in a department of statistics.
KEY WORDS: ANOVA; Design of experiments; Statistical ed-
ucation; Undergraduate curriculum
1. INTRODUCTION
The emergence of big data from every sector of the global
economy demands that future scientists, educators, managers,
and policy makers are equipped to manage, understand, and act
upon that data deluge. This increase in demand for data sci-
entists, the new catchphrase for statisticians and data-literate
analysts, preceded a huge growth in enrollments in statistics
courses and a displacement of calculus as the lingua franca of
science and commerce (Meng 2009; Utts 2015; Wasserstein
2015). Introductory statistics enrollments in the United States
more than doubled from 118,700 students in 1990 to 312,000
Natalie J. Blades (E-mail: blades@stat.byu.edu) is Associate Professor, G.
Bruce Schaalje (E-mail: schaalje@stat.byu.edu) is Professor, and William F.
Christensen (E-mail: william@stat.byu.edu) is Professor, Department of Statis-
tics, Brigham Young University, Provo UT 84602. The authors thank the editor,
the associate editor, and the reviewers for many useful suggestions. The authors
also thank H. Dennis Tolley and Del T. Scott, current and former chairs of their
department, for facilitating development of this second course.
326 2015 American Statistical Association DOI: 10.1080/00031305.2015.1086437
in 2010. In this same time period, enrollments in first semester
calculus dropped from 349,300 to 334,000 students; introduc-
tory statistics enrollments have increased from 34% of calculus
enrollments in 1990 to 93% of calculus enrollments in 2010
(Albers et al. 1992; Blair, Kirkman, and Maxwell 2013a, b).
This growth is also observed in the nations high schools: AP
Statistics lags behind only AP Calculus and AP Biology among
the STEM AP exams offeredand it is growing twice as fast as
the AP Calculus exam (The College Board 2011, 2013).
This growth in introductory statistics enrollments, together
with a developing understanding of pedagogy, has led to changes
in the introductory statistics curriculum. The introductory statis-
tics curriculum is delivered in many ways; however, the gen-
eral consensus converged upon over the last half century holds
that the first course in statistics should roughly cover descrip-
tive statistics, elementary sampling and design of experiments,
elementary probability, the sampling distribution of the sam-
ple mean, confidence intervals, hypothesis tests, one- and two-
sample problems for means and proportions, and inference for
simple linear regression. While this consensus is documented
with little variation in the contents of most standard textbooks
(Freedman, Pisani, and Purves 2007; Agresti and Franklin 2012;
DeVeaux, Velleman, and Bock 2012; Moore and McCabe 2012),
recent efforts have emphasized: use of active learning techniques
to focus on broadly defined conceptual understanding of statis-
tics; statistical literacy and statistical thinking using real data
rather than rote cookbook knowledge of statistical procedures
(ASA 2005); module-based approaches that use authentic prob-
lems to drive and organize the required statistical skills (Lock
and Meng 2010); and exposure to multivariate statistics (De
Veaux 2015).
The standard introductory statistics topics have served as an
introduction to an often unexpectedly useful subject for students
in a broad variety of majors and with a wide range of mathe-
matical competence. These introductory topics have spurred
interest in further study of statistics for those with quantita-
tive aptitudeand the media is promising rewards for people
trained in increasingly sophisticated statistical methods (Lohr
2009; Davidian and Louis 2012). This trend is already manifest
here at Brigham Young University (BYU) where record num-
bers of incoming first-year students are declaring statistics as a
major (46 and counting this year) and annual enrollment in our
second course has tripled from 60 to 180 since 2010; nation-
wide, statistics has been the fastest growing STEM major since
2010 (ASA 2015).
A consensus has not, however, developed about what to do
next with these students. What should be the content of a sec-
ond course? Even if first-year students have had AP Statis-
tics and AP Calculus, many programs still require additional
The American Statistician, November 2015, Vol. 69, No. 4
mathematical preparation for courses like mathematical statis-
tics or regression analysis. If students are excited by statistics, it
seems a pity to abandon them to a year or two of pure mathemat-
ics before education in statistics resumes. Even if their interest
in statistics does not wane during this period, students are liable
to develop the attitude that statistics is just a subset of mathe-
matics. The foundational fact that statistics forces a brave but
risky confrontation of mathematics with the real world can be
swamped or lost.
The purpose of this article is to propose design and analy-
sis of experiments (DAE) as an especially appropriate second
course for undergraduate statistics majors and to relate our ex-
periences with DAE as the second course over the last 4 years.
This proposal may appear to be either radical by its departure
from most programs or archaic in its foundational emphasis on
design. Undoubtedly some statistics programs use DAE as the
second course (though none of the top ten programs as defined
by number of statistics majors do) but the topic deserves fresh
eyes and fresh discussion.
2. CURRENT SECOND COURSES IN STATISTICS
While the introductory statistics course prepares students as
critical consumers of data and analyses, it is insufficient to pre-
pare students to apply statistical methods in their own work
(ASA 2005). Students may understand basic statistical concepts
after a first course but they have not mastered the application
of those ideas. The second course must begin to equip students
with tools to produce rather than merely consume statistical
analyses. The expected learning outcomes of the second course
are not as standardized as those for the first course in statistics;
these learning outcomes will depend on whether the second
course is the terminal statistics course of the students under-
graduate training or the next step in an undergraduate statistics
core for a statistics or analytics degree and they must reflect
whether advanced training in mathematics or another discipline
is presupposed.
Second courses in statistics have been developed for many
audiences: second courses for students in statistics majors, mi-
nors, and concentrations, second courses for math majors, sec-
ond courses for economics and psychology and sociology ma-
jors, and second courses for graduate students in a wide variety
of disciplines. The new ASA curriculum guidelines (2014) ac-
knowledge, There is not a single definition of what is appro-
priate as a second course in statistics, and a number of different
options can be found at many institutions. This is a big space
that is dominated by mathematical statistics: More than half
of nonintroductory statistics courses are taught in math depart-
ments and 72% of the nonintroductory statistics courses taught
in math departments are mathematical statistics or probability
(Blair, Kirkman, and Maxwell 2013a, b).
A terminal second course in statistics should be focused on
breadth, not depth; it should give students tools for building
models with quantitative or categorical responses and quantita-
tive or categorical predictors. The relatively small (but growing)
number of textbooks with the title or subtitle Second Course
in Statistics apparently without exception imply that the ap-
propriate topic for a second course is regression analysis. From
Lovedays (1961) Statistics a Second Course and Mosteller and
Tukeys eclectic and inimitable Data Analysis and Regression:
A Second Course in Statistics (1977) through Regression: A Sec-
ond Course in Statistics (Wonnacott and Wonnacott 1986) and
Regression with Graphics: A Second Course in Applied Statis-
tics (Hamilton 1991) to Dielmans Applied Regression Analysis:
A Second Course in Business and Economic Statistics (2004),
Lomaxs Statistical Concepts (2007), and the most recent edition
of Mendenhall and Sincichs A Second Course in Statistics: Re-
gression Analysis (2011), the titles and subtitles of the last three
decades point to the pervasiveness of this opinion; however,
these texts presuppose a potentially outdated view of the math-
ematical preparation of students taking a second coursethat
these students already have the mathematical maturity encapsu-
lated in integral and differential calculus, multivariate calculus,
and linear algebra.
The last few years have seen the publication of several
new textbooks updating the regression curriculum for second
courses with progressive active-learning approaches: Kuiper and
Sklars Practicing Statistics: Guided Investigations for the Sec-
ond Course (2012) aims to provide a case-based introduction to
statistical modeling and Cannon et al.s STAT2: Building Mod-
els for a World of Data (2012) provides a modular approach
to linear regression, analysis of variance (ANOVA) and exper-
imental design, and logistic regression. By focusing on broad
conceptual understanding these texts introduce simple and mul-
tiple regression, logistic regression, ANOVA, and experimental
design without including the mathematical details: both texts
present the material assuming students have only an exposure
to algebra.
In addition to these regression approaches there are many
other possibilities that could be considered for a second course:
introduction to probability, statistical computing, data science,
data scraping, or algorithmic predictive modeling. Some pro-
grams are now allowing for flexibility in the entry into the
major: the undergraduate statistics major at Harvard can choose
to take linear models or data science or finance after an intro-
ductory statistics courseor he or she may take probability as
the first course; at Berkeley a student may choose computing
with data, concepts of probability, or concepts of statistics as a
second course. In our own program, while most students take
the DAE course described here as the second course, students
could instead take discrete probability (an option popular among
students who would like to prepare for the actuarys probability
exam) or an introduction to statistical computing (in either R or
SAS) as his or her second course; Figure 1 briefly displays these
entry points into the statistics major at BYU. A second course
that focused on algorithmic modeling would address Breimans
(2001) concern regarding the heavy emphasis on data models
in our profession; Draper (1987, 1995) also called for more
predictive modeling to identify how the past and future are
connected by comparing predictions to observable reality and
Cobb (2015) further addressed this conflict between algorithmic
modeling and generative data models.
When the second course is not terminal, but rather part of
the statistics core for a B.S. in Statistics, the material could be
presented at greater depth. Rather than a survey of advanced
methods taught without accompanying mathematical detail, the
The American Statistician, November 2015, Vol. 69, No. 4 327
Figure 1. Entry points into the B.S. in Statistics at BYU. After the introductory course, students may enter the undergraduate core through
a methods sequence, a theory sequence, a computing sequence, or a mathematics sequence. Arrows represent prerequisites; the first classes
in methods, theory, and computing do not require prerequisite calculus. All tracks must be completed but the order can be flexibly adapted to
the interests, preferences, and experience of the student. After completion of the 27-credit core students take roughly 24 upper-division credits
that could include undergraduate courses in Bayesian statistics, survey sampling, experimental design, nonparametric statistics, quality control,
reliability, and survival analysis as well as classes from computer science, mathematics, and areas of application.

course could facilitate a deeper understanding in what will be
the second of many applied and theoretical statistics courses.
Such an approach would sacrifice breadth in the second course;
however, that breadth will be acquired by the end of the program.
Additionally, the appropriate second course must work within
the structure of the university. At BYU we are concerned with the
curriculum for the second course for statistics majors housed in
a department of statistics with 19 full-time tenure-track faculty.
Many of our majors have not finished calculus at the time they
declare a major in statistics. We require roughly 50 credits for
a B.S. Statistics degree (out of the 120 credits required for
graduation). This allows for more extensive study beyond the
introductory course: a required core of 6 credits of methods, 6
credits of theory, 6 credits of computing, 814 credits of math,
and 24 upper-division credits. In this context, we suggest DAE
has proven to be an appropriate second course.
3. DESIGN OF EXPERIMENTS AS THE
SECOND COURSE
calculus and linear algebra as part of the course, require concur-
rent enrollment, and so forth.
In 2010, a second course in DAE was developed to provide
a platform for students with only a single introductory statistics
course (possibly AP Statistics) to start understanding fundamen-
tal concepts of statistics. This DAE-based second course covers
the scientific method, statistical thinking, sources of variation,
randomized factorial designs, power, and sample size calcula-
tions. Students are encouraged to enroll concurrently in differen-
tial calculus; students who have already completed differential
calculus are encouraged to enroll in integral or multivariate cal-
culus, as appropriate. This second-course DAE covers the ma-
terial in sufficient depth that an undergraduate statistics major
would have appropriate familiarity with the design of experi-
ments without requiring additional courses; students who wish
to explore design of experiments further are well prepared for
an elective course (using, e.g., Lawson 2014).
The expected learning outcomes for second-course DAE
include

From the founding of BYUs Department of Statistics (with
concomitant creation of a B.S. in Statistics) in 1960 until 2010,
the second course in statistics at BYU was regression analysis.
The specific content of the second course drifted as textbooks
and software changed, but not in a major way. The question of
how to balance the introduction of interesting statistical topics
with the need for mathematical rigor was initially not a problem
because most statistics majors were sophomore- or junior-level
transfer students from engineering or mathematics who had al-
ready completed multivariate calculus and linear algebra. As
the major began to attract first-year students, this balance prob-
lem was approached in several ways (none of which were fully
satisfactory): enforce prerequisites that delayed students from
taking the second course until multivariate calculus and lin-
ear algebra had been completed, cover requisite concepts from
1. Defining the experimental unit, response variable, factor(s),
and level(s) of a basic experiment;
2. Understanding the role of randomization and replication in
inferring causation;
3. Performing a completely randomized design and construct-
ing the ANOVA table in SAS and R;
4. Computing the minimum number of replicates in a com-
pletely randomized design to achieve a given level of power;
5. Computing pairwise tests of differences in means in SAS
and R to understand a significant overall F-test;
6. Performing a randomized complete block design and con-
structing the ANOVA table in SAS and R;
7. Performing a factorial design and constructing the ANOVA
table in SAS and R;
8. Explaining a statistically significant interaction;

328 Statistics and the Undergraduate Curriculum

 9. Describing the differences between a split plot and a two-
way ANOVA; and
10. Working in teams to write a technical report and make a
technical presentation of a designed experiment.
Class begins with an intense two-week review of introduc-
tory statistics; the focus is on the two-sample t-test and con-
fidence intervalbut developing them in greater detail and
using them as a springboard for deeper discussion of power
and sample size. In weeks 3 and 4, the students cover prin-
ciples of experimentation and experimental design including
planning a three-factor experiment; this provides a platform for
discussing sources of variability and how to choose the response,
treatments, and experimental units. Week 5 is spent setting up
ANOVA tables, partitioning sums of squares, and performing
F tests. Weeks 6 and 7 focus on randomization and balanced
two-factor designs. Weeks 8 and 9 cover multi-way ANOVA
including interpreting interactions and main effects. Weeks 10
and 11 cover blockingincluding randomized blocks, split-plot
designs, nesting, crossing, and latin squares. Week 12 is spent
discussing multiple comparisons and contrasts.
While the ability to perform analyses in SAS and R is in-
cluded as a learning outcome, second-course DAE is not meant
to teach statistical software: software programs are used to offer
students exposure to SAS and R and to serve as tools for teach-
ing the structure of statistical problems. Students are given SAS
and R code snippets to use in their homework that require very
minor modification. This indirect, immersive introduction to
SAS and R takes students from zero to at-least-familiar in only
two months. At the end of second-course DAE, students can
analyze a continuous response variable from one-way, blocked,
crossed, and even balanced split-plot designs in both SAS and R;
this includes inputting and transforming the data and performing
model diagnostics. Statistical computing is explicitly covered in
a 6-credit sequence that students may begin while enrolled in
DAE, and it is expanded and reinforced in subsequent applied
courses.
One of the culminating experiences of DAE is a project where
the students synthesize the course material and gain experience
in the practice of statistics by designing, executing, analyzing,
and presenting the results of their own experiments. Students
see the practical issues of turning a research question into a
clearly defined experiment and grapple with amorphous issues
related to choosing factor levels, sample size, blocking, and so
forth. After refining these questions in several meetings with the
instructor, execution of the experiment is fairly straightforward.
Students report that they feel much more confident participating
in later research having had to wade through practical problems
on their own in DAE. We also employ preliminary structured
projects where the students use catapults, windmills, helicopters,
or activities from Stat2Labs (Kuiper 2015) to conduct their
own experiments. Sometimes we use one of these structured
experiments in class to solidify rudimentary concepts of ran-
domization, data collection, and moving data into software. On
occasion we have used the catapults multiple times throughout
the second course to demonstrate different designsas the stu-
dents gain familiarity with the devices they can focus on just the
newest layer of complexity. In addition to providing tangible
experience conducting and analyzing experiments, these
projects help students begin to develop statistical practice skills:
the group work with other statistics majors generates a nice ca-
maraderie and the project reports are refined in a subsequent
one-credit writing course that focuses on clear and ethical pre-
sentation of results.
Students are remarkably adept at finding interesting term
projects that are sufficiently simple and innocuous to obviate the
need for ethical review beyond the instructors required preap-
proval. Example projects include experiments that: (1) evaluate
the ability of subjects to reproduce a list of words after being as-
signed a specific time for studying and a specific font color for
the study materials; (2) compare respondents opinions about
quotes when they are reported to have originated from either a
political or religious group for which they may or may not have
affinity; and (3) evaluate the heating potential of a microwave
oven based on the presence/absence of a rotating table and the
centering of the heated object. Given the complexity of the ex-
perimental material (and in an effort to show their statistical
chops), most students opt to use blocking designs (blocking on
subject, on microwave oven, and so forth), and many students
use relatively sophisticated designs such as a split-plot design
with two between-block factors and one within-block factor.
Second-course DAE differs from the traditional design of ex-
periments (DOE) course in the sense that the course objective for
second-course DAE is not necessarily to develop students com-
petence in the field of experimental design. There are many top-
ics in DOE that are not covered in this second-course DAE: A/B
testing, triangle taste tests, random effects, multilevel models,
and adaptive designs. Our students gain facility with concepts
and tools associated with DAE, but developing such expertise
is not the motivation for using DAE per se.
4. ADVANTAGES AND DISADVANTAGES OF DAE AS
THE SECOND COURSE
DAE is an appealing second course for both pedagogical
and logistical reasons: DAE provides a framework for statistical
thinking and exposes students in their second course to a unified
approach subsuming the full spectrum of statistical problem
solving. But it is also mathematically accessible to first-year
university students.
DAE emphasizes to students the primacy of both design
and analysisteaching students that statistics is not just data
analysis but rather a framework for problem solving that in-
cludes formulating questions, collecting and organizing data,
analyzing data, and interpreting results (Franklin et al. 2005).
Studying DAE before regression places more emphasis on fun-
damentals of statistical thinking: the simple ANOVA model
yij = + i + ij introduces modeling while emphasizing no-
tions of components of variability, experimental units, and
statistical significance. These concepts are often lost in the
regression-first approach as the students try to explain y with
X. Once students have studied the linear model they often
believe y = X +  is the answer to everything statistical;
this leaves them less open to thinking about the structure of
datasubsampling versus sampling and split plots and intelli-
gent blocking. Students who go on to study multilevel modeling
The American Statistician, November 2015, Vol. 69, No. 4 329
or hierarchical modeling build on this DAE foundation to further
develop a complex appreciation of the data model.
DAE provides a convenient vehicle and an overarching narra-
tive for understanding many fundamental questions of statistical
thinking: How is the statistical model a tool for understanding
factors influencing an observed variable? How does a random
error component differ from a factor of direct interest? How can
we compare components of variability? What is an experimental
unit? Second-course DAE offers students a unique opportunity
to internalize these foundational concepts. For example, char-
acterization of variability in air quality is a motivating example
on the first day of the course. The students begin by trying to
characterize the variability of 12 measurements of local ambient
particulate matter. But then it is revealed that the 12 measure-
ments were obtained by taking four measurements on one day,
four measurements on a second day, and four measurements
on a third day. So the class attempts to incorporate that in-
formation in their characterization of variability of particulate
matter. Finally, it is revealed that the first two measurements
are two different measurements of the same filter using X-ray
fluorescenceand the discussion of characterization of vari-
ability continues. This forces students to think about the nature
of numbers, observations, and the genesis of variability. This
discussion of decomposition of variability is revisited at the
end of the course after studying the split plot design. Students
then understand that the definition of the experimental unit de-
pends on the chunk of material being randomly assigned to
different levels of a factor. Students do not seem to really un-
derstand experimental units until this pointand this is a valu-
able concept for all studentseven those in nonexperimental
fields.
Another major advantage of second course DAE within a
broad program of statistical training (i.e., a curriculum consist-
ing of many more than two applied statistics courses) is that
students begin their journey of statistical thinking with excep-
tionally high standards for data quality and exceptionally strin-
gent grounds for asserting causality. The gold standard of DAE
proves to be an invaluable tool when contextualizing analyses
of data arising from natural (but nonrandomized) experiments
or even purely observational studies. While we discuss the use
of ANOVA for the analysis of data arising from observational
studies in second-course DAE, together with the limitations of
the conclusions we can make in those settings, second-course
DAE is followed in the applied arm of our curriculum with and
prerequisite to a course focusing on models for observational
data, confounding, and causal inference. The experience of stu-
dents in DAE prepares these students to understand the limits of
observational data. This subsequent regression course begins by
considering a randomized experiment we would like to perform
but cannot. The students consider why we are unable to perform
some experiment and how a given observational study might
emulate a randomized experiment. Then they discuss what con-
ditions are required to identify causal effects in observational
studies (Hernan and Robins 2015). The students learn that if
observational data cannot reasonably emulate an interesting ex-
periment, the data may still be useful for prediction a la Breiman
(2001); however, their foundational training in DAE provides
grounds for considering results from such studies with nuanced
330 Statistics and the Undergraduate Curriculum
pragmatism and a heightened sensitivity to the need for ethical
communication.
Second-course DAE reviews and builds on what students
learned in an introductory statistics course without requiring
calculus. As statistics has grown from a graduate discipline
to include undergraduate majors, many undergraduate courses
(particularly regression courses) have retained the math prereq-
uisites of their graduate counterparts. This is not required: the
Ramsey and Schafer (2012) text successfully uses this approach
to teach multiple regression and ANOVA without requiring cal-
culus or linear algebra; graduate service courses often take this
approach. Gradually undergraduate programs have been ques-
tioning the importance of these prerequisites. The new ASA
guidelines (2014) assert both that [f]ew undergraduate statistics
students need the mathematics used to derive classical statistical
formulas, but also that undergraduate students should be able
to explain the interplay between mathematical derivations and
statistical applications. Many programs, including ours, still re-
quire calculus as a prerequisite for regression. But DAE can be
taught at an appropriate mathematical level for first-year univer-
sity students without obscuring mathematical detail or treating
output like black-box magic; consequently, students can take
DAE while they are concurrently developing their mathemati-
cal skills. Decoupling the mathematical prerequisites indicates
to students that statistical training is different from mathematical
training: It is not an extension of math. It is not a subdiscipline.
Statistics is an applied science that uses math. Concurrent sta-
tistical and mathematical training keeps students engaged with
statistics while they acquire the requisite mathematical tools.
Second-course DAE also provides many opportunities to con-
tinue emphasizing ethical elements of statistical practice that
are introduced in introductory courses (Verhoeven and Baker
2008; Hubert and Wainer 2012). Ethics are particularly ger-
mane when discussing experiments with human subjects: we
discuss whether it is ethical to use a placebo to treat sore throats
and whether it is ethical to randomize babies (or anyone) to
smoke. We talk about the (sometimes amusing) ethical aspects
of previous student project proposals that were submitted but
not approved: for example, a study of methods for asking some-
one on a date when the students had no intention of following
through with the dates. We discuss ethical considerations in test-
ing a priori hypotheses versus post hoc tests and data snooping.
We talk about reproducibility and documentation as guidelines
for statistical practice: in second-course DAE we explain that
if your description of your design has insufficient detail for
the instructor to replicate the experiment, it is not sufficiently
well documented; subsequent courses in our curriculum address
additional aspects of creating reproducible code.
While second-course DAE has many advantages for statis-
tics majors, it may not be the most obvious choice for all
studentsparticularly for students in nonexperimental fields.
Second-course DAE was designed for students who will take a
two-semester methods core: first, design and analysis of experi-
ments; then, regression. Minors who do not elect to take regres-
sion as well as students who discontinue the major after taking
DAE will have seen a fairly narrow view of the discipline. While
this may not disserve manufacturing and engineering students
who want a little more statistics, it may be too circumscribed for
students in nonexperimental fields: many economics, psychol-
ogy, and epidemiology students take courses in our department
beyond introductory statistics. For a student planning on only
one other statistics course after introductory statistics, we en-
courage enrollment in a different service course with a broader
scope (that begins with regression and incorporates point-and-
click software); however, for students who have indicated a de-
sire to become statisticians and who are enrolled in a program
offering a large suite of courses for a statistics major, we believe
that the foundational underpinnings of second-course DAE best
prepare a student for his or her educational career.
A minor disadvantage of teaching DAE as a second course
is the small set of suitable textbooks. We have used Cobbs
Introduction to Design and Analysis of Experiments (2008);
however, it is self-admittedly unorthodox in its conversational
style. While we like the book, many students find it difficult to
use as a reference for technical details. An alternative would
be Oehlerts (2010) freely available DAE digital textbook that
has been developed for students who have completed only an
introductory statistics course (though it is a bit heavy on formu-
lae and information and lighter on ideas). Montgomery (2012)
and Hicks and Turner (1999) could also be used for a second
course; however, both seem a bit inaccessible for a first-year stu-
dent armed only with AP Statistics or a sophomore with only an
introductory statistics course. Due to the sparse textbook options
(and lack of computing in the text we selected), the instructors
of this course have done more course preparation than may be
typical for a 200-level course (a course schedule and course
notes are available at the website, tofu.byu.edu/stat230/ ).
Another disadvantage is that DAE does not necessarily de-
velop the mathematical skills of students who arrive with well-
developed math skills. We do not capitalize on the beauty of
the general linear model; instead we use the pictorial approach
of Cobb (1998). It is unclear that this is unequivocally a dis-
advantage: students seem to gain a deeper understanding using
the pictorial approach; in contrast, when students are first in-
troduced to the matrix formulation of the linear model they
approach it more mathematically/algorithmically. Nonetheless,
for better or worse, reduction of the mathematical barriers to
the statistics major has been accompanied by a decline in the
average math competence of our B.S. in Statistics graduates.
Ultimately, we have found that DAE is a better choice for
the second course in a university where many students arrive
without having finished calculus. We are not avoiding topics
that need mathbut our first-year students often need to take
math, and DAE as the second course gives students the flexibility
to learn statistical ideas while acquiring math skills.
4.1 Evidence Supporting Second-Course DAE
To support claims of the benefits of second-course DAE, we
present evidence that student learning in second-course DAE is
acceptable and that students are more satisfied than students who
took our previous second applied course (a six-credit regression
course that required calculus as a prerequisite and introduced
enough matrix algebra and computing to cover the topic at the
level of Kutner et al. (2004).
Student performance on expected learning outcomes is con-
tinually assessed by measuring the proportion of possible points
earned by each student on exam questions corresponding to
each expected learning outcome. With the exception of block-
ing and factor crossing, all expected learning outcomes had me-
dian achievement above 0.80. For all learning outcomes, student
performance is skewed with large variation among students in
the lower tail. In our previous math-first-then-regression second
course, with the exception of the time series/logistic regres-
sion outcome, the median for each expected learning outcome
was also over 0.80. We also track performance in second-course
DAE as it relates to subsequent courses: overall grade in second-
course DAE is positively correlated with the three expected
learning outcomes in the subsequent regression course, which
are most related to topics covered in second-course DAE (ex-
plaining the difference between ANOVA and regression and
their computational methods, applying appropriate transforma-
tions to the response variable to improve agreement with re-
gression assumptions, and creating sets of indicator variables
for categorical explanatory variables).
While student learning is critically important, as we try to
nurture a students interest in statistics through the second course
we believe student enjoyment of the learning environment is also
important if we are to develop effective and engaged learners.
In our previous second course, between 10% and 25% of the
students either failed or withdrew after the add/drop deadline;
in second-course DAE only 6% to 8% of the students fail or
withdraw. Additionally, students in second-course DAE report
greater satisfaction with the course: average course satisfaction
is a full point higher (on an 8-point scale, comparison based on
sections of our old second course with new second-course DAE
taught by the same instructor); student perception of intellectual
skills developed is unchanged.
5. CONCLUSIONS AND RECOMMENDATIONS
The McKinsey Global Institute report (Manyika et al. 2011)
estimates that by 2018 the United States needs 140,000 to
190,000 more people with strong analytical skills and 1.5 mil-
lion more data-savvy managers and analysts to respond to the
tide of big data into every sector of the global economy. The
importance of meeting this demand is underscored by the NSF
announcement in March 2012 to dedicate $200 million to a
federal initiative to improve our ability to deal with big data
(Mervis 2012). To address this need we must develop a flexible
approach to providing statistical educationbeyond the intro-
ductory courseto increasingly diverse groups of students.
Given the competing options for a second course (as discussed
in Section 2), DAE may seem like the least progressive or mod-
ernizing approach. With the explosion of data that has never
heard of good experimental design (Efron 2001), focusing on
good experimental design in the second course may seem naive
or even archaic: as Horton (2015) explained, Students are gen-
erally taught that if data arise from well conducted randomized
trials, they can make causal conclusions using a two-sample
t-test. All too often, this is considered the pinnacle of statis-
tics.... But most data that students see are not derived from a
randomized trial with no dropout, full adherence, and sufficient
The American Statistician, November 2015, Vol. 69, No. 4 331
blinding. Focusing on DAE for the entire second semester of
study would seem to extend this limited view. And if the second
course were simultaneously the last course, second course DAE
would be regressive.
In fact, second-course DAE does not modernize the curricu-
lum by virtue of its subject matter: DAE is stodgy; it only
obliquely equips students to face-off with new and interesting
data types. But a modern curriculum is not just defined by the
coverage of modern data types within its coursework; rather, a
modern curriculum must also respond to changing student de-
mographics. The increasing demand for statisticians and data
scientists has led to a change in the profile of entering statistics
majors: these students no longer have the mathematical sophis-
tication of previous generations of statistics majors. Second-
course DAE trumpets the idea that a fundamental understand-
ing of the data-generating model provides a good foundation for
students of statisticsand is accessible to students with limited
mathematical training. So DAE is paradoxically a modernizing
influence because it addresses the new profile of statistics ma-
jors: not transfer students from mathematics but new first-year
students who have been educated in primary and secondary
schools that recognize and promote the importance of statistics.
We could respond to these majors in a traditional way: finish
your math, then come back. Or we can respond to these majors
in a way that nurtures their interest in statistics while concur-
rently developing their mathematical skills. So second-course
DAE is progressive because it explores how to take a traditional
topic that seems less cutting-edge than data science or analytics
and finds a place for ita place that recognizes some of the
foundational truths within its bounds.
We are not advocating design of experiments as the most
important statistical skillsuch a position encapsulates an an-
tiquated view of the discipline. In fact, the new ASA guidelines
(2014) do not even include classical design of experiments as
requisite for undergraduate training in statistics. The guidelines
only explicitly mention a few topics from design of studies (ran-
dom assignment, random selection, data collection, efficiency,
bias, causality, confounding, and coincidence, page 11). Rather,
we have found that DAE is a stable platform for introducing
general statistical methodology in the context of a simple prob-
lem that straddles the entire statistical analysis process from
question formulation and data acquisition through analysis and
communication of results while gently introducing students to
statistical computation along the way. In our program the DAE
is the first in a series of real data-based experiences culminat-
ing in elective courses that contain mini-capstones designed to
collectively prepare students for the workforceand it is acces-
sible to majors and minors and students from other disciplines
who want to take just one more statistics course.
The ASA guidelines (2014) assert that, No matter how inno-
vative the approach, we believe that it is not possible to develop
a comprehensive understanding of the range of key statistical
concepts after only two courses. Statistical training for under-
graduate statistics majors must go beyond this second course;
however, by the end of DAE our students have an encapsulated
view of statistical problem solving and the challenges of col-
lecting, analyzing, and interpreting data. Positioning DAE as the
second course in our curriculum allows us to give statistics ma-
332 Statistics and the Undergraduate Curriculum
jors the ability to answer authentic questions with real data early
in their programand this process is repeated as they continue
through the core and elective courses of the major.
Second-course DAE reflects the guiding principles of the new
ASA guidelines that emphasize the scientific method and its re-
lation to the statistical problem-solving cycle; real applications;
the need to compute with data in the context of answering a sta-
tistical question; flexibility; and a relationship with mathematics
that acknowledges undergraduate training in statistics has dif-
ferent demands than Ph.D. preparatory training. Ultimately, the
variety of second courses that are taught reflects the demands
of the students at each university and the interests and expertise
of the faculty; however, DAE should be considered a strong
contender among the set of possible second courses.
[Received December 2014. Revised July 2015.]
REFERENCES
Albers, D. J., Loftsgaarden, D. O., Rung, D. C., and Watkins, A. E. (1992), Sta-
tistical Abstract of Undergraduate Programs in the Mathematical Sciences
and Computer Science in the United States, Washington, DC: American
Mathematical Society. [326]
American Statistical Association (ASA) (2005), Guidelines for Assessment
and Instruction in Statistics Education: College Report, available at
http://www.amstat.org/education/gaise/. [326,327]
(2015), A Peek into the Largest, Fastest-Growing Undergraduate Statis-
tics Departments, Amstat News, 452, pp. 1822. [326]
American Statistical Association Undergraduate Guidelines Workgroup (2014),
2014 Curriculum Guidelines for Undergraduate Programs in Statistical
Science, Alexandria, VA: American Statistical Association, available at
http://www.amstat.org/education/curriculumguidelines.cfm. [327,330,332]
Blair, R., Kirkman, E. E., and Maxwell, J. W. (2013a), Con-
ference Board of Mathematical Sciences (CBMS) Survey Reports
2010, Providence, RI: American Mathematical Society, available at
http://www.ams.org/profession/data/cbms-survey/cbms2010 [326,327]
(2013b), Statistical Abstract of Undergraduate Programs in the Mathe-
matical Sciences in the United States: Fall 2010 CBMS Survey, Providence,
RI: American Mathematical Society. [326,327]
Breiman, L. (2001), Statistical Modeling: The Two Cultures, Statistical Sci-
ence, 16, 199231. [327,330]
Cannon, A. R., Cobb, G. W., Hartlaub, B. A., Legler, J. M., Lock, R. H., Moore,
T. L., Rossman, A. J., and Witmer, J. A. (2012), Stat 2: Building Models for
a World of Data, New York: Freeman. [327]
Cobb, G. W. (2008), Introduction to Design and Analysis of Experiments, Hobo-
ken, NJ: Wiley. [331]
(2015), Mere Renovation is Too Little Too Late: Its Time to Rebuild the
Undergraduate Curriculum from the Ground Up, The American Statistician,
69, this issue. [327]
Davidian, M., and Louis, T. A. (2012), Why Statistics? Science, 336, 12. [326]
De Veaux, D. (2015), Intro Stats in the 21st Century, Presented at the
U.S. Conference on Teaching Statistics, State College, Pennsylvania, 28
May. Available at https://www.causeweb.org/uscots/uscots15/presentations/
DickOpening.pptx [326]
Dielman, T. (2004), Applied Regression Analysis: A Second Course in Business
and Economic Statistics, Belmont, CA: Brooks/Cole. [327]
Draper, D. (1987), Comment: On Exchangeability Judgments in Predictive
Modeling and the Role of Data in Statistical Research, Statistical Science,
2, 454461. [327]
 (1995), Inference and Hierarchical Modeling in the Social Sciences,
Journal of Educational and Behavioral Statistics, 20, 115147. [327]
Efron, B. (2001), Comment on Breimans Statistical Modeling: The Two Cul-
tures, Statistical Science, 16, 218219. [331]
Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R.,
Perry, M., and Scheaffer, R. (2005), Guidelines for Assess-
ment and Instruction in Statistics Education (GAISE) Report,
Alexandria, VA: American Statistical Association, available at
http://www.amstat.org/education/gaise/GAISEPreK-12_Full.pdf [329]
Hamilton, L. C. (1991), Regression with Graphics: A Second Course in Applied
Statistics, Belmont, CA: Duxbury. [327]
Hernan, M. A., and Robins, J. M. (2015), Causal Inference, available
at http://cdn1.sph.harvard.edu/wp-content/uploads/sites/1268/2015/02/
hernanrobins_v1.10.27.pdf [330]
Hicks, C. R., and Turner, K. V. (1999), Fundamental Concepts in the Design of
Experiments, New York: Oxford University Press. [331]
Horton, N. J. (2015), Challenges and Opportunities for Statistics and Statistics
Education: Looking Back, Looking Forward, The American Statistician,
69, 138145. [331]
Hubert, L., and Wainer, H. (2012), A Statistical Guide for the Ethically Per-
plexed, Boca Raton, FL: Chapman and Hall. [330]
Kuiper, S. (2015), Stat2Labs, available at http://web.grinnell.edu/individuals/
kuipers/stat2labs/ [329]
Kuiper, S., and Sklar, J. (2012), Practicing Statistics: Guided Investigations for
the Second Course, Boston, MA: Pearson. [327]
Kutner, M., Nachtsheim, C., Neter, J., and Li, W. (2004), Applied Linear Sta-
tistical Models, Boston, MA: McGraw-Hill/Irwin. [331]
Lawson, J. S. (2014), Design and Analysis of Experiments With R, Boca Raton,
FL: Chapman Hall. [328]
Lock, K., and Meng, X.-L. (2010), Real-Life Module Statistics: A
Happy Harvard Experiment, in Data and Context in Statistics Ed-
ucation: Towards an Evidence-Based Society (Proceedings of the
Eighth International Conference on Teaching Statistics (ICOTS8, July,
2010), Ljubljana, Slovenia), ed. C. Reading, Voorburg, The Nether-
lands: International Statistical Institute. Available at http://iase-web.org/
documents/papers/icots8/ICOTS8_4D1_LOCK.pdf [326]
Lohr, S. (2009 August 5), For Todays Graduate, Just
One Word: Statistics, The New York Times. Available at
http://www.nytimes.com/2009/08/06/technology/06stats.html [326]
Lomax, R. G. (2007), Statistical Concepts: A Second Course, New York: Rout-
ledge. [327]
Loveday, R. (1961), A Second Course in Statistics, Cambridge, UK: Cambridge
University Press. [327]
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh,
C., and Byers, A. H. (2011), Big Data: The Next Frontier for Innova-
tion, Competition, and Productivity, McKinsey Global Institute. Available at
http://www.mckinsey.com/insights/business_technology/big_data_the_next_
frontier_for_innovation [331]
Mendenhall, W., and Sincich, T. (2011), A Second Course in Statistics: Regres-
sion Analysis, Upper Saddle River, NJ: Prentice Hall. [327]
Meng, X.-L. (2009), Desired and FearedWhat Do We Do Now
and Over the Next 50 Years, The American Statistician, 63, 202
210. [326]
Mervis, J. (2012), Agencies Rally to Tackle Big Data, Science, 336, 22. [331]
Montgomery, D. C. (2012), Design and Analysis of Experiments, Hoboken, NJ:
Wiley. [331]
Mosteller, F., and Tukey, J. W. (1977), Data Analysis and Regression: A Second
Course in Statistics, Reading, MA: Addison Wesley. [327]
Oehlert, G. W. (2010), A First Course in Design and Analysis of Experiments,
available at http://users.stat.umn.edu/gary/book/fcdae.pdf [331]
Ramsey, F., and Schafer, D. (2012), The Statistical Sleuth: A Course in Methods
of Data Analysis, Pacific Grove, CA: Duxbury. [330]
The College Board (2011), AP Examination Volume Changes (20012011),
available at http://media.collegeboard.com/digitalServices/pdf/research/
AP-Exam-Volume-Change-2011.pdf [326]
(2013), The 10th Annual AP Report to the Nation: Subject Supple-
ment, available at http://apreport.collegeboard.org [326]
Utts, J. (2015), The Many Facets of Statistics Education: 175 Years of Common
Themes, The American Statistician, 69, 100107. [326]
Verhoeven, P., and Baker, H. M. (2008), An Approach for Incorporating the
Topic Ethics in Research into an Introductory Statistics Course, Decision
Sciences Journal of Innovative Education, 6, 489493. [330]
Wasserstein, R. (2015), Communicating the Power and Impact of Our Pro-
fession: A Heads Up for the Next Executive Directors of the ASA, The
American Statistician, 69, 9699. [326]
Wonnacott, T. H., and Wonnacott, R. J. (1986), Regression: A Second Course
in Statistics, Malabar, FL: Krieger. [327]
The American Statistician, November 2015, Vol. 69, No. 4 333