dia and with government, modern ecological
Wilson Center Press,
thinking can perhaps become an ingredient in
the news on natural disturbances.
Literature Cited
Smith, C. In press. Gonzo flame video in
Yellowstone: how the media covered a
natural catastrophe. In P. S. Cook, D.
Gomery, and L. W. Lichty, editors. The
THE 10 MOST COMMON STATISTICAL ERRORS
Upon finishing his term as a member of the
Board of Editors of the ESA, Mack (1986)
provided some advice to authors about writ-
ing with preCision, clarity, and economy. Here
I follow his example with regard to common
statistical errors, in the hope that this infor-
mation will also be useful to authors. This ar-
ticle is based upon comments made by me
and by reviewers on the manuscripts I handled
as a member of the Board of Editors. The
errors are ordered roughly by relative com-
monness and expressed as suggestions.
1. Explain clearly what you did. Describe both
the experimental design and the statistical
analyses clearly and precisely, so that the
reader can determine exactly what you did.
Make sure that questions such as these are
answered:
How many replicates were there? What were
the sample sizes? How were plots, plants,
shoots, seeds located/chosen/arranged?
How many rows and how many columns
were there in each contingency table? What
variable is represented by the rows, what
variable by the columns? Was a chi-square
or a G test used to test the table for sig-
nificance? Were any cells pooled? If so,
which ones and why?
Exactly how were the sums of squares (SS)
tables and the Ftests constructed for each
analYSis of variance (ANOVA) and each
analysis of covariance (ANCOVA). Which
variables were used as effects in the model,
and which were used to partition the data
future of news.
Washington, D.C., USA.
Farb, N., and A. H. Malcom. 1988. Up from
the ashes. The New York Times Maga-
zine, 6 November 1988, 137:48-53.
S.TA Pickett
Institute of Ecosystem Studies
BoxAB
Millbrook, NY 12545
into subsets? Which effects were nested,
within which effects, and which effects were
crossed, with which effects? What were the
denominator mean squares? If the analyses
are at all complex, consider including a sam-
ple SS table, or at least specify the degrees
of freedom.
Was the dependent variable a character
measured on each individual, or a quadrat
mean, or a mean of three subsamples? Was
it the proportion surviving or the number
surviving? Was a ratio calculated for each
plant separately, or by dividing means?
If the editor and reviewers cannot answer
questions like these after reading the manu-
script, they do not have the information to
understand and evaluate what you did. Mis-
understood designs and analyses sometimes
lead reviewers and editors to decide that the
data do not support the conclusions drawn
from them. Frequently, unclear descriptions
suggest that the study may have other sta-
tistical problems (e.g., the degrees of freedom
may be apparently inconsistent with the de-
sign).
Standard terminology should always be
used. Avoid BMDP and SAS acronyms.
2. Avoid doing lots and lots of separate sta-
tistical tests. If you do enough statistical tests,
some are bound to show significance by
chance alone. In most cases this problem can
be avoided by replacing hordes of t tests with
the appropriate ANOVA, or by replacing a large
number of one-way ANOVAs with the appro-
161
priate multifactor ANOV A. If large numbers of
separate tests are unavoidable, adjust the
critical Pvalue(s) accordingly (e.g., Rice 1989,
Sokal and Rohlf 1981 : 779-782). Avoiding the
pitfalls awaiting those who attempt more com-
plex analyses is unfortunately not a sufficient
justification for dOing lots of simpler analyses.
3. When using ANOVAs, ANCOVA S, and
regressions, be aware of the assumptions and
test your data for them. ANOVA, in particular,
is robust to certain departures from assump-
tions, but not all departures from assumptions
are permissible in ANOV A or any other anal-
ysis. To avoid some of the commonest prob-
lems in this category, consider the following:
If the dependent variable only takes a few
different values, it may not be legitimate to
analyze it as a continuous variable. (A contin-
uous dependent variable is an assumption of
all of these analyses.) Be sure to check the
residuals, not the raw data, for normality. If
the dependent variable often takes the value
0, the distribution of the residuals may not be
sufficiently close to normal in many cases.
Transformations do not correct this problem,
nor do they correct the problem of too few
values, mentioned above. If a transformation
is used, for example to increase the homo-
geneity of variances, check the transformed
variable to see whether it meets each as-
sumption.
4. Interpret interaction effects in multi-way
ANOVAs correctly. In some cases, authors
discussed nonsignificant interaction effects as
if they were significant. A hypothetical ex-
ample of this error is the reporting of a non-
significant water· fertilizer effect, followed by
"fertilizer increased yield in the watered plots
but not in the unwatered plots."
5. Don't pool plots without justification. I use
"plots" in the technical sense; a "plot" can
be a Petri dish containing seeds, a portion of
a greenhouse bench, or a quadrat in the field.
A "plot" can be a block in a blocked design.
If plot is not included as a term in the analysis,
it has been pooled with some other term, and
the effects due to differences between differ-
ent plots will be confounded either with the
effects of an experimental treatment, or with
the residual variation. Both Type I (spurious
significance) and Type \I (overlooked signifi-
cance) errors can arise from such pooling.
Pooling plots can be justified in some circum-
162
stances (e.g., Sokal and Rohlf 1981: 284-
285), but it is a dangerous practice if done
without appropriate justification. If it is una-
voidable (as it may be in some contingency
table analyses), the potential problems should
at least be recognized.
6. Before you conclude that the null hypoth-
esis is true, determine the power of the test.
The absence of statistical significance does
not necessarily mean that the effect did not
occur. Perhaps the sample sizes were too
small, or the background variation too large,
to detect any but the largest differences be-
tween treatments. Determination of the power
of the test (Sokal and Rohlf 1981: 161-168,
Snedecor and Cochran 1989: 68-70, Steel
and Torrie 1980: 113-119) will give you an idea
of how small a difference you could have de-
tected as significant. Don't forget to check
your data for trends. A nonsignificant but ob-
vious trend suggests that the null hypothesis
should not be accepted. Of course, in this
case the null hypothesis cannot be rejected,
either.
7. Use multiple comparison tests correctly. A
posteriori multiple comparison tests should not
follow a nonsignificant main effect. Except in
special cases multiple comparison tests should
not be used as a substitute for a two-way
ANOV A with interaction term. See the recent
article by Day and Quinn (1989) for updated
advice on the selection and use of multiple
comparison tests.
8. Include replication in experimental designs
and observational schemes. Despite the at-
tention this issue has recently received (large-
ly due to the much-cited article by Hurlbert
[1984]), some manuscripts still report designs
without replication and do not recognize them
as such. This topic is closely related to the
following one:
9. Beware of confounded factors. In some
instances, especially in descriptive studies or
"natural experiments," factors are inherently
confounded. An experimental design can con-
found factors, for example by placing all in-
dividuals that receive treatment A in plot 1, all
individuals that receive treatment B in plot 2,
etc. Mortality or other losses during the ex-
periment may result in missing cells, which
cause factors to be confounded. A hypothet-
ical example of the latter: An experiment with
two levels of watering and two levels of fer-
tilizer was conducted. If all the low water/low
fertilizer plants died, leaving only three of the
four possible treatment combinations, it is not
possible to ascribe differences in final biomass
to the separate effects of water, or of fertilizer,
or to their interaction. Regardless of the source
of the confounding, problems arise if an au-
thor fails to recognize the situation.
10. Repeated measures of the same individual
are not independent, and cannot be analyzed
as if they were. This situation requires special
analyses (e.g., Gurevitch and Chester 1986).
Most of these suggestions have been word-
ed in terms of manipulative, controlled exper-
iments. Many of the manuscripts, however,
described descriptive studies. The frequency
and type of errors did not appear to differ
between the two types of studies. Each of the
suggestions applies to descriptive studies as
well as to manipulative experiments.
There was also a wide variety of less com-
mon errors. Sometimes the author appeared
to have ignored his/her own statistical results,
and asserted that effects were found when
the statistical tests gave nonsignificant re-
sults, or that no effect was found although the
statistical test was significant. Sometimes the
appropriate test of an assertion had not been
done. Most of the remaining errors involved
problems with the application, construction,
and interpretation of multi-way ANOVAs,
nested ANOVAs, or regressions.
No attempt has been made here to provide
complete explanations or references for any
topic. Most standard introductory texts cover
most of the topics mentioned here (e.g., Sokal
and Rohlf 1981 , Snedecor and Cochran 1989,
Steel and Torrie 1980). In some instances,
individual advice from a statistician may be
the best source of help.
I encourage all authors to make sure that
their statistical analyses are correct and that
their designs and analyses are clearly de-
scribed, before submitting a manuscript to
Ecology or Ecological Monographs. Avoiding
statistical errors, especially the common ones
mentioned above, will increase the likelihood
that your manuscript will eventually be ac-
cepted. By reducing the amount of revision
required, it will also decrease the time be-
tween submission and acceptance. Of course
good statistical methodology alone does not
make a good ecological study, but statistical
errors can fatally damage an otherwise ex-
cellent study.
Statistical techniques are some of the most
widely used, and most misused, of any tech-
niques utilized by ecologists. Over half of the
papers that involved a statistical analysis had
one or more statistical errors, even excluding
those errors classified in (1) above. That so
many of these errors involved relatively basic
statistical knowledge is worrying.
However, complex questions often require
complex designs to answer, and complex de-
signs usually require more complex statistical
analyses. Perhaps we are asking more com-
plex questions-and our statistical expertise
has not kept up. The high frequency of errors
in the more complex analyses may be a sign
of the rate at which the questions we are ask-
ing are becoming more complex and sophis-
ticated.
Acknowledgments
I thank my fellow editors D. Goldberg, L.
Miller, P. Morin, M. Rausher, and J. Travis,
as well as M. SciOli, for their suggestions and
comments on earlier versions of this manu-
script. I also thank the many reviewers who
generously gave their time to evaluate man-
uscripts and to provide comments for the au-
thors.
Literature Cited
Day, R. W., and G. P. Quinn. 1989. Com-
parisons of treatments after an analysiS
of variance in ecology. Ecological Mono-
graphs 59:433-463.
Gurevitch, J., and S. T. Chester. 1986. Anal-
ysis of repeated measures experiments.
Ecology 67:251-255.
Hurlbert, S. H. 1984. Pseudoreplication and
the design of ecological field experi-
ments. Ecological Monographs 54: 187-
211.
Mack, R. N. 1986. Writing with precision,
clarity, and economy. ESA Bulletin 67:
31-35.
Rice, W. R. 1989. Analyzing tables of sta-
tistical tests. Evolution 43:223-225.
Snedecor, G. W., and W. G. Cochran. 1989.
Statistical methods. Eighth edition. Iowa
State University Press, Ames, Iowa, USA.
Sokal, R. R., and F. J. Rohlf. 1981. Biom-
163
 etry. Second edition. Freeman, San Fran-
cisco, California, USA.
Steel, R. G. D., and J. H. Torrie. 1980. Prin-
ciples and procedures of statistics; a bio-
metrical approach. Second edition.
McGraw-Hili, New York, New York, USA.
ON THE RETIREMENT OF JOHN LANGDON BROOKS FROM NSF
After nearly twenty years of service and
leadership as an administrator of federal sup-
port for research in ecology and systematics,
John Brooks retired from the National Science
Foundation in June 1989. He was the consis-
tent source of sound judgment during a period
of dramatic changes as the field responded
to new tools and new ideas. Molecular meth-
ods, developed in microbiology and genetics,
were found to be useful in studies of popu-
lation genetics, systematics, and even behav-
ioral ecology. Systematic data analysis be-
came dominated by cladistic methods. Highly
complex models of ecosystems were shown
to be useful analytical tools. Team studies,
labelled Long-Term Ecological Research
(LTER) were incorporated into national net-
works of ecological reserves. The differences
between environmental science, applied ecol-
ogy, and basic research became less clear
and biologists realized that basic research in
conservation-related areas should have a
higher priority on their agendas. The job of
NSF has been to respond to these changing
needs, and to offer what support is available
to the most promising projects. Of course this
job has been made difficult by the ever-in-
creasing lag in the growth of funds available
for support compared to the need. The single
person who has had the major responsibility
for making wise decisions in this changing
scene, one that bears importantly on our abil-
ity to do basic research in ecology and sys-
tematics and on the health of our discipline,
has been John Brooks.
One way that NSF responds to changing
needs is by reorganization. When John came
to NSF in 1969, the Ecosystem Analysis Pro-
gram (now Ecosystem Studies) had just been
164
Norma Fowler
Department of Botany
University of Texas
Austin, TX 78713
formed. Its job was to oversee the adminis-
tration ofthe U.S. International Biological Pro-
gram. Facilities such as museum collections
were supported in the Systematic Biology
Program, and John became the Program Di-
rector for the core of ecology, the General
Ecology Program. By 1981 he had risen to
the directorship of the entire administrative
unit. In 1975 a new program was created to
support the population genetics that was for-
merly supported in SystematiC Biology, along
with the population ecology and physiological
ecology that was formerly supported in Gen-
eral Ecology, thereby allowing growth in all
three areas. The title of the new program was
Population Biology and PhYSiological Ecolo-
gy. John became Head of the Ecology and
Population Biology Section and then Deputy
Director (1975) and Director (1981) of the Di-
vision of Environmental Biology. During that
time a new program for support of facilities
was created, Biological Research Resources.
As Division Director, John was responsible for
the five present programs (Ecology, Popula-
tion Biology and Physiological Ecology, Eco-
system Studies, SystematiC Biology, and Bi-
ological Research Resources). These
programs provide most of the federal support
of basic research in whole organism biology.
(The exceptions are marine ecology, which is
supported in the Biological Oceanography
Program, and behavior, which is supported in
the Biological Basis of Behavior Program.)
I had the good fortune to work as a program
officer in this group from 1973 to 1977. That
was long enough to see several "rotators"
(now called "visiting scientists": program di-
rectors on leave from their universities who
worked at NSF for one or two years) come