Expanding the Role of Statistics to Areas Traditionally Dominated by

Expert Judgment
In a previous post, I wrote about how the field of statistics is more important now
than ever before due to the modern deluge of data. Because youre reading
Minitab's statistical blog, Ill assume that were in agreement that statistics allows
you to use data to understand reality. However, Id also bet that youre picturing
important but typical statistical studies, such as studies where Six Sigma
analysts determine which factors affect product quality. Or perhaps medical
studies, like determining the effectiveness of flu shots.
In this post, Im going to push that further, much further. So far that it might even
make you feel a bit uncomfortable. Its not that I want you to feel discomfort, but
I hope you can find new areas to apply statistical analyses. Im talking about
areas that are considered human specialties: intuition, experience, professional
judgment, decision-making, and even creativity.
Human Judgment Is Often Insufficient
Life is complicated and it forces people to process a large amount of information,
often in real time, to make complex decisions. Humans have developed mental
shortcuts, intuition, and accepted wisdom in order to avoid information overload.
Its often not possible to process all of the relevant variables, their interactions
and correlations, weight them all properly, and avoid personal biases. People are
also easily swayed by detailed stories, irrelevant information, and anecdotal
evidence.
Studies in the cognitive and behavioral sciences have consistently shown that our
brains are not up to the task. However, statistical analyses can solve these
problems. Studies that date back to at least the 1950s have repeatedly shown
that even simple statistical models can produce better predictions than expert
judgment. Let's check out some examples that show how statistical analyses
have outperformed experts.
Six Examples of Predictive Analytics and Statistical Decision-Making
Tools
Predicting wine vintage prices
Wine glassOrly Ashenfelter, a Princeton economist, fit regression models that
predict the price of wine vintages. These models include predictors such as
temperatures and rainfall, among others. Wine experts dismissed the regression
models until the models predictions beat the experts to the punch on identifying
several vintages of the century. (Use Regression in Minitab to Make Predictions)
Predicting cardiac risk in the emergency room
Cardiovascular complications pose one of the most significant risks to patients
undergoing major surgery. Ideally, cardiologists identify the patients with a high
risk before surgery so they can take steps to reduce the risk. Unfortunately, the
risk of complications is notoriously hard to predict even for cardiologists. Lee
Goldman, a cardiologist, used statistical methodology to create an easy-to-use
checklist that estimates cardiac risk. Over the years, this simple tool has proven
to outperform the assessment of emergency room doctors. It was originally
developed in 1977 but it was not implemented until the mid-1990s.
1

Predicting the correct diagnosis at the point of care

Isabel is a computer-based clinical diagnostic support system (DSS). This system
uses patient variables and symptoms to calculate the most probable diagnoses
from over 11,000 possible diagnoses, a task that simply isnt possible for
humans. A study found that when key features from 50 challenging cases,
reported in the New England Journal of Medicine, were entered into the system, it
provided the final diagnosis in 48 cases (96%).
The Journal of the American Medical Association has concluded that misdiagnosis
is increasing. An autopsy study found that 20% of fatal cases are diagnosed
incorrectly! So youd think that using software like this would be a good thing!
However, a 2007 study found...well, the title of the article in the Journal of
Medical Decision Making says it all: Patients derogate physicians who use a
computer-assisted diagnostic aid. Physicians are aware of this perception and, as
a result, many dont use these valuable tools.
Predicting academic achievement for college admissions
Academic admissions counselors attempt to predict the future academic
achievement of the candidates they review. However, decades of psychology
studies have found that high school rank and standardized test scores have a
higher correlation with their eventual academic achievement than expert
judgment.
Are you wondering if a reasonable compromise might be to use the numeric
measures in conjunction with human judgment? Researchers have studied this as
well. They found that academic achievement is better predicted by standardized
scores alone than by the scores plus expert judgment.
Predicting job performance to select job candidates
Human resource (HR) professionals try to predict the future job performance of
the applicants. A survey of HR executives reveals that the traditional
unstructured job interview is widely considered to be more effective than penciland-paper tests that assess aptitude, personality, and general mental ability.
However, a meta-analysis finds that these types of tests are consistently better
predictors than the unstructured interviews. Scholarly surveys have found that
HR professionals are aware of these persistent findings, but they dont think the
pattern applies to them!
Predicting movie profits to influence script writing
The last example comes from a recent New York Times article and it is a new one
for me. Youd think that script writing would remain a creative activity. However,
big money is riding on a movies success. Scripts are approved based on a
prediction that the movie will make money, and Im sure you see where this is
heading based on the other examples. Statistician Vinny Bruzzese has developed
a methodology to statistically analyze the story structure of the script in order to
predict profitability. Based on the analysis, Bruzzese suggests alterations to the
script that are designed to increase the predicted profits.
While the methodology is proprietary, it appears that Bruzzese maintains a very
detailed movie database which he uses to create a statistical model that relates
2

the characteristics of movies to their profits. Bruzzese hasnt released data that
would allow us to evaluate the results. However, he is reportedly expanding his
business and some industry executives are saying that everyone will be doing
this soon. Not surprisingly, this statistical intrusion into the creative world is
causing much controversy among the creative people!

Pushing the Boundaries

I mentioned that some of these examples might make you feel uncomfortable.
Does the idea that its better to select a job candidate based on a test score than
an interview make you squirm? Perhaps you dont like the idea that expert
opinion actually lowers the reliability of predictions for academic achievement?
How about the idea that your risk of complications may be determined by a
checklist rather than a doctors assessment in the ER? Personally, Im a writer, so
the script analysis example is a bit scary!
If you do feel discomfort, youre far from alone. In fact, there was great resistance
to all of the statistical models in the examples because they intrude on human
judgment. In some cases, the overwhelming results have changed minds. No one
is talking about getting rid of Goldmans cardiac risk assessment. But, thats not
always true.
After decades of research, many HR executives and admissions committees still
do not understand that test scores are proven to outperform human judgment.
And, patients still think that doctors who use computer tools are less competent
than those who dont. This occurs despite the fact that a meta-study, which
assessed 100 studies, found that an overwhelming majority concluded that
doctors who consult diagnosis predictions based on statistical analyses are
correct more often than unaided doctors.
Unfortunately, it is more socially acceptable to rely on judgment rather than test
scores, ratings, and formulas, even when the data suggest otherwise.
The Times They Are a-Changin!
Dont fret; there is a positive message behind all of this. Ultimately, this is a story
about playing to our strengths. It turns out that we arent always as good as
statistical models when it comes to predicting outcomes for complicated
scenarios. However, the good news is that we excel at creating the statistical
models and decision-making tools that produce better predictions! And, it has
been powerful statistical software, like Minitab, that has been instrumental in
unlocking this potential to produce the best predictions possible.
This blog is really a challenge to you. Find new and innovative ways to apply
statistics. Push the envelope. Dont let the old boundaries and discomfort hold
you back!