Critical Thinking:

What It Is and Why It Counts

Peter A. Facione

The late George Carlin worked
“critical thinking” into one of his comedic
monologue rants on the perils of trusting our
lives and fortunes to the decision-making of
people who were gullible, uninformed, and
unreflective. Had he lived to experience the
economic collapse of 2008 and 2009, he
would have surely added more to his
caustic but accurate assessments regarding
how failing to anticipate the consequences
of one’s decisions often leads to disastrous
results not only for the decision maker, but
for many other people as well.
weakness, public health problems, crime,
and avoidable poverty? Perhaps that
realization, along with its obvious
advantages for high level strategic decision
making, is what lead the Chairman of the
Joint Chiefs of Staff to comment on critical
thinking in his commencement address to a
graduating class of military officers.

After years of viewing higher

education as more of a private good which
benefits only the student, we are again
beginning to appreciate higher education as
being also a public good which benefits
society. Is it not a wiser social policy to
invest in the education of the future
workforce, rather than to suffer the financial
Teach people to make good
costs and endure the fiscal and social
decisions and you equip them to improve
burdens associated with economic
© 1992, 1998, 2004, 2006, 2011 Peter A. Facione, Measured Reasons and The California Academic Press, Millbrae, CA

Permission to Reprint for Non-Commercial Uses

This essay is published by Insight Assessment. The original appeared in 1992 and has been updated many times over the years.
Although the author and the publisher hold all copyrights, in the interests of advancing education and improving critical thinking,
permission is hereby granted for paper, electronic, or digital copies to be made in unlimited amounts, provided that their distribution
is free of charge provided that whenever material from this essay is cited or extracted in whole or in part that appropriate citation is
made by indicating this essay’s full title, author’s name, publisher’s name, year, and page or pages where it appears in this edition.
For permission for reprints intended for sale contact Insight Assessment by phone at 650-697-5628 or by email to
jmorante@insightassessment.com. ISBN 13: 978-1-891557-07-1.

To support the expenses of making this essay available free for non-commercial uses, the publisher has inserted information about
its critical thinking testing instruments. These tools assess the critical thinking skills and habits of mind described in this essay. To
build critical thinking skills and habits of mind use Dr. Facione’s newest book THINK_Critically, Pearson Education 2011.
their own futures and become contributing
members of society, rather than burdens on
society. Becoming educated and practicing
good judgment does not absolutely
guarantee a life of happiness, virtue, or
economic success, but it surely offers a
better chance at those things. And it is
clearly better than enduring the
consequences of making bad decisions and
better than burdening friends, family, and all
the rest of us with the unwanted and
avoidable consequences of those poor
choices.
Defining “Critical Thinking”
Yes, surely we have all heard
business executives, policy makers, civic
leaders, and educators talking about critical
thinking. At times we found ourselves
wondering exactly what critical thinking was
and why is it considered so useful and
important. This essay takes a deeper look at
these questions.
But, rather than beginning with an
abstract definition – as if critical thinking
were about memorization, which is not the
case – give this thought experiment a try:
Imagine you have been invited to a movie
by a friend. But it’s not a movie you want to
see. So, your friend asks you why. You
give your honest reason. The movie
offends your sense of decency. Your friend
asks you to clarify your reason by explaining
what bothers you about the film. You reply
that it is not the language used or the
sexuality portrayed, but you find the
violence in the film offensive.
Sure, that should be a good enough
answer. But suppose your friend, perhaps
being a bit philosophically inclined or simply
curious or argumentative, pursues the
matter further by asking you to define what
you mean by “offensive violence.”
Take a minute and give it a try.
How would you define “offensive violence”
as it applies to movies? Can you write a
characterization which captures what this
Facione, PA, “Critical Thinking: What It is and Why it Counts”
commonly used concept contains? Take
care, though, we would not want to make
the definition so broad that all movie
violence would be automatically “offensive.”
And check to be sure your way of defining
“offensive violence” fits with how the rest of
the people who know and use English
would understand the term. Otherwise they
will not be able to understand what you
mean when you use that expression.
Did you come up with a definition
that works? How do you know?
What you just did with the
expression “offensive violence” is very much
the same as what had to be done with the
expression “critical thinking.” At one level
we all know what “critical thinking” means —
it means good thinking, almost the opposite
of illogical, irrational, thinking. But when we
test our understanding further, we run into
questions. For example, is critical thinking
the same as creative thinking, are they
different, or is one part of the other? How
do critical thinking and native intelligence or
scholastic aptitude relate? Does critical
thinking focus on the subject matter or
content that you know or on the process you
use when you reason about that content?
It might not hurt at all if you formed
some tentative preliminary ideas about the
questions we just raised. We humans learn
better when we stop frequently to reflect,
rather than just plowing from the top of the
page to the bottom without coming up for
air.
Fine. So how would you propose we
go about defining “critical thinking.” You do
not really want a definition plopped on the
page for you to memorize, do you? That
would be silly, almost counterproductive.
The goal here is to help you sharpen your
critical thinking skills and cultivate your
critical thinking spirit. While memorization
definitely has many valuable uses, fostering
critical thinking is not among them. So, let’s
look back at what you might have done to
define “offensive violence” and see if we
can learn from you. Did you think of some
2011 update Page 2
scenes in movies that were offensively
violent, and did you contrast them with other
scenes that were either not violent or not
offensively violent? If you did, good. That is
one (but not the only) way to approach the
problem. Technically it is called finding
paradigm cases. Happily, like many things
in life, you do not have to know its name to
do it well.
Back to critical thinking – let’s ask
ourselves to come up with possible
examples of strong critical thinking? How
about the adroit and clever questioning of
Socrates or a good attorney or interviewer?
Or, what about the clever investigative
approaches used by police detectives and
crime scene analysts? Would we not want
to also include people working together to
solve a problem as they consider and
discuss their options? How about someone
who is good at listening to all sides of a
dispute, considering all the facts, and then
deciding what is relevant and what is not,
and then rendering a thoughtful judgment?
And maybe too, someone who is able to
summarize complex ideas clearly with
fairness to all sides, or a person who can
come up with the most coherent and
justifiable explanation of what a passage of
written material means? Or the person who
can readily devise sensible alternatives to
explore, but who does not become
defensive about abandoning them if they do
not work? And also the person who can
explain exactly how a particular conclusion
was reached, or why certain criteria apply?
Or, considering the concept of
critical thinking from the opposite direction,
we might ask what the consequences of
failing to use our critical thinking might be.
Imagine for a moment what could happen
Facione, PA, “Critical Thinking: What It is and Why it Counts”
when a person or a group of people decides
important matters without pausing first to
think things through.
Expert Opinion
An international group of experts
was asked to try to form a consensus about
the meaning of critical thinking.1 One of the
first things they did was to ask themselves
the question: Who are the best critical
thinkers we know and what is it about them
that leads us to consider them the best?
So, who are the best critical thinkers you
know? Why do you think they are strong
critical thinkers? Can you draw from those
examples a description that is more
abstract? For example, consider effective
trial lawyers, apart from how they conduct
their personal lives or whether their client is
really guilty or innocent, just look at how the
lawyers develop their cases in court. They
use reasons to try to convince the judge and
jury of their client’s claim to guilt or
innocence. They offer evidence and
evaluate the significance of the evidence
presented by the opposition lawyers. They
interpret testimony. They analyze and
evaluate the arguments advanced by the
other side.
1
Many useful characterizations of critical thinking by noted
theorists and teachers are captured in Conversations with
Critical Thinkers , John Esterle and Dan Clurman (Eds.).
Whitman Institute. San Francisco, CA. 1993
2011 update Page 3
 Now, consider the example of the
team of people trying to solve a problem.
The team members, unlike the courtroom’s
adversarial situation, try to collaborate. The
members of an effective team do not
compete against each other. They work in
concert, like colleagues, for the common
goal. Unless they solve the problem, none
of them has won. When they find the way
to solve the problem, they all have won. So,
from analyzing just two examples we can
generalize something very important: critical
thinking is thinking that has a purpose
(proving a point, interpreting what
something means, solving a problem), but
critical thinking can be a collaborative,
noncompetitive endeavor. And, by the way,
even lawyers collaborate. They can work
together on a common defense or a joint
prosecution, and they can also cooperate
with each other to get at the truth so that
justice is done.
We will come to a more precise
definition of critical thinking soon enough.
But first, there is something else we can
learn from paradigm examples. When you
were thinking about “offensive violence” did
you come up with any examples that were
tough to classify? Borderline cases, as it
were — an example that one person might
consider offensive but another might
reasonably regard as non-offensive. Yes,
well, so did we. This is going to happen
with all abstract concepts. It happens with
the concept of critical thinking as well.
There are people of whom we would say, on
certain occasions this person is a good
thinker, clear, logical, thoughtful, attentive to
the facts, open to alternatives, but, wow, at
other times, look out! When you get this
person on such-and-such a topic, well it is
all over then. You have pushed some kind
of button and the person does not want to
hear what anybody else has to say. The
person’s mind is made up ahead of time.
New facts are pushed aside. No other point
of view is tolerated.
Do you know any people that might
fit that general description?
Facione, PA, “Critical Thinking: What It is and Why it Counts”
Good. What can we learn about
critical thinking from such a case? Maybe
more than we can learn from just looking at
the easy cases. For when a case is on the
borderline, it forces us to make important
distinctions. It confronts us and demands a
decision: In or Out! And not just that, but
why? So, our friend who is fair-minded
about some things, but close-minded about
others, what to do? Let’s take the parts we
approve of because they seem to us to
contribute to acting rationally and logically
and include those in the concept of critical
thinking, and let’s take the parts that work
against reason, that close the mind to the
possibility of new and relevant information,
that blindly deny even the possibility that the
other side might have merit, and call those
poor, counterproductive, or uncritical
thinking.
2
Now, formulate a list of cases —
people that are clearly strong critical
thinkers and clearly weak critical thinkers
and some who are on the borderline.
Considering all those cases, what is it about
them that led you to decide which were
which? Suggestion: What can the strong
critical thinkers do (what mental abilities do
they have), that the weak critical thinkers
have trouble doing? What skills or
approaches do the strong critical thinkers
2
Spoken by the Vampire Marius in Ann Rice’s book The
Vampire Lestat Ballantine Books. New York, NY. 1985.
2011 update Page 4
habitually seem to exhibit which the weak
critical thinkers seem not to possess?
Core Critical Thinking Skills
Above we suggested you look for a
list of mental skills and habits of mind, the
experts, when faced with the same problem
you are working on, refer to their lists as
including cognitive skills and dispositions.
As to the cognitive skills here is what
the experts include as being at the very core
of critical thinking: interpretation, analysis,
evaluation, inference, explanation, and self-
regulation. (We will get to the dispositions
in just a second.) Did any of these words or
ideas come up when you tried to
characterize the cognitive skills — mental
abilities — involved in critical thinking?
Quoting from the consensus
statement of the national panel of experts:
interpretation is “to comprehend and
express the meaning or significance of a
wide variety of experiences, situations, data,
events, judgments, conventions, beliefs,
rules, procedures, or criteria.”3 Interpretation
3
The findings of expert consensus cited or reported
in this essay are published in Critical Thinking: A
Statement of Expert Consensus for Purposes of
Educational Assessment and Instruction. Peter A.
Facione, principle investigator, The California
Academic Press, Millbrae, CA, 1990. (ERIC ED 315
423). In 1993/94 the Center for the Study of Higher
Facione, PA, “Critical Thinking: What It is and Why it Counts”
includes the sub-skills of categorization,
decoding significance, and clarifying
meaning. Can you think of examples of
interpretation? How about recognizing a
problem and describing it without bias?
How about reading a person’s intentions in
the expression on her face; distinguishing a
main idea from subordinate ideas in a text;
constructing a tentative categorization or
way of organizing something you are
studying; paraphrasing someone’s ideas in
your own words; or, clarifying what a sign,
chart or graph means? What about
identifying an author’s purpose, theme, or
point of view? How about what you did
above when you clarified what “offensive
violence” meant?
Again from the experts: analysis is
“to identify the intended and actual
inferential relationships among statements,
questions, concepts, descriptions, or other
forms of representation intended to express
belief, judgment, experiences, reasons,
information, or opinions.” The experts
include examining ideas, detecting
arguments, and analyzing arguments as
sub-skills of analysis. Again, can you come
up with some examples of analysis? What
about identifying the similarities and
differences between two approaches to the
solution of a given problem? What about
picking out the main claim made in a
newspaper editorial and tracing back the
various reasons the editor offers in support
of that claim? Or, what about identifying
unstated assumptions; constructing a way
to represent a main conclusion and the
various reasons given to support or criticize
it; sketching the relationship of sentences or
paragraphs to each other and to the main
Education at The Pennsylvania State University
undertook a study of 200 policy-makers, employers,
and faculty members from two-year and four-year
colleges to determine what this group took to be the
core critical thinking skills and habits of mind. The
Pennsylvania State University Study, under the
direction of Dr. Elizabeth Jones, was funded by the
US Department of Education Office of Educational
Research and Instruction. The Penn State study
findings, published in 1994, confirmed the expert
consensus described in this paper.
2011 update Page 5
purpose of the passage? What about
graphically organizing this essay, in your
own way, knowing that its purpose is to give
a preliminary idea about what critical
thinking means?
The experts define evaluation as
meaning “to assess the credibility of
statements or other representations which
are accounts or descriptions of a person’s
perception, experience, situation, judgment,
belief, or opinion; and to assess the logical
strength of the actual or intended inferential
relationships among statements,
descriptions, questions or other forms of
representation.” Your examples? How
about judging an author’s or speaker’s
credibility, comparing the strengths and
weaknesses of alternative interpretations,
determining the credibility of a source of
information, judging if two statements
contradict each other, or judging if the
evidence at hand supports the conclusion
being drawn? Among the examples the
experts propose are these: “recognizing the
factors which make a person a credible
witness regarding a given event or a
credible authority with regard to a given
topic,” “judging if an argument’s conclusion
follows either with certainty or with a high
level of confidence from its premises,”
“judging the logical strength of arguments
based on hypothetical situations,” “judging if
a given argument is relevant or applicable
or has implications for the situation at hand.”
Do the people you regard as strong
critical thinkers have the three cognitive
skills described so far? Are they good at
interpretation, analysis, and evaluation?
What about the next three? And your
examples of weak critical thinkers, are they
lacking in these cognitive skills? All, or just
some?
To the experts inference means “to
identify and secure elements needed to
draw reasonable conclusions; to form
conjectures and hypotheses; to consider
relevant information and to educe the
consequences flowing from data,
Facione, PA, “Critical Thinking: What It is and Why it Counts”
statements, principles, evidence,
judgments, beliefs, opinions, concepts,
descriptions, questions, or other forms of
representation.” As sub-skills of inference
the experts list querying evidence,
conjecturing alternatives, and drawing
conclusions. Can you think of some
examples of inference? You might suggest
things like seeing the implications of the
position someone is advocating, or drawing
out or constructing meaning from the
elements in a reading. You may suggest
that predicting what will happen next based
what is known about the forces at work in a
given situation, or formulating a synthesis of
related ideas into a coherent perspective.
How about this: after judging that it would
be useful to you to resolve a given
uncertainty, developing a workable plan to
gather that information? Or, when faced
with a problem, developing a set of options
for addressing it. What about, conducting a
controlled experiment scientifically and
applying the proper statistical methods to
attempt to confirm or disconfirm an
empirical hypothesis?
Beyond being able to interpret,
analyze, evaluate and infer, strong critical
thinkers can do two more things. They can
explain what they think and how they
arrived at that judgment. And, they can
apply their powers of critical thinking to
themselves and improve on their previous
opinions. These two skills are called
“explanation” and “self-regulation.”
The experts define explanation as
being able to present in a cogent and
coherent way the results of one’s reasoning.
This means to be able to give someone a
full look at the big picture: both “to state and
to justify that reasoning in terms of the
evidential, conceptual, methodological,
criteriological, and contextual considerations
upon which one’s results were based; and
to present one’s reasoning in the form of
cogent arguments.” The sub-skills under
explanation are describing methods and
results, justifying procedures, proposing and
defending with good reasons one’s causal
2011 update Page 6
and conceptual explanations of events or
points of view, and presenting full and well-
reasoned, arguments in the context of
seeking the best understandings possible.
Your examples first, please... Here are
some more: to construct a chart which
organizes one’s findings, to write down for
future reference your current thinking on
some important and complex matter, to cite
the standards and contextual factors used
to judge the quality of an interpretation of a
text, to state research results and describe
the methods and criteria used to achieve
those results, to appeal to established
criteria as a way of showing the
reasonableness of a given judgment, to
design a graphic display which accurately
represents the subordinate and super-
ordinate relationship among concepts or
ideas, to cite the evidence that led you to
accept or reject an author’s position on an
issue, to list the factors that were
considered in assigning a final course
grade.
Maybe the most remarkable
cognitive skill of all, however, is this next
one. This one is remarkable because it
allows strong critical thinkers to improve
their own thinking. In a sense this is critical
thinking applied to itself. Because of that
some people want to call this “meta-
cognition,” meaning it raises thinking to
another level. But “another level” really
does not fully capture it, because at that
next level up what self-regulation does is
look back at all the dimensions of critical
thinking and double check itself. Self-
regulation is like a recursive function in
mathematical terms, which means it can
apply to everything, including itself. You
can monitor and correct an interpretation
you offered. You can examine and correct
an inference you have drawn. You can
review and reformulate one of your own
explanations. You can even examine and
correct your ability to examine and correct
yourself! How? It is as simple as stepping
back and saying to yourself, “How am I
doing? Have I missed anything important?
Let me double check before I go further.”
Facione, PA, “Critical Thinking: What It is and Why it Counts”
The experts define self-regulation
to mean “self-consciously to monitor one’s
cognitive activities, the elements used in
those activities, and the results educed,
particularly by applying skills in analysis,
and evaluation to one’s own inferential
judgments with a view toward questioning,
confirming, validating, or correcting either
one’s reasoning or one’s results.” The two
sub-skills here are self-examination and
self-correction. Examples? Easy — to
examine your views on a controversial issue
with sensitivity to the possible influences of
your personal biases or self-interest, to
check yourself when listening to a speaker
in order to be sure you are understanding
what the person is really saying without
introducing your own ideas, to monitor how
well you seem to be understanding or
comprehending what you are reading or
experiencing, to remind yourself to separate
your personal opinions and assumptions
from those of the author of a passage or
text, to double check yourself by
recalculating the figures, to vary your
reading speed and method mindful of the
type of material and your purpose for
reading, to reconsider your interpretation or
judgment in view of further analysis of the
facts of the case, to revise your answers in
view of the errors you discovered in your
work, to change your conclusion in view of
the realization that you had misjudged the
importance of certain factors when coming
to your earlier decision. 4
4
The California Critical Thinking Skills Test, and the
Test of Everyday Reasoning, the Health Science
Reasoning Test, the Military and Defense Reasoning
Profile, The Business Critical Thinking Skills Test,
and the Legal Studies Reasoning Profile along with
other testing instruments authored by Dr. Facione and
his research team for people in K-12, college, and
graduate / professional work target the core critical
thinking skills identified here. These instruments are
published in English and several authorized
translations exclusively by Insight Assessment.
2011 update Page 7
The Delphi Research Method
The panel of experts we keep
referring to included forty-six men and
women from throughout the United States
and Canada. They represented many
different scholarly disciplines in the
humanities, sciences, social sciences, and
education. They participated in a research
project that lasted two years and was
conducted on behalf of the American
Philosophical Association. Their work was
Facione, PA, “Critical Thinking: What It is and Why it Counts”
published under the title Critical Thinking: A
Statement of Expert Consensus for
Purposes of Educational Assessment and
Instruction. (The California Academic
Press, Millbrae, CA, 1990). You may
download the executive summary of that
report free. Visit
http://www.insightassessment.com
You might be wondering how such a
large group of people could collaborate on
2011 update Page 8
this project over that long a period of time
and at those distances and still come to
consensus. Good question. Remember
we’re talking the days before e-mail.
Not only did the group have to rely
on snail mail during their two-year
collaboration; they also relied on a method
of interaction, known as the Delphi Method,
which was developed precisely to enable
experts to think effectively about something
over large spans of distance and time. In
the Delphi Method a central investigator
organizes the group and feeds them an
initial question. [In this case it had to do
with how college level critical thinking
should be defined so that people teaching at
that level would know which skills and
dispositions to cultivate in their students.]
The central investigator receives all
responses, summarizes them, and transmits
them back to all the panelists for reactions,
replies, and additional questions.
Facione, PA, “Critical Thinking: What It is and Why it Counts”
Wait a minute! These are all well-
known experts, so what do you do if people
disagree? And what about the possible
influence of a big name person? Good
points. First, the central investigator takes
precautions to remove names so that the
panelists are not told who said what. They
know who is on the panel, of course. But
that is as far as it goes. After that each
experts’ argument has to stand on its own
merits. Second, an expert is only as good
as the arguments she or he gives. So, the
central investigator summarizes the
arguments and lets the panelists decide if
they accept them or not. When consensus
appears to be at hand, the central
investigator proposes this and asks if
people agree. If not, then points of
disagreement among the experts are
registered. We want to share with you one
important example of each of these. First
we will describe the expert consensus view
of the dispositions which are absolutely vital
to strong critical thinking. Then we will note
a point of separation among the experts.
2011 update Page 9
The Disposition Toward Critical
Thinking
What kind of a person would be apt
to use their critical thinking skills? The
experts poetically describe such a person
as having “a critical spirit.” Having a critical
spirit does not mean that the person is
always negative and hypercritical of
everyone and everything.
The experts use the metaphorical
phrase critical spirit in a positive sense. By
it they mean “a probing inquisitiveness, a
keenness of mind, a zealous dedication
to reason, and a hunger or eagerness for
reliable information.”
Almost sounds like Supreme Court
Justice Sandra Day O’Connor or Sherlock
Holmes The kind of person being described
here is the kind that always wants to ask
“Why?” or “How?” or “What happens if?”.
The one key difference, however, is that in
fiction Sherlock always solves the mystery,
while in the real world there is no guarantee.
Critical thinking is about how you approach
problems, questions, issues. It is the best
way we know of to get to the truth. But!
There still are no guarantees — no answers
in the back of the book of real life. Does
this characterization, that strong critical
thinkers possess a “critical spirit, a probing
inquisitiveness, a keenness of mind...” fit
with your examples of people you would call
strong critical thinkers?
But, you might say, I know people
who have skills but do not use them. We
cannot call someone a strong critical thinker
just because she or he has these cognitive
skills, however important they might be,
because what if they just do not bother to
apply them?
One response is to say that it is hard
to imagine an accomplished dancer who
never dances. After working to develop
those skills it seems such a shame to let
Facione, PA, “Critical Thinking: What It is and Why it Counts”
them grow weak with lack of practice. But
dancers get tired. And they surrender to the
stiffness of age or the fear of injury. In the
case of critical thinking skills, we might
argue that not using them once you have
them is hard to imagine. It’s hard to
imagine a person deciding not to think.
Considered as a form of thoughtful
judgment or reflective decision-making, in a
very real sense critical thinking is
pervasive. There is hardly a time or a
place where it would not seem to be of
potential value. As long as people have
purposes in mind and wish to judge how to
accomplish them, as long as people wonder
what is true and what is not, what to believe
and what to reject, strong critical thinking is
going to be necessary.
And yet weird things happen, so it is
probably true that some people might let
their thinking skills grow dull. It is easier to
imagine times when people are just too
tired, too lax, or too frightened. But imagine
it you can, Young Skywalker, so there has
to be more to critical thinking than just the
list of cognitive skills. Human beings are
more than thinking machines. And this
brings us back to those all-important
attitudes which the experts called
“dispositions.”
The experts were persuaded that
critical thinking is a pervasive and
purposeful human phenomenon. The ideal
critical thinker can be characterized not
merely by her or his cognitive skills but also
by how she or he approaches life and living
in general. This is a bold claim. Critical
2011 update Page 10
thinking goes way beyond the classroom.
In fact, many of the experts fear that some
of the things people experience in school
are actually harmful to the development and
cultivation of strong critical thinking. Critical
thinking came before schooling was ever
invented, it lies at the very roots of
civilization. It is a corner stone in the
journey human kind is taking from beastly
savagery to global sensitivity. Consider
what life would be like without the things on
this list and we think you will understand.
The approaches to life and living which
characterize critical thinking include:
* inquisitiveness with regard to a wide range of
issues,
* concern to become and remain well-informed,
* alertness to opportunities to use critical thinking,
* trust in the processes of reasoned inquiry,
* self-confidence in one’s own abilities to reason,
* open-mindedness regarding divergent world views,
* flexibility in considering alternatives and opinions
* understanding of the opinions of other people,
* fair-mindedness in appraising reasoning,
* honesty in facing one’s own biases, prejudices,
stereotypes, or egocentric tendencies,
* prudence in suspending, making or altering
judgments,
* willingness to reconsider and revise views where
honest reflection suggests that change is
warranted.
What would someone be like who
lacked those dispositions?
It might be someone who does not
care about much of anything, is not
interested in the facts, prefers not to think,
mistrusts reasoning as a way of finding
things out or solving problems, holds his or
her own reasoning abilities in low esteem, is
close-minded, inflexible, insensitive, cannot
understand what others think, is unfair when
it comes to judging the quality of arguments,
denies his or her own biases, jumps to
conclusions or delays too long in making
judgments, and never is willing to
reconsider an opinion. Not someone
Facione, PA, “Critical Thinking: What It is and Why it Counts”
prudent people would want to ask to
manage their investments!
The experts went beyond
approaches to life and living in general to
emphasize that strong critical thinkers can
also be described in terms of how they
approach specific issues, questions, or
problems. The experts said you would find
these sorts of characteristics:
* clarity in stating the question or concern,
* orderliness in working with complexity,
* diligence in seeking relevant information,
* reasonableness in selecting and applying criteria,
* care in focusing attention on the concern at hand,
* persistence though difficulties are encountered,
* precision to the degree permitted by the subject and
the circumstances.
So, how would a weak critical thinker
approach specific problems or issues?
Obviously, by being muddle-headed about
what he or she is doing, disorganized and
overly simplistic, spotty about getting the
facts, apt to apply unreasonable criteria,
easily distracted, ready to give up at the
least hint of difficulty, intent on a solution
that is more detailed than is possible, or
being satisfied with an overly generalized
and uselessly vague response. Remind you
of anyone you know?
Someone positively disposed toward
using critical thinking would probably agree
with statements like these:
“I hate talk shows where people shout their
opinions but never give any reasons at all.”
“Figuring out what people really mean by
what they say is important to me."
“I always do better in jobs where I'm
expected to think things out for myself.”
“I hold off making decisions until I have
thought through my options.”
“Rather than relying on someone else's
notes, I prefer to read the material myself.”
“I try to see the merit in another’s opinion,
even if I reject it later.”
“Even if a problem is tougher than I
expected, I will keep working on it.”
“Making intelligent decisions is more
important than winning arguments.”
2011 update Page 11

A person disposed to be averse or
hostile toward using critical thinking
would probably disagree with the
statements above but be likely to agree with
these:
“I prefer jobs where the supervisor says
exactly what to do and exactly how to do it."
“No matter how complex the problem, you
can bet there will be a simple solution.”
"I don't waste time looking things up."
“I hate when teachers discuss problems
instead of just giving the answers.”
“If my belief is truly sincere, evidence to the
contrary is irrelevant."
“Selling an idea is like selling cars, you say
whatever works."
We used the expression “strong
critical thinker” to contrast with the
expression “weak critical thinker.” But you
will find people who drop the adjective
“strong” (or “good”) and just say that
someone is a “critical thinker” or not. It is
like saying that a soccer (European
“football”) player is a “defender” or “not a
defender”, instead of saying the player’s
skills at playing defense are strong or weak.
People use the word “defender” in place of
the phrase “is good at playing defense.”
Similarly, people use “critical thinker” in
Facione, PA, “Critical Thinking: What It is and Why it Counts”
place of “is a strong critical thinker” or “has
strong critical thinking skills.” This is not
only a helpful conversational shortcut, it
suggests that to many people “critical
thinker” has a laudatory sense. The word
can be used to praise someone at the same
time that it identifies the person, as in “Look
at that play. That’s what I call a defender!”
“If we were compelled to make a
choice between these personal
attributes and knowledge about
the principles of logical
reasoning together with some
degree of technical skill in
manipulating special logical
processes, we should decide for
the former.”
John Dewey, How We Think, 1909. Republished as
How We Think: A Restatement of the Relation of
Reflective Thinking to the Educational Process. D. C.
Heath Publishing. Lexington, MA. 1933.
We said the experts did not come to
full agreement on something. That thing
has to do with the concept of a “strong
critical thinker.” This time the emphasis is
on the word “good” because of a crucial
2011 update Page 12
ambiguity it contains. A person can be
good at critical thinking, meaning that the
person can have the appropriate
dispositions and be adept at the cognitive
processes, while still not being a good (in
the moral sense) critical thinker. For
example, a person can be adept at
developing arguments and then, unethically,
use this skill to mislead and exploit a gullible
person, perpetrate a fraud, or deliberately
confuse and confound, and frustrate a
project.
The experts were faced with an
interesting problem. Some, a minority,
would prefer to think that critical thinking, by
its very nature, is inconsistent with the kinds
of unethical and deliberately
counterproductive examples given. They
find it hard to imagine a person who was
good at critical thinking not also being good
in the broader personal and social sense.
In other words, if a person were “really” a
“strong critical thinker” in the procedural
sense and if the person had all the
appropriate dispositions, then the person
simply would not do those kinds of
exploitive and aggravating things.

This self-rating form also appears in Chapter 3 of Think Critically, Pearson Education, 2011. For a fuller and more
robust measure of critical thinking dispositions see the California Critical Thinking Disposition Inventory (CCTDI) by
Facione and Facione, published in 1992, by Insight Assessment.

Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 13
 The large majority, however, hold
the opposite judgment. They are firm in the
view that strong critical thinking has nothing
to do with any given set of cultural beliefs,
religious tenants, ethical values, social
mores, political orientations, or orthodoxies
of any kind. Rather, the commitment one
makes as a strong critical thinker is to
always seek the truth with objectivity,
integrity, and fair-mindedness. The majority
of experts maintain that critical thinking
conceived of as we have described it above,
is, regrettably, not inconsistent with abusing
one’s knowledge, skills, or power. There
have been people with superior thinking
skills and strong habits of mind who,
unfortunately, have used their talents for
ruthless, horrific, and immoral purposes.
Would that it were not so! Would that
experience, knowledge, mental horsepower,
and ethical virtue were all one and the
same. But from the time of Socrates, if not
thousands of years before that, humans
have known that many of us have one or
more of these without having the full set.
Any tool, any approach to situations,
can go either way, ethically speaking,
depending on the character, integrity, and
principles of the persons who possess
them. So, in the final analysis the majority
of experts maintained that we cannot say a
person is not thinking critically simply
because we disapprove ethically of what the
person is doing. The majority concluded
that, “what ‘critical thinking’ means, why it is
of value, and the ethics of its use are best
regarded as three distinct concerns.”
Perhaps this realization forms part of
the basis for why people these days are
demanding a broader range of learning
outcomes from our schools and colleges.
“Knowledge and skills,” the staples of the
educational philosophy of the mid-twentieth
century, are not sufficient. We must look to
a broader set of outcomes including habits
of mind and dispositions, such as civic
engagement, concern for the common
good, and social responsibility.
Facione, PA, “Critical Thinking: What It is and Why it Counts”
“Thinking” in Popular Culture
We have said so many good things
about critical thinking that you might have
the impression that “critical thinking” and
“good thinking” mean the same thing. But
that is not what the experts said. They see
critical thinking as making up part of what
we mean by good thinking, but not as being
the only kind of good thinking. For example,
they would have included creative thinking
as part of good thinking.
Creative or innovative thinking is the
kind of thinking that leads to new insights,
novel approaches, fresh perspectives,
whole new ways of understanding and
conceiving of things. The products of
creative thought include some obvious
things like music, poetry, dance, dramatic
literature, inventions, and technical
innovations. But there are some not so
obvious examples as well, such as ways of
putting a question that expand the horizons
of possible solutions, or ways of conceiving
of relationships which challenge
presuppositions and lead one to see the
world in imaginative and different ways.
The experts working on the concept
of critical thinking wisely left open the entire
question of what the other forms good
thinking might take. Creative thinking is
only one example. There is a kind of
purposive, kinetic thinking that instantly
coordinates movement and intention as, for
example, when an athlete dribbles a soccer
ball down the field during a match. There is
a kind of meditative thinking which may
lead to a sense of inner peace or to
profound insights about human existence.
In contrast, there is a kind of hyper-alert,
instinctive thinking needed by soldiers in
battle. In the context of popular culture one
finds people proposing all kinds of thinking
or this-kind of intelligence or that-kind of
intelligence. Some times it is hard to sort out
the science from the pseudo-science – the
kernel of enduring truth from the latest
cocktail party banter.
2011 update Page 14
“Thinking” in Cognitive Science
Theories emerging from more scientific
studies of human thinking and decision-
making in recent years propose that thinking
is more integrated and less dualistic than
the notions in popular culture suggest. We
should be cautious about proposals
suggesting oversimplified ways of
understanding how humans think. We
should avoid harsh, rigid dichotomies such
as “reason vs. emotion,” “intuitive vs. linear,”
“creativity vs. criticality,” “right brained vs.
left brained,” “as on Mars vs. as on Venus.”
There is often a kernel of wisdom in
popular beliefs, and perhaps that gem this
time is the realization that some times we
decide things very quickly almost as
spontaneous, intuitive, reactions to the
situation at hand. Many accidents on the
freeways of this nation are avoided
Facione, PA, “Critical Thinking: What It is and Why it Counts”
precisely because drivers are able to see
and react to dangerous situations so
quickly. Many good decisions which feel
intuitive are really the fruit of expertise.
Decisions good drivers make in those
moments of crisis, just like the decisions
which practiced athletes make in the flow of
a game or the decisions that a gifted
teacher makes as she or he interacts with
students, are borne of expertise, training,
and practice.
At the same time that we are
immersed in the world around us and in our
daily lives, constantly making decisions
unreflectively, we may also be thinking quite
reflectively about something. Perhaps we’re
worried about a decision which we have to
make about an important project at work, or
about a personal relationship, or about a
legal matter, whatever. We gather
information, consider our options, explore
2011 update Page 15
possibilities, formulate some thoughts about
what we propose to do and why this choice
is the right one. In other words, we make a
purposeful, reflective judgment about what
to believe or what to do – precisely the kind
of judgment which is the focus of critical
thinking.
Recent integrative models of human
decision-making propose that the thinking
processes of our species is not best
described as a conflictive duality as in
“intuitive vs. reflective” but rather an
integrative functioning of two mutually
supportive systems “intuitive and reflective.”
These two systems of thinking are present
in all of us and can act in parallel to process
cognitively the matters over which we are
deciding.
One system is more intuitive,
reactive, quick and holistic. So as not to
confuse things with the notions of thinking in
popular culture, cognitive scientists often
name this system, “System 1.” The other
(yes, you can guess its name) is more
deliberative, reflective, computational and
rule governed. You are right, it is called
“System 2.”
In System 1 thinking, one relies
heavily on a number of heuristics (cognitive
maneuvers), key situational characteristics,
readily associated ideas, and vivid
memories to arrive quickly and confidently
at a judgment. System 1 thinking is
particularly helpful in familiar situations
when time is short and immediate action is
required.
While System 1 is functioning,
another powerful system is also at work,
that is, unless we shut it down by abusing
alcohol or drugs, or with fear or indifference.
Called “System 2,” this is our more
reflective thinking system. It is useful for
making judgments when you find yourself in
unfamiliar situations and have more time to
figure things out. It allows us to process
abstract concepts, to deliberate, to plan
ahead, to consider options carefully, to
Facione, PA, “Critical Thinking: What It is and Why it Counts”
review and revise our work in the light of
relevant guidelines or standards or rules of
procedure. While System 2 decisions are
also influenced by the correct or incorrect
application of heuristic maneuvers, this is
the system which relies on well articulated
reasons and more fully developed evidence.
It is reasoning based on what we have
learned through careful analysis, evaluation,
explanation, and self-correction. This is the
system which values intellectual honesty,
analytically anticipating what happens next,
maturity of judgment, fair-mindedness,
elimination of biases, and truth-seeking.
This is the system which we rely on to think
carefully trough complex, novel, high-
stakes, and highly integrative problems.5
Educators urge us to improve our
critical thinking skills and to reinforce our
disposition to use those skills because that
is perhaps the best way to develop and
refine our System 2 reasoning.
System 1 and System 2 are both
believed to be vital decision-making tools
when stakes are high and when uncertainty
is an issue. Each of these two cognitive
systems are believed to be capable of
functioning to monitor and potentially
override the other. This is one of the ways
our species reduces the chance of making
foolish, sub-optimal or even dangerous
errors in judgment. Human thinking is far
from perfect. Even a good thinker makes
both System 1 and 2 errors. At times we
misinterpret things, or we get our facts
wrong, and we make mistakes as a result.
5
Chapters 9 and 10 of Think Critically, Pearson
Education, 2011, locate critical thinking within this
integrative model of thinking. The cognitive heuristics,
which will be described next, and the human capacity
to derive sustained confidence decisions (right or
wrong),-- known as “dominance structuring,” – are
presented there too. There are lots of useful exercises
and examples in that book. You may also wish to
consult the references listed at the end of this essay.
The material presented in this section is derived from
these books and related publications by many of
these same authors and others working to
scientifically explain how humans actually make
decisions.
2011 update Page 16
But often our errors are directly related to
the influences and misapplications of
cognitive heuristics. Because we share the
propensity to use these heuristics as we
make decisions, let’s examine how some of
them influence us.
Cognitive heuristics are thinking
maneuvers which, at times, appear to be
almost hardwired into our species. They
influence both systems of thinking, the
intuitive thinking of System 1 and the
reflective reasoning of System 2. Five
heuristics often seem to be more frequently
operating in our System 1 reasoning are
known as availability, affect, association,
simulation, and similarity.
Availability, the coming to mind of a
story or vivid memory of something that
happened to you or to someone close to
you, inclines a person make inaccurate
estimates of the likelihood of that thing’s
happening again. People tell stories of
things that happened to themselves or their
friends all the time as a way of explaining
their own decisions. The stories may not be
scientifically representative, the events may
be mistaken, misunderstood, or
misinterpreted. But all that aside, the power
of the story is to guide, often in a good way,
the decision toward one choice rather than
another.
The Affect heuristic operates when
you have an immediate positive or an
Facione, PA, “Critical Thinking: What It is and Why it Counts”
negative reaction to some idea, proposal,
person, object, whatever. Sometimes called
a “gut reaction” this affective response sets
up an initial orientation in us, positive or
negative, toward the object. It takes a lot of
System 2 reasoning to overcome a powerful
affective response to an idea, but it can be
done. And at times it should be, because
there is no guarantee that your gut reaction
is always right.
The Association heuristic is
operating when one word or idea reminds
us of something else. For example, some
people associate the word “cancer” with
“death.” Some associate “sunshine” with
“happiness.” These kinds of associational
reasoning responses can be helpful at
times, as for example if associating cancer
with death leads you not to smoke and to go
in for regular checkups. At other times the
same association may influence a person to
make an unwise decision, as for example if
associating “cancer” with “death” were to
lead you to be so fearful and pessimistic
that you do not seek diagnosis and
treatment of a worrisome cancer symptom
until it was really too late to do anything.
The Simulation heuristic is working
when you are imagining how various
scenarios will unfold. People often imagine
how a conversation will go, or how they will
be treated by someone else when they
meet the person, or what their friends or
boss or lover will say and do when they
have to address some difficult issue. These
simulations, like movies in our heads, help
us prepare and do a better job when the
difficult moment arrives. But they can also
lead us to have mistaken expectations.
People may not respond as we imagined,
things may go much differently. Our
preparations may fail us because the ease
of our simulation misled us into thinking that
things would have to go as we had
imagined them. And they did not.
The Similarity heuristic operates
when we notice some way in which we are
like someone else and infer that what
2011 update Page 17
happened to that person is therefore more
likely to happen to us. The similarity
heuristic functions much like an analogical
argument or metaphorical model. The
similarity we focus on might be fundamental
and relevant, which would make the
inference more warranted. For example, the
boss fired your coworker for missing sales
targets and you draw the reasonable
conclusion that if you miss your sales
targets you’ll be fired too. Or the similarity
that comes to mind might be superficial or
not connected with the outcome, which
would make the inference unwarranted. For
example you see a TV commercial showing
trim-figured young people enjoying fattening
fast foods and infer that because you’re
young too you can indulge your cravings for
fast foods without gaining a lot of excess
unsightly poundage.
Heuristics and biases often
appearing to be somewhat more associated
with System 2 thinking include: satisficing,
risk/loss aversion, anchoring with
adjustment, and the illusion of control.
Satisficing occurs as we consider
our alternatives. When we come to one
which is good enough to fulfill our objectives
we often regard ourselves as having
completed our deliberations. We have
satisficed. And why not? The choice is,
after all, good enough. It may not be
perfect, it may not be optimal, it may not
even be the best among the options
available. But it is good enough. Time to
decide and move forward.
The running mate of satisficing is
temporizing. Temporizing is deciding that
the option which we have come to is “good
enough for now.” We often move through
life satisficing and temporizing. At times we
look back on our situations and wonder why
it is that we have settled for far less than we
might have. If we had only studied harder,
worked out a little more, taken better care of
ourselves and our relationships, perhaps we
would not be living as we are now. But, at
Facione, PA, “Critical Thinking: What It is and Why it Counts”
the time each of the decisions along the
way was “good enough for the time being.”
We are by nature a species that is
averse to risk and loss. Often we make
decisions on the basis of what we are too
worried about losing, rather than on the
basis of what we might gain. This works out
to be a rather serviceable approach in many
circumstances. People do not want to lose
control, they do not want to lose their
freedom, they do not want to lose their lives,
their families, their jobs, their possessions.
High stakes gambling is best left to those
who can afford to lose the money. Las
Vegas didn’t build all those multi-million
dollar casino hotels because vacationers
are winning all the time! And so, in real life,
we take precautions. We avoid
unnecessary risks. The odds may not be
stacked against us, but the consequences
of losing at times are so great that we would
prefer to forego the possibilities of gain in
order not to lose what we have. And yet, on
occasion this can be a most unfortunate
decision too. History has shown time and
time again that businesses which avoid
risks often are unable to compete
successfully with those willing to move more
boldly into new markets or into new product
lines.
Any heuristic is only a maneuver,
perhaps a shortcut or impulse to think or act
in one way rather than another, but certainly
not a failsafe rule. It may work out well
much of the time to rely on the heuristic, but
it will not work out for the best all of the
time.
For example, people with something
to lose tend toward conservative choices
politically as well as economically. Nothing
wrong with that necessarily. Just an
observation about the influence of Loss
Aversion heuristic on actual decision
making. We are more apt to endure the
status quo, even as it slowly deteriorates,
than we are to call for “radical” change.
Regrettably, however, when the call for
change comes, it often requires a far
2011 update Page 18
greater upheaval to make the necessary
transformations, or, on occasion, the
situation has deteriorated beyond the point
of no return. In those situations we find
ourselves wondering why we waited so long
before doing something.
The heuristic known as Anchoring
with Adjustment is operative when we find
ourselves making evaluative judgments.
The natural thing for us to do is to locate or
anchor our evaluation at some point along
whatever scale we are using. For example,
a professor says that the student’s paper is
a C+. Then, as other information comes our
way, we may adjust that judgment. The
professor, for example, may decide that the
paper is as good as some others that were
given a B-, and so adjust the grade upward.
The interesting thing about this heuristic, is
that we do not normally start over with a
fresh evaluation. We have dropped anchor
and we may drag it upward or downward a
bit, but we do not pull it off the bottom of the
sea to relocate our evaluation. First
impressions, as the saying goes, cannot be
undone. The good thing about this heuristic
is that it permits us to move on. We have
done the evaluation; there are other papers
to grade, other projects to do, other things in
life that need attention. We could not long
endure if we had to constantly re-evaluate
every thing anew. The unfortunate thing
about this heuristic is that we sometimes
drop anchor in the wrong place; we have a
hard time giving people a second chance at
making a good first impression.
The heuristic known as Illusion of
Control is evident in many situations. Many
of us over-estimate our abilities to control
what will happen. We make plans for how
we are going to do this or that, say this or
that, manipulate the situation this way or
that way, share or not share this information
or that possibility, all the time thinking that
some how our petty plans will enable us to
control what happens. We act as if others
are dancing on the ends of the strings that
we are pulling, when in actuality the
influences our words or actions have on
Facione, PA, “Critical Thinking: What It is and Why it Counts”
future events may be quite negligible. At
times we do have some measure of control.
For example we may exercise, not smoke,
and watch our diet in order to be more fit
and healthy. We are careful not to drink if
we are planning to drive so that we reduce
the risks of being involved in a traffic
accident. But at times we simply are
mistaken about our ability to actually
exercise full control over a situation. Sadly
we might become ill even if we do work hard
to take good care of ourselves. Or we may
be involved in an accident even if we are
sober. Our business may fail even if we
work very hard to make it a success. We
may not do as well on an exam as we might
hope even if we study hard.
Related to the Illusion of Control
heuristic is the tendency to misconstrue our
personal influence or responsibility for past
events. This is called Hindsight Bias. We
may over-estimate the influence our actions
have had on events when things go right, or
we may underestimate our responsibility or
culpability when things go wrong. We have
all heard people bragging about how they
did this and how they did that and, as a
result, such and such wonderful things
happened. We made these great plans and
look how well our business did financially.
Which may be true when the economy is
strong; but not when the economy is failing.
It is not clear how much of that success
came from the planning and how much
came from the general business
environment. Or, we have all been in the
room when it was time to own up for some
thing that went wrong and thought to
ourselves, hey, I may have had some part in
this, but it was not entirely my fault. “It
wasn’t my fault the children were late for
school, hey I was dressed and ready to go
at the regular time.” As if seeing that the
family was running late I had no
responsibility to take some initiative and
help out.
2011 update Page 19
“Insanity is doing the same
thing over and over again while
expecting a different outcome.”
Albert Einstein
Research on our shared heuristic
patterns of decision-making does not aim to
evaluate these patterns as necessarily good
or bad patterns of thinking. I fear that my
wording of them above may not have been
as entirely neutral and descriptive as
perhaps it should have been. In truth,
reliance on heuristics can be an efficient
ways of deciding things, given how very
complicated our lives are. We cannot
devote maximal cognitive resources to
every single decision we make.
Those of us who study these
heuristic thinking phenomena are simply
trying to document how we humans do
think. There are many useful purposes for
doing this. For example, if we find that
people repeatedly make a given kind of
mistake when thinking about a commonly
experienced problem, then we might find
ways to intervene and to help ourselves not
repeat that error over and over again.
This research on the actual patterns
of thinking used by individuals and by
groups might prove particularly valuable to
those who seek interventions which could
improve how we make our own heath care
decisions, how we make business
decisions, how we lead teams of people to
work more effectively in collaborative
settings, and the like.
Popular culture offers one other
myth about decision-making which is worth
questioning. And that is the belief that when
we make reflective decisions we carefully
weigh each of our options, giving due
consideration to all of them in turn, before
deciding which we will adopt. Although
perhaps it should be, research on human
decision-making shows that this simply is
Facione, PA, “Critical Thinking: What It is and Why it Counts”
not what happens.6 When seeking to
explain how people decide on an option with
such conviction that they stick to their
decision over time and with such confidence
that they act on that decision, the concept
that what we do is build a Dominance
Structure has been put forth. In a nutshell
this theory suggests that when we settle on
a particular option which is good enough we
tend to elevate its merits and diminish its
flaws relative to the other options. We raise
it up in our minds until it becomes for us the
dominant option. In this way, as our
decision takes shape, we gain confidence in
our choice and we feel justified in
dismissing the other options, even though
the objective distance between any of them
and our dominant option may not be very
great at all. But we become invested in our
dominant option to the extent that we are
able to put the other possibilities aside and
act on the basis of our choice. In fact, it
comes to dominate the other options in our
minds so much that we are able to sustain
our decision to act over a period of time,
rather than going back to re-evaluate or
reconsider constantly. Understanding the
natural phenomenon of dominance
structuring can help us appreciate why it
can be so difficult for us to get others to
change their minds, or why it seems that our
reasons for our decisions are so much
better than any of the objections which
others might make to our decisions. This is
not to say that we are right or wrong.
Rather, this is only to observe that human
beings are capable of unconsciously
building up defenses around their choices
which can result in the warranted or
unwarranted confidence to act on the basis
of those choices.
6
Henry Montgomery, “From cognition to action: The
search for dominance in decision making.” In Process
and Structure in Human Decision-Making,
Montgomery H, Svenson O (Eds). John Wiley & Sons:
Chichester, UK, 1989. For a more accessible
description along with reflective exercises on how to
avoid becoming “locked in” to a poor decision
prematurely, see chapter 10 of Think Critically.
2011 update Page 20
 Realizing the power of dominance
structuring, one can only be more
committed to the importance of education
and critical thinking. We should do all that
we can to inform ourselves fully and to
reflect carefully on our choices before we
make them, because we are, after all,
human and we are as likely as the next
person to believe that we are right and they
are wrong once the dominance structure
begins to be erected. Breaking through that
to fix bad decisions, which is possible, can
be much harder than getting things right in
the first place.
There are more heuristics than only
those mentioned above. There is more to
learn about dominance structuring as it
occurs in groups as well as in individuals,
and how to mitigate the problems which
may arise by prematurely settling on a
“good enough” option, or about how to craft
educational programs or interventions which
help people be more effective in their
System 1 and System 2 thinking. There is
much to learn about human thinking and
how to optimize it in individuals of different
ages; how to optimize the thinking of groups
of peers and groups where organizational
hierarchies influence interpersonal
dynamics. And, happily, there is a lot we
know today about human thinking and
decision-making that we did not know a few
years ago.
Why critical thinking?
Let us start with you first. Why
would critical thinking be of value to you to
have the cognitive skills of interpretation,
analysis, evaluation, inference, explanation,
and self-regulation?
Apart from, or maybe in light of, what
we said at the beginning of this essay about
the utility of positive critical thinking and
about the problems that failures of critical
thinking contribute to, why would it be of
value to you to learn to approach life and to
approach specific concerns with the critical
thinking dispositions listed above? Would
Facione, PA, “Critical Thinking: What It is and Why it Counts”
you have greater success in your work?
Would you get better grades?
Actually the answer to the grades
question, scientifically speaking, is very
possibly, Yes! A study of over 1100 college
students shows that scores on a college
level critical thinking skills test significantly
correlated with college GPA.7 It has also
been shown that critical thinking skills can
be learned, which suggests that as one
learns them one’s GPA might well improve.
In further support of this hypothesis is the
significant correlation between critical
thinking and reading comprehension.
Improvements in the one are paralleled by
improvements in the other. Now if you can
read better and think better, might you not
do better in your classes, learn more, and
get better grades. It is, to say the least,
very plausible.
Learning, Critical Thinking, and Our
Nation’s Future
“The future now belongs to societies
that organize themselves for learning...
nations that want high incomes and full
employment must develop policies that
emphasize the acquisition of knowledge
and skills by everyone, not just a select
few.”
Ray Marshall & Marc Tucker, Thinking For A Living:
Education And The Wealth of Nations, Basic Books. New
York. 1992.
But what a limited benefit — better
grades. Who really cares in the long run?
Two years after college, five years out, what
does GPA really mean? Right now college
7
Findings regarding the effectiveness of critical
thinking instruction, and correlations with GPA and
reading ability are reported in “Technical Report #1,
Experimental Validation and Content Validity” (ERIC
ED 327 549), “Technical Report #2, Factors
Predictive of CT Skills” (ERIC ED 327 550), and
“Gender, Ethnicity, Major, CT Self-Esteem, and the
California Critical Thinking Skills Test” (ERIC ED 326
584). These findings remain consistent in research
using the tools in the California Critical Thinking Skills
Test family of instruments published by Insight
Assessment.
2011 update Page 21
level technical and professional programs
have a half-life of about four years, which
means that the technical content is
expanding so fast and changing so much
that in about four years after graduation
your professional training will be in serious
need of renewal. So, if the only thing a
college is good for is to get the entry level
training and the credential needed for some
job, then college would be a time-limited
value.
The APA Delphi Report,
Critical Thinking: A Statement of Expert
Consensus for Purposes of Educational
Assessment and Instruction
1990 ERIC Doc. NO.: ED 315 423
Is that the whole story? A job is a
good thing, but is that what a college
education is all about, getting started in a
good job? Maybe some cannot see its
further value, but many do. A main
purpose, if not the main purpose, of the
collegiate experience, at either the two-year
or the four-year level, is to achieve what
people have called a “liberal education.”
Not liberal in the sense of a smattering of
this and that for no particular purpose
except to fulfill the unit requirement. But
liberal in the sense of “liberating.” And who
is being liberated? You! Liberated from a
kind of slavery. But from whom?
From professors. Actually from
dependence on professors so that they no
longer stand as infallible authorities
delivering opinions beyond our capacity to
Facione, PA, “Critical Thinking: What It is and Why it Counts”
challenge, question, and dissent. In fact,
this is exactly what the professors want.
They want their students to excel on their
own, to go beyond what is currently known,
to make their own contributions to
knowledge and to society. [Being a
professor is a curious job — the more
effective you are as a teacher, less your
students require your aid in learning.]
Liberal education is about learning
to learn, which means learning to think for
yourself on your own and in collaboration
with others.
Liberal education leads us away
from naive acceptance of authority, above
self-defeating relativism, and beyond
ambiguous contextualism. It culminates in
principled reflective judgment. Learning
critical thinking, cultivating the critical spirit,
is not just a means to this end, it is part of
the goal itself. People who are weak critical
thinkers, who lack the dispositions and skills
described, cannot be said to be liberally
educated, regardless of the academic
degrees they may hold.
Yes, there is much more to a liberal
education, than critical thinking. There is an
understanding of the methods, principles,
theories and ways of achieving knowledge
which are proper to the different intellectual
realms. There is an encounter with the
cultural, artistic and spiritual dimensions of
life. There is the evolution of one’s decision
making to the level of principled integrity
and concern for the common good and
social justice. There is the realization of the
ways all our lives are shaped by global as
well as local political, social, psychological,
economic, environmental, and physical
forces. There is the growth that comes from
the interaction with cultures, languages,
ethnic groups, religions, nationalities, and
social classes other than one’s own. There
is the refinement of one’s humane
sensibilities through reflection on the
recurring questions of human existence,
meaning, love, life and death. There is the
sensitivity, appreciation and critical
2011 update Page 22
appraisal of all that is good and all that is
bad in the human condition. As the mind
awakens and matures, and the proper
nurturing and educational nourishment is
provided, these others central parts of a
liberal education develop as well. Critical
thinking plays an essential role in achieving
these purposes.
Any thing else? What about going
beyond the individual to the community?
scale economic disaster would become
extremely likely. So, given a society that
does not value and cultivate critical thinking,
we might reasonably expect that in time the
judicial system and the economic system
would collapse. And, in such a society, one
that does not liberate its citizens by teaching
them to think critically for themselves, it
would be madness to advocate democratic
forms of government.

The experts say critical thinking is

fundamental to, if not essential for, “a
rational and democratic society.” What
might the experts mean by this?

Well, how wise would democracy be

if people abandoned critical thinking?
Imagine an electorate that cared not for the
facts, that did not wish to consider the pros
and cons of the issues, or if they did, had
not the brain power to do so. Imagine your
life and the lives of your friends and family
Is it any wonder that business and
placed in the hands of juries and judges
civic leaders are maybe even more
who let their biases and stereotypes govern
interested in critical thinking than
their decisions, who do not attend to the
educators? Critical thinking employed by an
evidence, who are not interested in
informed citizenry is a necessary condition
reasoned inquiry, who do not know how to
for the success of democratic institutions
draw an inference or evaluate one. Without
and for competitive free-market economic
critical thinking people would be more easily
enterprise. These values are so important
exploited not only politically but
that it is in the national interest that we
economically. The impact of abandoning
should try to educate all citizens so that they
critical thinking would not be confined to the
can learn to think critically. Not just for their
micro-economics of the household checking
personal good, but for the good of the rest
account. Suppose the people involved in
of us too.
international commerce were lacking in
critical thinking skills, they would be unable
Generalizing, imagine a society, say,
to analyze and interpret the market trends,
for example, the millions of people living in
evaluate the implications of interest
the Los Angeles basin, or in New York and
fluctuations, or explain the potential impact
along the east coast, or in Chicago, or
of those factors which influence large scale
Mexico City, Cairo, Rome, Tokyo, Baghdad,
production and distribution of goods and
Moscow, Beijing, or Hong Kong. They are,
materials. Suppose these people were
de facto, entirely dependent upon one
unable to draw the proper inferences from
another, and on hundreds of thousands of
the economic facts, or unable to properly
other people as well for their external
evaluate the claims made by the
supplies of food and water, for their survival.
unscrupulous and misinformed. In such a
Now imagine that these millions permitted
situation serious economic mistakes would
their schools and colleges to stop teaching
be made. Whole sectors of the economy
people how to think critically and effectively.
would become unpredictable and large

Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 23
Imagine that because of war, or AIDS, or
famine, or religious conviction, parents
could not or would not teach their children
how to think critically. Imagine the social
and political strife, the falling apart of
fundamental systems of public safety and
public health, the loss of any scientific
understanding of disease control or
agricultural productivity, the emergence of
paramilitary gangs, strong men, and petty
warlords seeking to protect themselves and
their own by acquiring control over what
food and resources they can and destroying
those who stand in their path.
Look at what has happened around
the world in places devastated by economic
embargoes, one-sided warfare, or the
HIV/AIDS epidemic. Or, consider the
problem of global climate change, and how
important it is for all of us to cooperate with
efforts to curtail our uses of fossil fuels in
order to reduce emissions of harmful
greenhouse gases.
Facione, PA, “Critical Thinking: What It is and Why it Counts”
Consider the “cultural revolutions”
undertaken by totalitarian rulers. Notice how
in virtually every case absolutist and
dictatorial despots seek ever more severe
limitations on free expression. They label
“liberal” intellectuals “dangers to society”
and expel “radical” professors from teaching
posts because they might “corrupt the
youth.” Some use the power of their
governmental or religious authority to crush
not only their opposition but the moderates
as well -- all in the name of maintaining the
purity of their movement. They intimidate
journalists and those media outlets which
dare to comment “negatively” on their
political and cultural goals or their heavy
handed methods.
The historical evidence is there for
us to see what happens when schools are
closed or converted from places of
education to places for indoctrination. We
know what happens when children are no
longer being taught truth-seeking, the skills
of good reasoning, or the lessons of human
history and basic science: Cultures
disintegrate; communities collapse; the
machinery of civilization fails; massive
numbers of people die; and sooner or later
social and political chaos ensues.
Or, imagine a media, a religious or
political hegemony which cultivated, instead
of critical thinking, all the opposite
dispositions? Or consider if that hegemony
reinforced uncritical, impulsive decision
making and the “ready-shoot-aim” approach
to executive action. Imagine governmental
structures, administrators, and community
leaders who, instead of encouraging critical
thinking, were content to make knowingly
irrational, illogical, prejudicial, unreflective,
short-sighted, and unreasonable decisions.
How long might it take for the people
in this society which does not value critical
thinking to be at serious risk of foolishly
harming themselves and each other?
The news too often reports about
hate groups, wanton shooting, terrorists and
2011 update Page 24
violently extreme religious zealots.
Education which includes a good measure
of critical thinking skills and dispositions like
truth-seeking and open-mindedness, is a
problem for terrorists and extremists of
every stripe because terrorists and
extremists want to control of what people
think. They are ideologists of the worst kind.
Their methods include indoctrination,
intimidation, and the strictest authoritarian
orthodoxy. In the “black-and-white” world of
“us vs. them” a good education would mean
that the people might begin to think for
themselves. And that is something these
extremists do not want.
History shows that assaults on
learning, whether by book burning, exile of
Facione, PA, “Critical Thinking: What It is and Why it Counts”
intellectuals, or regulations aimed at
suppressing research and frustrating the
fair-minded, evidence-based, and unfettered
pursuit of knowledge, can happen wherever
and whenever people are not vigilant
defenders of open, objective, and
independent inquiry.
Does this mean that society should place a
very high value on critical thinking?
Absolutely!
Does this mean society has the right to
force someone to learn to think critically?
Maybe. But, really, should we have to?
2011 update Page 25
 EXPERT CONSENSUS STATEMENT REGARDING CRITICAL
THINKING AND THE IDEAL CRITICAL THINKER

“We understand critical thinking to be purposeful, self-regulatory judgment

which results in interpretation, analysis, evaluation, and inference, as well
as explanation of the evidential, conceptual, methodological, criteriological,
or contextual considerations upon which that judgment is based. CT is
essential as a tool of inquiry. As such, CT is a liberating force in education
and a powerful resource in one’s personal and civic life. While not
synonymous with good thinking, CT is a pervasive and self-rectifying
human phenomenon. The ideal critical thinker is habitually inquisitive, well-
informed, trustful of reason, open-minded, flexible, fair-minded in
evaluation, honest in facing personal biases, prudent in making judgments,
willing to reconsider, clear about issues, orderly in complex matters,
diligent in seeking relevant information, reasonable in the selection of
criteria, focused in inquiry, and persistent in seeking results which are as
precise as the subject and the circumstances of inquiry permit. Thus,
educating strong critical thinkers means working toward this ideal. It
combines developing CT skills with nurturing those dispositions which
consistently yield useful insights and which are the basis of a rational and
democratic society.”

READINGS and REFERENCES

American Philosophical Association, Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment
and Instruction. "The Delphi Report," Committee on Pre-College Philosophy. (ERIC Doc. No. ED 315 423). 1990

Brookfield, Stephen D.: Developing Critical Thinkers: Challenging Adults to Explore Alternative Ways of Thinking and Acting. Josey-
Bass Publishers. San-Francisco, CA. 1987.

Browne, M. Neil, and Keeley, Stuart M.: Asking the Right Questions. Prentice-Hall Publishers. Englewood Cliffs, NJ. 2003.

Costa, Arthur L., & Lowery, l Lawrence F.: Techniques for Teaching Thinking. Critical Thinking Press and Software. Pacific Grove,
CA. 1989.

Facione, Noreen C, and Facione Peter A.: Critical Thinking and Clinical Judgment in the Health Sciences - An International
Teaching Anthology. The California Academic Press, Millbrae CA. 2008.

Facione, Noreen C. and Facione, Peter A. : Critical Thinking Assessment and Nursing Education Programs: An Aggregate Data
Analysis. The California Academic Press. Millbrae, CA 1997.

Facione, Noreen. C., and Facione, Peter A., Analyzing Explanations for Seemingly Irrational Choices, International Journal of
Applied Philosophy, Vol. 15 No. 2 (2001) 267-86.

Facione, Peter A and Noreen C,: Thinking and Reasoning in Human Decision Making. The California Academic Press. Millbrae CA,
2007

Facione, Peter A, Think Critically, Pearson Education: Englewood Cliffs, NJ, 2011.

Facione, P.A., Facione, N.C., Talking Critical Thinking, Change: The Magazine of Higher Education, March-April, 2007.

Facione, P.A., Facione N. C., and Giancarlo, C: The Disposition Toward Critical Thinking: Its Character, Measurement, and
Relationship to Critical Thinking Skills, Journal of Informal Logic, Vol. 20 No. 1 (2000) 61-84.

Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 26
Gilovich, Thomas; Griffin, Dale; and Kahneman, Daniel: Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge
University Press. 2002.

Goldstein, William, and Hogarth, Robin M. (Eds.): Research on Judgment and Decision Making. Cambridge University Press. 1997.

Esterle, John, and Clurman, Dan: Conversations with Critical Thinkers. The Whitman Institute. San Francisco, CA. 1993.

Janis, I.L. and Mann, L: Decision-Making. The Free Press, New York. 1977.

Kahneman, Daniel; Slovic, Paul; and Tversky, Amos: Judgment Under Uncertainty: Heuristics and Biases. Cambridge University
Press. 1982.

Kahneman Daniel: Knetsch, J.L.; and Thaler, R.H.: The endowment effect, loss aversion, and status quo bias. Journal of Economic
Perspectives. 1991, 5;193-206.

King, Patricia M. & Kitchener, Karen Strohm: Developing Reflective Judgment. Josey-Bass Publishers. San Francisco, CA. 1994

Kurfiss, Joanne G., Critical Thinking: Theory, Research, Practice and Possibilities, ASHE-ERIC Higher Education Report # 2,
Washington DC, 1988.

Marshall, Ray, and Tucker, Marc, Thinking for a Living: Education and the Wealth of Nations, Basic Books. New York, NY. 1992.

Resnick, L. W., Education and Learning to Think, National Academy Press, 1987.

Rubenfeld, M. Gaie, & Scheffer, Barbara K., Critical Thinking in Nursing: An Interactive Approach. J. B. Lippincott Company.
Philadelphia PA, 1995.

Siegel, Harvey: Educating Reason: Rationality, CT and Education. Routledge Publishing. New York. 1989.

Sternberg, Robert J.: Critical Thinking: Its Nature, Measurement, and Improvement. National Institute of Education, Washington DC,
1986.

Toulmin, Stephen: The Uses of Argument. Cambridge University Press, 1969.

Wade, Carole, and Tavris, Carol: Critical & Creative Thinking: The Case of Love and War. Harper Collins College Publisher. New
York. NY 1993.

GOVERNMENT REPORTS

U.S. Department of Education, Office of Educational Research and Improvement, National Center for Educational
Statistics (NCES) Documents National Assessment of College Student Learning: Getting Started, A Summary of Beginning
Activities. NCES 93-116.

National Assessment of College Student Learning: Identification of the Skills to Be Taught, Learned, and Assessed, A
Report on the Proceedings of the Second Design Workshop, November 1992. NCES 94-286.

National Assessment of College Student Learning: Identifying College Graduates' Essential Skills in Writing, Speech and
Listening, and Critical Thinking. NCES 95-001.

About the Author

Dr. Peter A. Facione and his co-investigators have been engaged in research and teaching about reasoning,
decision-making, and effective individual and group thinking processes since 1967. Over the years they developed
instruments to measure the core skills and habits of mind of effective thinking, these instruments are now in use in
many different languages throughout the world. Since 1992 Dr. Facione has presented hundreds of workshops about
effective teaching for thinking and about leadership, decision-making, leadership development, planning and
budgeting, and learning outcomes assessment at national and international professional association meetings and on
college and university throughout the nation. Dr. Facione, is a principal of the research and consulting firm,
Measured Reasons, and a Senior Researcher with Insight Assessment. He earned his Ph.D. at Michigan State in
1971, and in subsequent years chaired the Department of Philosophy at Bowling Green State University, served as
Dean of the School of Human Development and Community Service at California State University Fullerton, Dean of
the College of Arts and Sciences at Santa Clara University, and Provost of Loyola University Chicago. In 1999-2000
Dr. Facione was Chair of the American Conference of Academic Deans. He has been on many boards and panels,
including the California Commission on Teacher Credentialing and the ACE Presidents’ Task Force on Education. He
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 27
has contributed articles to The Chronicle of Higher Education, Change - The Magazine of Higher Education, and
Liberal Education. With Dr. Noreen Facione he co-authored Thinking and Reasoning in Human Decision Making
(2007) and Critical Thinking and Clinical Judgment in the Health Sciences (2008). From 1988 through 1990 Dr.
Facione was the principal investigator for the American Philosophical Association research project which culminated
in the Delphi Report – An Expert Consensus Conceptualization of Critical Thinking. The executive summary of that
report is available free of charge from Insight Assessment. His email is pfacione@measuredreasons.com Visit his
website www.measuredreasons.com and the Insight Assessment website, www.insightassessment.com

Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 28
Developing critical thinking Page 1 of 3

Developing critical thinking

The ability to think critically is a key skill for academic success.

It means not taking what you hear or read at face value, but using your critical faculties to weigh up the
evidence, and considering the implications and conclusions of what the writer is saying.

Imagine two situations. On the first, you are on a country walk and you come across a notice which tells
you not to attempt to climb a fence because of risk of electrocution. Would you pause to consider before
obeying this instruction? On the other hand, suppose you were to receive a letter from a local farmer
announcing that he proposed to put up an electric fence to protect a certain field. In this case, would you
not be more likely to think about his reasons for doing so and what the implications would be for you and
your family? In the first case, you are thinking reactively and in the second, you are thinking critically.

An allied skill is the ability to analyse – that is, to read or listen for the following points:

• How robust are the points presented as evidence?

• Does the author have a coherent argument, and do the points follow through logically from one
another or are their breaks in the sense?
Can you spot flaws?
• Is the conclusion clearly presented?
• Are there signs of bias or persuasion in the language, such as use of emotional appeal, or
indications that the author adheres to a particular school of thought or methodological perspective
(an example here might be that of someone whose methodological approach was strongly
quantitative, or qualitative)?
• How do the views presented differ from those of others in the field?

The key to critical thinking is to develop an impersonal approach which looks at arguments and facts and
which lays aside personal views and feelings. This is because academic discourse is based according to
key principles which are described as follows by Northedge (2005):

• Debate: arguing different points of view.

• Scholarship: awareness of what else has been written, and citing it correctly.
• Argument: developing points in a logical sequence which leads to a conclusion.
• Criticism: looking at strengths and weaknesses.
• Analysis: taking the argument apart, as described above.
• Evidence: ensuring that the argument is backed by valid evidence.
• Objectivity: the writing should be detached and unemotional and without direct appeal to the
reader.
• Precision: anything that does not assist the argument should be omitted.

Critical and analytical thinking should be applied at all points in academic study - to selecting information,
reading, writing, speaking and listening. Of these, learning to read and evaluate information critically is
perhaps the most important skill, which if acquired can then be applied to other areas.

Selecting information critically

The first stage in reading critically is to exercise care in the information you use - how trustworthy is it? For
printed material, consider:

• For books, who is the publisher? Is it a reputable academic publisher? Is the book part of a series
(in which case it will also have another layer of ‘quality control’, from the Series Editor).
• For journal articles, does the article appear in an academic journal? (Your tutor should be able to
tell you what the leading journals are in your field.)
• For both, who is the author and does he or she come from a respectable academic organization?
• How recent it the publication date, and are you using the latest edition of a textbook?

http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
Developing critical thinking Page 2 of 3

Particular care needs to be exercised when using information from the Internet. This will be the topic of
another article on this site, but you need to consider relevance and in particular:

• What is its source? Is it from a commercial or academic organization, and if the latter, is it from well
-known one? (For example, when I looked up ‘critical thinking’ I got a lot of commercial sites who
were trying to sell particular services such as software or consultancy.)
• Is it written in an academic style, with references,
substantiated claims etc.?
• There are many journals which are published on the Internet. Not all of these are subject to the
process of peer review, which involves the content being checked by people of standing.
• When was it posted/updated?

Reading critcally

When reading academic texts, you need to employ certain procedures.

1. Identify the argument – what is the author’s main line of reasoning?

2. Analyse and criticize the argument:

• Are the reasons sufficient, and are they valid to the argument, in other words do they support it, or
would it be possible to draw other conclusions from them?
• Does the author develop the argument in a logical and coherent fashion, i.e. premise/point A/point
B/conclusion, avoiding confusing breaks in the logical flow?
• Is the author’s logic always valid, or does he/she draw arguments from false premises, or are there
flaws in the reasoning assuming a causal connection where none is justifiable or generalizing from
too few examples?
• Is the author’s style objective, or does he/she use emotive language, designed to get the reader’s
sympathy, for example, words or phrases such as cruel, inhuman, Golden Age?

3. Assess the evidence:

• What is it – statistics, surveys, case studies, findings from experiment are all examples of evidence
that may be presented.
• Is it valid? Validity may be affected by external criteria such as the source (for example an article
from an academic journal is likely to be more reliable than one from a newspaper) or by the
particular bias of the party concerned (for example if a women’s hospital is resisting closure, look
carefully at evidence of other women’s services in the area). You should also examine the intrinsic
qualities of the evidence, for example how recent are case studies? How robust are experiments?
How large and representative is the survey? Is evidence anecdotal (for example, stories of one
person being cured from a particular treatment are less impressive than clinical trials)?

4. What are the conclusions, and are they supported by the evidence? It may be possible to present what
appears to be flawless research, which may yet not justify the conclusions. A good example here is the
ongoing debate on child care, and whether mothers are better off at home looking after their children
themselves. In the 1950s, John Bowlby presented good arguments why mothers should stay at home,
which was subsequently reputed by later researchers, whilst the stay at home argument is now making a
return. The studies themselves may not present valid evidence and need to be seen against other trends,
such as the need to ensure full male employment after the war, the rise of feminism, and women’s desire
for choice over whether or not they work.

5. What are the alternatives? Look at the author’s work from different perspectives - how does the view
presented differ from others? Does the author have a particular agenda, revealed (as in the case of a
particular view of research, see above) or hidden (for example, particular reasons, political or other, for
arguing a case)? Does the evidence really lead to the conclusions offered or might there be other
explanations (see the example in 4 above).

Air traffic in the Southeast of the country is becoming increasingly great, there are
three airports and the plan is to expand the airport at Lutwick to ease congestion at
the other airports and help with the expected tripling of demand by 2030 (1). Pollution
from aircraft is one of the biggest problems of our times. (2) A recent survey of local
residents by the Lutwick Times showed that 60 per cent of local residents would
oppose the plan. (3) Lutwick was severely bombed in the war (4), and has suffered
enough (5) without the further incursion of gridlocked motorways which would result
from the enlarged airport. (6) Forecasts show that many more houses will be affected
by noise pollution than with other airports, that the projected increase in jobs is

http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
Developing critical thinking Page 3 of 3

dubious, and that flooding may result from hard surface run-off (7). We must therefore
oppose the plans and press for an environmental impact tax on aviation (8).

A critique of the above passage

1. The first sentence is a descriptive statement: unclear what the author’s premise is.
2. The second sentence is a non sequitur - there is no explicit link with the first statement.
3. A survey is a piece of evidence, but how reliable is the source, 60 per cent of what number and
when was the survey carried out?
4. That Lutwick was bombed during the war is a non sequitur, and is not essential to the case.
5. This is emotive language.
6. This assumes a causal connection between the enlarged airport and congested motorways, but
there may be other reasons why motorways are congested.
7. These statements constitute evidence, but they are not substantiated, referenced or quantified.
What forecasts, how many more houses over what area, where will the flooding be and why will it
result from hard surface run-off?
8. The conclusion is clearly stated, but its first part (that plans must be opposed) clearly shows the
bias of the writer, and the second part (the environmental impact tax) does not necessarily follow
from the evidence, which is specific to a particular location. In order to arrive at that conclusion it is
necessary to report evidence on a more general link between flights and the environment, to argue
for the practicality of the tax looking at opposing arguments such as would it price air travel out of
some people’s reach and would this be a bad thing?

Writing Critically

Much has been written elsewhere on this site about the writing process, so we will only make brief
reference here. Planning is the key: if you organize your ideas carefully in your plan, you will be clearer
what you have to write.

You need to employ the same critical judgement to your own writing as you do to that of other people,
although it can be more difficult to assess your own work! In particular:

• Check your line of reasoning is clear – start out by stating what you propose to do, organize your
information in a logical pattern, and reach a clear and substantiated conclusion.
• Ensure that the evidence you use is valid according to the criteria set out above, under Reading
critically.
• Be aware of the difference between descriptive writing, which tells a story, using statements,
explanation and lists etc., and analytical writing, which presents an argument, giving reasons,
weighing up information, and drawing conclusions.

Listening and speaking critically

Much of learning is carried out by dialogue, and by tossing ideas around, but you can’t expect yourself or
others always to substantiate with the same degree of rigour as in writing! However, listen out in yourself
and other people for inconsistencies and contradictions; if you are in a seminar, notice how ideas are
‘developed’ through dialogue, how your ideas fit in or contradict with those of others, etc. Be as prepared
to ask questions as you are to listen, for example if someone offers a point of view about a particular
author or text, don’t be afraid to challenge them to substantiate their claim. You will also need to put
forward a reasoned argument, which will help develop your thinking skills, particularly as verbal debate
proceeds at a more rapid pace than writing or reading, which are mostly solitary activities. Remember, too,
that you have extra ‘data’ in the form of body language - does the latter fit in with what they are saying, or
are you noticing contradictory signals, for example, a raised eyebrow?

Reference

• Northedge, A. (2005), The Good Study Guide, Open University Press, Buckingham, UK

Printed from: http://0-

www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_thinking.htm on Thursday
November 4th, 2010
© Emerald Group Publishing Limited

http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
 http://logical-critical-thinking.com/critical-thinking/10-ways-or-techniques-to-help-improve-critical-thinking/

Critical Thinking – 10 Ways to Improve your Critical Thinking Skills

February 23, 2011
Here is a list of 10 ways or techniques you can try to help improve critical thinking.
1. Question the Evidence 6. Learn about Cognitive Bias
2. Establish Criteria 7. Identify your Emotions
3. Learn about Logic 8. Identify your Genetic Instincts
4. Learn about Logical Fallacies 9. Question your indoctrination
5. Identify the Premise 10. Try to Prove yourself Wrong
1. Question the Evidence
Whenever you are analyzing something try to to question the validity of the evidence. Is the evidence
correct? What reasons do we have for trusting the evidence? Claims should be backed up with some
evidence that can stand up to skepticism.
2. Establish Criteria
It is good to establish a set criteria when analyzing subjects. As an example if we are choosing which car to
buy we might set certain criteria on gas mileage, mileage, brand, and price. We would then apply these
criteria fairly to each car in our evaluations. The same might go for philosophies or religions. We would need
to establish criteria by which to judge them evenly and fairly.
3. Learn Logic
Understanding logic will help us process information better and formulate better beliefs, claims, and
arguments. Learning about logic goes along way to help us improve our critical thinking, the two are like
peanut butter and jelly.
4. Learn about Logical Fallacies
Learning about logical fallacies will help you identify others claims that are built in a logically fallacious
manner and structure. Another reason to learn about logical fallacies is to prevent your self from making
claims and arguments that are structurally fallacious.
5. Identify the Premise
When hearing a claim or analyzing a statement it is important to identify the premise. When you identify the
premise you can decide if it is valid or agreeable. If you do not agree with the premise of a statement or
claim you cannot continue further. Instead, you must regress back and try to figure out a valid premise, if
one exists.
6. Learn about Cognitive Bias
A Cognitive Bias is a weird trick where the mind distorts reality. Having a knowledge of what cognitive biases
are will help you identify them when you do them and when others do them.
7. Identify your Emotions
Knowing your emotions will help you understand your urges and decisions better. Being able to recognize
and identify your emotions will help you key into actions and behaviours that seem good at the time but are
actually not appropriate.
8. Identify Genetic Instincts
Deep genetic instincts might drive us to behaviours and actions that are not critically thought out. The
loyalty to the tribe, a deep genetic instinct, might make us violent toward others where this type of
behaviour is completely unwarranted, unjust, and immoral.
9. Question your Indoctrination
We are all indoctrinated into things growing up. Whether it is jingoistic patriotism, religion, racial, or political
almost all of us have been indoctrinated into a belief. In order to think critically we need to question and test
the validity of these indoctrinations.
10. Try to Prove yourself Wrong
In order to think critically we have to admit we are wrong a lot. No one has all the answers. If you try to
prove your beliefs and ideas wrong and test them you will help yourself admit error more. The more errors
we get rid of in our beliefs the more likely we are to hold good logical beliefs.
 http://www.forbes.com/sites/work-in-progress/2012/03/27/how-to-develop-5-critical-thinking-types/

How to Develop 5 Critical Thinking Types

Holly Green , Contributor - 27 Mar 2012
Great leaders think strategically.
They can understand and appreciate the current state as well as see possibilities. When dealing with today’s issues, they
operate from a broad, long-term perspective rather than focusing only on short-term implications. And they can gather
information and make decisions in a timely manner.
Most of all, strategic leaders know how to strike a balance between visualizing what might or could be and an effective day-
to-day approach to implementation. They can look into the future to see where the company needs to go and what it will
look like once they get there. And they can do this while making sure the right things get done on a daily basis.
This type of strategic leadership requires five different types of thinking. Knowing when and how much to utilize each one is
the hallmark of great leaders.
1. Critical thinking is the mental process of objectively analyzing a situation by gathering information from all possible
sources, and then evaluating both the tangible and intangible aspects, as well as the implications of any course of
action.
2. Implementation thinking is the ability to organize ideas and plans in a way that they will be effectively carried out.
3. Conceptual thinking consists of the ability to find connections or patterns between abstract ideas and then piece
them together to form a complete picture.
4. Innovative thinking involves generating new ideas or new ways of approaching things to create possibilities and
opportunities.
5. Intuitive thinking is the ability to take what you may sense or perceive to be true and, without knowledge or
evidence, appropriately factor it in to the final decision.
Until recently, most leaders could get by with critical and implementation thinking. But in today’s hyper-fast world,
conceptual, innovative and intuitive thinking have becoming increasingly important, especially in industries where frenetic
change represents the rule rather than the exception.
Business leaders still need to gather and analyze data, make decisions, and implement them well. But now they have to take
in vast amounts of data from a more diverse array of sources. They have to make decisions much more quickly. And they
have to do it knowing that everything could change overnight.
In such an environment, the ability to ponder possibilities, see patterns and connections that others don’t see, and look at
the same data in new and different ways represents a formidable competitive advantage.
Some leaders seem to be born with these intuitive types of thinking skills. But since most of us are not so naturally gifted,
here are some suggestions for developing these essential leadership skills.
• Take time to look around. Browse business websites and read related publications to learn how other organizations
have implemented various strategies in order to increase their competitive advantage.
• Be willing to change directions and/or pursue new goals when strategic opportunities arise. Think about what is
keeping you on the same path and force yourself to ponder whether or not you should shift plans. Consider worst-
case scenarios.
• When problems arise, don’t settle for a quick fix. Instead, carefully look at the problem and take the time to analyze
all possible solutions. Create a checklist for yourself to trigger thoughts on long-term consequences and possibilities.
• Help others in the organization feel that they are part of the overall mission and strategies by discussing it with them
frequently and involving them as much as possible.
• Pause and view your situation from another perspective – that of an employee, customer, supplier, etc.
• Research and analyze your company’s major competitors. Create a detailed profile of each one and share it with
your team. Constantly look for first-hand data rather than relying on anecdotal information.
• Engage in “what-if” thinking. For example, “If we do this, how will our competitors respond? What will our
customers think? What impact will this have on our suppliers and distributors? What if there is something we have
not considered?”
• Expand your data sources to include areas totally outside your business or industry. Analyze other industries to see
what they’re doing well and how that could be adapted to your business.
Most of all, get in the habit of stimulating your mind by not thinking about your business. From time to time, go outside your
office and take a walk. Turn off your processing and just soak in the sights, sounds, and scents of the environment. Let your
mind wander, and allow yourself the luxury of daydreaming. You’ll be amazed at what you can come up with simply by
shifting out of the critical/implementation thinking modes from time to time.
The human brain is a powerful leadership tool. It works even better when you use all five thinking types!
 An Article Published in ‘Nature’, International Weekly Journal of Science
http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183

Policy: Twenty tips for interpreting scientific claims

By William J. Sutherland, David Spiegelhalter & Mark Burgman
20 November 2013
Science and policy have collided on contentious issues such as bee declines, nuclear power and the role
of badgers in bovine tuberculosis.
Calls for the closer integration of science in political decision-making have been commonplace for
decades. However, there are serious problems in the application of science to policy — from energy to
health and environment to education.
One suggestion to improve matters is to encourage more scientists to get involved in politics. Although
laudable, it is unrealistic to expect substantially increased political involvement from scientists. Another
proposal is to expand the role of chief scientific advisers1, increasing their number, availability and
participation in political processes. Neither approach deals with the core problem of scientific ignorance
among many who vote in parliaments.
Perhaps we could teach science to politicians? It is an attractive idea, but which busy politician has
sufficient time? In practice, policy-makers almost never read scientific papers or books. The research
relevant to the topic of the day — for example, mitochondrial replacement, bovine tuberculosis or
nuclear-waste disposal — is interpreted for them by advisers or external advocates. And there is rarely,
if ever, a beautifully designed double-blind, randomized, replicated, controlled experiment with a large
sample size and unambiguous conclusion that tackles the exact policy issue.
In this context, we suggest that the immediate priority is to improve policy-makers' understanding of
the imperfect nature of science. The essential skills are to be able to intelligently interrogate experts and
advisers, and to understand the quality, limitations and biases of evidence. We term these interpretive
scientific skills. These skills are more accessible than those required to understand the fundamental
science itself, and can form part of the broad skill set of most politicians.
To this end, we suggest 20 concepts that should be part of the education of civil servants, politicians,
policy advisers and journalists — and anyone else who may have to interact with science or scientists.
Politicians with a healthy scepticism of scientific advocates might simply prefer to arm themselves with
this critical set of knowledge.
We are not so naive as to believe that improved policy decisions will automatically follow. We are fully
aware that scientific judgement itself is value-laden, and that bias and context are integral to how data
are collected and interpreted. What we offer is a simple list of ideas that could help decision-makers to
parse how evidence can contribute to a decision, and potentially to avoid undue influence by those with
vested interests. The harder part — the social acceptability of different policies — remains in the hands
of politicians and the broader political process.
Of course, others will have slightly different lists. Our point is that a wider understanding of these 20
concepts by society would be a marked step forward.
Differences and chance cause variation. The real world varies unpredictably. Science is mostly about
discovering what causes the patterns we see. Why is it hotter this decade than last? Why are there more
birds in some areas than others? There are many explanations for such trends, so the main challenge of
research is teasing apart the importance of the process of interest (for example, the effect of climate
change on bird populations) from the innumerable other sources of variation (from widespread
changes, such as agricultural intensification and spread of invasive species, to local-scale processes, such
as the chance events that determine births and deaths).
No measurement is exact. Practically all measurements have some error. If the measurement process
were repeated, one might record a different result. In some cases, the measurement error might be
large compared with real differences. Thus, if you are told that the economy grew by 0.13% last month,
there is a moderate chance that it may actually have shrunk. Results should be presented with a
precision that is appropriate for the associated error, to avoid implying an unjustified degree of
accuracy.
Bias is rife. Experimental design or measuring devices may produce atypical results in a given direction.
For example, determining voting behaviour by asking people on the street, at home or through the
Internet will sample different proportions of the population, and all may give different results. Because
studies that report 'statistically significant' results are more likely to be written up and published, the
scientific literature tends to give an exaggerated picture of the magnitude of problems or the
effectiveness of solutions. An experiment might be biased by expectations: participants provided with a
treatment might assume that they will experience a difference and so might behave differently or report
an effect. Researchers collecting the results can be influenced by knowing who received treatment. The
ideal experiment is double-blind: neither the participants nor those collecting the data know who
received what. This might be straightforward in drug trials, but it is impossible for many social studies.
Confirmation bias arises when scientists find evidence for a favoured theory and then become
insufficiently critical of their own results, or cease searching for contrary evidence.
Bigger is usually better for sample size. The average taken from a large number of observations will
usually be more informative than the average taken from a smaller number of observations. That is, as
we accumulate evidence, our knowledge improves. This is especially important when studies are
clouded by substantial amounts of natural variation and measurement error. Thus, the effectiveness of
a drug treatment will vary naturally between subjects. Its average efficacy can be more reliably and
accurately estimated from a trial with tens of thousands of participants than from one with hundreds.
Correlation does not imply causation. It is tempting to assume that one pattern causes another.
However, the correlation might be coincidental, or it might be a result of both patterns being caused by
a third factor — a 'confounding' or 'lurking' variable. For example, ecologists at one time believed that
poisonous algae were killing fish in estuaries; it turned out that the algae grew where fish died. The
algae did not cause the deaths2.
Regression to the mean can mislead. Extreme patterns in data are likely to be, at least in part,
anomalies attributable to chance or error. The next count is likely to be less extreme. For example, if
speed cameras are placed where there has been a spate of accidents, any reduction in the accident rate
cannot be attributed to the camera; a reduction would probably have happened anyway.
Extrapolating beyond the data is risky. Patterns found within a given range do not necessarily apply
outside that range. Thus, it is very difficult to predict the response of ecological systems to climate
change, when the rate of change is faster than has been experienced in the evolutionary history of
existing species, and when the weather extremes may be entirely new.
Beware the base-rate fallacy. The ability of an imperfect test to identify a condition depends upon the
likelihood of that condition occurring (the base rate). For example, a person might have a blood test that
is '99% accurate' for a rare disease and test positive, yet they might be unlikely to have the disease. If
10,001 people have the test, of whom just one has the disease, that person will almost certainly have a
positive test, but so too will a further 100 people (1%) even though they do not have the disease. This
type of calculation is valuable when considering any screening procedure, say for terrorists at airports.
Controls are important. A control group is dealt with in exactly the same way as the experimental
group, except that the treatment is not applied. Without a control, it is difficult to determine whether a
given treatment really had an effect. The control helps researchers to be reasonably sure that there are
no confounding variables affecting the results. Sometimes people in trials report positive outcomes
because of the context or the person providing the treatment, or even the colour of a tablet3. This
underlies the importance of comparing outcomes with a control, such as a tablet without the active
ingredient (a placebo).
Randomization avoids bias. Experiments should, wherever possible, allocate individuals or groups to
interventions randomly. Comparing the educational achievement of children whose parents adopt a
health programme with that of children of parents who do not is likely to suffer from bias (for example,
better-educated families might be more likely to join the programme). A well-designed experiment
would randomly select some parents to receive the programme while others do not.
Seek replication, not pseudoreplication. Results consistent across many studies, replicated on
independent populations, are more likely to be solid. The results of several such experiments may be
combined in a systematic review or a meta-analysis to provide an overarching view of the topic with
potentially much greater statistical power than any of the individual studies. Applying an intervention to
several individuals in a group, say to a class of children, might be misleading because the children will
have many features in common other than the intervention. The researchers might make the mistake of
'pseudoreplication' if they generalize from these children to a wider population that does not share the
same commonalities. Pseudoreplication leads to unwarranted faith in the results. Pseudoreplication of
studies on the abundance of cod in the Grand Banks in Newfoundland, Canada, for example, contributed
to the collapse of what was once the largest cod fishery in the world4.
Scientists are human. Scientists have a vested interest in promoting their work, often for status and
further research funding, although sometimes for direct financial gain. This can lead to selective
reporting of results and occasionally, exaggeration. Peer review is not infallible: journal editors might
favour positive findings and newsworthiness. Multiple, independent sources of evidence and replication
are much more convincing.
Significance is significant. Expressed as P, statistical significance is a measure of how likely a result is to
occur by chance. Thus P = 0.01 means there is a 1-in-100 probability that what looks like an effect of the
treatment could have occurred randomly, and in truth there was no effect at all. Typically, scientists
report results as significant when the P-value of the test is less than 0.05 (1 in 20).
Separate no effect from non-significance. The lack of a statistically significant result (say a P-value >
0.05) does not mean that there was no underlying effect: it means that no effect was detected. A small
study may not have the power to detect a real difference. For example, tests of cotton and potato crops
that were genetically modified to produce a toxin to protect them from damaging insects suggested that
there were no adverse effects on beneficial insects such as pollinators. Yet none of the experiments had
large enough sample sizes to detect impacts on beneficial species had there been any5.
Effect size matters. Small responses are less likely to be detected. A study with many replicates might
result in a statistically significant result but have a small effect size (and so, perhaps, be unimportant).
The importance of an effect size is a biological, physical or social question, and not a statistical one. In
the 1990s, the editor of the US journal Epidemiology asked authors to stop using statistical significance
in submitted manuscripts because authors were routinely misinterpreting the meaning of significance
tests, resulting in ineffective or misguided recommendations for public-health policy6.
Study relevance limits generalizations. The relevance of a study depends on how much the conditions
under which it is done resemble the conditions of the issue under consideration. For example, there are
limits to the generalizations that one can make from animal or laboratory experiments to humans.
Feelings influence risk perception. Broadly, risk can be thought of as the likelihood of an event
occurring in some time frame, multiplied by the consequences should the event occur. People's risk
perception is influenced disproportionately by many things, including the rarity of the event, how much
control they believe they have, the adverseness of the outcomes, and whether the risk is voluntarily or
not. For example, people in the United States underestimate the risks associated with having a handgun
at home by 100-fold, and overestimate the risks of living close to a nuclear reactor by 10-fold7.
Dependencies change the risks. It is possible to calculate the consequences of individual events, such as
an extreme tide, heavy rainfall and key workers being absent. However, if the events are interrelated,
(for example a storm causes a high tide, or heavy rain prevents workers from accessing the site) then
the probability of their co-occurrence is much higher than might be expected8. The assurance by credit-
rating agencies that groups of subprime mortgages had an exceedingly low risk of defaulting together
was a major element in the 2008 collapse of the credit markets.
Data can be dredged or cherry picked. Evidence can be arranged to support one point of view. To
interpret an apparent association between consumption of yoghurt during pregnancy and subsequent
asthma in offspring9, one would need to know whether the authors set out to test this sole hypothesis,
or happened across this finding in a huge data set. By contrast, the evidence for the Higgs boson
specifically accounted for how hard researchers had to look for it — the 'look-elsewhere effect'. The
question to ask is: 'What am I not being told?'
Extreme measurements may mislead. Any collation of measures (the effectiveness of a given school,
say) will show variability owing to differences in innate ability (teacher competence), plus sampling
(children might by chance be an atypical sample with complications), plus bias (the school might be in an
area where people are unusually unhealthy), plus measurement error (outcomes might be measured in
different ways for different schools). However, the resulting variation is typically interpreted only as
differences in innate ability, ignoring the other sources. This becomes problematic with statements
describing an extreme outcome ('the pass rate doubled') or comparing the magnitude of the extreme
with the mean ('the pass rate in school x is three times the national average') or the range ('there is an x-
fold difference between the highest- and lowest-performing schools'). League tables, in particular, are
rarely reliable summaries of performance.
Nature 503, 335–337 (21 November 2013) doi: 10.1038/503335a
References
1. Doubleday, R. & Wilsdon, J. Nature 485, 301–302 (2012).
o Article
o PubMed
o ISI
o ChemPort
Show context
2. Borsuk, M. E., Stow, C. A. & Reckhow, K. H. J. Water Res. Plan. Manage. 129, 271–282 (2003).
o Article
Show context
3. Huskisson, E. C. Br. Med. J. 4, 196–200 (1974)
o Article
o PubMed
o ChemPort
Show context
4. Millar, R. B. & Anderson, M. J. Fish. Res. 70, 397–407 (2004).
o Article
Show context
5. Marvier, M. Ecol. Appl. 12, 1119–1124 (2002).
o Article
o ISI
Show context
6. Fidler, F., Cumming, G., Burgman, M., Thomason, N. J. Socio-Economics 33, 615–630 (2004).
o Article
Show context
7. Fischhoff, B., Slovic, P. & Lichtenstein, S. Am. Stat. 36, 240–255 (1982).
o ISI
Show context
8. Billinton, R. & Allan, R. N. Reliability Evaluation of Power Systems (Plenum, 1984).
Show context
9. Maslova, E., Halldorsson, T. I., Strøm, M., Olsen, S. F. J. Nutr. Sci. 1, e5 (2012).
o Article
o PubMed
Author information
Affiliations
1. William J. Sutherland is professor of conservation biology in the Department of Zoology, University of
Cambridge, UK.
2. David Spiegelhalter is at the Centre for Mathematical Sciences, University of Cambridge.
3. Mark Burgman is at the Centre of Excellence for Biosecurity Risk Analysis, School of Botany, University
of Melbourne, Parkville, Australia.

Corresponding author
Correspondence to: William J. Sutherland
Comments (19 comments)
Tal Galili • 2013-12-08 06:39 PM
While the article offer nice simplifications of many important topics in understanding scientific
claims, the description of significance (e.g: p-value), is severely lacking. To quote Wikipedia's
section "misunderstandings of p-value" on this: 1) The p-value is not the probability that the null
hypothesis is true, nor is it the probability that the alternative hypothesis is false – it is not
connected to either of these. In fact, frequentist statistics does not, and cannot, attach
probabilities to hypotheses. Comparison of Bayesian and classical approaches shows that a p-
value can be very close to zero while the posterior probability of the null is very close to unity (if
there is no alternative hypothesis with a large enough a priori probability and which would
explain the results more easily). This is Lindley's paradox. But there are also a priori probability
distributions where the posterior probability and the p-value have similar or equal values. 2) The
p-value is not the probability that a finding is "merely a fluke." As calculating the p-value is based
on the assumption that every finding is a fluke (that is, the product of chance alone), it cannot be
used to gauge the probability of a finding being true. The p-value is the chance of obtaining the
findings we got (or more extreme) if the null hypothesis is true. https://en.wikipedia.org/wiki/P-
value#Misunderstandings
Patrick Elliott • 2013-12-06 06:15 AM
I would add in one more: #21: Understand what placebo actually does mean. Placebo isn't "does
nothing". Rather, it can be very effective for pain, and even has been shown to have other
physiological effects, which can even include improvements in immune responses (though, this
and other effects are not as great as for pain). The reason being that its literally the "mind over
matter" effect. The default assumption is, among those that don't believe in "mind over matter"
is that all this stuff kind of runs on automatic, and the brain has no control over it. Ironically, the
"true believers" in mind over matter seem to think that the mind (not the brain, duh...) has some
sort of mystical, super-power to effect everything, if you just spend enough time meditating. The
reality is a bit more.. complicated, and vastly less mystical. Pain needs to react on a "huge" scale
to mental state. For a wounded animal, the ability to, unconsciously, fade out "existing" pain
from their wound, such as a badly injured leg, for the duration of a fight, or escape, can be life or
death. Even the ability to do things that are familiar, and thus comforting, and thus reduce the
pain, might be critical to handling the stress of such injuries. In short, there is already a
mechanism there, which can **selectively** knock out pain, and humans have way more
complex, not just conscious perceptions, but layers of sub-conscious responses. Its hardly a
surprise that, some where in there, is a way for a placebo to selectively have a major effect on
pain, even over long periods of time (like for acupuncture, a practice based on the very silly idea
that the body contains nine rivers of power in it, with tributaries, just like ancient China, where it
was invented). Obviously, state of mind "might" be useful to enhance, or suppress, intentionally
or otherwise, immunity, digestion, and a whole lot of other semi-automatic things. Things that,
when upset by certain conditions, or even conditioned responses (I feel like running away every
time someone gets sick at work, and I am the one "assigned" to clean the floors that day, for
example), which may have perfectly reasonable causes (if everyone else is sick, it might be
something you ate, so maybe you need to be sick too?, for example), but which can be
unlearned, or otherwise modified by experience. So, rather than placebo being "does nothing", a
more accurate description is, "Takes advantage of preexisting triggers and experience, to cause
**limited** changes in what the body is doing." If you think about that for a moment, you can
see why its huge headache for medicine research, and a major problem for the credibility of
people selling things that are *not* tested as carefully. In both cases, trust in the person giving it
to you, trust that it will work, belief in the method being employed, and other factors, will all
"trigger" this placebo effect. Someone taking "big pharma" drugs might feel sicker taking them,
but better taking something given to them by a guru, but its the lack of trust in the former, and
the unreasonable trust in the later, as well as the products being given, which is having the
effect, not the actual drugs themselves. Its a major pain for drug makers, to actually get provable
results, which are not **purely** due to everyone in the study believing in the medicine, and the
doctor. For the guru/herbalist, etc.... its less of a problem, since the people going to them are
probably completely certain it will help them, and trust them, even when, from the stand point
of actually treating the disease, it might be totally worthless. And, that can kill people, who
would have otherwise *been* cured, by proven medications. Note: this isn't a denial of there
being problems with Big Pharma, or with medicine as its practiced. The former is about money
(and so it big-CAM, which is quickly creeping up, closer and closer to being worth as many
billions), while doctors are all too often paid to offer new drugs, instead of proven ones, paid by
number of procedures, instead of based on outcomes, and almost everything we rely on that
*does* undeniably work, is no longer even made by Big Pharma, but by smaller, less exacting,
more prone to errors, shortfalls, or price gouging, because those, often life saving drugs, are, "no
longer cost effective to produce." The same reason why some promising drugs, for rare,
sometimes even fatal, but possibly treatable, manageable, or even curable, diseases, never make
it to market at all. Its undeniable we have problems with the industry, but.. its also undeniable
that the only reason that doctors "generally" are not allowed to push placebos, but insurance
companies are willing to pay for unlicensed, not recognized by the AMA, "specialists" for them
instead, is because insurance companies can't have their licenses pulled for lying to their patients
about the chances of something curing them. Instead, they just decide which one will cost less to
give you, something that isn't proven, but a lot of people like, but it vastly cheaper, or something
that might not help you anyway, but makes the doctor a lot of money, involves lots of ,
sometimes, unnecessary and inconclusive, lab work and tests, and then, might not help you, if
you are ill enough, any more than the placebo would have. Of course the insurance companies
are willing to pay for that as an option...
Mike Taylor • 2013-12-03 01:33 PM
See also the excellent article in response by Chris Tyler (director of the UK's Parliamentary Office
of Science and Technology) on Top 20 things scientists need to know about policy-making. These
two articles make a very helpful complementary pair.
Edward Powell • 2013-11-30 07:19 PM
To these twenty, I would add two more: 21) The predictive ability of computer simulations is
extremely limited, and should only be relied on when substantial and detailed agreement of the
simulation results with experimental data is achieved, *and* when a detailed and thorough
understanding of the elements of uncertainty in the models and simulations is achieved. A true
scientist tries his best to falsify, disprove, or even "break" his models and simulations in any
possible way, before relying on the results. John von Neuman said, "With four parameters I can
fit an elephant, and with five I can make him wiggle his trunk." Any simulation with *any* free
parameters is suspect, and detailed, conclusive, and scientifically justified causal relationships
must be well understood for each and every free parameter before a simulation can even begin
to be considered reliable. 22) No scientist shall, under any circumstances, refer to a simulation
run as an "experiment". Simulations are not experiments, only data taken from actual reality-
based experiments can be called real data.
Oliver H. • 2013-12-07 06:22 PM
What is "reality-based"? The vast majority of basic research is not "reality-based", but based on
lab experiments that do not present reality but are actively controlled to keep certain factors fix
that would vary in "reality", so as to isolate an effect. Tissue culture is nothing BUT a simulation
as to what happens in actual tissue.
Chris Atkins • 2013-11-28 08:30 PM
This is an excellent article and a subject close to my heart. It could (and should) be used with
secondary (high) school students, particularly (but not exclusively) of science. For that purpose
each 'tip' would benefit from a short (one page) article which describes the issue in more detail
with one or two examples to bring it to life.
Lee De Cola • 2013-11-27 05:48 PM
this is a great list, but to be widely read it needs to be shortened...i've tried and come up with 3
groupings: psychology (of scientists), experiments, and statistics - could we get along with 3 or 4
tips in each group?
James Scanlan • 2013-11-27 05:29 PM
The is a useful list in many respects. But it overlooks what may be the most serious problem in
the interpretation of data on differences between outcome rates. Virtually all efforts to interpret
data ongroup differences in outcome rates, including data on rates of treated and control
subjects in clinical trials, are undermined by failure to recognize patterns by which standard
measures of differences between outcome rates tend to be systematically affected by the
prevalence of an outcome. The most notable of these patterns is that by which the rarer an
outcome the greater tends to be the relative difference in experiencing it and the smaller tends
to be the relative difference in avoiding it. By way of example pertinent to item 20 on the list,
lowering test cutoffs (or generally improving test performance) tends to reduce relative
differences between rates at which two groups pass a test while increasing relative differences
rates at which they fail the test. Among other examples, reducing poverty tends to increase
relative differences in poverty rates while reducing relative differences in rates of avoiding
poverty; improving health tends to increase relative differences in mortality and other adverse
health outcomes while reducing relative differences in survival and other favorable health
outcomes; improving healthcare tends to reduce relative differences in receipt of appropriate
care while increasing relative differences in rates of failing to receive appropriate care; reducing
adverse lending outcomes or school suspension rates tends to increase relative difference in
experiencing those outcomes while reducing relative differences in avoiding those outcomes. It
is not possible to interpret data on group differences or advise policy makers on such issues
without knowing these things. Yet very few people interpreting data on differences in outcome
rates know that it is even possible for the two relative differences to change in opposite
directions, much less that they tend to do so systematically. A number of references explaining
these and related patterns and their implications in varied contexts are listed below.
Reference 1 provides a fairly succinct explanation of the pattern of relative differences described
above and does so in the course of explaining that, as a result of the failure to understand the
pattern, the US government encourages lenders and schools to take actions that make it more
likely that the government will sue them for discrimination. Reference 8 explains some of the
clinical implications of the failure to understand the pattern, explaining as well that the rate ratio
mentioned as a measure of effect in item 20 of the list is not merely a flawed measure of effect,
but an illogical one. References 9 and 10 are discussions of the above-described pattern of
relative differences by other persons. The latter reference observes that governments that
ignore the pattern “run the risk of guaranteeing failure, largely for conceptual and
methodological reasons rather than social welfare reasons.” The observation was focused on the
meeting of health inequalities reduction goals cast in relative terms. But, as reflected in
references 1 through 8, failure to understand the pattern, and relative patterns by which
measures tend to be affected by the prevalence of an outcome, undermines society’s
understanding of a great many things.
1. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies” (Amstat News,
Dec. 2012):
http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
2. “Measuring Health and Healthcare Disparities,” Federal Committee on Statistical Methodology
2013 Research Conference, Nov. 5-7, 2013.
http://jpscanlan.com/images/2013_Fed_Comm_on_Stat_Meth_paper.pdf
http://jpscanlan.com/images/2013_FCSM_Presentation.ppt
3. “The Mismeasure of Discrimination,” Faculty Workshop at the University of Kansas School of
Law, Sept. 20, 2013:
http://jpscanlan.com/images/Univ_Kansas_School_of_Law_Faculty_Workshop_Paper.pdf
http://jpscanlan.com/images/University_of_Kansas_School_of_Law_Workshop.pdf
4. “The Mismeasure of Group Differences in the Law and the Social and Medical Sciences,”
Applied Statistics Workshop at the Institute for Quantitative Social Science at Harvard University,
Oct. 17, 2012:
http://jpscanlan.com/images/Harvard_Applied_Statistic_Workshop.ppt
5. “Can We Actually Measure Health Disparities?” (Chance, Spring 2006):
http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf
6. “The Misinterpretation of Health Inequalities in the United Kingdom,” British Society for
Populations Studies Conference 2006, Sept. 18-20, 2006:
http://www.jpscanlan.com/images/BSPS_2006_Complete_Paper.pdf
7. “Race and Mortality” (Society, Jan./Feb. 2000):
http://www.jpscanlan.com/images/Race_and_Mortality.pdf
8. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com
http://www.jpscanlan.com/scanlansrule/subgroupeffects.html
9. Lambert PJ, Subramanian S. Disparities in Socio-Economic outcomes: Some positive
propositions and their normative implications. Society for the Study of Economic Inequality
Working Paper Series, ECINEQ WP 2012 – 281:
http://www.ecineq.org/milano/WP/ECINEQ2012-281.pdf
10. Bauld L, Day P, Judge K. Off target: A critical review of setting goals for reducing health
inequalities in the United Kingdom. Int J Health Serv 2008;38(3):439-454:
http://baywood.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,4,1
1;journal,7,157;linkingpublicationresults,1:300313,1
Benjamin Allen • 2013-11-26 11:04 PM
An excellent article - well done! Translation from science into policy is made even more difficult
when scientists themselves produce and promote unreliable science. Sutherland et al's 20 tips
are well worth the read. For an ecological example that epitomises many of the 20 points they
raise, see 'http://www.sciencedirect.com/science/article/pii/S0006320712005022'.
Douglas Duncan • 2013-11-26 06:38 PM
Almost all those politicians have been to university, but these aspects of science are rarely
discussed with non-science majors there. They need to be! I have been doing so for about 6
years, and have published the effects it has on students. Remember, these people have probably
never been to a scientific meeting; they have little idea how scientists interact, and they sure
don't learn that in a classroom, because most of us behave entirely differently there than we do
with colleagues. The way scientists interact, how they arrive at consensus, what causes them to
doubt (even the basic distinction between peer reviewed and non-peer reviewed publication)
are things they have never encountered (according to our interviews of non-science majors).
Anyone is welcome to use my curriculum, which is here (along with reference to the published
results): http://casa.colorado.edu/~dduncan/pseudoscience/
Bart Penders • 2013-11-26 09:44 AM
The public credibility of science and the trust non-scientists place in scientific claims and the
institution of science is not primarily derived from the methodological quality of scientific
inquiry. The list above is worthwhile to determine that quality (or any threats to it), but not to
help establish the public credibility (or trust). Let me quote Harvard historian of science Steven
Shapin who wrote, eloquently: "When King Lear decided to take early retirement, he announced
his intention to divide up the kingdom among his three daughters, each to get a share
proportioned to the genuine love she bore him. Each is asked to testify to her love. For Goneril
and Regan that presents no problem, and both use the oily art of rhetoric to good effect.
Cordelia, however, trusts to the authenticity of her love and says nothing more than the simple
truth. For Lear this will not do. Truth is her dower but credibility has she none. Cordelia, we
should understand, was a modernist methodologist. The credibility and the validity of a
proposition ought to be one and the same. Truth shines by its own light. And those claims that
need lubrication by the oily art thereby give warning that they are not true". (Shapin 1995).
Cordelia acts from the conviction that contained in her claims is the truth and that it can be
accessed by those who hear it. The credibility of her claim is evident to herself - but only to
herself. For a scientist to convince non-scientists, persuading them to see the truth as (s)he does,
requires - next to that truth - a construction of credibility. Scientists have very strict and shared
rules to establish the credibility of their claims, these include notions of peer review, citations,
transparant methodologies and much more. They are part of the well-known academic
credibilising strategy in which the work scientists do to make their data, arguments, theories or
claims become stable and uncontested truths (i.e. to achieve absolute credibility). "The same
work enables them to conduct further research by strengthening their reputation and attracting
new funding. The process of gathering credibility is never-ending and cyclical: it drives the
credibility cycle. Important in this process are moments of conversion, or translation. Latour and
Woolgar show how time, money and effort are translated into data; data is translated into
arguments, which are subsequently written down in publications. Peer scientists will read or cite
them, resulting in recognition for their claim or argument. This recognition can be mobilised to
support new funding requests. The new funding, translated into new personnel, projects or
machineries, will produce new data, continuing the cycle. Every translation that takes place,
contributes to the credibility of a given scientific claim, which extends to the researchers,
laboratory or institute making the claim. If the ‘scientific wheel of credibility’ continues to turn, it
will uphold or heighten the credibility of claims and claimants. Academic conventions, such as
scientific publications, citations and peer review are part of this credibility cycle. The credibility
cycle as a strategy to construct credibility is powerful only within the scientific community:
outside the university walls, peer review and citation cultures are relatively unknown and cannot
accrue similar credibility." (Penders 2013). Non-scientists (as well as scientists, when it comes to
non-scientific claims) use other criteria, standards and strategies to evaluate the credibility of
claims. What matters most, in real life, are the shape or form (visually, or rhetorically) of a
message, the medium or location of a claim (e.g. quality newspaper vs. casual water-cooler
chat), the degree in which a claim specifically addresses a problem (or question) the audience
identifies with, the source of the claim (a trusted colleague or friend vs. an unknown online
forum) and the relationship between the claim and the audience. The route a scientific claim
takes before it reaches any given policy maker, may matter more than its content. Who whispers
it into her ear may matter more than its content and the dominant political climate may matter
more than the content of the claim. The authors wisely acknowledge that they "are not so naive
as to believe that improved policy decisions will automatically follow". The reason for that is that
any scientific claim wildly varies in its credibility over time, location and much more. That
credibility is not derived from its content, but it is the reason why a claim is takes seriously, acted
upon, insert into or excluded from policy.
Gavin Cawley • 2013-11-25 11:35 AM
For policy related issues, the tip I would add would be: Always interpret a scientific finding in
the way that provides the least support for ones current position. This helps to guard against
the sort of confirmation biases etc. to which we are all susceptible. If the finding genuinely
supports your position, it will still do so under its least favourable interpretation, and adopting
this interpretation will demonstrate your self-skepticism.
Anand Ramanathan • 2013-11-21 07:11 PM
The article is good about warning policy makers about new scientific claims. However, it is a little
worrisome that the same logic can be applied to any science to reject established results such as
the warming over the last few decades or the effect of smoking. I would add one more point.
Science typically does not change overnight and results that have been tested and confirmed
over years (or decades) are unlikely to be wrong. Any new result that seeks to overthrow some
bit of well-established science should be backed by a much higher standard of proof, such as
independent replications, or carefully controlled studies with large sample sizes.
Gavin Cawley • 2013-11-25 11:42 AM
In the case of the warming of the last few decades, I would say this is addressed by the points
"Separate no effect from non-significance" and "Data can be dredged or cherry picked" as such
arguments are normally based on the lack of a statistically significant warming trend since some
cherry picked start (e.g. 1998 El-Nino) and ignore the statistical power of the test. A test for the
existence of a change in the underlying rate of warming would likely also give a non-significant
result.
Christopher Tong • 2013-11-21 06:37 AM
A few other tips might include an awareness of Simpson's Paradox, the Curse of Dimensionality,
and the ease with which data models can overfit one data set and fail to generalize to others.
The distinction between exploratory and confirmatory findings should also be emphasized.
Exploratory research seeks to generate new hypotheses, and confirmatory research aims to
evaluate pre-specified hypotheses. Sometimes a single study will have both confirmatory and
exploratory findings, as follows. The confirmatory claims are pre-specified before the data was
collected, and their consistency with the subsequent data are then evaluated. Exploratory
findings are tentative findings of other, unanticipated patterns in the same data. The
epistemological status of both kinds of findings must be kept distinct. Finally, data analysis
methods must be chosen for fitness for purpose, and not by pattern matching, as is too often the
case.
Brad Louis • 2013-11-21 03:40 AM
Great list. Can I suggest a few more? We don't know everything (Science is never settled. Any
scientist can be wrong.) - that's why we keep investing in scientific research. We will know more
in another hundred years, by which time a lot of what we think we know now will be proven to
be nonsense. Proper experiments produce the most robust scientific evidence - good
experimental design can eliminate many of the problems mentioned in earlier tips.
Observational studies and computer models are not a patch on proper experiments. Science
funding supports a process not an outcome - well done science doesn't pretend to know the
outcome of research, diversity of research perspectives is a good thing.
Ian Campbell • 2013-11-20 11:00 PM
This article provides a valuable summary of issues that need to be appreciated in interpreting
science. However, in the environmental decision making arena, the science is never sole factor to
be considered - there is always a value judgement as well. Science tells us the consequences of
greenhouse gas emissions, or the likelihood that logging a particular area of forest will cause the
extinction of a particular species, but then we have to make a value judgement. Do I value the
profits I make from my power station more than the contribution it makes to global warming?
Do I value the timber products I can get by logging this forest more than the possum that will
become extinct? As value judgements these decisions should be informed by the science, but are
made by politicians who are expected to reflect the values of their society. Politicians are always
averse to making value judgement calls that will be unpopular with a substantial proportion of
their community, or an influential group within the community, so they often try to deflect that
responsibility. Two common deflection techniques are appeals to (often dodgy) short term
economics (it would be too expensive, cost too many jobs), and raising doubts about the science
either by citing poor science, or by suggesting that there is a lack of scientific consensus on the
issue. It is difficult to tell whether a true lack of scientific understanding, as addressed in the
article, or a misunderstanding of convenience to deflect criticism is a greater problem.
Fred Sachs • 2013-11-21 05:16 PM
The article does emphasize some important issues but at the core is how one might evaluate the
contributions of all these factors. The probability of success in a project cannot be calculated and
a legislator would have to assign a weighting factor to each of the 20 criteria, let alone the
societal consequences, to decide on the level of support and it can't be done. Real answers like
this depend on numbers and we don't have the information to generate the numbers. I suspect
that we are going to have to put up with intuition on the part of the policy makers. I have a few
specific issues. One is that 5% significance is an entirely subjective choice by the researcher and
has no significance if the distribution is not Gaussian. That should be the first test. Even at 5%
there is a 1 in 20 chance that the hypothesis is wrong. Does this data represent that time?
Especially in clinical data one should know the probability of getting better compared to the
probability of getting worse. You should not do a one sided test to ask if subjects got better and
ignore the times they got worse. The outcome should also be weighted by the payoff; a number
not easily evaluated. Did the positive patients feel better while the negative ones died? Finally,
for the cause and effect issue I like to bring up something seen a billion times a day: the roosters
crowed and then the sun came up; clearly the roosters caused sunrise.