USING MULTIPLECHOICE QUESTIONS

TO MEASURE
PROBLEM SOLVING
AND CRITICAL
THINKING

A step-by-step procedure
for creating problemsolving measures:

Step 1: Decide on the principle or

principles to be tested. Criteria to
be considered
1. Should be known principles but the situation in which
the principles are to be applied should be new.
2. Should involve significantly important principles
3. Should be pertinent to a problem or situation common
to all students.
4. Should be within the range of comprehension of all
students.
5. Should use only valid and reliable sources from which
to draw data.
6. Should be interesting to students.

Step 2: Determine the phrasing of

the problem situation so as to require
the student in drawing conclusions to
be one of the following:
1. Make a prediction.
2. Choose a course of action.
3. Offer an explanation for an observed

phenomenon.
4. Criticize a prediction of explanation made by

others

Step 3:
Set up the problem
situation in which the principle or
principles
selected
operate.
Present the problem to a class
with
directions
to
draw
a
conclusion or conclusions and give
several supporting reasons for
their answer

Step 4:
Edit the students
responses, selecting those that
are most representative of their
thinking.
These will include
conclusions
and
supporting
reasons that are both acceptable
and unacceptable

Step 5: To the conclusions and

reasons
obtained
from
the
students, the teacher now adds
any others that he or she feels are
necessary to cover the salient
points. The total number of items
should be at least 50 percent
more than is desired in the final
form to allow for elimination of
poor items. The following list is a
guide to the type of statements
that can be used:

1. True statements of principles and facts

2. False statements of principles and facts
3. Acceptable an unacceptable analogies
4. Appeal to acceptable or unacceptable
authority
5. Ridicule
6. Assumption of the conclusion
7. Teleological explanations

Step 6: Submit test to other

judges for criticisms. Revise
test in view of criticisms
Step 7:
Follow with
discussion

Administer test.
thorough class

Step 8: Conduct an item

analysis
Step 9: In the light of steps 7
and
8, revise the test.

Norris and Ennis (1989) write about four

kinds of critical thinking questions:

Deduction
Credibility
Induction
Assumption-identification
Following are examples of these types
of items:

DEDUCTION
1. If these beings are from Earth, then another
space ship must have landed on Nicoma. But no
other space ship has landed on Nicoma.
Suppose that what the speaker says is true.
Then must this be true?
These beings are not from Earth.
A. Yes, it must be true.
B. No, it cannot be true.
C. We cannot tell for sure from the information
given

CREDIBILITY
2. Which underlined statement is more
believable?
A. The health officer says, This water is safe
to drink.
B. Several others are soldiers. One of them
says, This water supply is not safe.
C. A and B are equally believable.

INDUCTION
Mr. Brown, who lives in the town of Salem, was
brought before the Salem municipal court for the
sixth time in the past month on a charge of
keeping his pool hall open after 1A.M. He again
admitted his guilt and was fined the maximum ,
$500, as in each earlier instance.
3. On some nights it was to Mr. Browns
advantage to keep his pool hall open after 1
A.M., even at the risk of paying a $500 fine.
This statement is
A.
B.
C.
D.
E.

True
Probably true
Probably false
False
Insufficient data

ASSUMPTION IDENTIFICATION
4. Since about half the villagers have very short
hair, I think that at least half are male.
A. Half are female
B. All males have short hair
C. Only males have short hair

Fat Mice

Black Tail

White Tail

Thin Mice

GENERAL TYPES OF ESSAY

QUESTIONS
1. Extended Versus Restricted Response

Complex learning outcomes can be measured effectively with questions.

There are the abilities, for example, to

explain cause-effect relationships

describe applications of principles

present relevant arguments

Formulate tenable hypotheses

Formulate valid conclusions

State necessary assumptions

Describe the limitations of data

Explain methods and procedures

Produce, organize, and express ideas

Create original forms (e.g., design an experiment)

Evaluate the worth of ideas

2. Content Versus Mechanics of

Expression
The essay question or test allows the student to present his or her
knowledge and understanding and to organize the material in a
unique form and style

If the instructor has attempted to develop students expressive skills,

and if this learning outcome is included in his or her table of
specifications, evaluating such skills is legitimate

If expressive skill are not part of the instructional program, it is

unethical to evaluate them.

If the

score of each essay item includes an evaluation of the

mechanics of English, this should obviously, be brought to the

students attention.

Separate scores should be computed for content and expression

The decision to include either or both of these elements in a score,

and the relative weighting of each, should be dictated by the table of
specifications.

3. Specific Types of Essay Questions

The following set of essay questions
illustrates how the phrasing of an essay item
could be framed to elicit particular behaviors
and levels of response.

1. Recall
A. Simple recall
1.

What is the chemical formula for

hydrogen
2.

peroxide?

Who wrote the Emergence of Lincoln?

B. Selective recall in which a basis for evaluation

or judgment is suggested
1. Which three individuals in the 19th century
had

the

most

contemporary life?

profound

effect

on

2. UNDERSTANDING
A. Comparison of two phenomena on a single designated basis
1. Compare the writers of the English Renaissance to those of
the 19th century in their abilities to describe nature.
B. Comparison of two phenomena in general
1. Compare the French and Russian Revolutions
C. Explanation of the use or exact meaning of a phrase or
statement
1. The Book of John begins In the beginning was the word
From what philosophical system does this statement derive?

2. UNDERSTANDING
D. Summary of a text or some portion of it
1.

State the central thesis of the Communist Manifesto.

E. Statement of an artists purpose in the selection or organization

of material
1.

why did Hemingway describe in detail the episode in which

Gordon, lying wounded, engages the oncoming enemy?

2. What was Beethovens purpose in deviating from the

orthodox symphony form in Symphony No. 6?

3. APPLICATION
It should be clearly understood that whether or not a question elicits
application depends on the preceding educational experience. If a
particular analysis has been taught explicitly, a question involving that
analysis is a matter of simple recall.

A. Causes or effects
1. Why can too frequent reliance on penicillin for the
treatment of minor ailments eventually result in its
diminished effectiveness against major invasion of body
tissues by infectious bacteria?
2. Why did fascism flourish in Italy and Germany but not in
England and the United States?

3. APPLICATION
B. Analysis (it is advisable not to use the word analysis in the
question itself)
1.

Why was Hamlet torn by conflicting desires?

2. Why is the simple existence of slavery an insufficient

explanation for the outbreak of the American Civil War?
C. Statement of relationship
1. Intelligence is said to correlate with school achievement at
about .65. Explain this relationship.
D. Illustrations or examples of principles
1.

Name three examples of uses of the lever in typical American

homes

3. APPLICATION
E. Application of rules or principles in specified situations
1. I would you weigh more or less on the moon? On the sun?
Explain.
F. Reorganization of facts
1. Some writers have concluded that the American
Revolution was not merely a political revolution against
England bur
also a social revolution, within the colonies, of
the poor against the wealthy. Using the available evidence
to support this assertion, what other conclusion is possible

4. JUDGMENT
A. Decision for or against
1. Should members of the Communist Party be allowed to
teach in American colleges? Why or why not?
2. Is nature or nurture more influential in determining human
behavior? Why?
B. Discussion
1. Discuss the likelihood that four-year private liberal arts
colleges will gradually be replaced by junior colleges and
state universities

4. JUDGMENT
C. Criticism of the adequacy, correctness, or relevance of a
statement
1.

The discovery of penicillin has often been called an accident.

Comment on the adequacy of this explanation.

D. Formulation of new questions

1. What should one find out to explain why some students of
high
intelligence fail in school?
2.

What questions should a scientist ask to determine why more

smokers than non-smokers develop lung cancer?