Creating Multiple Choice Tests

to Assess Critical Thinking Skills

Bonnie H. Armstrong, Ph.D. Assessment Services Academic & Professional Program Services

October 2006

Objectives
{ { {

Name the student performance levels that test critical thinking skills. Analyze multiple choice items to determine their performance levels. Describe question formats that are appropriate for multiple choice questions testing critical thinking skills. Develop one multiple choice item for your subject area that tests a critical or higher order thinking skill.

October 2006

What Effective Assessments Do:

{

Measure intended learning outcomes.

z

What students should know and/or be able to do.

Measure knowledge and skills that were included in the instructional experience.
z

Information, examples, practices or other activities

Enable generalization of learning from observed to unobserved performances.

z z

Critical points/essential content Appropriate range of content

October 2006

Comparison of Objectives & Assessments

{

Objectives state the desired learning outcomes and describe what the students should be able to do. Assessments measure student achievement of the desired learning outcomes.

October 2006

Relationship of Objectives & Assessments

{

Objectives provide the foundation for selecting and developing the assessments by providing a verb that
z z

Describes the behaviors students should perform. Indicates the desired performance levels.

Assessments
z z

Should include student performances at the same levels as those stated in the objectives. May include lower levels if appropriate for testing the objective.

October 2006

Levels of Performance
{ { { {

{ {

Information (recall) Concepts (comprehension) Rules (application) Problem Solving (analysis/synthesis/evaluation) Psychomotor Skills Attitudes

October 2006

Is the level of performance appropriate?

z z

Information: Know that, know how, define Concepts: Discriminate, classify, know why, understand how, describe/provide examples Rules: Apply rules to solve problems, interpret, use procedures Problem Solving: Compare/contrast, explain, break apart, produce, develop, create, generate, critique, judge, recommend. Psychomotor Skills: Demonstrate, perform, produce Attitudes: Appreciate, value, choose, believe

October 2006

What is the performance level?

Examples of Objectives
{

Identify specific forms of European architecture from late Roman times through the 17th Century. Define animal, mineral, and vegetable as used in biology. Name the three stages of a society.

October 2006

Information
Examples of Objectives
{

Identify specific forms of European architecture from late Roman times through the 17th Century. Define animal, mineral, and vegetable as used in biology. Name the three stages of a society.

October 2006

What is the performance level?

Examples of Objectives
{

Interpret graphic results from a variety of sources. Summarize the effects of three models of educational reform on the present education system in the US. Describe why objectives should be considered when selecting appropriate course assessments.
October 2006 10

Concepts
Examples of Objectives
{

Interpret graphic results from a variety of sources. Summarize the effects of three models of educational reform on the present education system in the US. Describe why objectives should be considered when selecting appropriate course assessments.
October 2006 11

What is the performance level?

Examples of Objectives
{

Apply development steps to create web pages in html. Solve quadratic equations. Prepare a seven course meal following standard culinary procedures and using principles of haute cuisine.

October 2006

12

Apply rules or procedures

Examples of Objectives
{

Apply development steps to create web pages in html. Solve quadratic equations. Prepare a seven course meal following standard culinary procedures and using principles of haute cuisine.

October 2006

13

What is the performance level?

Examples of Objectives
{

{ {

Using a family case description, explain the family dynamics and the type and degree of family dysfunction existing. Use relevant research findings to develop conclusions about optimum educational environments. Using the articles provided and the editorial and professional standards checklist, determine which articles should be included in the next issue of the journal.
October 2006 14

Problem Solving (analysis/synthesis/evaluation)

Examples of Objectives
{

{ {

Using a family case description, explain the family dynamics and the type and degree of family dysfunction existing. Use relevant research findings to develop conclusions about optimum educational environments. Using the articles provided and the editorial and professional standards checklist, determine which articles should be included in the next issue of the journal.
October 2006 15

Is the content of the item appropriate?

{

Topic: Is the content

z

An important topic that supports one or more objectives? An important subtopic for an objective?

Teaching Points: Is the content an important teaching point from

z z z

The lectures? The readings? Other resources?

October 2006

16

Objective: Identify elements of sound building design.

What performance level is being tested?

1. Which of the following groups of items would best contribute to a sound building design? A) heavy walls, large windows, tall structure B) thin walls, large windows, external supports C) small windows, external supports, many doorways
October 2006 17

Objective: Determine sound building design elements as presented in blueprints and specifications.

What performance level is being tested? 1. Building Plan A is an example of: A) Sound building design because it has thick walls,
small windows, and no external supports. B) Sound building design because it has thin walls, large windows, and external supports. C) Unsound building design because it has thick walls, small windows, and external supports. D) Unsound building design because it has thin walls, large windows, and no external supports.
October 2006

18

Objective: Using specified data, solve simple genetics problems. What performance level is being tested? What performance level is being tested?

Given the following data, what fraction of the offspring from Plants A and B would be expected to have both red flowers and yellow seeds? Data set: Plant A is heterozygous for flower color and homozygous for yellow seeds while Plant B is heterozygous for seed color and homozygous for red flowers. A) 1/4 B) 1/2 C) 3/4 D) 7/8
1.
October 2006 19

Objective: Explain how and why information seeking problems occur.

What performance level is being tested?

Using the information seeking case study provided, which element seems to be most important in explaining the outcome?
a. User characteristics. b. Controlled vocabularies c. Information organization d. Flexibility of information systems
October 2006 20

Objective: Make appropriate editorial decisions regarding various types of news and other types submissions to the newspaper.

What performance level is being tested?

Read the following letter to the editor concerning campus parking problems. What action would you take regarding this letter? (Insert letter to the editor.)
a. b. c. d.

Write to author for more information. Place letter in publish as soon as possible file. Place letter into might publish file. Discard letter.
October 2006 21

Our Objectives What are the PLs?

{

Name the student performance levels that test critical thinking skills. Analyze multiple choice items to determine their performance levels. Describe question formats that are appropriate for MC questions testing CTS. Develop one multiple choice item for your subject area that tests a critical or higher order thinking skill.

October 2006

22

Question Formats
{ { { { {

Premise - Consequence Analogy Case Study Incomplete Scenario Problem/Solution Evaluation

October 2006

23

Internet Information
{

Techniques for writing critical thinking skill questions: http://tep.uoregon.edu/resources/assess ment/multiplechoicequestions/sometechni ques.html Condensed Bloom: http://www.ceap.wcu.edu/Houghton/Lear ner/think/thinkhigherorder.html#skills Higher-order thinking skills question templates. http://sbci.cps.k12.il.us/HOTS_templates. pdf
October 2006 24