Problem Solving Assessment in Games and Simulation Environments

Richard E. Mayer University of California, Santa Barbara Collaborators Eva Baker Roxana Moreno Gayle Dow Harold F. ONeil, Jr. Jennifer Dyck William Prothero Anne Fay Jill Quilici James Lester Valerie Sims Patricia Mautone Hiller Spires Sarah Mayer William Vilberg Supported by National Science Foundation, Andrew Mellon Foundation

Problem Solving Assessment in Games and Simulation Environments

Introduction What Is the Rationale for Games and Simulations? What Are Instructional Games and Simulations? What Is Problem-Solving Assessment in Games and Simulations? Example 1: Assessment of Problem Solving in Computer Programming Example 2: Assessment of Problem Solving in Playing Educational Games Example 3: Assessment of Problem Solving in the Bunny Game Example 4: Assessment of Problem Solving in the Design-A-Plant Game Example 5: Assessment of Problem Solving in the Profile Game Example 6: Assessment of Problem Solving in the Dr. Phyz Simulation Example 7: Assessment of Problem Solving in an Aircraft Simulation Conclusion

What Is the Rationale for Games and Simulations? Bransford, Brown, & Cocking (1999) Visual modes of presentation: technologies can help people visualize difficult-to-understand concepts[so] students are able to work with visualizationsoftware Interactive modes of presentation: new technologies are interactive [so] it is easier to create environments in which students learn by doing

What Are Instructional Games and Simulations? An instructional simulation: 1. Is a multimedia environment that models a to-be-learned system. 2. Allows for user interaction by responding in rule-based ways to user input. 3. Is intended to promote learner understanding of the to-belearned system. An instructional game: 4. Also involves one or more players competing to achieve some goal which is accomplished through understanding the subject matter.

What Is Problem-Solving Assessment in Games and Simulations?

Types of Knowledge

Name Facts
Concepts Procedures Strategies

Definition basic elements

models or principles

Example magnets are made of metal

how an electric motor works

step-by-step processes how to add numbers general methods how to break a problem into parts

Criteria for Problem-Solving Assessments

Interactivity criterion: involves interaction between the learner and the computer-based system Multimedia criterion: involves verbal and non-verbal forms of representation (such as graphics) Problem-solving criterion: involves solving problems

Seven Games and Simulations for Assessing the Learners Knowledge

1. 2. 3. 4. 5. 6. 7. Computer programming Educational games Bunny game Design-a-plant game Profile game Dr. Phyz simulation Aircraft simulation

Programming in Logo

RT 90

FD 5

Purpose: To determine the cognitive consequences of learning Logo or BASIC programming

Test Knowledge Description Score(s)

Map test

Strategic

Given a neighborhood map and list of locations draw a path connecting them. Given a neighborhood map showing a path connecting various locations, give verbal directions.
Given a Logo command, draw what happens when it is executed.

Number correct
Number correct

Command prediction

Conceptual

Number correct, misconception type

Problem translation Following procedures

Conceptual

Given a word problem, select a corresponding equation.

Number correct

Strategic

Given a list of verbal commands,

Number correct

describe the final output.

From: Fay & Mayer (1987), Mayer & Fay (1987), Mayer, Dyck, & Vilberg (1986)

Command prediction problem

RT 90

Problem translation A car rental service charges 20 dollars a day and 15 cents a mile to rent a car. Find the expression for total cost C, in dollars, of renting a car for D days to travel M miles. a. C = 20D + 0.15M b. C = 15D + 0.20M c. C = 20D + 15 M d. C = 0.15D + 20M

Following procedures 1. Put 5 in Box A. 2. Put 4 in Box B 3. Add the number in Box A and the number in Box B, put the result in Box C. 4. Add the number in Box A and the number in Box C, put the result in Box A. 5. Write down the numbers from Box A, B, and C. What is the output of this program? a. 5, 4, 9 b. 14, 4, 9 c. 14, 9, 9 d. 9, 4, 9 e. none of the above

Purpose: To determine the cognitive consequences of playing educational computer games.

Test
Sentence translation

Knowledge
Conceptual

Description

Score(s)

Given a sentence from a word Number correct problem, select the corresponding number sentence.

Necessary numbers
Necessary operations

Conceptual

Given a word problem, select the numbers that are needed to solve the problem.

Number correct

Strategic

Given a word problem, select Number correct the arithmetic operations that are needed to solve the problem. Given a new game, make each necessary move. Number of excess moves, type of strategy

Puzzle tank

Strategic

Mental rotation
Form board

Strategic
Strategic

Judge whether two shapes are same or different.

Judge whether shapes fit together.

Response time, best fitting model

Response time

From Mayer et al. (1997), Mayer et al. (1999), Sims & Mayer (2002)

Sentence translation Ann and Rose have 20 books altogether. Which number sentence is correct? a. Anns books = Roses books + 20 b. Anns books + 20 = Roses books c. Anns books + Roses books = 20 d. Anns books = Roses books

Necessary numbers Which numbers are needed to solve this problem? A package of 3 toys costs 88 cents. Richie bought 2 packages. How many toys did he buy? a. 3, 88, 2 b. 3, 88 c. 88, 2 d. 3, 2

Necessary operations What should you do for this problem? The 200 children at River View School are going on a bus trip. Each bus holds 50 children. How many buses are needed? a. divide, then add b. subtract only c. multiply only d. divide only

Purpose: To assess knowledge of addition and subtraction of signed numbers in the bunny game.

Test
Bunny game

Knowledge
Conceptual

Description
Given an arithmetic problem, use virtual joystick to simulate movement of bunny along number line. Given an arithmetic problem, type in correct answer.

Score(s)
Number correct, misconception type

Arithmetic solving

Strategic, conceptual

Number correct, misconception type

From Moreno & Mayer (1999)

Bunny game 4 - -5 =___

-9 -8 -7 -6 -5 -4 -3 -2 -1 0

1 2 3 4 5 6 7 8 9
JUMP FORWARD

FACE LEFT

FACE RIGHT

JUMP BACK

Purpose: To assess knowledge of how plants grow in the design-a-plant game.

Test
Design-aPlant game

Knowledge
conceptual

Description
Select roots, stem, and leaves for a plant to grow in a specified environment. Given a plant with specified roots, stem, and leaves, describe the environment it is best suited for.

Score(s)
Number correct

Number correct

From Moreno et al. (2001)

Design-a-plant game
Design a plant to live in an environment with low sunlight. [Select leaves, stem, and roots.] Describe the environment that would be best for a plant with thick, small, think-skinned leaves; think, long, and dark bark stem; and branching, shallow, and thin roots.

Purpose: To assess knowledge of geology sonar detection in the profile game.

Test
Profile game

Knowledge
strategic

Description
Detect geological features in a geology simulation.

Score(s)
Number correct, solution time

From Mayer, Mautone, & Prothero (2002)

Section of earth

Draw a line here

Profile line

See its profile here

Purpose: Assess knowledge of how an electric motor works in the Dr. Phyz simulation. Test Troubleshooting problem Redesign problem Knowledge conceptual Description List possible reasons for motor to not work when turned on. Describe how to change a motor for a certain purpose. Explain why an event occurs. Tell what would happen for a specific set of actions. Tell actions to take to make the motor accomplish a Score(s) Number of correct answers

conceptual

Number of correct answers

Principle problem Prediction problem Action problem

conceptual

Number of correct answers Number of correct answers Number of correct answers specific goal.

conceptual.

conceptual

From Mayer et al. (2003)

Troubleshooting problem Suppose you switch on an electric motor but nothing happens. What could have gone wrong? Redesign problem What could you do to increase the speed of the electric motor, that is, to make the wire loop rotate more rapidly? Principle problem Why does the wire loop move? Prediction problem What happens if you connect the negative terminal to the red wire and the positive terminal to the yellow wire? Action problem What can you do to reverse the movement of an electric motor, that is, to make the wire loop rotate in the opposite direction?

Purpose: To assess knowledge of an aircraft fuel system in the aircraft simulation.

Test
Troubleshooting problem

Knowledge
conceptual

Description
List possible reasons for fuel system to malfunction.

Score(s)
Number correct

Redesign problem
Prescription problem

conceptual

Describe how to change the fuel system for a certain purpose

Tell what would happen for a specific set of actions

Number correct

conceptual

Number correct

From ONeil et al. (2000)

Troubleshooting problem Suppose you try to transfer fuel from one tank to another but nothing happens. What could have gone wrong? Redesign problem What could be done to make fuel transfer more reliable, that is, to minimize the chances of something going wrong when fuel is transferred from one tank to the next?

Prediction problem What would happen if the ejector pump stopped working?

Conclusion: Problem-solving performance in any domain depends on the learners knowledge, which is measurable. Conceptual and strategic knowledge can be assessed. Assess conceptual knowledge by asking the learner to: Specify what happens when a command is executed Translate a word problem into an equation Tell what numbers are needed to solve a word problem Ask a learner to role play what happens for a numerical computation Answering open-ended questions involving troubleshooting, redesign, predicting, explaining, and prescribing Assess strategic knowledge by asking the learner to: Generate or follow directions Describe the output of a procedure Specify the operations needed to solve a word problem Learn to play a new game Make judgments