Planning Tests and Assessments

Four Steps in Planning an Assessment

1. Deciding its purpose

2. Developing test specifications
3. Selecting best item types
4. Preparing items

Step 1: Decide the Purpose

What location in instruction? Or, the role of time in assessment!

1. pre-testing
o readiness
i. limited in scope
ii. low difficulty level
iii. serve as basis of remedial work, adapting instruction
o pretest (placement)
i. items similar to outcome measure
ii. but not the same (like an alternative form)
2. during instruction
o formative
i. monitor learning progress
ii. detect learning errors
iii. feedback for teacher and students
iv. limited sample of learning outcomes
v. must assure that mix and difficulty of items sufficient
vi. try to use to make correction prescriptions (e.g., review for whole
group, practice exercises for a few)
o diagnostic
i. enough items needed in each specific area
ii. items in one area should have slight variations
3. end of instruction
o mostly summative –broad coverage of objectives
o can be formative too

Step 2: Develop Test Specifications

• Why? Need good sample!

• How? Table of specifications (2-way chart, "blueprint")
1. Prepare list of learning objectives
2. outline instructional content
3. prepare 2-way chart
4. or, use alternative to 2-way chart when more appropriate
5. doublecheck sampling

Planning assessments 1
Sample of a Content Domain (such as this course)

1. trends/controversies in assessment
2. interdependence of teaching, learning, and assessment
3. purposes and forms of classroom assessment
4. planning a classroom assessment (item types, table of specs)
5. item types (advantages and limitations)
6. strategies for writing good items
7. compiling and administering classroom assessments
8. evaluating and improving classroom assessments
9. grading and reporting systems
10. uses of standardized tests
11. interpreting standardized test scores

Sample Table of Specifications (For content from this course)

Sample SLOs (you Bloom Levels

would typically have
more) Remember Understand Apply Analyze Evaluate Create

Identifies definition of X
key terms (e.g., validity)

Identifies examples of X
threats to test reliability
and validity

Selects best item type for X

given objectives

Compares the pros and X

cons of different kinds of
tests for given purposes

Evaluates particular X
educational reforms
(e.g., whether they will
hurt or help instruction)

Create a unit test X

Total number of items

Planning assessments 2
Spot the Poor Specific Learning Outcomes (use previous table of specifications)

Which entries are better or worse than others? Why? Improve the poor ones.

1. Knowledge
a. Knows correct definitions
b. Able to list major limitations of different types of items
2. Comprehension
a. Selects correct item type for learning outcome
b. Understands limitations of true-false items
c. Distinguishes poor true-false items from good ones
3. Application
a. Applies construction guidelines to a new content area
b. Creates a table of specifications
4. Analysis
a. Identifies flaws in poor items
b. Lists general and specific learning outcomes
5. Synthesis
a. Lists general and specific content areas
b. Provides weights for areas in table of specifications
6. Evaluation
a. Judges quality of procedure/product
b. Justifies product
c. Improves a product

Why are These Better Specific Learning Outcomes?

1. Knowledge
a. Selects correct definitions
b. Lists major limitations of different item types
2. Comprehension
a. Selects proper procedures for assessment purpose
b. Distinguishes poor procedures from good ones
c. Distinguishes poor decisions/products from good ones
3. Application
a. Applies construction guidelines to a new content area
4. Analysis
a. Identifies flaws in procedure/product
b. Lists major and specific content areas
c. Lists general and specific learning outcomes
5. Synthesis
a. Creates a component of the test
b. Provides weights for cells in table of specifications
6. Evaluation
a. Judges quality of procedure/product
b. Justifies product
c. Improves a product

Planning assessments 3
Step 3: Select the Best Types of Items/Tasks

What types to choose from? Many!

1. objective--supply-type
a. short answer
b. completion
2. objective--selection-type
a. true-false
b. matching
c. multiple choice
3. essays
a. extended response
b. restricted response
4. performance-based
a. extended response
b. restricted response

Which type to use? The one that fits best!

1. most directly measures learning outcome

2. where not clear, use selection-type (more objective)
a. multiple choice best (less guessing, fewer clues)
b. matching only if items homogeneous
c. true-false only if only two possibilities

Strengths and Limitations of Objective vs. Essay/Performance

Objective Items

• Strengths
o Can have many items
o Highly structured
o Scoring quick, easy, accurate

• Limitations
o Cannot assess higher level skills (problem formulation, organization,
creativity)

Essay/Performance Tasks

• Strengths
o Can assess higher level skills
o More realistic
• Limitations
o Inefficient for measuring knowledge

Planning assessments 4
 o Few items (poorer sampling)
o Time consuming
o Scoring difficult, unreliable

Step 4: Prepare Items/Tasks

Strategies to Measure the Domain Well—Reliably and Validly

1. specifying more precise learning outcomes leads to better-fitting items

2. use 2-way table to assure good sampling of complex skills
3. use enough items for reliable measurement of each objective
o number depends on purpose, task type, age
o if performance-based tasks, use fewer but test more often
4. keep in mind how good assessment can improve (not just measure) learning
o signals learning priorities to students
o clarifies teaching goals for teacher
o if perceived as fair and useful

Strategies to Avoid Contamination

1. eliminate barriers that lead good students to get the item wrong
2. don’t provide clues that help poor students get the item correct

General Suggestions for Item Writing

1. use table of specifications as guide

2. write more items than needed
3. write well in advance of testing date
4. task to be performed is clear, unambiguous, unbiased, and calls forth the intended
outcome
5. use appropriate reading level (don’t be testing for ancillary skills)
6. write so that items provide no clues (minimize value of "test-taking skills")
a. a/an
b. avoid specific determiners (always, never, etc.)
c. don’t use more detailed, longer, or textbook language for correct answers
d. don’t have answers in an identifiable pattern
7. write so that item provides no clues to other items
8. seeming clues should lead away from the correct answer
9. experts would agree on the answer
10. if item revised, recheck its relevance

Planning assessments 5