Creating a Table of Specifications

A table of specifications is simply a means of connecting learning

objectives, instructional activities and assessment. The following steps will guide you in the creation of a table of specifications: Develop learning objectives based on the taxonomy of educational objectives Identify instructional activities that target the learning objectives Implement instructional activities Reflect on instructional activities and identify relevant learning objectives that will be assessed based on the instructional experience Determine the relative importance and weighting of each objective Generate test items based on the designated learning objectives
http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%2 0Taxonomy%20Action%20Verbs.pdf

Day 2 Session 1

Developing Test Items and Rubrics

Warmer: Find someone who

You cannot use the same name twice! Find someone who: 1. drank coffee this morning. 2. watched a TV movie yesterday. _____________ _____________

3. grew up in another country.

4. played badminton last week. 5. skipped lunch today. 6. sang at a karaoke last Saturday. 7. cut his/her hair in the last month. 8. had a date in last weekend. 9. took a trip out of town last holiday

_____________
_____________ _____________ _____________ _____________ _____________ _____________

10. bought a mobile phone a few weeks ago. _____________

Review/Preview (5 min)
1.

Name one type of closed-ended item and list the pros and cons for this type of item.
_____________________________________________

2.

3.
4.

Two components of a test blueprint are __________ and _________________. The degree to which the test actually measures what it is intended to measure is called reliability (T/F) The systematic assignment of numbers to attributes is called A. Test B. Evaluation C. Assessment D. Measurement

Types of Test Items

subjective

objective Closedended

Openended

Writing Objective Test Items

Multiple-choice Questions (MCQ) True/False (TF) Matching

Supply-Type

Multiple-choice questions
are the hardest type of objective questions to write for

classroom teachers Although many people believe MCQs are simplistic, actually the format can be used for intellectually challenging tasks Multiple-choice test items consist of
a stem or question and three or more alternative answers the correct answer sometimes called the keyed response and the incorrect answers called distractors.

Always keep in mind the guiding principles when

writing MCQs

The good and the bad of MCQs

Pros Cons

more answer options (4-5) reduce the chance of guessing that an item is correct many items can aid in student comparison and reduce ambiguity greatest flexibility in type of outcome assessed: knowledge goals, application goals, analysis goals, etc.

reading time increased with more answers reduces the number of questions that can be presented difficult to write four or five reasonable choices takes more time to write questions

Tips for Writing MCQs

Avoid responses that are interrelated. Avoid negatively stated items. It is easy to miss the negative word "not." If you use negatives, bold-face the negative qualifier to ensure people see it. Avoid making your correct response different from

the other responses, grammatically, in length, or otherwise.

Tips for Writing MCQs

Avoid the use of "none of the above." When a students guesses "none of the above," you still do not know if they know the correct answer. Avoid repeating words in the question stem in your

responses.
For example, if you use the word "purpose" in the

question stem, do not use that same word in only one of the answers, as it will lead people to select that specific response.
Use plausible, realistic responses.

Tips for Writing MCQs

Create grammatically parallel items to avoid giving

away the correct response.

For example, if you have four responses, do not start

three of them with verbs and one of them with a noun.

Always place the "term" in your question stem and

the "definition" as one of the response options.

Example 1
Stem Presented as a Question. This form is generally better than the incomplete stem because it is simpler and more natural.

Who is primarily responsible for maintaining an aircraft in an airworthy condition? A. Pilot in command or operator. B. Owner or operator of the aircraft. C. The lead mechanic responsible for that aircraft.

Example 2
Stem as an Incomplete Statement. When using this form, care must be exercised to avoid ambiguity, giving clues, and using unnecessarily complex or unrelated alternatives.

VFR cruising altitudes are required to be maintained when flying ..

A. at 3,000 feet or more AGL, based on true course. B. more than 3,000 feet AGL, based on magnetic course. C. at 3,000 feet or more above MSL, based on magnetic heading.

Example 3
Stem Supplemented by an Illustration. This form is useful for measuring the ability to read instruments, or identify objects.

(Refer to figure 1.) The acute angle A is the angle of

A. Attack. B. Dihedral. C. Incidence.

Example 4

Multiple Response is Required. This form is a variation of the previous forms in that it contains more than one correct answer, and students are instructed to select all correct answers.

Which of the following statements is/are generally true regarding the charging of several aircraft batteries together? 1. Batteries of different voltage (but similar capacities) can be connected in series with each other across the charger, and charged using the constant current method. 2. Batteries of different ampere-hour capacity and same voltage can be connected in parallel with each other across the charger, and charged using the constant voltage method. 3.Batteries of the same voltage and same ampere-hour capacity must be connected in series with each other across the charger, and charged using the constant current method. A. 3. B. 1 and 2. C. 2 and 3.

Example 5
Negative Variety Type. This form is not suggested but, if used, always emphasize the negative word.

Which of the following is NOT considered a method of heat transfer? A. Diffusion. B. Conduction. C. Convection.

Example 6
Association Type. This form is useful if a limited number of associations are to be made.

Which aircraft has the right-of-way over the other aircraft listed? A. Airship. B. Gyroplane. C. Aircraft towing other aircraft.

Example 7
Definition Type.
This form is used to determine knowledge of a specific definition.

Aspect ratio of a wing is defined as the ratio of the A. wingspan to the wing root. B. wingspan to the mean chord. C. square of the chord to the wingspan.

True/False (TF)
Advantage: Easier and quicker to write than MCQs Remember the guidelines for writing effective test

items when creating these T/F items

The good and the bad of TF

Pros Cons

can present many items most difficult question at once to write objectively easy to score ambiguous terms can used to assess popular confuse many misconceptions, cause- few answer options (2) effect reactions increase the chance of guessing that an item is correct; need many items to overcome this effect

Tips for Writing True/False Items

Do not use definitive words such as "only," "none,"

and "always," that lead people to choose false, or uncertain words such as "might," "can," or "may," that lead people to choose true. Do not write negatively stated items, as they are confusing to interpret: "Thomas Jefferson did not write the Declaration of Independence." True or False?

Tips for Writing True/False Items

People have a tendency to choose "true," so

design at least 60% of your T/F items to be "false" to further minimize guessing effects. Use precise words (100, 20%, half), rather than vague or qualitative language (young, small, new, beautiful, many). Avoid making the correct answer longer than the incorrect answer (a give-away).

Matching
Matching is an extended form of MCQ that draws

upon the students ability to make connections between ideas, vocabulary and structure advantage over MCQs is that the student has more distractors per item. writing items in the matching format is somewhat easier for teachers than either MCQs or TFNs some important points to bear in mind

The good and the bad of Matching Items

Pros Cons

efficient difficult to assess used to assess student higher-order outcomes understanding of (i.e., analysis, associations, synthesis, evaluation relationships, goals) definitions

Tips for Writing Matching Items

Include more items in the answer group than in the

question group Never write items that rely on direct 1-on-1 matching.
The consequence is that if a student gets one item

wrong, at least two are wrong by default. By contrast, all previous items right, the last item is a process of elimination freebie.
Matching can be used very effectively with related

items for gap-fill paragraphs instead of two lists.

In this way, students focus on meaning in context and

attend to features such as collocation.

Tips for Writing Matching Items

If a two-column format is used for matching,

number the questions and letter the answer options.

Leave a space for students to write the letter of the

chosen answer. This prevents lines drawn from Q to A columns.

Two-column matching formats should be used

sparingly for word association tasks.

Tips for Writing Matching Items

When this is the specific testing objective, be sure

that the syntax between the two columns is correct and unambiguous. Avoid extraneous clues such as using an when the correct answer starts with a vowel.

Matching-Equal Columns
When using this form, providing for some items in the response column to be

used more than once, or not at all, can preclude guessing by elimination.
Directions: In the blank before each electrical term in the left-hand column, write

the letter corresponding to the unit of measurement which is most closely associated with that term. Each unit of measurement may be used more than once and some units may not be used at all. 1. ____ Electromotive force 2. ____ Electrical power, apparent 3. ____ Electrical power, true 4. ____ Resistance 5. ____ Capacitance 6. ____ Inductance 7. ____ Current 8. ____ Impedance A. Watt B. Volt C. Ampere D. Coulomb E. Ohm F. VAR G. Farad H. Henry

Matching-Unequal Columns

Generally preferable to equal columns.

Directions: In the blank before each phrase in the left-hand column, write the letter(s) corresponding to the type(s) of drag which is/are most closely associated with that phrase. Each type of drag may be used more than once, and some types may not be used at all. 1. ____ Occurs when varied currents over an airplane meet and interact. 2. ____ Results from the turbulent wake caused by the separation of airflow from the surface of a structure. 3. ____ Caused by the roughness of the airplane's surfaces. E. Interference drag 4. ____ Generated by the airflow circulation around the airfoil as F. Rolling drag it creates lift. G. Sliding drag A. Form drag B. Induced drag C. Skin friction drag D. Static drag

Supply-Type
may be required where a selection-type cannot be

devised to properly measure student knowledge valuable in measuring the students' generalized understanding of a subject

The good and the bad of SupplyType

Pros Cons

chances of guessing scoring is not objective reduced can cause difficulty for measures knowledge computer scoring and fact outcomes well, terminology, formulas

Writing Subjective Test Items

Short-answer Problem sets Oral exams

Performance tests
Essay tests Guided and Free Writing

Short-answer Qs
depending on objectives set, these

questions can call for one or two sentences or a long paragraph easier to write, though they take longer to score compared to multiple-choice tests

Problem sets
Normally used in Mmathematics and the

Sciences Tip = allow students ten minutes to solve a problem you can do in two minutes

Oral exams
common at the graduate level, rarely used for

undergraduates except in foreign language classes are usually time-consuming, too anxiety provoking for students, and difficult to score unless the instructor tape-records the answers
However, a math professor has experimented with individual thirtyminute oral tests in a small seminar class. Students receive the questions in advance and are allowed to drop one of their choosing. During the oral exam, the professor probes students' level of understanding of the theory and principles behind the theorems. He reports that about eight students per day can be tested.

Performance tests
ask students to demonstrate proficiency in
conducting an experiment, executing a series of steps in a reasonable amount of

time, following instructions, creating drawings, manipulating materials or equipment, or reacting to real or simulated situations
can be administered individually or in groups seldom used in colleges and universities logistically difficult to set up, hard to score, and the content of most courses does not necessarily lend

itself to this type of testing

Tips for performance tests

in classes that require students to demonstrate their

skills (for example, health fields, the sciences, education). Anderson (1987, p. 43) recommends the following: 1. Specify the criteria to be used for rating or scoring (for example, the level of accuracy in performing the steps in sequence or completing the task within a specified time limit). 2. State the problem so that students know exactly what they are supposed to do (if possible, conditions of a performance test should mirror a real-life situation). 3. Give students a chance to perform the task more than once or to perform several task samples.

Essay tests - Pros

relatively easy to write enable you to judge students' abilities to organize, integrate, interpret material, and express themselves in their own words Research indicates that (McKeachie, 1986) students study more efficiently for essay-type examinations than for selection (MCQ) tests students preparing for essay tests focus on broad issues, general concepts, and interrelationships rather than on specific details, and this studying results in somewhat better student performance regardless of the type of exam they are given also give you an opportunity to comment on students' progress, the quality of their thinking, the depth of their understanding, and the difficulties they may be having

Essay tests - Cons

because essay tests pose only a few questions, their content validity may be low. the reliability of essay tests is compromised by subjectivity or inconsistencies in grading

difficulty also lies in clearly specifying the task for

the student so that grading is fair and equitable to all students

Tips for essay tests

writing good subjective items is an interactive,

negotiated process one best approach is to write a sample answer and then analyze the elements of that answer OR ask a colleague to write a sample answer and critique the prompt

Guided vs. Free Writing

literature generally addresses two types of writing: free writing
requires students to read a prompt that poses a situation and write a planned response based on a combination of background knowledge and knowledge learned from the course
guided writing

requires students to manipulate content that is provided in the prompt, usually in the form of a chart or diagram a bridge between objective and subjective formats requires teachers to be very clear about what they expect students to do decide in advance whether mechanical issues like spelling, punctuation and capitalization matter when the task focuses on comprehension

Guided vs. Free Writing

goal for teachers is to elicit comparable products

from students of different ability levels

Writing Assessment Scales

literature generally recognizes two

different types of writing scales for assessing student written proficiency:

1. holistic marking and
2. analytical marking

Alternative Testing Modes

Take-home tests
Open-book tests Group exams

Paired testing
Portfolios

Administering Tests

Timing
Item Type
True/False MCQ

Time Allocated
30 seconds 1 min

MCQ with Higher Learning Objectives

Short Answer Completion Matching Short Essay Extended Essay Visual Image

1.5 mins
2 minutes 1 min 30 secs per response 10-15 mins 30 min 30 seconds

Checklist
Is this item an appropriate measure of my learning

objective? Is the item format the most effective means of measuring the desired knowledge? Is the item clearly worded and easily understandable by the target student population? Are items of the same format grouped together? Are various item types included in the assessment? Do students have enough time to answer all test items? Are test instructions specific and clear? Does the number of questions targeting each objective match the importance weighting of that objective? Are scoring guidelines clearly available to students?

In closing
Tests are just tools
(Hanna & Dettmer, 2004; Kubiszyn & Borich, 2007)

Test items are part of those tools Like the tools in a carpenters toolbox, you

need to choose from tests toolbox what works best for the task at hand Remember, "If the only tool you have is a hammer, you tend to see every problem as a nail