Development of Two Measures of Learner Motivation

John Keller Florida State University

Notice This is a partial draft of a manuscript in progress. It contains sufficient information about two situational measures of the motivation to learn to allow you to make use of them. These are copyrighted instruments. Please contact the author for permission to use them. There is no fee if they are used for research, but there are conditions that apply to their use. For further information, contact John Keller at arcsman@comcast.net.

1. Introduction
The purpose of this study was to develop and validate two motivational measurement instruments. The first instrument, called the Course Interest Survey (CIS), was designed to measure students' reactions to classroom instruction. The second, called the Instructional Materials Motivation Survey (IMMS), was designed to measure students' motivational reactions to self-directed instructional materials. Both surveys are situational measure of students' motivation to learn with reference to a specific learning condition such as an instructor-facilitated learning environment (the CIS), a self-paced print module (the IMMS), or a self-directed e-learning course. Furthermore, they were designed to be in correspondence with a theoretical foundation represented by a specific model of learner motivation which is called the ARCS Model (Keller, 1987a, 1987b). This theory is derived from the current literature on human motivation (Keller, 1979, 1983, 1999); hence, many of the items in the CIS are similar in intent (but not in wording) to items in established measures of psychological constructs such as need for achievement, locus of control, and self-efficacy, to mention three examples. As situational instruments, the CIS and IMMS are not intended to measure students' generalized levels of motivation toward school learning (i.e. they are not trait- or construct-type measures). The goal with these instruments is to find out how motivated students are, were, or expect to be, by a particular course. The expectation is that these surveys can be used with undergraduate and graduate students, adults in non-collegiate settings, and with secondary students. They can also be used with younger students who have appropriate reading levels. With younger students or ones who are not sufficiently literate in English, some of the items may have to be read aloud and paraphrased to relate them to the classroom experiences of the audience. Furthermore, both instruments can be adapted to fit specific situations. That is, the default wording of items contains phrases such as this course, or this lesson. These can be changed to fit the specific situation that is being assessed, such as this lecture, this computer-based instruction, etc. Also, it is possible to change the tense of the items to use it as a pretest. However, the substance of the items can not be changed because they are based of specific components of motivation.

John Keller, copyright () 2006.

Version 060222

Page 1 of 9

Development of Two Measures of Learner Motivation

Draft in progress

Following is information about each instrument followed by some recommendations about how to use them..

2. Development Process

Item development: After reviewing the concepts and strategies that comprise the ARCS model and a variety of instruments used to measure motivational constructs, a pool of items was prepared. These were reviewed by 10 graduate students who were well versed in the motivational literature. They responded to each item then discussed the items which seemed ambiguous, unrelated to the appropriate higherorder concept (i.e. attention, relevance, confidence, or satisfaction), or otherwise difficult to respond to. Ambiguity check (faking it!): The original item pool was reduced, revised, and administered to ten adults (mostly graduate students) who were told to respond to it twice. The first time they were to "fake good," and the second time to "fake bad." That is, they were to assume they were taking a course which was highly motivating, and to answer each item in a way that would indicate their highly positive motivation. Secondly, they were to assume that the course was totally unmotivating, and to answer accordingly. This test revealed a few items that could "go either way," i.e. potentially poor discriminators. These items were then revised or deleted and retested.

3. Course Interest Survey

The instructions and items are presented below (Tables 1 and 2): Instructions Course Interest Survey
John M. Keller Florida State University 1. There are 34 statements in this questionnaire. Please think about each statement in relation to the instructional materials you have just studied, and indicate how true it is. Give the answer that truly applies to you, and not what you would like to be true, or what you think others want to hear. 2. Think about each statement by itself and indicate how true it is. Do not be influenced by your answers to other statements. 3. Record your responses on the answer sheet that is provided, and follow any additional instructions that may be provided in regard to the answer sheet that is being used with this survey. Thank you.

Table 1. Instructions for the Course Interest Survey

John Keller, copyright () 2006.

Version 060222

Page 2 of 9

Development of Two Measures of Learner Motivation

Draft in progress

Course Interest Survey

John M. Keller Florida State University 1 (or A) = Not true 2 (or B) = Slightly true 3 (or C) = Moderately true 4 (or D) = Mostly true 5 (or E) = Very true 1. The instructor knows how to make us feel enthusiastic about the subject matter of this course. 2. The things I am learning in this course will be useful to me. 3. I feel confident that I will do well in this course. 4. This class has very little in it that captures my attention. 5. The instructor makes the subject matter of this course seem important. 6. You have to be lucky to get good grades in this course. 7. I have to work too hard to succeed in this course. 8. I do NOT see how the content of this course relates to anything I already know. 9. Whether or not I succeed in this course is up to me. 10. The instructor creates suspense when building up to a point. 11. The subject matter of this course is just too difficult for me. 12. I feel that this course gives me a lot of satisfaction. 13. In this class, I try to set and achieve high standards of excellence. 14. I feel that the grades or other recognition I receive are fair compared to other students. 15. The students in this class seem curious about the subject matter. 16. I enjoy working for this course. 17. It is difficult to predict what grade the instructor will give my assignments. 18. I am pleased with the instructor's evaluations of my work compared to how well I think I have done. 19. I feel satisfied with what I am getting from this course. 20. The content of this course relates to my expectations and goals. 21. The instructor does unusual or surprising things that are interesting. 22. The students actively participate in this class. 23. To accomplish my goals, it is important that I do well in this course. 24. The instructor uses an interesting variety of teaching techniques. 25. I do NOT think I will benefit much from this course. 26. I often daydream while in this class. 27. As I am taking this class, I believe that I can succeed if I try hard enough. 28. The personal benefits of this course are clear to me. 29. My curiosity is often stimulated by the questions asked or the problems given on the subject matter in this class. 30. I find the challenge level in this course to be about right: neither too easy not too hard. 31. I feel rather disappointed with this course. 32. I feel that I get enough recognition of my work in this course by means of grades, comments, or other feedback. 33. The amount of work I have to do is appropriate for this type of course. 34. I get enough feedback to know how well I am doing.

Table 2. Items of the Course Interest Survey

Scoring The survey can be scored for each of the four subscales or the total scale score (Table 3).The response scale ranges from 1 to 5 (see Table 2). This means that the minimum

John Keller, copyright () 2006.

Version 060222

Page 3 of 9

Development of Two Measures of Learner Motivation

Draft in progress

score on the 34 item survey is 34, and the maximum is 170 with a midpoint of 102. The minimums, maximums, and midpoints for each subscale vary because they do not all have the same number of items. An alternate scoring method is to find the average score for each subscale and the total scale instead of using sums. For each respondent, divide the total score on a given scale by the number of items in that scale. This converts the totals into a score ranging from 1 to 5 and makes it easier to compare performance on each of the subscales. There are no norms for the survey. As it is a situation specific measure, there is no expectation of a normal distribution of responses. As data become available from a variety of applications of the scales, descriptive statistical information will be published. Scores are determined by summing the responses for each subscale and the total scale. Please note that the items marked reverse (Table 3) are stated in a negative manner. The responses have to be reversed before they can be added into the response total. That is, for these items, 5 = 1, 4 = 2, 3 = 3, 2 = 4, and 1 = 5. Attention 1 4 (reverse) 10 15 21 24 26 (reverse) 29 2 5 8 (reverse) 13 20 22 23 25 (reverse) 28 Table 3. CIS scoring guide Relevance 3 6 (reverse) 9 11 (reverse) 17 (reverse) 27 30 34 Confidence Satisfaction 7 (reverse) 12 14 16 18 19 31 (reverse) 32 33

Psychometric testing: The survey was administered to a class of 45 undergraduates, and the internal consistency estimates were satisfactorily high. A pretest version was prepared by rewriting items in the future tense and was administered in an undergraduate class of 65 students. The internal consistency estimates were high, but further revisions were made to improve the instrument. It was then administered to 200 undergraduates and graduate students in the School of Education at the University of Georgia at Valdosta. Information was also obtained about the student's course grades and grade point averages. The internal consistency estimates, based on Cronbachs alpha, were satisfactory (Table 3). Scale Attention Relevance Reliability Estimate (Cronbach E) .84 .84

John Keller, copyright () 2006.

Version 060222

Page 4 of 9

Development of Two Measures of Learner Motivation

Draft in progress

Confidence Satisfaction Total scale Table 4. CIS reliability estimates

.81 .88 .95

CIS Situational Validity CIS scores from the 200 Georgia undergraduates and graduates were correlated with their course grades and grade point averages (Table 5). All of the correlations with course grade are significant at or beyond the .05 level, and none of the correlations with grade point average are significant at the .05 level. This supports the validity of the CIS as a situation specific measure of motivation, and not as a generalized motivation measure, or construct measure, for school learning. ARCS Categories Course Grade .19 .43 .51 .49 .47 GPA

Attention Relevance Confidence Satisfaction Total Scale

.01 .08 .01 .03 .04

Table 5. CIS correlations with course grade and GPA

John Keller, copyright () 2006.

Version 060222

Page 5 of 9

Development of Two Measures of Learner Motivation

Draft in progress

4. Instructional Materials M. Scale

The instructions and items for the IMMS are as follows (Tables 6 and 7): Instructions Instructional Materials Motivation Survey
John M. Keller Florida State University 1. There are 36 statements in this questionnaire. Please think about each statement in relation to the instructional materials you have just studied, and indicate how true it is. Give the answer that truly applies to you, and not what you would like to be true, or what you think others want to hear. 2. Think about each statement by itself and indicate how true it is. Do not be influenced by your answers to other statements. 3. Record your responses on the answer sheet that is provided, and follow any additional instructions that may be provided in regard to the answer sheet that is being used with this survey. Thank you.

Table 6. Instructions for the Instructional Materials Motivation Survey

Instructional Materials Motivation Survey John M. Keller Florida State University 1 (or A) = Not true 2 (or B) = Slightly true 3 (or C) = Moderately true 4 (or D) = Mostly true 5 (or E) = Very true 1. When I first looked at this lesson, I had the impression that it would be easy for me. 2. There was something interesting at the beginning of this lesson that got my attention. 3. This material was more difficult to understand than I would like for it to be. 4. After reading the introductory information, I felt confident that I knew what I was supposed to learn from this lesson. 5. Completing the exercises in this lesson gave me a satisfying feeling of accomplishment. 6. It is clear to me how the content of this material is related to things I already know. 7. Many of the pages had so much information that it was hard to pick out and remember the important points. 8. These materials are eye-catching. 9. There were stories, pictures, or examples that showed me how this material could be important to some people. 10. Completing this lesson successfully was important to me. 11. The quality of the writing helped to hold my attention. 12. This lesson is so abstract that it was hard to keep my attention on it. 13. As I worked on this lesson, I was confident that I could learn the content. 14. I enjoyed this lesson so much that I would like to know more about this topic. 15. The pages of this lesson look dry and unappealing. 16. The content of this material is relevant to my interests. 17. The way the information is arranged on the pages helped keep my attention. 18. There are explanations or examples of how people use the knowledge in this lesson. 19. The exercises in this lesson were too difficult. 20. This lesson has things that stimulated my curiosity. 21. I really enjoyed studying this lesson. 22. The amount of repetition in this lesson caused me to get bored sometimes. 23. The content and style of writing in this lesson convey the impression that its content is worth knowing.

John Keller, copyright () 2006.

Version 060222

Page 6 of 9

Development of Two Measures of Learner Motivation

Draft in progress

24. I learned some things that were surprising or unexpected. 25. After working on this lesson for awhile, I was confident that I would be able to pass a test on it. 26. This lesson was not relevant to my needs because I already knew most of it. 27. The wording of feedback after the exercises, or of other comments in this lesson, helped me feel rewarded for my effort. 28. The variety of reading passages, exercises, illustrations, etc., helped keep my attention on the lesson. 29. The style of writing is boring. 30. I could relate the content of this lesson to things I have seen, done, or thought about in my own life. 31. There are so many words on each page that it is irritating. 32. It felt good to successfully complete this lesson. 33. The content of this lesson will be useful to me. 34. I could not really understand quite a bit of the material in this lesson. 35. The good organization of the content helped me be confident that I would learn this material. 36. It was a pleasure to work on such a well-designed lesson.

Table 7. Items of the Instructional Materials Motivation Survey

Scoring As with the CIS, the survey can be scored for each of the four subscales or the total scale score (Table 8).The response scale ranges from 1 to 5 (see Table 7). This means that the minimum score on the 36 item survey is 36, and the maximum is 180 with a midpoint of 108. The minimums, maximums, and midpoints for each subscale vary because they do not all have the same number of items. An alternate scoring method is to find the average score for each subscale and the total scale instead of using sums. For each respondent, divide the total score on a given scale by the number of items in that scale. This converts the totals into a score ranging from 1 to 5 and makes it easier to compare performance on each of the subscales. There are no norms for the survey. As it is a situation specific measure, there is no expectation of a normal distribution of responses. As data become available from a variety of applications of the scales, descriptive statistical information will be published. Scores are determined by summing the responses for each subscale and the total scale. Please note that the items marked reverse (Table 8) are stated in a negative manner. The responses have to be reversed before they can be added into the response total. That is, for these items, 5 = 1, 4 = 2, 3 = 3, 2 = 4, and 1 = 5.

John Keller, copyright () 2006.

Version 060222

Page 7 of 9

Development of Two Measures of Learner Motivation

Draft in progress

Attention 2 8 11 12 (reverse) 15 (reverse) 17 20 22 (reverse) 24 28 29 (reverse) 31 (reverse) 6 9 10 16 18 23

Relevance 1

Confidence 5 14 21 27 32 36

Satisfaction

3 (reverse) 4 7 (reverse) 13 19 (reverse) 25 34 (reverse) 35

26 (reverse) 30 33

Table 8. IMMS scoring guide

Psychometric testing: The survey was administered to a total of 90 undergraduate students in two undergraduate classes for preservice teachers at Florida State University. The internal consistency estimates, based on Cronbachs alpha, were satisfactory (Table 9). Reliability estimates Scale Attention Relevance Confidence Satisfaction Total scale Table 9. IMMSS reliability estimates Reliability Estimate (Cronbach E) .89 .81 .90 .92 .96

IMMS Validity Test Validity was established by preparing two sets of instructional materials covering the concept of behavioral objectives. These materials were part of a unit of work on lesson planning and instructional design. Both lessons had the same objectives and technical content. The lesson for the control group was prepared according to standard principles of instructional design, but was not enhanced in any way to make it interesting. The experimental lesson was enhanced with strategies to stimulate curiosity, illustrate the

John Keller, copyright () 2006.

Version 060222

Page 8 of 9

Development of Two Measures of Learner Motivation

Draft in progress

practical relevance of the content, build confidence, and provide satisfying outcomes. Students were randomly assigned to the two lessons which they completed during one class period, including testing. Scores on the experimental lesson were significantly higher than for the control lesson.

5. Summary

Both surveys are robust

y y y y

Wording can be adapted to specific courses Subscales can be used Length can be modified with cautions

Both surveys are available (free) for R&D

John Keller, copyright () 2006.

Version 060222

Page 9 of 9