Education 353C Problems in Measurement: Generalizability Theory

Richard J. Shavelson & Edward H. Haertel Tuesdays 9:00AM 11:50AM Cubberley 207 Generalizability Theory (G theory) provides a powerful class of psychometric models for behavioral and social measurement data. G theory was developed beginning in the 1960s by Lee J. Cronbach here at Stanford University with his collaborators Goldine Gleser, Harinder Nanda, and Nageswari Rajaratnam (Cronbach, Rajaratnam, & Gleser, 1963; Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It encompasses classical test theory (CTT) as a special case, but goes far beyond CTT in clarifying conceptual confusions and providing more powerful statistical tools. Most importantly, G theory systematizes distinctions among the many reliability coefficients of CTT, and enables the calculation of reliability coefficients and/or standard errors of measurement tailored to specific measurement applications and/or interpretations. G theory also provides a powerful extension of the Spearman-Brown "prophecy" formula, enabling the design of efficient data collection plans for specific measurement applications. In this course, we will provide a systematic introduction to the concepts and models of G theory, as well as special-purpose software useful in carrying out the requisite calculations. The goals of the course are to provide a working knowledge of common Gtheory models and applications, to enable students to read critically journal articles employing G theory, and to introduce some unsolved problems and current research topics related to G-theory. The class will typically be organized as follows. The instructors might begin with a minilecture on the topic of the day with discussion. Then, a problem with data will be introduced and students will work in teams to solve the problem within the G theory framework. Toward the end of the quarter, the problems will be replaced by student-led presentation of self-chosen paper that uses G theory. Occasional homework will consist of students working on a brief problem with data. There will be a final project in which either a student will apply G theory to a somewhat novel set of data and report her/his findings to the class or will read up on a topic not covered in class (e.g., newly developed standard errors and confidence intervals for variance component estimates) and teach the class about the topic. There will be no final examination. The primary text for the course is Generalizability Theory: A Primer by Richard J. Shavelson and Noreen M. Webb. In addition, Generalizability Theory by Robert L. Brennan is recommended. Additional readings will be provided during the quarter. Note that the Primer is just thatan introductory text on G theory. Brennans is the currently authoritative text. So, when you complete a chapter in the Primer and you want more

G Theory Seminar

Shavelson & Haertel

statistical background, turn to Brennan. Brennan covers roughly the same materials in his Chapters 1-4 as the Primer doesbut in a somewhat different structure. You can purchase both textbooks online so they are not in the bookstore. For convenience, we have copied the first two chapters of the Primer and will email them to you.
Some References Brennan, R. L. (1997). A perspective on the history of generalizability theory. Educational Measurement: Issues and Practice, 16(4), 14-20. Brennan, R. L. (2000a). (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice, 19(1), 5-10. Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag. Cardinet, J., Tourneur, Y., & Allal, L. (1976). The symmetry of generalizability theory: Application to educational measurement. Journal of Educational Measurement,13, 119-135 Cardinet, J., Tourneur, Y., & Allal, L. (1981). Extension of generalizability theory and its applications in educational measurement. Journal of Educational Measurement, 18, 183-204. Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391-418 Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley. Cronbach, L. J., Linn, R. L., Brennan, R. L, & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399. Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: American Council on Education/Praeger. Shavelson, R. J. (2004). Editors preface to Lee J. Cronbachs My Current Thoughts on Coefficient Alpha and Successor Procedures. Educational and Psychological Measurement, 64(3), 389-391. Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36, 61-71. Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973-1980. Mathematical and Statistical Psychology, 34, 133-166. British Journal of

Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage. Shavelson, R.J., Webb, N.M., & Rowley, G. (1989). Generalizability theory. American Psychologist, 44(6), 922-932. Webb, N. M., Shavelson, R. J., & Maddahian, E. (1983). Multivariate generalizability theory. In L.J. Fyans (Ed.), Generalizability theory: New directions for testing and measurement (pp. 67-82). San Francisco: Jossey-Bass. Webb, N. M., Shavelson, R. J., & Haertel, E.H. (2007). Reliability coefficients and generalizability theory. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics: Psychometrics (Vol. 26, pp. 81-124). Amsterdam: Elsevier B. V. Wiley, E. W. (2000). Bootstrap strategies for variance component estimation: Theoretical and empirical results. Unpublished doctoral dissertation, Stanford University.

Spring 2007

Tentative Course Schedule1 Date 4-3 Topic Introduction to G Theory & Statistical Demo Software (GENVA & EduG) SW Tables 1.1 and 1.3 AERA WeekNo class meeting Readingsa SW Chapter 1 CGNR Preface & Pp 1-14 Homework #1 A. Handout P x I G study B. Run Exercise 3 in SW Ch. 3 with GENOVA & EduG Homework #2Brief proposal for final project (bring handouts to class 4/24) Read Cardinets Why EduG Homework #3Read & prepare to lead discussion: Group 1: Cardinet et al 1981 Group 2: CLBH Group 3: SRW Homework #4 Prepare final project description; hand in at class on 5-8 Homework #5 Cardinet Exercise #2 & Cardinet Complex Exercise Assignmentb

4-10

See 4-3 Readings SW Chapter 2 Brennan Appx. F EduG Users Guide SW Chapters 2 & 3

4/17

Discuss Homework #1 Statistical Model for G Theory G Studies with crossed facets One-facet G Study simulation

4/24

Discuss final project ideas Discuss Why EduG In Class Exercise #1 G studies with nested facets Group presentations of papers Random & Fixed Facets Decision Studies Exercise#2_Cardinet #1 Collect project descriptions

SW Chapter 4

5-1

SW Chapter 5

5-8

SW Chapter 6

This schedule is tentative and may change depending on coverage in class or other unexpected events.

G Theory Seminar

Shavelson & Haertel

5-15

Discuss Homework #5 G & D Studies for same and different Designs Multivariate G Theory mGenova Exercise Presentations of student projects

SW Chapter 7 & 8

Work on projects

5/22

WS Brennan 267-285 (Optional)

5-29

6-5 Presentations of student projects ___________________________________________

a

Readings to be done before class on date given CGNR refers to Cronbach, Gleser, Nanda, & Rajaratnam (1972). CLBH refers to Cronbach, Linn, Brennan, & Haertel (1997). SW refers to Shavelson & Webb (1991). SRW refers to Shavelson, Ruiz-Primo, and Wiley (1999). WS refers to Webb & Shavelson (1983). b Assignments due the week following their assignment

Spring 2007