Best Assessment Processes IV A Working Symposium ~ April 7 - 8, 2001

Diagnostic vs. Prognostic Assessment: Getting to the Good Stuff

Gwen Lee-Thomas, Ph.D. Director of Assessment Rose-Hulman Institute of Technology Terre Haute, Indiana

 Assessment: The process of gathering and

analyzing purposeful data that provides insight into the quality or effectiveness of a process, outcome, or objective for continuous improvement.

Evaluation: Information derived from analyzed

data used to implement improvements in the quality or effectiveness of a process or outcome.
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Forms of Assessment
Formative Assessment: The collection of data and the feedback of the results on an ongoing basis (G. Rogers & J. Sando, 1996) Summative Assessment: Designed to produce information that can be used to make decisions about the overall success of the project or process. (G. Rogers & J. Sando, 1996)
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Quantitative (Objective)
Written Exams & Assignments Group Projects Performance Appraisals Capstone Courses Surveys/Questionnaires Institutional Data Student Self-Assessments* Classroom Assessment Techniques

Qualitative: (Subjective)
Behavioral Observations Exit Interviews Simulations Focus Groups Classroom Assessment Techniques Oral Exams Presentations* Portfolios*

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Diagnostic Assessment The use of certain datagathering techniques that identify the condition or the what in an occurrence that may inform continuous improvement practices.

Prognostic Assessment The use of multiple processes that complementarily provide insight into the why of an occurrence to adequately predict future occurrence for continuous improvement.

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Principle #2:
Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.
Involves cognitive, attitudinal, and behavioral assessment. Ongoing assessment to reveal growth, change, and increasing degrees of integration.
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Principle #5:
Assessment works best when it is ongoing, not episodic.
Cumulative information that is not one shot. Continuous improvement of student learning as well as the assessment process itself.

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Start with Basic Assessment Questions

What is it I want to know? Why do I want to know that? What answers do I think the information will give me? Wheres the best place to get the information or data? How will I get the information or data? What will I do with it once I get it?
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

What do I want to know? What will I do with the information? Assessment Plan How can I get the information? Wheres the best place to get the information?
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Why do I want to know that?

What will the information tell me?

Design the Assessment Process

What is it that I want to know?
This is when you will need to think more specifically about whether or not you will need to know more about what occurred by finding out why something occurred.
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 When deciding on the data to gather, consider what assessment tools are appropriate. Keep in mind whether or not why is just as important as what.

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

External Client Project Assessment Project Objectives Reduce the Brain Drain in the state of Indiana Assist current businesses in becoming more productive and competitive. Provide growth in the number of new businesses in Indiana

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Project Objective: Reduce the Indiana Brain Drain of engineering students. Student Surveys S99 & S00 Project Objective External Client Surveys F99
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Focus Groups S00

Employer Interviews S00

(Objective: Reduce Indiana Brain Drain)

Student Survey Questions: Q7: I will seek employment in Indiana upon graduating from RH. Q9: I believe I will eventually start my own business in Indiana. External Client Survey Questions: Q6: The project was a good way of evaluating and recruiting technical talent.

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Student Surveys: (N=21)

Q7: I will seek employment in Indiana upon graduating from RH.
10% Yes 33% No 57% Undecided

External Client Surveys: (N=46)

Q6: The project was a good way of evaluating and recruiting technical talent. *
16% Strongly Agree 36% Agree 18% Neutral 0% Disagree or strongly disagree 39% Not Applicable
Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

Sunday, April 8, 2001 Session #40

(Focus Groups & Interviews)

Interviews with Employers: The company recognized technical talent for recruiting purposes, and were pleased at students initiative, knowledge, and performance. Focus Groups with Students: Reasons for not staying in Indiana were based on the students varied personalities and preferences for a city. As engineers the students could work in almost any city and most companies had sites in other states.

Sunday, April 8, 2001 Session #40

Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Diagnostic Assessment:
Allows you to discover the what in an occurrence. We must be careful when determining if the what is enough to make an informed decision for improvement.

Prognostic Assessment:
Allows you to go beyond the what and identify the why in an occurrence. This next step includes

the use of multiple assessment methods over incremental points in time.

Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment

 Angelo, Thomas A. and Patricia K. Cross, 2nd ed. (1993). Classroom assessment techniques: A handbook for college teachers. Jossey-Bass. San Francisco. Astin, Alexander W. (1993). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. American Council on Education. Oryx Press. Phoenix, AZ 85012-3397. Johnson, David W. and Roger T. Johnson, (1996). Meaningful and manageable assessment through cooperative learning. Interaction Book Company. Edina, MN. Leat, David & Nichols, Adam. (2000). Assessment in Education. Brains on the Table: diagnostic and formative assessment through observation. Vol. 7, no. 1. ISSN 0969-594X.
Sunday, April 8, 2001 Session #40 Gwen Lee-Thomas, Ph.D. Diagnostic vs. Prognostic Assessment