Académique Documents
Professionnel Documents
Culture Documents
Draft
Spring 2008
May 8, 2008
Introduction
This report of Food Science’s assessment focuses on the Program’s use of its Critical
and Integrative Thinking rubric as applied in a set of courses taken by Food Science
students and other students. The Food Science faculty have a variety of goals. To
address departmental diversity, meet process strategies for accountability required by
the professional accrediting association (The Institute of Food Technologists), WSU and
UI strategic plan and WSU’s regional accrediting body’s requirements, and, most
importantly, to improve the education of future Food Scientists, faculty in the Food
Science programs at WSU and at the UI have identified a single, comprehensive
outcome: Students who complete the program will demonstrate professional level
competence in scientific reasoning in Food Sciences. This is the fifth assessment cycle
for the program.
3. The assessment will help identify weaknesses and strengths to help improve and
promote the Food Science program.
Faculty agreed that the double review process used by the Educational Testing
Services would be appropriate for establishing and monitoring inter-rater reliability and
consensus necessary for improving program coherency and faculty community. To
establish reliability, raters participated in a norming session, rating an identical paper
based on the scoring criteria, and then negotiated a consensus on how the criteria
would be applied. Further, by adopting the rigor embodied in the double review
approach faculty establish expert validity in the assessment process.
Performance Criteria
The six criteria can be mapped to the WSU’s Six Goals of the Baccalaureate by
association as illustrated in Figure 1 and Figure 2:
The outcomes assessed in FS implicitly map to WSU’s six goals of the baccalaureate.
The correspondence between WSU’s six goals and the FS assessment process has
been used here to provide a view of how FS’ students are performing on institutional
goals. Table 1 reflects a distribution of rubric dimensions to the WSU Six. Figure 3
represents the results of that distribution.
Inter-rater Reliability
Inter-rater reliability was determined using the ETS method—a calculation of the
percentage of overall agreement—which was used to monitor the reliability of the raters.
Acceptable levels of reliability are indicated by 75% or better using the ETS method.
The overall percentage agreement among raters in this session was 74%.
Course
FS 470 FS 303 FS 220 Overall
Rubric Dimension WSU and UI
(n=13) (n=15) (n=20) (n=48)
Mean Mean Mean Mean
Problem 3.81 3.10 3.57 3.49
Hypothesis 3.65 3.27 3.10 3.34
Observations 4.04 2.97 3.12 3.38
Data 4.00 2.73 2.97 3.23
Conclusions 3.77 2.77 3.00 3.18
Communication 4.00 2.97 3.23 3.40
Table 3. 2008 Critical and Integrative Thinking Outcomes by Dimension and Course
Assignment Assessment
In addition to assessing student performance in written papers, raters used the same
criteria to assess the assignments that students were responding to. The scoring
method assessed the extent to which the assignment expectations related to the six
dimensions of the rubric: explicit, implicit, or absent. These ratings are necessarily
Dimension 1: Problem
Dimension 2: Hypothesis
Dimension 3: Observations
Dimension 4: Data
• A little hard to figure out how to do this in light of the first sentence (R3)
• More detail of expectations is needed. (R5)
Dimension 5: Conclusions
Dimension 6: Communication
• I would have trouble trying to organize my thoughts for this, the structure is clear,
but would be difficult to implement. (R3)
• This really lacks detail on expectations. I could not do this assignment without
instructor explaining verbally. (R5)
Dimension 1: Problem
• Written English is a bit difficult to follow and some instructions are not all that clear
(R3)
Dimension 2: Hypothesis
• Critique does not usually require that there be a hypothesis (for sure). An outline
should include objectives or a roadmap for the paper, but whether this would be
understood by students would depend on what level they are at in college.(R3)
Dimension 3: Observations
Dimension 4: Data
o This implies that the paper will involve an analysis of primary literature and not
review literature and also that the paper will focus on scientific reports rather
than policy studies.(R3)
Dimension 5: Conclusions
Dimension 6: Communication
More detail here would help. Depending on a student’s perspective of what this
assignment is supposed to be about (e.g. science vs policy)they could have
major problems with trying to write a paper with the proposed structure. The
assignment is "to critically analyze information about a topic related to food
safety and quality and to write a paper which critiques, summarizes and
articulates the current level of knowledge concerning that topic". This is pretty
vague. (R3)
Dimension 1: Problem
Dimension 2: Hypothesis
Dimension 4: Data
Dimension 6: Communication
Figure 9. Comparison of Ratings for Student Work and the Assignment Prompt for FS
202
Figure 11. Comparison of Ratings for Student Work and the Assignment Prompt for FS
470
The program has made solid progress in formalizing the assessment of the three
goals—student competence in scientific reasoning, relating the assessment of that
competence to the WSU Big Six, and implementing a process that will help establish
Food Science.
Program Innovations
Throughout the study, noteworthy innovations have been integrated into Food Science:
4. The assessment of student work and assignment prompts was completed online
using the Skylight survey tool. This innovation offers the option of more convenient
asynchronous rating after a brief face to face norming session.
We make suggestions below for next steps but without investment from faculty progress
is unlikely. Ultimately the Food Science faculty will have to make a commitment to
improvements of their choice, identify the players needed to implement them, and
create a timeline with milestones. Such choices, even if limited in number, will guide
future assessment efforts in the classroom settings experienced by Food Science
students.
Measure what you value and value what you measure. If the learning outcomes
identified by faculty are important enough to guide the assessment of the program, they
are important enough to communicate to students as a guide for their learning.
There are many ways to communicate the value of the learning outcomes to the
students.
• Give the rubric to students when they enter the program. Make it clear to them that
these criteria will be the measure of their success in the program and of their
readiness to enter the professional community. . The grading scale is absolute—
professional competence—faculty, professionals in the industry, peers, and they
will continuously assess their performance, using that absolute standard, as they
move through the program
• Let students know that through ongoing and consistent feedback they will
improve in their understanding and practice of these criteria. Coach the students
in using the rubric to self assess and assess peers. (CTLT can help with this)
• Involve Students in the Process: On the first day of classes, give a copy of the
rubric to your students along with the syllabus.
o Conduct a short norming session with students early in the semester.
• Coach faculty in using the rubric to self assess, assess peers, and students.
(CTLT can consult with faculty individually or in group workshops)
• Meet regularly to identify and implement changes. Identify trends and problem
areas in student performance. Solicit faculty, professional and student reflections
on strategies for revising assignments to improve program outcomes (The
reflections are invaluable evidence of quality enhancement essential for
accreditation as well as program improvement).
• Develop a formalized timeline for rating sessions and the use of their results.
The model below repeats as part of an ongoing process