Angela Yarnell

MEMT 898
Daugherty
October 1, 2009

Research Analysis 2

Ciorba, C. (2009). Measurement of instrumental and vocal undergraduate performance juries

using a multidimensional assessment rubric. Journal of Research in Music Education.
Vol. 57 (1), p. 5-15.

Purpose of the study: The purpose of this study was to investigate the effectiveness of a
multidimensional assessment rubric when administered across all university-level instrumental
and vocal performance juries during one semester.

Research questions posed:

• Is there reliability between judges of a multidimensional assessment rubric used in all the
various performance areas of brass, woodwinds, strings, voice, piano, guitar, and percussion?
• How effective is a multidimensional assessment rubric at rating disparities in performances
of students at different levels of music study?
• What similarities are there between points awarded on a multidimensional assessment rubric
and through typical single score grading of juries?

Primary methodology of this study: quantitative (non-experimental)

For quantitative (non-experimental) studies:

Participants N=359, later reduced to 325
Judges N=37
Number of Groups: 28, later reduced to 20

Type(s) of Data Collection:

Statistical Analyses (list the tests): Cronbach’s alpha, reliability rating and reliability
coefficients, means and standard deviations, one-way multivariate analysis of variance
(MANOVA), Analyses of variance (ANOVA) using the Bonferroni method at the .025 level,
Pearson correlations

Methodology Summary:
A panel of experienced university faculty developed an assessment rubric that could be applied
to all vocal and instrumental performance areas for use in the study. The resulting rubric was
piloted over the course of two semesters of jury performances and adjusted prior to use in this
study.

Faculty panels (judges) of 2, 3, and 4 members listened to undergraduate performance juries and
rated student achievement on the rubric without discussion between faculty members. There
were 20 different panels of judges and 325 participants in the study in one semester of juries at a
small university.

Conclusions:
Of all the panels, 18 out of 20 showed a significant level of agreement among judges. Two
judges on separate panels showed a low level of reliability and were subsequently removed.
When these two judges were removed, the reliability coefficients for subcategories on the rubric
were above .70. Reliability coefficients for the overall score from the rubric ranged from .66 to
.99, while the reliability coefficients for holistic grading ranged from .56 to 1.0. MANOVA
revealed that there were significant differences in scores by year in school. The ANOVA test
showed significance on scale scores, but not on grades by year. Correlations among
subcategories and overall scores were high (.81 - .89), but when correlations among scale
dimensions and grades were tested, they only had moderate (.64 - .72) agreement.

The above tests revealed that rubric scores were both significantly and positively related to year
in school, while typical holistic grades assigned by the same judges’ panels were not related to
school year. A multidimensional assessment rubric may show student achievement gain each
year and help track a student’s real progress in target areas. Single letter grades are nearly equal
to rubrics in their ability to sufficiently describe student achievement in a single performance.
However, rubrics provide more feedback than a single letter grade can and may encourage
students to attend to lower ranked subcategories.

Recommendations for future research:

A longitudinal study of undergraduate jury performances of the same students over four years is
needed to verify the validity of this rubric. This tool also needs to be applied at other universities
to determine content validity. The authors further recommend examination of relationships
between scores from this assessment tool and other extant records of student achievement,
including standardized test scores.

Evaluation/assessment of this research study:

Authors of this study were interested in a single rubric that could be used with reliability and
validity in all performance areas. While they were able to demonstrate a high degree of
interjudge reliability, some performance media may benefit from separate categories specific to
that area. This study’s greatest contribution is the significance of rubric scores as they pertain to
year in school, which demonstrates that achievement can be tracked from year to year and
specific feedback can be given to the student on their growth. The fact that holistic grades were
not significantly different than rubric composites is not unexpected. The additional advantages a
multidimensional assessment tool can provide are worth the effort in administering such a rubric.

Generalization/transfer possibilities:
Specific feedback is the goal of every teaching cycle in a private lesson or ensemble rehearsal.
Rubrics developed for use in Kansas choir environments include a state honor choir audition
rubric, a large group music festivals rubric, daily rehearsal rubrics, and others. In Kansas, choir
directors have advocated for slightly different rubrics for use in vocal situations than the
instrumental rubrics first developed for our use. We have found the extra areas address items
that are unique to singing and have been verified for their interjudge reliability. Regardless of
similarity between holistic grading and rubric composite scores, students and directors gain
much from the increased specific feedback that rubrics provide. They help communicate
consistency in grading and provide data for later comparisons. Performance rubrics can also be
used in the development of a student portfolio to monitor growth and achievement.