Académique Documents
Professionnel Documents
Culture Documents
A FACET-FACTORIAL
APPROACH TO RATING
HIGH SCHOOL CHORAL
MUSIC PERFORMANCE
John M. Cooksey
The purpose of this study was to construct and test a rating scale for the
evaluation of high school choral music performance, using a facet-factoria I
approach. A scale was developed by collecting descriptions of high school
choral performances, transforming them into items, and pairing them with
a Likert-typescale. Fiftyjudges used the scale to rate one hundred high school
choral performances. The ratings were factor analyzed and inter-judge
reliability estimates were obtained. Two criterion-related studies, using a
global performance rating and the NIMACadjudication scale as criteria,
were completed. Seven factors of choral performance were produced by the
analysis: diction, precision, dynamics, tone control, tempo, balance/blend,
and interpretation/musical effect. Thirty-six criteria (dimensions-items)
were selected to form subscales to measure these seven factors. The final
scale achieved high inter-judge reliability and high criterion-related validity.
The Problem
The problem was to construct a rating scale for the evaluation of high
school choral music performance using a facet-factorial approach. Analy-
sis of the problem led to the development of the following subproblems:
TheScale
The first step in the development of the CPRS was to determine what
evaluative criteria should be used in the description of high school
choral performance. Evaluative statements were collected from three
major sources: (1) 618 adjudication sheets containing judges' comments
about actual high school choral performances at district and state vocal
contests; (2) fifty-two critiques written by choral teachers on recorded per-
formances of high school choruses; and (3) twelve essays written by choral
102/JRME
experts on aural aspects of high school choral performances. A content
analysis produced more than 500 statements about the evaluative aspects
of high school choral performance. The statements were pooled and
placed under the following categories of the National Interscholastic
Music Activities Commission (NIMAC) choral evaluation structure:
balance, diction, intonation, technique, tone, interpretation, and musical
effect.8 During succeeding revisions of the list, statements which dupli-
cated others or seemed vague or irrelevant were eliminated. After several
revisions, 147 statements were chosen for the initial item pool.
The next step was to determine the factors that seemed central to rat-
ing high school choral performance as identified by facet-factorial analy-
sis. Facet-factorial scales were used by Butt and Fiske (1968)4 to measure
dominance in personality:
Table 1
Sample Items from the Performance Description Item Pool
Judges'Responses SampleItems
SD D NN A SA 130. Top voice flat at times. (-)
SD D NN A SA 131. Excellentfeeling of ensemble.(+)
SD D NN A SA 132. Phraseslack consistent contour and arch. (-)
SD D NN A SA 133. Excellentarticulationof pitches.(+)
13
Joy P. Guilford, Psychometric Methods, second edition (New York: McGraw-Hill
Book Co., Inc., 1954), p. 383.
106/JRME
Table2
VarimaxFactorLoadingsfor ItemsSelectedforthe CPRS*
00
?C C C
X, c c o
-"- 0L E
_'E m no k- a a
0-
Items I II III IV V V Vl VIII
A. Diction
1. Articulation was clear
and precise .375 .378 .550
2. Words clearly under-
standable .721
3. The diction of this
group is excellent .316 .638
4. Diction is muddy .755
5. Initial consonants need
more emphasis .618
B. Unity
1. Excellent unity of style .450 .605
2. Overall effect is choppy
and over-sectionali7ed .382 .606
3. Too much emphasis on
unimportant words and
syllables .632
4. Excellent feeling of
ensemble .436 .572
5. Good overall blend of
all parts .386 .537
C. Balance
1. Top voices cover up
lower voices .819
2. Men's voices (Sop. II/
Alto for Women's Ch.)
balance the choir very
well .768
3. Excellent balance
between all parts .742
4. Inner parts balance the
outer voices very well .647
5. Lowest part balances
upper parts very well .669
D. Precision
1. Sloppy rhythms .334 .610
2. All part entrances are
very precise .301 .692
3. Attacks and releases of
many notes are imprecise .321 .562
Cooksey/107
4. Some poor entrances by
different parts .724
5. Attacks are consistently
weak .514 .427
E. Tempo
1. Tempo unsteady in some
sections .596
2. Excellent control of
tempo .425 .592
3. Tendency to rush the
tempo .636
4. Unsteady rhythmic
sections .482 .537
F. Dynamics
1. Needs wider dynamic
contrasts .630
2. Lovely changes in
dynamics .373 .685
3. Delicate, expressive
shading in dynamics .467 .574
4. Excellent use of "pp" .514
5. Dynamics handled well
in relation to phrase
development .398 .494
G. Tone Control
1. The tone quality is too
harsh in forte passages .678
2. Sopranos (Ten. I for
Men's Ch.) sound forced
in upper pitch and dy-
namic ranges .741
3. The tone quality is often
forced in this choir .655
4. Excellent control of in-
tonation at forte levels .396 .644
5. Intonation in all parts
excellent throughout .438 .545
H. Interpretation/Musical Effect
1. Performance exhibits the
proper stylistic interpre-
tation .674
2. This choir projects the
mood of the selection
very well .843
3. Emotional concept of
word meanings very
well expressed .740
4. A musical and artistic
effect... fluid and vital .663 .363
5. Excellent forward
rhythmic drive .714
* All item loadings above .30 included across factors.
108/JRME
Table 3
The Choral Performance Rating Scale
The purpose of the following questions is to have you as accurately as possible
describe the performance which you have just heard. Respond to each statement
on the basis of how much you agree or disagree that the statement is descriptive
of the performance. Use the following five-point scale:
SD - Strongly disagree that the statement is descriptive
D - Disagree that the statement is descriptive
NN- Neither disagree nor agree that the statement is descriptive
A - Agree that the statement is descriptive
SA - Strongly agree that the statement is descriptive
+ sign indicates positive statement; - sign indicates negative statement.
Please choose only one response to each question. Please attempt to answer every
question. Circle responses.
SD D NN A SA 1. Excellent forward rhythmic drive +
SD D NN A SA 2. Attacks and releases of many notes are imprecise -
SD D NN A SA 3. Excellent unity of style +
SD D NN A SA 4. Dynamics handled well in relation to phrase develop-
ment +
SD D NN A SA 5. Lowest part balances upper parts very well +
SD D NN A SA 6. Excellentuse of "pp" +
SD D NN A SA 7. Tendency to rush the tempo -
SD D NN A SA 8. Excellentfeeling of ensemble +
SD D NN A SA 9. The tone quality is too harshin forte passages-
SD D NN A SA 10. Lovely changes in dynamics +
SD D NN A SA 11. Sloppy rhythms -
SD D NN A SA 12. All part entrancesare very precise +
vised CPRS scores for the first three groups were above .98. The student
group of judges achieved a reliability estimate of .97. Inter-judge reli-
ability coefficients for each of the subscales of the revised CPRS were
generally above .95 for the first three groups of judges, and above .92
for the student group. Table 5 contains the inter-judge reliability esti-
mates for the revised CPRS subscale scores.
To estimate inter-judge reliability for adjudication panels of different
sizes, a generalized Spearman-Brown prophecy formula was applied to
the reliability estimates of the total and subscale revised CPRS scores.14
For the first three groups of judges, reliability estimates for the total
revised CPRS scores were above .84 for as few as two judges; the reli-
ability estimate for the student group was .76 for as few as two judges.
The subscale reliability coefficients for the first three groups were above
.72 for as few as three judges, and for the student group were above .70
for three judges.
To examine the criterion-related validity of the revised CPRS, two
14 Harold Gulliksen, Theory of Mental Tests (New York: John Wiley and Sons, 1950).
Cooksey/109
SD D NN A SA 13. Unsteady rhythmic sections -
SD D NN A SA 14. Needs wider dynamic contrasts -
SD D NN A SA 15. The tone quality is often "forced" in this choir -
SD D NN A SA 16. Words clearly understandable +
SD D NN A SA 17. Top voices cover up lower voices -
SD D NN A SA 18. A musical and artistic effect... fluid and vital +
SD D NN A SA 19. Good overall blend of all parts +
SD D NN A SA 20. Initial consonants need more emphasis -
SD D NN A SA 21. This choir projects the mood of the selection very well +
SD D NN A SA 22. Excellent control of tempo +
SD D NN A SA 23. Excellent control of intonation at forte levels +
SD D NN A SA 24. Overall effect is choppy and over-sectionalized -
SD D NN A SA 25. Emotional concept of word meanings very well ex-
pressed +
SD D NN A SA 26. Excellent balance between all parts +
SD D NN A SA 27. Attacks are consistently weak -
SD D NN A SA 28. Performance exhibits the proper stylistic interpretation +
SD D NN A SA 29. Articulation was clear and precise +
SD D NN A SA 30. Men's voices (Sop. 2/Alto for Women's Ch.) balance
the choir very well +
SD D NN A SA 31. Delicate, expressive shading in dynamics +
SD D NN A SA 32. Diction is muddy -
SD D NN A SA 33. Too much emphasis on unimportant words and syl-
lables -
SD D NN A SA 34. Tempo unsteady in some sections -
SD D NN A SA 35. Intonation in all parts excellent throughout +
SD D NN A SA 36. Some poor entrances by different parts -
SD D NN A SA 37. Soprano (Ten. 1 for Men's Ch.) sound forced in upper
-
pitch and dynamic ranges
SD D NN A SA 38. Inner parts balance the outer voices very well +
SD D NN A SA 39. The diction of this group is excellent +
-C. c-
O
0 - )-
o C -
U; E
C C
C E X
ItsI 11 IV V V Vl
A. Diction
1. Articulation was clear
and precise .678
2. Words clearly under-
standable .776
3. The diction of this group
is excellent .780
4. Diction is muddy .789
5. Initial consonants need
more emphasis .589
B. Precision
1. Sloppy rhythms .334 .376 .306 .389
2. All part entrances are
very precise .338 .601
3. Attacks and releases of
many notes are imprecise .343 .498
4. Some poor entrances by
different parts .327 .585
5. Attacks are consistently
weak .371 .488 .304
C. Dynamics
1. Needs wider dynamic
contrasts .744
2. Lovely changes in dy-
namics .737
3. Delicate, expressive shad-
ing in dynamics .665 .321
4. Excellent use of "pp" .716
5. Dynamics handled well
in relation to phrase de-
velopment .578 .382
D. Tone Control
1. The tone quality is too
harsh in forte passages .657 .316
2. Soprano (Ten. I for
Men's Ch.) sound forced
in upper pitch and dy-
namic ranges .577 .351
Cooksey/111
3. The tone quality is often
forced in this choir .688
4. Excellent control of into-
nation at forte levels .520 .378 .343
5. Intonation in all parts
excellent throughout .466 .385 .345
E. Tempo
1. Tempo unsteady in some
sections .702
2. Excellent control of
tempo .679 .414
3. Tendency to rush the
tempo .593
4. Unsteady rhythmic sec-
tions .344 .577
F. Balance/Blend
1. Top voices cover up lower
voices .647
2. Men's voices (Sop. II/
Alto for Women's Ch.)
balance the choir very
well .743
3. Excellent balance between
all parts .321 .660
4. Inner parts balance the
outer voices very well .654
5. Lowest part balances up-
per parts very well .735
6. Good overall blend of all
parts .377 .525 .407
G. Interpretation/Musical Effect
1. Performance exhibits the
proper stylistic interpre-
tation .346 .685
2. This choir projects the
mood of the selection
very well .319 .741
3. Emotional concept of
word meanings very well
expressed .304 .348 .623
4. A musical and artistic
effect... fluid and vital .323 .353 .322 .315 .552
5. Excellent forward rhyth-
mic drive .629
6. Excellent unity of style .547
112/JRME
Table 5
Inter-JudgeReliabilityEstimates for the Revised CPRSSubscale
Scores
Group4
Group1 Group2 Group3 (students)
Subscale Categories (N = 20) (N = 20) (N = 20 (N = 20)
Diction .951 .972 .949 .879
Precision .971 .976 .959 .935
Dynamics .961 .967 .974 .952
Tone Control .984 .980 .975 .959
Tempo .953 .957 .935 .923
Balance/Blend .976 .964 .971 .933
Interpretation/Musical Effect .980 .971 .976 .950
scores for the revised CPRS evaluations in each set and both ranks and Z
scores for the global performance criterion were generally above .80 for
the first three groups of judges, and above .70 for the student group of
judges. Correlation coefficients between the subscale scores and the ranks
and Z scores ranged from .623 to .923 for the first three groups of judges,
and from .442 to .901 for the student judges.
To examine the contributions of the subscale scores of the revised
CPRS in predicting the global criterion, a step-wise multiple regression
analysis was performed on the data generated by the revised CPRS. The
subscale scores for each of the four groups of judges who used the CPRS
in evaluating the three sets of choral performances served as the inde-
pendent variables in predicting the global performance rating criterion.
The corrected R2's for the revised CPRS subscale scores of the four
groups of judges for performance sets I-III and the criterion of ranks
ranged from .630 to .993, and from .787 to .955 for the criterion of
normalized scores. Since the results of the multiple regression analysis
showed R2's which were highly inflated, multiple R coefficients were
computed from a pooled within-groups correlation matrix. The values
of the two criteria were .874 (for ranks) and .876 (for Z scores). The sub-
scale scores of the revised CPRS for the four groups of judges therefore
produced substantial evidence of success in predicting the global per-
formance rating criterion.
Finally, to test the stability of the weights for the subscale scores for
each of the four groups of judges generated by the step-wise multiple
regression across the three sets of performances, the weights for each
group of subscale scores were applied to the subscale scores of the other
three sets. The data indicated that the weights for sets I, III, and IV were
Cooksey/113
comparable, but the weights for set II produced poor results when
applied to the other three sets.
In the final study concerning the criterion-related validity of the re-
vised CPRS, a new group of twenty judges (group 5) used the NIMAC
adjudication scale in evaluating set III of the high school choral per-
formances, previously judged by groups 3 and 4 using the CPRS. The
same procedures used for comparing the revised CPRS scores and the
global performance ratings were applied. The results of this test were:
(1) The inter-judge reliability estimates for the NIMAC total and
subscale scores of group 5 were above .94.
(2) The zero-order correlation coefficients between the revised CPRS
total scores, subscale scores (generated by the ratings of the judges in
groups 3 and 4 on the set III performances), and the NIMAC criterion
were generally above .85.
(3) A step-wise multiple regression analysis for the revised CPRS sub-
scale scores (generated by the ratings of the judges in groups 3 and 4 on
the set II performances) and the NIMAC scale criterion produced sub-
stantial evidence (R2 = .970 for group 3 subscale scores: group 5 NIMAC
total scores; R2 = .900 for group 4 subscale scores: group 5 NIMAC
total scores) that the revised CPRS served as a good predictor for the
criterion.
Summaryand Conclusions
(1) The facet-factorial approach produced a seven-factor structure of
choral performance. These factors were diction, precision, dynamics, tone
control, tempo, balance/blend, and interpretation/musical effect.
(2) Thirty-six items-dimensions were selected to form the subscales to
measure the seven factors.
(3) The revised CPRS achieved high inter-judge reliability.
(4) The revised CPRS achieved high criterion-related validity when a
global performance rating was used as the criterion for comparison.
(5) The revised CPRS achieved high criterion-related validity when
the NIMAC scale was used as the criterion for comparison.
These results suggest that some of the difficulties involved in measur-
ing choral performance achievement can be overcome. Some of the basic
components of choral performance were successfully identified by factor
analysis. These components were defined by the items selected to measure
them. Since the selection of these items rested upon musical, statistical,
and empirical considerations, a significant degree of objectivity was
maintained in both the selection and definition of the essential criteria
needed to develop a structure of choral performance. Some of the items
selected for the revised CPRS included "expressive" aspects of choral
performance as well as the more easily measurable technical items.
114/JRME
CaliforniaStateUniversity
Fullerton,California