Académique Documents
Professionnel Documents
Culture Documents
Jason E. King
2004.
Abstract
Theoretical Framework
po pe
K , (1)
1 pe
k k
where po pii , pe pi . p.i , and p = the proportion of
i 1 i 1
ratings by two raters on a scale having k categories.
Fleiss’ extension of kappa, called the generalized
kappa, is defined as
n k
nm 2 x 2
ij
i 1 j 1
K 1 k , (2)
nm m 1 p j q j
j 1
P E 2m 3 P E 2 m 2 p 3j
2
2
SE ( K ) , (3)
Nm(m 1) 1 P E 2
m m
where P( E ) p 2j and p p 3j . Fleiss, Nee, and Landis
3
j
j 1 j 1
(1979) corrected the standard error formula to be
2
k
p j q j p j q j q j p j
k
2
SE K k . (4)
p q j j nm m 1 j 1 j 1
j 1
Matrix
Run MATRIX procedure:
Report
Estimated Kappa, Asymptotic Standard Error,
and Test of Null Hypothesis of 0 Population Value
%Inter_Rater(InputData=a,
DataType=c,
VarianceType=c,
CategoryFile=CatFile,
OutFile=a2);
Standard
Category Kappa Error Z Prob>Z
AC1 Standard
Category statistic Error Z Prob>Z
BY CATEGORY
gen kappa_cat1 = 0.070
gen kappa_cat2 = 0.117
gen kappa_cat3 = 0.466
gen kappa_cat4 = 0.427
gen kappa_cat5 = 0.558
OVERALL
gen kappa = 0.236
a
This approximate standard error formula based on Fleiss (Psychological Bulletin, 1971, Vol. 76, 378-382)
b
This approximate standard error formula based on Fleiss, Nee & Landis (Psychological Bulletin , 1979, Vol. 86, 974-977)
Conclusion
Footnote
1
Gwet (2002a) notes that Fleiss’ generalized kappa was based
not on Cohen’s kappa but on the earlier pi () measure of
inter-rater agreement introduced by Scott (1955).
Generalized Kappa 10
References
Table 1