Académique Documents
Professionnel Documents
Culture Documents
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
Assessment of
Competencies in
Educational Contexts
HOGREFE
Assessment of
Competencies in
Educational Contexts
Johannes Hartig
Eckhard Klieme
Detlev Leutner
(Editors)
HOGREFE
Johannes H a iti*. Eckh ard K lic m c. Dctlcv Lcutncr: A ssessm ent o f Com petencies in Educational Contexts. H o grcfc P u b lish in g G m b H . Gottingen 20t>8
L i b r a r y o f C o n g r e s s C a t a l o g i n g in P u b lic a tio n
is available v ia the L ibrary o f C o n g ress M arc D atab ase u n d er the
LC C on tro l N u m b er 2 0 0 6 9 3 3 0 0 6
L ib r a r y a n d A rc h iv e s C a n a d a C a ta lo g u in g in P u b lic a tio n
A ssessm en t o f co m p ete n cie s in ed ucational co n tex ts /J o h a n n e s
H artig , E ck h ard K licm e. D ctlcv L cutncr. editors.
Includes bibliographical references.
IS B N 9 7 8 -0 -8 8 9 3 7 -2 9 7 -9
1 C om petency b ased ed u catio n al tests. 2. E d u catio n al evaluation.
I. K liem e, E ck h ard , 1954- II L cutncr, Dot lev III H artig , Jo h an n es. 1970LC 1034.A 88 2007
371.26
C 2 0 0 6 -9 0 4 9 38-6
Contents
Preface
........................................................................................................................................ v
C o n te n ts .................................................................................................................................................ix
Con trib u to rs ...................................................................................................................................... xi
T h e C o n c e p t of C o m p e te n c e in Ed u catio n al Contexts
E ck h a rd K liem e, Johannes H artig, a n d D om in iqu e R a u c h ........................................................... 3
Psychometric Modeling
4
R epo rtin g Test O u tco m e s Using M o d els for C o g n itiv e D iag n o sis
M a tth ia s von D avier, Lou D i Bello, a n d K en ta ro Y a m a m o to ................................................
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n c riu u b te W c itc r g a b c o d e r V c rv ic lfa tig u n g .
151
C ontents
10
11
12
253
14
15
16
2 0 0 8 H ogrefe Publishing G m bH
K c in e u n e riu u b fe W c itc r g a b e o d e r V c r v ic lf a tig u n g .
Contributors
Eva L. Baker
N ational C e n te r for R esearch on E valuation, S ta n d a rd s , a n d S tu d e n t Testing
(C R E S S T ), U niversity o f C alifo rn ia, L o s A ngeles, C alifo rn ia, USA.
William L. Bewley
N ational C e n te r for R esearch on E v alu atio n , S ta n d a rd s , a n d S tu d e n t Testing
(C R E S S T ), U niversity o f C alifo rn ia, L o s A ngeles, C alifo rn ia, USA.
( 7aits // . ( 'arstensen
D e p a r tm e n t o f E ducational S cie n ce and R esearch M ethodology,
L e ib n iz -In s titu te for S cience E d u ca tio n (IP N ), Kiel, G erm any .
Gregory K. W. K. Chung
N ation al C e n te r for R esearch on E valuation , S ta n d a rd s , and S tu d e n t Testing
(C R E S S T ), U niversity o f C alifo rn ia, L o s A ngeles, C alifo rn ia, USA.
Paul De Boeck
D e p a r tm e n t o f Psychology, K ath o liek e Universiteit L eu ven, B elgium .
Lou DiBello
L e a r n i n g Sciences R e s e a r c h Institute, U niversity o f Illinois at Chicago,
Illinois, USA.
Theo J H. M Eggen
Cito, P s y c h o m e tric R e s e a rc h C en tre, A r n h e m , N e th e rla n d s /
D e p a r tm e n t o f R e s e a rc h M eth odo logy , M e a s u r e m e n t , and D ata A nalysis,
U niversity o f Tw ente, E n sch ede, N etherlands.
I Hrike Enders
In stitute o f Psychology, F ried rich S chiller University, Jena, G erm an y .
Judith Gluck
In stitute o f Psychology, A lp e n - A d r ia U niversity K la g e n fu rt, A ustria.
Johannes Hartig
C e n te r for E d ucatio nal Q u a lity and E valuation, G e r m a n In stitu te for
International E d ucatio nal R esearch (D IPF), F r a n k f u r t, G erm any.
M arcus Hasselhorn
C e n t e r for E d u ca tio n a n d D ev elo pm ent, G e r m a n Institute for International
E d ucatio nal R e s e a rc h (D IP F ), F r a n k f u r t, G erm any .
Ingmar Hosenfeld
F aculty o f Psychology, U n iv ersity o f K o b le n z -L a n d a u ,
C a m p u s L a n d a u , G erm any.
Nina Jude
C e n te r for E ducational Q u a lity and E valuation, G e r m a n Institu te for
International E ducational R e s e a rc h (D IP F ), F r a n k f u r t, G erm an y.
2 D08
H u g ic fc PuM iJiing G m b H
K e in e u n e r la u b te W c itc r g a b e o d c r V c r v ic lf a tig u n g .
xii
C on trib uto rs
A strid Jurecka
C e n t e r for E d ucational Q u a lity a n d Evaluation, G e r m a n In stitute for
International E d ucational R esearch (D IPF), F r a n k f u r t, G erm any.
Eckhard Klieme
C e n t e r for E d ucatio nal Q u a lity a n d E valuation, G e r m a n In stitute for
International E d ucatio nal R esearch (D IP F ), F ra n k f u rt, G erm any.
U lfKrdhne
In stitute o f Psychology, F riedrich S chiller University, Jena, G erm an y .
De/lev Leutner
D e p a r tm e n t o f Instru ctio n al Psychology, School o f E d u ca tio n ,
U n iv e rs ity o f D u is b u rg -E s s e n , Essen, G e rm a n y .
( h risto f NachtigaU
In stitute o f Psychology, F riedrich S ch iller U niversity, Jena, G erm an y .
HaroldF. O'Neil
Rossier School o f E d u ca tio n , U n iv ersity o f S o u th e rn C alifo rn ia / National
C e n t e r for R esearch on E valuation, S ta n d a rd s , and S tu d e n t Testing
(C R E S S T ), U n iv e rs ity o f C alifornia, L o s A ngeles, C alifo rn ia, USA.
M anfred Prenzel
D e p a r tm e n t o f E d u ca tio n al Science, L e ib n iz -In s titu te for S cience E d ucatio n
(IPN ), Kiel, G erm an y .
Dominique Rauch
C e n t e r for E d u ca tio n al Q u a lity a n d E valuation, G e r m a n In stitute for
In ternation al E ducational R esearch (D IP F ), F r a n k f u r t, G erm an y .
Tina Seidel
In stitute o f E d u ca tio n al Science, D e p a r t m e n t o f E ducational Psychology,
F riedrich S chiller University, Jen a, G erm any .
( 1hristiane Spiel
F aculty o f Psychology, U niv ersity o f V ienna, A ustria.
R o/fSteyer
In stitute o f Psychology, F ried rich S chiller University, Jena, G erm an y .
M ark Wilson
G r a d u a t e School o f E ducation, U niversity o f C alifornia,
Berkeley, C alifornia, USA.
Joachim Wirth
D e p a r tm e n t o f R e s e a rc h on L e a r n i n g a n d In stru ctio n , R u h r-U n iv e rs ita t
B o c h u m , G e rm a n y .
Kentaro Yamamoto
E d ucatio nal Testing S ervice, P rin ceto n , N e w Jersey, U SA .
2 0 0 8 H ogrefe Publishing G m bH
K c in e u n e riu u b fe W c itc r g a b c o d e r V c r v ic lf a tig u n g .
Part I
Theoretical Perspectives and
Developmental Models
H o g re fe P u b lish in g G m b H
K e in e u n e rla u b te W c itc rg a b c o d e r V c rv ic lfa tig u n g .
2008
Chapter i
The Concept of Competence in Educational
Contexts1
Eckhard Klieme, Johannes Hartig, and Dominique Rauch
2008
H u g ic fc PuM iJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
2008
H o g ic fc PuM iJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
Bildung
2008
H u g ic fc PuM iJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
2008
H u g ic fc PuM iJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
10
11
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
12
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
13
14
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
15
2008
H ugtcfc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
17
References
A d am s. R (Ed.). (2 005) PISA 2003 tech n ica l report. Paris: O E C D
A d am s. R ., & W u , M (E ds.) (2002) PISA 2 0 0 0 tech n ica l re p o rt P aris: O ECD .
A m elan g , M .,& F u n k e . J. (2005). E nt w ick lung u n d Im p lem en tieru n g e in e sk o m b in ie rte n B eratu n g sun d A u sw a h lv e rfa h re n s fu r d ie w ic h tig ste n S tu d ien g an g e an d e r U n iv ersitat H eidelberg.
P sych o lo g isch e Rundschau, 56, 135137.
B a n d u ra , A . (1990). C onclusion: R eflectio n s on n o n ability d e te rm in a n ts o f co m petence. In R.
S tern b erg & J. K o llig ian Jr. (E d s ), C om peten ce c o n sid e re d (pp. 315-362). N ew H aven, CT: Yale
U niversity Press.
B eck. B , & K lie m e, E (2007). E in lc itu n g . In B. B eck & E. K liem e (E d s ), DESl: Sprachliche
K om petenzen. K o n zep te u n d M e ssu n g (p p 1-8) W cinhcim : Beltz.
B inkley, M R., S tern b erg . R . Jo n es, S., & N o h ara, D (1999). A n o verarch in g fra m e w o rk f o r
u n derstan din g a n d a sse ssin g life skills. U n p u b lish ed In tern atio n al L ife S kills Survey (IL S S )
F ram ew o rk s W a sh in g to n .
B y b ee, R. W. (1997). T ow ard an u n d e rsta n d in g o f scientific literacy. In W. Gr&ber & C. B olte (E ds.),
Scientific L iteracy, an in tern a tio n a l Sym posium (pp. 3 7 -6 8 ). Kiel: IPN.
C h en , L. (2004). O n text stru c tu re . la n g u ag e proficiency, and re a d in g co m p re h en sio n test fo rm at
interactio n s: a reply to K o b ay ash i. 2002. L anguage Testing. 21. 2 2 8 -2 3 4 .
C heng. L .. W atan ab e, Y . & C u rtis. A (E d s ). (2004). W ashback in language testing, research
con texts a n d m ethods. M aliw ah L aw ren ce E rlbaum
C hom sky, N (1968) L angu age a n d M in d N ew York H a rc o u rt. B ra c e & W orld. Inc
C h u n g , Ci. K W. K ., O 'N e il, H F , B ew ley, W L ., & B aker, E. L. (2008). C o m p u ter-b ased a ssess
m en ts to su p p o rt d ista n c e learn in g . In J I la rtig , E K liem e, & 1) I.e u tn e r (E d s ), A ssessm en t o f
com p eten cies in ed u ca tio n a l con texts (pp. 25 3 -2 7 6 ). G o ttin g en : H ogrefe & Huber.
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g u b c o d e r V c r v ic lf a tig u n g .
18
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
21
22
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
24
M. H asselh orn
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
26
M. H asselh orn
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
30
M. H asselh orn
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
34
M. H asselh orn
2D0S H o g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
38
M. Hasselhorn
le a rn in g (for f u r t h e r details, see S od ian, 2002). T h e y are left aside, how ever, in the
fo llo w in g outline.
T h e m a in fe a tu re o f th e first turning point is an e n o r m o u s in c re ase in the effi
ciency o f working memory , esp ecially o f phonological w o r k in g m e m o r y - th e stor
a g e system respo nsib le for th e p ro c e s s in g o f verbal a n d aco ustic inform atio n. M o s t
c h ild ren a rriv e at th is critical m o m e n t at so m e p o in t in the sixth y e a r o f life, w h en
th e subvocal rehearsal p ro cess o r in n e r sp e e c h " b e c o m e s a u to m atically activated
w ith in th e phonological loop A lth o u g h in principle children are cap able o f e n g a g
ing in in n e r speech at an even y o u n g e r age, th e y d o not d o so s p o n tan eo u sly or
automatically. W ith d ire c t g u id a n c e , y o u n g e r c h ild ren can be in d u c ed to e n g a g e in
s im ila r i n n e r s p eech processes, but it is im p o r ta n t to r e m e m b e r th a t this re q u ire s a
significant a m o u n t o f in struction . W ith o u t i n n e r speech, th e fu n ctio n al cap acity o f
th e phonological w o r k in g m e m o r y is v e r y limited. T h e phonological store o f w ork
ing m e m o r y h as a cap acity o f only a b o u t one and a h a l f to tw o seconds, a f te r w hich
it is o v erw ritten .
At the age o f approxim ately eight y ears - usually so m ew h at earlier in girls than
in boys - the second turning point in developm ent can be discerned. Its m a in char
acteristic is an overall ch an g e in motivational preconditions. T h e over-optim ism o f
y o u n g children that en g end ers the feeling o f being highly com p etent is lost. Social
c o m p a ris o n s - above all w ith peers - increasingly d e te rm in e self-perception and selfevaluation. Increasingly realistic self-appraisals are the result. Yet a tra c e o f the you ng
child's over-optim ism rem ains, visible in the tendency - still perceptible in adulthood
- to distort inform atio n in the interests o f o n e s o w n self-image and overestim ate o n e
self in relation to the facts (e.g., Taylor & Brow n, 1988). T h is m otivational self-defense
b e c o m e s very fragile from th e ninth y e a r o f life. T h u s one finds that individuals with
low self-esteem often lack th e motivationally beneficial tendency to lift th eir self-es
teem b ecau se they doubt that a m o re positive self-concept w ould stand up to reality
(Baum eister, Tice, & Hutton, 1986).
At ap p ro x im a te ly 10 y e a rs o f age th e child a rriv e s at a third turning point. It is
c h a ra c te r iz e d by the first o c c u r r e n c e o f ab stract self-reflection" (Piaget, 1971).
A b stra c t self-reflection is ex tre m ely im p o r ta n t for th e central executive a s well as the
m e ta c o g n itiv e facets o f l e a r n i n g preconditions. P resu m ab ly, it is the s a m e co g n itiv e
p ro c e s s that trig g ers m otivational c h a n g e s at the a g e o f eight and th e n b rin g s a b o u t a
significant in c re ase in metacognitions b e tw e e n th e ag es o f eight and 10. At this point,
c h ild ren b e g in t h i n k i n g m o re a n d m o re a b o u t the m selv es, th e ir o w n k n o w led g e, and
th eir o w n le a r n in g processes. A s a c o n s e q u e n c e , they s u d d e n ly b e g in to s p o n ta n e
ously em ploy strategies. T h e p rod uction deficits o f m a n y strateg ies are lost at the end
o f e le m e n ta ry school, p rob ably a s a result o f a b s tra c t self-reflection, w h ich lead s to
a significant in c re ase in m e ta c o g n itiv e know ledge.
In the eleventh o r tw e lfth y e a r o f life th e in d iv id u a ls performance motive system
stabilizes, c h a ra c te r iz e d by an o rientation to w a r d s e ith e r s u ccess o r failure. H ere w e
can id e ntify the fourth turning point in the d ev e lo p m e n t o f individual p reco n d itio n s
for successful learning. It o c c u rs in a p h a s e o f d ev elo p m en t c h a r a c te riz e d by in-
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
42
M. H asselh orn
N icholls, J G. (1979). D ev elo p m en t o f p erc e p tio n o f o w n a tta in m e n t and cau sal a ttrib u tio n s for
su ccess and failu re in read in g . J o u rn a l o f E d u ca tio n a l P sychology, 71. 9 4 -9 9 .
N icholls. J G ..& M iller. A T. (1985). D ifferen tiatio n o f th e c o n c e p ts o f luck and skill. D evelo p m en ta l
P sych ology, 21, 7 6 -8 2 .
N isan . M . & K o riat, A (1977). G u ld e n 's actu al ch o ices and th e ir co n cep tio n o f the w ise ch o ice in
a delay -o f-g ratificatio n situ atio n . C h ild D evelopm en t, -IS, 4 8 8 -4 9 4 .
Piaget. .1. (1971). B io lo g y a n d kn ow ledge C h icag o C h icag o U niversity P ress
P ickerin g . S. J. (2001). T h e d ev elo p m en t o f v isu o -sp a tial w o rk in g m em ory. M em ory. 9, 4 2 3 -4 3 2 .
P ick erin g , S. J., (ia th e rc o le , S. E ., H all. M . & L loyd, S. A. (2001). D evelopm ent o f m em ory for
p a tte rn and path: F u rth e r ev id en ce for th e fractio n a tio n o f v isu o -sp a tial m em ory. Q u a rterly
J o u rn a l o f E xperim en tal Psychology'. 54A, 3 9 7 -4 2 0 .
P in trich . P. R ., & Z u sh o . A. (2002). T h e developm ent o f ac ad e m ic self-regulation: T h e role
o f co g n itiv e and m o tiv atio n al factors. In A W igfield & J S. E c c lc s (E ds.). D evelopm en t o f
ach ievem en t m o tiva tio n (pp 2 4 9 -2 8 4 ) San D iego. CA A cad em ic P ress.
Pressley. M ., B orkow ski, J. G , & S chneider. W (1989) G ood in fo rm atio n p ro c e ssin g W hat it is and
how ed u catio n can p ro m o te it J o u rn a l o f E d u ca tio n a l R esearch, 2, 8 5 7 -8 6 7
R h e in b erg , F., S ehm alt. 11-I).. & W asser. J. (1978). Rin L e h re ru n te rsc h ie d , d e r e tw a s ausm acht.
Z eitsch rift f iir E n tw icklu n gspsych ologie und P a d a g o g isch e P sych ologie. 10. 3-7.
S chneid er, W . <&BO ttner, G. (2002). E n tw ick lu n g d es G e d a c h tm sse s bei K in d e rn und Ju g en d lich en
In R O e rte r & L. M o n tad a (E d s ), E n tw icklu n gspsych ologie (filth ed itio n , pp. 495-516).
W ein h cim P sychologie V erlags U nion
S chneid er, W . G ru b e r. H.. G old. A .. & O p w is. K (1993). C h e ss e x p e rtise and m em o ry fo r ch ess
p o sitio n s 111 c h ild re n an d adults. J o u rn a l o f E xperim en tal C h ild P sych ology, 56, 3 2 8 -3 4 9
S ch u m an n -H en g steler, R. (1995) D ieE n tw ick lu n g d esvisu ell-rJ u m lich en G ed d ch tn isses. G ottingen
I Iogrefe.
S hod a. Y.. M ischel. W.. & P eake, P. K. (1990). P re d ic tin g adolescent co g n itiv e and self-regulatory
co m p ete n cie s fro m preschool delay o f g ratificatio n . D evelo p m en ta l P sych ology, 26. 9 7 8 -9 8 6 .
Siegel. L. (1994). W o rk in g m em o ry and read in g : A lifesp an p ersp ectiv e. In tern ation al J o u rn a l o f
B eh a vio u ra l D evelo p m en t. 17. 109-124
S ieglcr. R. S. (1996). E m ergin g minds: The p ro c e ss o f ch an ge in children's thinking New York. N Y
O xford U n iv ersity Press.
S kinner. E . A (1995) P e rc e iv e d con trol, m otivation , a n d co p in g N ew bury Park, CA Sage.
S k in n er, E. A.. Z im m er-G em b e ck , M. .!., & C o n n ell, J. P. (1998). In d iv id u al d iffe re n ce s and the
developm ent o f p erceiv ed control. M on ograph s o f the S o c ie ty fo r R esearch in C h ild D evelopm ent,
63 (2 -3 , S erial No. 254).
S o d ian . B. (2002). E n tw ic k lu n g b eg rifflich e n W issen s. In R. O e rte r & L. M ontada (E d s ),
E n tw icklu n gspsych ologie (fifth ed itio n , pp. 4 4 3 -4 6 8 ). W cinheim : P sychologic V erlags U nion
Taylor, S E . & B ro w n . J. D (1988) Illu sio n and w ell-b ein g A so cial-psychological p ersp e ctiv e on
m ental health P syc h o lo g ica l Bulletin, 103, 193-210
T ru d ew in d , C\. & K o h n e, W. (1982). B e z u g sn o rm o rie n tie ru n g d e s L e h re rs und M otiv en tw ick lu n g :
Z u sa m m e n h a n g e m it S ch u lleistu n g , In tellig en z u n d M erk m alen in d e r hauslichen U m w elt in der
G run d sch u le. In F. R h e in b erg (Ed.), J a h rh u ch fiir E m pirische E rzieh u n gsw issen sch aft 1982 (pp.
115-142). D u sseld o rf: S chw ann,
van A k en . M A. G . H elm k e, A ., & S chneider, W (1997). S elb stk o n zep t und L eistu n g - D y n am ik
ih res Z u sa m m c n sp ie ls E rg cb n isse aus d em S C H O L A S T IK -P rojekt. In F. E. W ein ert & A.
H e lm k e (E ds.), E n tw icklun g nn G ru n d sch u la lter (pp. 341-350) W einhcim B eltz.
W einer, B (1979) A th e o ry o f m o tiv atio n fo r so m e classro o m ex p erien ces. J o u rn a l o f E du cation al
P sychology, 71, 3-25.
W ein ert, F. E. (2001). C o n cep t o f co m p eten ce: A c o n c e p tu a l clarification. In I) L .R ychen & L. II
S alg am ik (E ds.), D efining a n d selectin g k e y co m p eten cies (pp. 4 5 -6 5 ) G ottin g en : 1Iogrefe.
2008
H u g ic fc PubliJiing G m b H
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
46
Theoretical Background
Deductive Reasoning
R e a s o n in g is a p ro cess o f th o u g h t that p ro d u c e s con clu sio n s on the basis o f p e r c e p
tions, tho u g h ts, or assu m p tio n s. D e d u c tio n yie ld s valid conclu sio n s that m u s t b e tr u e
giv en that th eir p re m is e s are tru e ( J o h n so n -L a ird , 1999). T h e m o s t p r o m in e n t theory
o f d e d u c tiv e r e a s o n in g is P ia g e ts c o g n itiv e -d e v e lo p m e n ta l th e o ry (Piaget, 1971).
A c c o r d in g to this theory, c h ild re n m ove th r o u g h four d ev elo p m en tal s tag es w h ich
d iffe r qualitatively in the c o g n itiv e p r o c e s s e s that they c a n handle: the sen sum otor,
th e preoperatio nal, the c o n c re te -o p e ra tio n a l a n d th e fo rm al-o p eratio n a l. T h e tr a n s i
tions b e tw e e n th e se stages o c c u r a ro u n d ages 2, 6, a n d 12. A c c o r d in g to Piaget, the
tran sition from the c o n c re te -o p e ra tio n a l to the f o r m a l- o p e r a tio n a l stag e is th e m ost
im p o r ta n t step in the d ev e lo p m e n t o f ded u ctiv e reaso ning. T h u s , th e D R V fo c u s e s on
th e transition from c o n c re te -o p e ra tio n a l to fo rm a l-o p e ra tio n a l think in g.
A c c o rd in g to Piaget, th e re are tw o m a jo r c h a ra c te ris tic s o f fo rm al-o p eratio n a l
th i n k i n g in c o m p a r is o n to c o n c re te -o p e ra tio n a l th in king: h y p o th e tic o - d e d u c tiv e
r e a s o n in g a n d propositional th o u g h t ( I n h e ld e r & Piaget, 1958; Piaget, 1972; see also
Berk, 1993). W h e n faced w ith a problem , fo rm a l-o p e ra tio n a l in d iv id u als th in k o f all
possible factors that could affect the outcom e. T h e y are also able to ta k e into a c c o u n t
factors that are not im m e d ia te ly s u g g e s te d by c o n c re te fe a tu re s o f th e situation. In
addition, fo rm a l-o p e ra tio n a l in d iv id u als can ev a lu a te th e logic o f sta te m e n ts by r e
flecting on th e s ta te m e n ts them selves; th e y do not need to relate them to real-world
c irc u m s ta n c e s .
H o w ev e r, a s em pirical results show, th e se theoretical classifications reflect w h a t
is possible ra th e r th a n w h a t is typical o f adolescents. W h e n p resen ted w ith c o n d i
tional re a s o n in g problem s, a d o le sc e n ts a n d even adults rarely s p o n ta n e o u s ly d is tin
g u ish d e d u c tiv e a r g u m e n ts fro m n o n -d e d u c tiv e a r g u m e n ts (B u llo c k & S odian, 2003;
2 0 0 8 H o g rtfe Publishing G m bH
K c in c u n e riu u b fe W c itc r g u b c o d e r V c r v ic lf u tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
50
to all studies, th e m a tic (i.e., f a m ilia r and concrete) c o n d itio n a ls are e a s ie r to solve
than a b s tra c t a n d c o u n te rfa c tu a l ones. B ased on Piaget ( In h e ld e r & Piaget, 1977),
c o u n te r fa c tu a l ta s k s should be best suited to prove w h e th e r p a rticip an ts are able to
d is ta n c e th e m s e lv e s fro m real-world know ledge. H ow ever, the em pirical e v id en ce
is m ix e d : w h ile so m e stud ies fo u n d d iffe re n c e s in difficulty b e tw e e n a b s tra c t and
c o u n te rfa c tu a l ta sk s (Ziegler, 1996), o th e rs fo u n d no such d iffe re n c e s (M a r k o v its &
V achon, 1989; R o b e r g e & Paulus, 1971).
M o d e of Presentation of th e A n te ce d e n t
T h e a n te c e d e n t can be p resented w ith ( I f th e sun do es not shine, M ic h a e la w e a r s a
red T-shirt ) and w ith o u t negation (see e x a m p le above). W h i l e n eg atio n s are, in p r in
ciple, co m p re h e n sib le for c o n c re te operational in d iv id u a ls (Gray, 1990; M o s h m a n ,
1977), d u r i n g th e tran sition to f o rm a l-o p e ra tio n a l stage n eg ations c a n h in d e r th e
tr a n s f o rm a tio n o f c o m p e te n c e into p e r f o r m a n c e ( M o s h m a n , 1977). Em pirical in
vestig atio n s sh o w system a tic in c re a se s in ta sk difficulty w h e n n eg ations w e re used
in a n teced en ts, w h ile n o effects o f n eg atio n s in the c o n s e q u e n t w ere o b s e rv e d
( M o s h m a n , 1979; R o b e rg e & M a s o n , 1978; W i l d m a n & Fletcher, 1977).
2D08 H u g ic fc PuMiJiing G m b H
K c in e u n e r lu u b te W c ite r g a b c o d e r V c r v ic lf u tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
54
C o n te n t o f C onditional
C oncrete
C/>
p*
or
o
cr
100 %
80 %
60 %
40 %
20%
A bstraet
C ounterfactual
u.
Ur-
.2
ja
o
CO
0%
Type o f
M T /M P N A /A C
Inference L
C o n crete
M T /M P N A /A C
M T /M P N A /A C
M T /M P NA /A C
J
Interm ediate
A dvanced
D e v e lo p m e n ta l Stages
J o h a n n e s llu rtig , K ck h ard K licm e. D c tle v I.e u tn c r: A sse ssm e n t o f C o m p e te n c ie s in E d u c a tio n a l C o n te x ts , H o g re fc P u b lish in g G m b H . G o ttin g e n 200R
H u g ic fc PuM iJiing G m b H
K e in e u n erlu u b te W c ite rg a b c o d e r V c rv ic lfa tig u n g .
2008
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
58
the three classes with respect to grade: 27.0% o f the advanced intermediate participants
were only in the 7th grade, while 18.8% o f the concrete-operational participants w ere in the
12th grade. Thus, the levels o f deductive reasoning performance are only partly age- and
grade-dependent.
Relation Betw een D R V P erform ance and M arks in M athem atics
A s d ed u ctiv e r e a s o n in g c o m p e te n c e is a s s u m e d to b e a n e c e s s a ry prerequ isite for
u n d e r s ta n d i n g and solving p ro b le m s in m a th e m a tic s , w e ex pected a significant re
lationship b e tw e e n m a r k s in m a th e m a tic s and D R V p e rfo rm a n c e . A 2 x 2 analysis o f
v a r ia n c e w ith the facto rs o f g r a d e (7th to 12lh) a n d co g n itiv e stage ( c o n c r e te - o p e ra
tional, in te rm e d ia te , a d v a n c e d in term ed iate) sh o w ed a significant effect o f c o g n i
tive stage, 7 (2,398) = 5.285, p = .005), n o g r a d e effect, and n o interaction. In the
A u stria n g r a d i n g sy ste m , w ith 1 b e in g the b e s t m a rk and 5 b e i n g the w o rst m a rk ,
c o n c re te -o p e ra tio n a l s tu d en ts had an averag e m a th e m a tic s m a rk o f 3.13 (SD = .97),
in te rm e d ia te students, 2.88 (SD = 1.02), and a d v a n c e d in te r m e d ia te stud ents, 2.83
(SD = 1.18). A g ain, the d iffe re n c e w a s largely b e tw e e n the c o n crete-o p eratio n al
a n d th e tw o in te rm e d ia te stages. C o n c re te -o p e ra tio n a l s tu d en ts h a d low er m a r k s in
m a th e m a tic s than the o th e r tw o groups.
F ro m a theoretical point o f v ie w relations b e t w e e n c o m p e te n c e in d e d u c tiv e r e a s o n
ing and m a th e m a tic s can only b e e x p e c te d o n ce s tu d en ts have reached a n a d v a n c e d
level in d e d u c tiv e reaso ning. To test this a s s u m p tio n w e calculated the S p e a r m a n
co rrelation coefficient b e tw e e n m a th e m a tic s m a r k s and d e d u c tiv e r e a s o n in g per
f o r m a n c e w ith in each latent class. R esults s u p p o rte d th e a s s u m p tio n . C orrelation
coefficients w e re - . 0 5 (/; = .521) in the c o n c re te -o p e ra tio n a l stage, - .1 0 (p = .245) in
th e in te rm e d ia te stage, and - . 2 9 in the a d v a n c e d in te rm e d ia te stag e (p = .001).
2008
H ugtcfc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
62
References
A kaike. II. (1973). In fo rm atio n theory and an ex ten sio n o f th e m a x im u m lik elih o o d principle. In B.
N. P etro v & F. C sak i (E d s ). 2 n d In tern ation al Sym posium on Inform ation Theory. B udapest:
A k a d e m ia i K iado.
B e c k m a n n , J 1\, & G u th k e, J (1999) P sy c h o d ia g n o stik d es S ch lu ssfo lg ern d cn D enkens: H an d b u ch
z u r A d ap tiv en C o m putergestU tztcn In tc llig e n z -L e rn tc stb a ttc rie fu r S ch lu ssfo lg crn d es D enken
(ACIL). G o ttin g en H ogrefe
B eller, S., & S p ad a, II (2003) T h e logic o f co n ten t effects in p ro p ositional reaso n in g : T h e case o f
cond itio n al re aso n in g w ith a p o in t o f v ie w Thinking a n d R eason ing, 9, 335-378.
B erk. L. E . (1993). Infants, children, a n d adolescen ts. N eed h am H eig h ts, M A : A lly n & Bacon.
B ullock. M . & S o d ian , B (2003). E n tw ic k lu n g d e s w isse n sch a ftlich e n D en k en s. In W. S ch n eid er &
M K n o p f (H rs g ). E ntw icklung, L eh ren und Lernen. Ztint G eden ken an F ran z E m anuel We in ert
(pp. 75-91). G o ttin g en : H ogrefe.
B y rn es. J P . & O v erto n . W F. (1986). R e aso n in g ab o u t c e rta in ty an d uncertain ty in co n crete, causal,
and p ro p o sitio n al contexts. D evelo p m en ta l P sychology, 22, 793-799.
C h azan , I) (1993). H igh school g eo m etry stu d e n ts' justification for th e ir v ie w s o f e m p iric a l ev id en ce
and m ath em atical p ro o f E d u ca tio n a l S tudies in M ath em atics, 24, 359-387
d e R ib a u p ie rre , A .R ie b e n . L .,& L autrey. J. (1991). D ev elo p m en tal ch a n g e and indiv id u al differences:
A lo n g itu d in al stu d y u sin g P iag etian tasks. P syc h o lo g y M on ograph s, III, 287-311
D eL isi, R .. <& S taudt, J. (1980). In d iv id u al d iffe re n ce s in co lleg e s tu d e n ts p e rfo rm a n c e on form al
o p eratio n al ta sk s Jo u rn a l o f A p p lie d D ev elo p m e n ta l P sychology. I. 201-208.
D u g an , C. M . & R cv lin . R (1990). R esp o n se o p tio n s an d p rese n tatio n form at as c o n trib u to rs to
co n d itio n al re aso n in g J o u rn a l o f E x p erim en ta l P sychology, 42, 8 2 9 -8 4 8
E v an s, J. S B T , N ew stead , S E ., & B y rn e . R M J (1993). H um an reasoning: The p sy c h o lo g y o f
dedu ction. Hove: L aw ren ce E rlbaum
Fischer, G. H (1995). T he L in e a r L o g istic Test M odel In G. H. F ischer & I. W. M o len aar (E d s ),
R asch M o d els F oundations, recen t developm en ts, a n d a p p lica tion s (pp. 131-155). N ew York:
Springer.
F ischer. G. H ., & M o len aar, I. W (Eds.). (1995). R asch m odels F oundations, recen t developm en ts,
a n d application s. N ew York S p rin g er
F isk. J. E.. & S h arp , C. (2002). S y llo g istic re a so n in g an d co g n itiv e ageing. Q u a rte rly J o u rn a l o f
E xperim en ta l P sych ology: H um an E xperim en tal Psychology', 55A, 1273-1293
Gnlhooly, K J . L ogie, R. 11. & W y n n , V (1999). S y llogistic re a so n in g task s, w o rk in g m em ory, and
skill. E u ropean J o u rn a l o f C o g n itive P sych ology, 11, 4 7 3 -4 9 8 .
G luck. J., & In d u rk h y a. A. (2001). A sse ssin g ch an g es in th e lo n g itu d in al salien ce o f item s in
co n stru c ts. J o u rn a l o f A d o lesc en t R esearch , 16, 169-187.
G luck. J.. M ach at. R . Jirasko. M.. & R o llett. B. (2002). T rain in g -related ch a n g e s in solution strateg y
in a sp atial test: A n ap p licatio n o f item re sp o n se m odels. L earn in g a n d In divid u a l D ifferences,
13. 1- 2 2 .
G ltick, J . & Spiel. C (1997) Item re sp o n se m o d e ls fo r rep ea ted m e a su re s d e sig n s A p p licatio n and
lim ita tio n s o f fo u r d ifferen t ap p ro ach es. M ethods o f P sych o lo g ica l R esearch Online, 2 (O nline
Journal: h ttp ://w w w . m pr-online.dej.
G luck, J., & Spiel, C (in press). U sing item resp o n se m o d els to a n a ly z e change: A dv an tag es and
lim itatio n s. In M. von D av ie r & C .H C a rsten se n (E d s ). M u ltiva ria te a n d m ixture distribu tion
R asch m o d els - extensions a n d application s. N ew York: Springer.
G ray. W. M (1990) F o rm al o p eratio n al th o u g h t In W. F. O v e rto n (E d ), R easoning, N ecessity, a n d
Logic: D evelo p m en ta l P ersp ective s (pp. 227-253). H illsd ale: E rlb au m
G rig g s, R A., & C ox, J R. (1982) T he elu siv e th em atic m a te ria ls effect in W ason's selection task.
B ritish J o u rn a l o f P sychology, 73, 4 0 7 -4 2 0
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
70
J. H artig
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
74
J. H artig
2008
H ugtcfc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
78
J. H artig
2008
H u g ic fc Publish in g G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
82
J. H artig
T h e D I N O Model
T h e D I N O model (deterministic input . noisy or, Tempi in & H e n so n , 200 6, see also
Tem plin, 2 0 0 6 ) is a latent class model that c o n ta in s o n e d ic h o to m o u s latent abil
ity for each ta sk d e m a n d in the m odel, i.e. 0 pv e {0, l}. A s in the m u ltid im e n s io n a l
R asch m odel, th e re is a c o r r e s p o n d in g latent v a ria b le for each ta sk d e m a n d (only the
ind ex k is used for b oth latent v a ria b le s a n d ta s k dem ands). T h u s , an indiv id ual p
either m a s te r s a ta sk d e m a n d k ( 0 ( = 1) or he or she d o e s n 't (0 |k 0). T h e total ability
profile 0^ o f an in d iv id u al is re p re s e n te d by a v e c to r o f d ic h o to m o u s elem ents, d efin
ing w h ich d e m a n d s an in dividu al d o e s m a s t e r a n d w h ich he o r she d o e s not. T h e
D I N O m odel c o n ta in s o n e latent class for each possible p a tte rn o f abilities 0 , i.e. 2 k
classes. A m odel w ith four task d e m a n d s , for instance, c o n ta in s 16 latent classes for
all c o m b in a tio n s o f abilities from 0 p = { 0 ,0 ,0 , 0 j to 0 p = [l, 1,1,1
T h e D I N O m od el c o n ta in s a d ic h o to m o u s latent solution p ro b a b ility co that
only ta k e s v a lu e s o f z e ro a n d o n e ( a ) p; e [0,1}; h e n c e the d en o tatio n d e te rm in is tic
input"). T h e D I N O model is a compensatory m odel; abilities are c o m b in e d in a
disjunctive fiin c tio n :
< v = i - f l ( '- e j *
k=l
(3 )
2008 H o g ic fc PubliJiing G m b H
K c in e u n e r lu u b te W c ite r g u b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
8 6
J. H artig
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
90
J. H artig
S tout. W. (2007). S k ills d ia g n o sis u sin g IR T -based co n tin u o u s latent trail m odels. Jou rn al o f
E du ca tio n a l M easurem ent, 44. 313-324
T em plin, J. (2 006) C D M C o g n itive d ia g n o sis m odelin g w ith M plus. U ser G uide. R etrieved
N o v em b e r 2006 fro m http://\Y\\ w .iq g rad s n et/jtc m p lin /d o \v n lo a d s/C D M _ u scr_ g u id e p d f
T em plin. J L., & H en so n . R A. (2006). M easu rem en t o f psy ch o lo gical d iso rd e rs u sin g co g n itiv e
d ia g n o sis m o d e ls P sych o lo g ica l M ethods, 11, 2 8 7 -3 0 5
T h u rsto n e, I.. L . (1947). M u ltiple f a c to r analysis. C hicago: U niversity o f C h icag o P ress.
Van den N o o rtg ate. W . l)e B o eck , P., & M eulders. M. (2003) C ro ss-classificatio n m ultilevel logistic
m o d e ls in p sy ch o m etrics. J o u rn a l o f E d u ca tio n a l a n d B eh a vio ral S ta tistics. 28, 3 69-386.
von D avier, M ., D iB ello. L . & Y am am oto, K (2008). R e p o rtin g test o u tco m es u sin g m o d els for
co g n itiv e d iag n o sis. In J. H artig . E. K liem e. & D L e u tn e r (Eds.X^sse-ss/wew/ o f com peten cies in
ed u ca tio n a l con texts (pp. 151-174). G o ttin g en : H ogrefe.
W ilso n , M (1998) Saltus: a m o d el o f d isc o n tin u ity in co g n itiv e developm ent P sych o lo g ica l
Bulletin, 105, 276-289.
W ilso n , M . & D e B oeck. P. (2004). D escrip tiv e and ex p lan ato ry item resp o n se m odels. In P.
I)e B oeck & M W ilson (E d s ). E xplan atory item resp o n se m odels: A g e n e ra lize d lin ea r a n d
n on lin ear a p p ro a ch (pp. 43-74). N ew York: Springer.
W ilson. M.. D e B oeck. P . & C a rsten se n , C. II. (2008). E x p lan ato ry item re sp o n se m odels: a b rief
intro d u ctio n . In J. H a rtig , E. K liem e, & D. L e u tn e r (E d s ). A ssessm en t o f co m p eten cies in
e d u ca tio n a l con texts (pp. 91-120). G o ttin g en : H o g refe & H u b er
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
94
-2
-i
3
19
(A b stra ct M o d e l l i n g in A r ith m e t ic )
D ifferen ce
Put th e digits 3, 6, 1, 9, 4, 7 in the b o x e s s o that the d ifference
b e tw e e n the t w o th re e -d ig it n u m b e r s is m a x im ised .
11
11
2. n u m b e r: 1
||
Figure 1. Two examples o f items from the German M athematical Literacy Test.
2008
H u g ic fc P u b lish in g G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
98
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
102
2 0 0 8 H ogrefe Publishing G m bH
K c in e u n e r lu u b te W c ite r g a b c o d e r V e r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
106
%pt f ( 1
Kp ) =
e X P ( 0 ;.)
1 eX P (P >
(4 )
(5)
i.e.. the ratio o f the success probability (7t ) to the failure probability (1 - n _).
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
114
2 0 0 8 H ogrefe Publishing G m bH
K c in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
122
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
126
S am ple
V
V
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
130
and m a th em a tics or science and m athem atics, so that cov arian ces betw een th ese subject
m atters can be estimated. T his obviously m e a n s increasing the n u m b e r o f different
booklets.
T h e logistics o f a d m i n i s t e r i n g b o o k le ts c o n s tr u c te d u s in g th e B IB design is roughly
as follows: T h e set o f b o o k le ts is spiralled th r o u g h o u t th e sam ple, so that c la ss ro o m s
te sted w ith in th e a s s e s sm e n t will receive a (p s e u d o -) r a n d o m selection o f the b o o k le ts
(a s im ila r ap p ro ach as in an EG design). In th is way, w h e n the set o f b o o k le ts are
giv en o ut to the e x a m in e e s d u r i n g test a d m in is tr a tio n , c lu ste rin g o f b o o k le ts will be
m i n i m i z e d and ap p ro x im a te ly th e s a m e n u m b e r o f b o o k le ts is given to ap p ro x im a te ly
eq u iv alen t sub-sam ples.
B oth IR T and traditional e q u a tin g m e th o d s can be alternativ ely used for m o s t o f the
e q u a tin g designs, in c lu d in g th e N E A T design. H owever, i f th e data is collected fol
lo w in g a B IB design, then the large m iss in g -b y -d e sig n d a ta fe a tu re can be ad d re s s e d
o nly b y the IR T m ethods. Similarly, in vertical l i n k in g settings, w h e r e th e differen c es
b e tw e e n the abilities o f th e g r o u p s o f te st ta k e rs are large, the IR T m e th o d o lo g ie s are
m o re a p p ro p ria te for li n k in g p u r p o s e s th a n the o b s e rv e d -s c o re m e th o d s , even tho ugh
th e re are c i r c u m s ta n c e s w h e r e o b s e rv e d - s c o r e e q u a t in g m e th o d s are used.
In this section w e d e s c rib e d th e data collection d e sig n s u sed in differen t t y p e s o f
l i n k in g processes. In the next section w e p ro vide a descrip tio n o f the statistical tools
used to c o n d u c t th e actu al lin k in g o n ce the data h as b een collected.
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
134
2 0 0 8 H ogrefe Publishing G m bH
K c in e u n e r lu u b te W c ite r g a b c o d e r V e r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
13X
(3)
GT(y).
Several im p o r ta n t classes o f o b s e r v e d -s c o r e e q u a tin g m e th o d s m ay be v ie w e d
a s only d iffe rin g in th e w a y th a t the c o n tin u iza tio n o f F T(x) a n d GT(y) is achieved.
T h e traditional e q u ip e rc e n tile e q u a t in g m e th o d (also called percen tile r a n k m e th o d )
u s e s lin e a r in terp o latio n o f th e d iscrete d istribu tion to m a k e it p ie c e w is e lin e a r and
therefore, co ntinuous. T h e kernel e q u a t in g (K E ; von D av ier et al., 2004; H olland &
Thayer, 1989) m e th o d u s e s a G a u s s ia n kernel s m o o th i n g to a p p r o x im a te the d iscrete
h is to g r a m by a c o n tin u o u s d ensity function.
T h e e q u ip e rc e n tile e q u a tin g f u n c tio n lead s to lin e a r e q u a tin g if one a s s u m e s that
F t(x) and Gr(y) are c o n tin u o u s and have th e s a m e s h a p e w h ile d iffe rin g in m e a n and
variance. T h e lin e a r e q u a tin g fu n ctio n , Lin, T(x), is defined by
Lilly j
( x ) =
[I
yT + G
f ( x - V xt T
YT
(4)
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
142
R esearch Q u e s t i o n s
Von D av ier and v o n D av ier (in press) identified fo u r re s e a rc h q u e s tio n s th a t m otivate
th e s tu d y o f c h a n g e (regardless o f the d o m a in , in psychology, ed u cation, o r other
fields):
1) W h a t is the c h a n g e that each p e rs o n e x p e r ie n c e s o v e r t i m e ?
(individual change)
2) D o th e rates at w h ich each individual c h a n g e s differ by v a lu e s /o u tc o m e s o f
b a c k g r o u n d v a ria b le s ? (inter-individual s y ste m a tic change)
3) D o e s a specific tr e a tm e n t h a v e a n effect on h o w an indiv id ual c h a n g e s ?
(causal inferences)
4) H o w do es a c o h o r t c h a n g e o ver ti m e ?
O bviously, an a n s w e r to the first question on th e in dividu al trajectories over tim e is
a prerequisite for a n s w e r s to th e s u b s e q u e n t tw o questions. In o th e r w o rd s, m o d e lin g
th e c h a n g e that an ind ividual person e x p e rie n c e s w ith ti m e is at th e c o re o f th e study
o f change. H ow ever, th e a n s w e r to th e fourth research question m a y o r m a y not rely
on the indiv id ual trajectories.
A ssum ptions
W h e n w e talk abou t p ro c e s s a s s u m p tio n s , w e refer to th o se a s s u m p ti o n s n ec e s s a ry
to a n s w e r the q u estio n s above. Von D av ier and von D a v ie r (in press) identified three
ty p e s o f p ro c e s s assum p tio n s: a) a s s u m p tio n s abou t the d ata, b) a s s u m p tio n s about
th e i n s tr u m e n t/ o u tc o m e variable, a n d c) a s s u m p tio n s ab out the model(s).
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
146
Discussion
T h is c h a p te r rev ie w s th e e x istin g m e th o d o lo g ie s fo r e q u a t in g a n d l i n k i n g tests that
m e a s u r e the s a m e c o n s tr u c t o ver time. T h e first p a r t o f the c h a p te r d e s c r ib e s the
horizontal e q u a tin g , w h e r e in ter-ch an g eab ility o f the scores is desired. T h e n , v e r t i
cal scaling is d is c u s s e d as it is u sed in ed u cational assessm en ts, w h e r e m e a s u r i n g
g ro w th in a p a rtic u la r d o m a in a n d c o m p a r a b ility o f sco res on test f o r m s th at m e a
sure th e s a m e c o n s tr u c t but d iffe r in difficulty is desired. W e also a d d r e s s th e c h a l
lenges o f c o v e r in g large content d o m a in s in educational s u rv e y a s s e s s m e n ts over
m a n y cycles and d is c u s s so m e o f th e solutions th a t w e r e d ev eloped u s in g exten sio n s
o f item re s p o n s e m o d e ls for r e p o rtin g s u b g ro u p distributions. In the p rev io u s sec
tion, w e d is c u s s e d s o m e e x istin g e x p la n a to ry m o d e ls for inter- a n d intra-in dividu al
g row th.
E a c h o f th e se a r e a s is a large field in itself, and it potentially has a stro n g i m
pact on the ed u cational policies, on the life o f stu d e n ts a n d p a re n ts or on the life o f
professionals.
N o w a d a y s , w h e n m o re and m o re s ta n d a r d iz e d te stin g is used nationally and inter
nationally, w e are also d is c o v e rin g m o re c h a lle n g e s in e n s u r i n g that th e p ro c e s s and
th e results are fair and accu rate. For e x am p les, a m o n g the c h a lle n g e s and research
o p p o r tu n itie s for test l i n k in g w e easily can m entio n the definition o f g ro w th , the
c o n s tr u c tio n o f the a n c h o r sets, the choice o f the re p o rtin g scale, a n d the c h a ra c te r
istics o f the sam p les used for e s ta b lis h in g th e scale, w h ich ideally should be re p re
sentative for the p o p u la tio n o f test takers. In addition, p s y c h o m e tr ic ia n s w o r r y about
th e m a i n te n a n c e o f th e scale: h o w to in tro d u c e n e w fo rm s , h o w to m o n ito r the scale
o ver tim e, a n d h o w to ad ju st to c h a n g e s in the a d m in is tr a t io n m ode.
In conclusion, w e notice that m a n y re s e a rc h e rs and p ractitio n ers alre ad y w ork
to g e th er in a d d r e s s in g th e se challenges, and w e h o p e that m a n y u n iv e rsities will
c o n s id e r im p le m e n tin g tr a i n in g c u rr ic u la that p re p a re th e fu tu re g e n e ra tio n o f p sy
c h o m e tric ia n s for the c h a lle n g e s in the field o f ed u c a tio n in th e 21st century.
References
ACT. Inc. (2000). A C T ed u ca tio n a l pla n n in g a n d a ssessm en t system . Iow a City, I A: Author.
A llen. N. L., Je n k in s, I; . & S ch o eps, T. L. (2004). The N AE P 1997 a rts tech n ical an alysis rep o rt
(E T S -N A E P 04-T01). P rin ceto n . NJ: ETS.
A n g o ff, W H (1971). Scales, n o rm s, and equivalent scores. In R L T h o rn d ik e (E d ), E du cation al
m easu rem en t (2nd e d . pp 5 0 8 -6 0 0 ). W ash in g to n , DC: A m eric an C o u n cil o n E ducation
B o ck, R D , & A itk in . M (1981). M arg in al m a x im u m likelihood e stim a tio n o f item param eters:
A pplication o f an E M algorithm . P sych om etrika, 46, 4 4 3 -4 4 5 .
C om preh en sive tests o f b a sic skills, fo rm s I a n d I '(P re lim in a ry Technical Report). (1982). M onterey,
CA: C T B /M eG raw -I Ii 11
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
154
M . v o n D a v i e r , L. Di B e l l o , & K. Y a m a m o t o
I'ahlc I. Fictitious O -matrix fo r six items (A to F), three skills (Add, Sub, Mult) and
two examinees y and z with different skill sets.
Q -m atrix:
T a s k by Skill
Skill:
Add
Sub
E x am in ee y
M ult
Task
A
Add
Sub
Mult
Add
Sub
M ult
no
no
y es
yes
Yes
yes
1
1
F
G
+
+
+
+
+
1
I
E xam inee z
+
4-
Note. A d d = A d d i t i o n : S u b = S u b t r a c t i o n : M u l t = M u l t i p l i c a t i o n .
T h e im plied ru le w h e n c o n v e rtin g a Q - m a t r i x and a skill p a tte rn to a set o f e x p ected
r e s p o n s e s is: the m o r e re q u ire d skills present, the h ig h e r the probability o f s u c c e s s .
T h is assists in d e t e r m i n i n g th e m ost probable r e s p o n s e s fo r each set o f skills. For ex
a m i n e e y in th e e x a m p le one m a y a r g u e that (A = 0, B = 0, C = 0, D = 0, E = 0, F = 1,
G = 0) is the m o s t plausible v e c to r o f r e s p o n s e s i f the p re s e n c e o f all re q u ire d skills
is n e c e s s a r y to solve a specific task. T h is v i e w w o u ld re p re s e n t a non-compensatory
a p p ro a c h u n d e r ly i n g the w a y in w h ich skills are exp ressed o r tra n sla te d in to success
rates. A s o m e w h a t m o r e fo rg iv in g v ie w cou ld a r g u e that e x a m in e e m a y either show
th e above pattern o f r e s p o n s e s or m ay p ro d u c e at least o n e o th e r re s p o n s e pattern,
n am ely (A = 0, B = 1, C = 0, D = 0, E = 1, F = 1, G = 1), since at least a fraction o f
th e re q u ire d skills are present. T h i s r e p re s e n ts a compensatory a s s u m p tio n o f how
skill p resen ce is ex p re s s e d in h ig h e r o r low er probabilities o f s u c c e e d in g in tasks.
For e x a m in e e z, how ever, all re q u ire d skills are present, so the typical re s p o n s e fro m
th is e x a m in e e should b e (A = 1, B = 1, C = 1, D = 1, E = 1, F = 1, G = 1).
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
166
M . v o n D a v i e r , L. D i B e l l o , & K. Y a m a m o t o
0 if a A=0,1,...,/, -1
1
if a k = /,
if a k = 0,
1 / / a , = 1,...,/,.-!,/,
)=(000...00),.(000...01),...,(011...11),(1 II...11)
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
170
M . v o n D a v i e r , L. Di B e l l o , & K. Y a m a m o t o
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
Hogrefe P u b lish in g G m b H
K e in e u n erlu u b te W c ite rg a b c o d e r V c rv ie lfa tig u n g .
2008
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
1 8 0
D. L e u t n e r , J. H a r t i g , & N. J u d e
O bjectivity
T h e objectivity o f a test m e a n s above all th at th e test results reflect o n ly c h a r a c te ris
tics o f th e individual test subject and not c h a ra c te ris tic s o f th e p erson a d m in is te r i n g
th e test o r c h a ra c te ris tic s o f the test situation. O fte n the c o n c e p t o f objectivity also
e n c o m p a s s e s th e idea that the test results th e m selv es, a n d th e in terp retatio n o f th ese
results, are in d e p e n d e n t o f the p erson e v a lu a tin g and in te r p r e tin g them . In line with
th e different criteria used for e v a lu a tin g a te s ts objectivity, it is c o m m o n to d is tin
g u ish b e tw e e n objectivity o f im p lem en tatio n , analysis, and interpretation.
T h e m o s t c o m m o n strategy for e n s u r in g the objectivity o f a d ia gno stic p r o c e d u re
is to s ta n d a rd iz e and d o c u m e n t each step re q u ire d in a d m in is te r i n g the test and in
a n a l y z in g a n d in te r p r e tin g its results. I f th e test p r o c e d u r e has b een s ta n d a rd iz e d ,
a n d i f the test h as b een a d m in is te r e d by a tra in e d tester, objectivity is usually c o n
sidered as given. A lth o u g h objectivity c a n be c o n sid ered a n e c e s s a r y p recond ition
for the reliability and validity o f a s s e s sm e n t p ro c e d u r e s (R ost, 2004), this quality
criterio n is fa r m o r e rarely the subject o f d iscu ssio n and analysis in ed u catio nal and
psychological a sse ssm e n t. For s ta n d a r d iz e d psychological tests, w h ich usu ally only
c o n ta in closed a n s w e r form ats, it m a y in d e ed b e tr u e th at objectivity is not a critical
issue. In ed u catio nal contexts, however, o ther d a ta s o u rces are o ften used a s the basis
to id e ntify inter-individual d iffe re n c e s in c o m p e te n c ie s , in c lu d in g beh avio ral obser
vations, j u d g m e n t s o f freely fo rm u la te d texts, o r portfolios. S ince c o m p e te n c ie s are
defined as c o n te x t-d e p e n d e n t realistic c o n s tru c ts, this b road a ra n g e o f m e th o d s
for em pirical an aly sis a p p e a r s appropriate. A s th e evaluation o f o p e n a n s w e r fo rm a ts
usu ally r e q u ire s the appraisal o f o b s e rv e d b e h a v io r by raters, however, th e q u estio n
o f the objectivity o f test results is significant in this case. I f an a s s e s sm e n t pro cess
d o e s involve j u d g m e n t s by a rater, it is all the m o re in d is p e n s a b le that th o ro u g h
d o c u m e n ta tio n and in s tr u c tio n s for th e a s s e s sm e n t be provided and that raters be
tra in e d on this basis. It is equally in d isp e n sa b le th a t th e co n siste n cy o f the raters be
a n a ly z e d , o r at least a ra n d o m sam p le o f th em . A lack o f co n siste ncy a m o n g raters
w o u ld indicate th e need to revise th e evaluation criteria, the ev alu atio n process, or
th e d o c u m e n ta tio n .
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
186
D. L e u t n e r , J. H a r t i g , & N. J u d e
Strategies for Test Construction: Generation and Selection of Test and Task
Content
T h e central and p r i m a r y question in the c o n s tr u c tio n o f any n e w a s s e s sm e n t i n s t r u
m e n t is h o w the test and th e co n ten t o f ite m s are to be defined, delim ited , and select
ed. T h e selection o f test co n tents u ltim ately d e t e r m in e s the n a tu re o f th e m e a s u re d
co n stru ct. In the following, four general strategies used in psychology to derive test
co n te n ts will b e presented: ex tern al test c o n s tru c tio n , ded u ctiv e test c o n stru ctio n ,
ind uctive test c o n stru ctio n , and criterion sam pling. In practice, m i x tu r e s o f th ese
four strategies are used as well.
External Test C o n stru ctio n
T h e p r im a r y goal o f extern al c o n s tr u c tio n is to predict a p a rtic u la r external criterion,
e.g., w h e th e r th e test-tak er will fall in to a p a rtic u la r catego ry o f individuals. T h e
origin o f item co n te n ts is o f s e c o n d a r y im p o r ta n c e here, and u su ally the ra n g e o f
ite m s chosen is as b ro ad and h e te ro g e n e o u s as possible (b o th in tests and q u e s tio n
naires). T h e item s are inv estig ated em p irically to see w h e th e r the g r o u p s o f interest
d iffe r in any respect. T h o s e item s that d iffe r m o s t w id ely by g r o u p are c o m b in e d
into a m e a s u r e m e n t i n s t r u m e n t . T h is in s t r u m e n t can th e n b e used w ith in d iv id u
als in f u tu r e stud ies to e s tim a te the g r o u p to w h ich they belong. T h is p r o c e d u r e is
used, for ex am p le, to assign psychiatric patients to specific d ia g n o stic g r o u p s (see,
for exam ple, H a th a w a y & M cK in ley, 1951) o r to d is tin g u is h b e tw e e n successful and
u n s u c c e s s fu l applicants for a jo b or u n iv e rs ity en ro llm en t.
In the case o f extern al test c o n stru ctio n , the p ra g m a tic q u estio n o f h o w to pred ict
a criterion b est is o f sole interest, and theoretical a s s u m p tio n s a b o u t th e c o n s tr u c ts
th at a r e m e a s u r e d are usu ally disregarded. For c o m p e te n c e assessm en ts, however,
this ap p ro ach is hardly appropriate.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
191
References
B o rsb o o m , D.. M ellen b erg h , G. J.. & van H ce rd c n . J (2004) T h e C oncept o f Validity. P sych o lo g ica l
R eview . I l l , 1061-1071.
B row n , J D., & H u d so n , T. (2002). C riterio n -re fe ren ce d lan gu age resting. C a m b rid g e C am b rid g e
U niversity P ress.
C ronbach . L. J., & M eehl. P. E. (1955). C o n stru c t v alid ity in psychological tests. P sych ological
B ullet in. 52. 281-302.
E m b retso n . S. (2006). T h e co n tin u ed search for non arb itrary m e tric s in psychology. A m erican
P sych ologist, 61, 50-55.
Flcxcr. R W . & B aer, R M . (2005). D esc rip tio n and ev alu atio n o f a u n iv ersity -b ased tran sitio n
en d o rse m en t p ro g ra m C a r e e r D evelo p m en t f o r E x cep tio n a l Individuals, 28, 80-91
G augler. B. B. R o sen th al, D B , T h o rn to n , G C .. & B e n tso n , C (1987). M e ta-an a lv sis o f assessm en t
c e n te r validity. J o u rn a l o f A p p lie d P sych ology, 72, 493-511.
G oldberg, L. R (1990). A n alte rn a tiv e d escrip tio n o f p e rso n a lity : T h e B ig-F ive fa c to r stru ctu re.
J o u rn a l o f P erso n a lity a n d S o c ia l P sychology, 59, 1216-1229.
H acker. D. J . B ol. L .. H o rg an , D. D.. & R akow , E. A. (2000). Test pred ictio n and p e rfo rm a n c e in a
classro o m context. J o u rn a l o f E d u ca tio n a l P sychology. 92, 160-170
H am b lcto n . R. K , & Z em sk y . A (2003). A d v an ces in crite rio n -re fe re n c e d te stin g m e th o d s and
practices. In C. R. R e y n o ld s & R W. K a m p h a u s (E d s ), H an dbook o f p s y c h o lo g ic a l a n d
e d u ca tio n a l a ssessm en t o f ch ildren (2nd cd ., p p 3 7 7 -4 0 4 ). N ew York G u ilfo rd P ress.
H artig , J (2006). S k alie ru n g u n d K o m p eten zn iv eau s [S caling and co m p ete n ce levels] In B B eck
& E K liem e (E d s ), S prach liche K om ptenzen. K on zepte u n d M e ssu n g (pp. 8 3 -9 9 ). W einheim :
B eltz.
1Iarlig. J. (2008). P sy ch o m etric M o d els for th e A sse ssm e n t o f C o m p eten cies. In J. I Iartig. E. K liem e,
& D. L e u tn e r (E ds.), A ssess m ent o f co m p eten cies in e d u ca tio n a l con texts (pp. 6 9 -9 0 ). G ottingen:
I logrefe & H u b er
H a rtig , J.. & K liem e, E. (2006). K o m p cten z und K o m p e te n z d ia g n o stik [C om petencies and
co m p ete n ce assessm ent | In K. S ch w eizer (E d ), L eistu n g u n d L eistu n g sd ia g n o stik (pp. 127-143)
B e rlin Springer.
H athaw ay. S. R & M e K in ley . J. C. (1951). The M in n eso ta m u ltiph asic p e r so n a lity in ven tory
(revised). M in n eap o lis: U niversity o f M in n eso ta.
K lauer. K. J. (1987). K riteriu m so rien tierte Tests [C riterio n -referen ced tests]. G ottin g en : H ogrefe.
K oep p en . K .. H a rtig , J.. K liem e, E ., & L eu tn er. D. (2008). C u rre n t issues in co m p ete n ce m o d elin g
and assessm en t. Z eitsch rift f u r P sych o lo g ie Jo u rn a l o f P sychology. 216, 61-73
K ru g er. J . & D u n n in g . D. (1999). U n sk illed and u n aw a re o f it ho w d ifficu lties in rec o g n iz in g
o n e 's o w n in co m p eten ce lead to in flated se lf-assessm en ts J o u rn a l o f P erso n a lity a n d S ocia l
P sych ology, 77, 1121-1134
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g u b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
195
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
2 0 1
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
206
A.Jurecka
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
2 1 2
A.Jurecka
In tern atio n al Test C o m m issio n (2005). In tern ation al G u id e/in d es on C o m p u ter-B a sed a n d InternetD e liv e r e d Testing. R etriev ed A p ril 28, 2008 fro m http://\vvvvv.intestcom .org/D o\vnloads/IT C % 2
0 G u id elin es% 2 0 o n % 2 0 C o m p u ter% 2 0 -% 2 0 v ersio n % 2 0 2 0 0 5 % 2 0 ap p ro v ed .p d f
Irv in e . S.. K u tsch k e. T., & W alker, R (2000) S creen in g c o n s c rip ts in G erm any using Item
G en erativ e -Tests. A rb eitsb eric h l Nr. I 2 0 0 0 d e s P sych o lo g isch en D ien stes d e r B undesw ehr.
B o n n B u n d e sm in iste riu m d e r V erteid ig u n g F SZ III 4
Jo d o in , M. G. (2003). M easu rem en t E fficiency o f In n o v ativ e Item F o rm a ts in C o m puter-B ased
Testing. J o u rn a l o f E d u ca tio n a l M easurem ent, -10. 1-15.
K irb a c h , C. .& M ontel. C. (2003). D as In te rn e tre c ru itm g to o l P E R L S. In: .1 E rp e n b e c k & L. von
R o sen stiel (E ds.), Ila n d b u ch K om peten zm essu n g (pp. 4 6 0 -4 7 0 ). S tu ttg art: S chaffer-P oeschel.
K o n ak , 0 .. D u in d a m . T.. & K am p h u is. F. (2005). C IT O -S prach test. W issen sch aftlicher Bericht.
R etriev ed J a n u a n 28. 2006 from h ttp //\vw w .cito.com /de/sprachtcst/rcsourccs/C T to-S prachtcstW issen sch aftlich er-B erich t PDF
K u b in g er, K D . & F ark as, M G. (1991) D ie B rau ch b ark eit d e r N o rm en d es P apier-B leistift-T ests fu r
die C om puter-V orgabe: E in E x p erim e n t am B eispiel d e r SPM von R av en als k ritisc h e r ITeitrag.
Z eitsch rift f i t r D ifferen tielle u n d D ia g n o stisch e P sycb o lo g ie, 12, 257266.
K ubinger. K. D (1993). T estth eo retisch e P ro b lem e d e r C o m p u terd ia g n o stik . Z eitsch rift f i t r A rb eitsu n d O rg a m sa tio n sp sych o lo g ie, 3 7. 130-137.
L u m sd en . J. A ., S am p so n . J. P., R eard o n . R. C\, & L en z, J. G. (2002). A co m p a riso n stu d y o f
the p a p er, p e r so n e l co m p u ter (PC), a n d in tern et version o f H o lla n d 's S elf-D irected Search:
Technical R ep o rt No. 30. T allah assee. C en ter for th e S tudy o f T echnology in C o u n selin g and
C a re er D ev elo p m en t. T h e C a re e r C enter. F lo rid a S tate U n iv ersity
M ead, A D , & D rasgow , F (1993). E q u iv alen ce o f co m p u terize d a n d p ap er-an d -p en cil co g n itiv e
ability tests: a m e ta-an aly sis. P sych o lo g ica l Bulletin. 114, 4 4 9 -4 5 8
M eijer. R. R., & N ering. M I.. (1999) C o m p u teriz ed ad ap tiv e te stin g O v erv iew and introduction.
A p p /ie d P sych o lo g ica l \le a su re m e n t, 23. 187-194.
M elter. A .. I la m an n . I.. K u tseh k e. T.. & S to rm , E. G. (2002) M o d e m isie ru n g d e r E ig n u n g sd ia g n o slik
im P sy ch o lo g isch en D ien st d e r B u n d e sw e h r - E rg eb n isse und P e rsp e k tiv e n Z eitsch rift f itr
P erso n a lp sych o lo g ie. J. 35-41
M orelan d . K. L (1992). C o m p u ter-assisted psy ch o lo g ical assessm en t. In M Z e id n e r& R M ost (Eds.),
P sych o lo g ica l testing: A n in side v ie w (p p 3 4 3 -3 7 6 ) Palo A lto. CA C o n su ltin g P sychologists
P ress.
N ag lieri, J A., D rasgbow . F , S ch m it, M . H an d ler, L., P rifitera, A., M argolis, A.. & V elasquez, A.
(2004). P sychological T estin g on th e In tern et. N ew P ro b lem s, Old Issues. A m erican P sych ologist.
59, 150-162.
N eu m a n n , G.. <fc B ay d o u n , R (1998). C o m p u teriz atio n o f p ap er-an d -p en cil tests: w h en a re they
eq u iv alen t? A p p lie d P sych o lo g ica l M easurem ent, 22. 7 1 -8 3
O cn n in g . S (2003). D as E l.IG O -S y stem . In J. E rp e n b e c k & L. von R o sen stiel (E ds.), Ilan dbu ch
K om petenzm essu n g. (pp 4 5 4 -4 5 9 ). S tu ttg a rt SchalTer-Poschel Verlag
O rd in a te (2004). Set 10 Test D escrip tio n . R etriev ed N o v em b er 11,2007 fro m h ttp /Avww versan ttest.
d e /p d f/V a lid atio n R ep o rt.p d f
P A S S -IT (2003). P hase I interim resea rch re p o rt sum m ary. R etriev ed A pril 2008 from h ttp ://
w w w .p ass-it.o rg .u k /reso u rces/0 3 1 1 1 7 -research p l-su m m ary .p d f
PA SS-IT (2005). D ev elo p in g the fu tu re p o te n tia l o f a ssessm en t through technology. R etriev ed A p ril
2008 fro m http://\vw \v.pass-it.o rg .u k /re so u rc e s/p a ss-it_ le a lle t_ p d f
Pater. E (2004). S p raeh sta n d serh cb u n g fu r d ie K in d e r d e s E in sc h u lu n g sja h rg a n s 2004 in D uisburg.
W eiteren tw ick lu n g und A n w en d u n g d e s zw e isp rach ig en C ito-T ests D eu tsch als Z w eitsprache,
2, 26-33.
P oggio, J , G lasn ap p , D. R ., Yang, X., & P oggio. A J. (2005) A co m p a ra tiv e ev alu atio n o f score
resu lts from c o m p u te riz e d and p ap e r & pencil m a th em atic s testin g in a large scale state
assessm en t p ro g ram The J o u rn a l o f Technology, Learning, a n d A ssessm en t, 3, 4 -3 0 .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
216
T. H. J. M. E ggen
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
2 2 0
T. H. J. M. E ggen
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
225
Start
In this phase, th e s ta r tin g item(s) in th e C A T is (are) indicated. A s th e re is, in general,
no in fo rm a tio n available on th e proficiency o f th e student, o n e could start with:
- one or m o re r a n d o m ly selected item s fro m th e item b a n k ;
- o ne o r m o re r a n d o m ly selected item s from c ertain s u b - d o m a i n s o f th e item bank.
O n e cou ld decid e to start w ith item s w h ich h a v e a c ertain c o m m o n in s tru c tio n for
th e students. A n e x a m p le w o u ld be to s ta rt w ith a test o f m a th e m a tic s w ith a few
item s not a llo w in g the test-tak er to u s e p a p e r a n d pencil to m a k e co m p u tatio n s;
- o ne o r m o re ra n d o m ly selected item s from a c e rta in subset o f the item b a n k , b ased
on psy ch o m etrical c h a ra c te ris tic s o f the items. A n e x a m p le w ould be to start w ith
a ran d o m selection o f a fe w easy items.
I f th e re is in fo rm a tio n available on th e proficiency o f th e s tu d e n t before testing, s e
le ctin g th e first item s o n the basis o f that in fo rm a tio n is o f c o u rs e possible.
Select
B efore th e a d m in is tr a tio n o f every item, a n item selection p r o c e d u r e is carried out.
F rom the item b a n k an item is chosen that is in a c c o r d a n c e w ith the a n s w e rs g iv e n by
th e stu d en t th u s far: the test is a d a p te d or tailored to th e proficiency o f the stu d e n t b e
ing tested. T h e item selection p r o c e d u r e is respo nsib le for the gain in efficiency that
c a n be a ch iev ed w ith a CAT. It is b a s e d on a n in fo rm a tio n c o n c e p t o f w h ich the basic
idea is that th e item, w h ich p ro m ise s to giv e the m ost in fo rm a tio n on the s tu d e n t s
proficiency, d e m o n s tr a te d thus far is a d m in is te r e d next. In a s e p a ra te section o f this
chapter, the criteria for item selection will b e d e s c rib e d in m o re detail.
Compute
In the c o m p u ta tio n phase, the scores o f the stu d en t are processed . B ased on the
scores, statistical p r o c e d u r e s (discussed in m o r e detail in the next section) d e t e r m i n e
th e s tu d e n t s proficiency a n d an ind ication o f its accuracy.
Stop
A f t e r the a d m in is tra tio n o f each item a decision is m a d e w h e th e r a n e w item is to be
selected o r w h e th e r testin g can be te rm in a te d . C rite ria for s to p p in g are:
- the
a c c u r a c y o f th e
e s tim a te o f th e s tu d e n t s proficiency;
- the
a c c u r a c y o f the
decision for cla ss ify in g the student;
- the
m a x i m u m (and
possibly also the m i n i m u m ) practically available te stin g tim e
o r n u m b e r o f item s th at can be a d m in is te re d
T h e last criterion in co m b in a tio n w ith o ne o f the first tw o criteria is often chosen.
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
230
T. H. J. M. E ggen
Content Control
W h e n o nly m a x i m u m in fo rm a tio n selection ta k e s place, it cou ld provide results th at
a r e in conflict w ith the desired content specification o f th e test. A test c o n s tru c to r
could d e m a n d th a t s u b - d o m a i n s o f the m e a s u r e d proficiency are re p resen te d in a
c e rta in p ro p o rtio n in each CAT. O n e reason for this d e m a n d could be the content
a n d fa c e valid ity o f the test, a n o th e r th e r e q u ire m e n t to re p o rt s e p a ra te e s tim a te s on
th e s u b - d o m a i n s for d ia g n o s tic p u rp o ses. For ex am p le, a test on e le m e n ta r y a r it h
m e tic should h av e an equal n u m b e r o f a d d itio n and s u b tr a c tio n items. T h e r e are
several possibilities for re a liz in g such a specification, tw o o f w hich are:
T h e item b a n k is p artitio n e d into s u b -b a n k s . For each s u b - d o m a i n to be d is tin
g u is h e d th e re is a s u b - b a n k a n d a stu d en t t a k i n g a C A T for each s u b -b a n k . I f one
uses a variab le le n g th for each sub-C A T , there is no c o m p le te control on the n u m
ber, a n d th u s on th e p r o p o rtio n s b e tw e e n the n u m b e r o f item s, in the s u b -d o m a in s.
N e v erth eless, this a p p ro a c h is often applied. O n e m a in reason to d o this is that for
c e rta in s u b - d o m a i n s s o m e t im e s a specific s tim u lu s o r item t y p e is used. A lte rn a tin g
item s o f differen t s u b - d o m a i n s in one C A T then m ay ca u s e problem s. A C A T on
la n g u a g e could, for ex am p le, consist o f re a d in g and listening items. It is practical not
to m i x th e se item ty p e s d u r i n g th e a d m in is tr a tio n o f th e C A T
I f it is re q u ire d to have item s on s u b - d o m a i n s in fixed p r o p o rtio n s in every CAT, it
is possible to ad ap t the C A T a lg o rith m to ac h ie v e that. T h e idea is that the alg o rith m
ta k e s care o f th e b est ap p ro x im a tio n o f th e desired specification d u r in g th e a d m i n
istration o f the CAT. A n elegant and sim ple m e th o d to ac h ie v e this w a s p r o p o s e d by
K i n g s b u r y a n d Z a r a (1991), w hich o p e ra te s a s follows: a fte r each item, th e p e rc e n t
a g e s o f ite m s in the s u b - d o m a i n s o f all item s a d m in is te r e d th u s fa r are su b tracted
fro m th e d esired p e rc e n ta g e s o f ite m s in th e s u b - d o m a in s . From th e s u b - d o m a i n th at
h as the low est p ercen tage, an item that h as m a x i m u m in fo rm a tio n is a d m in is te r e d
next. So, in th e alg o rith m first th e s u b - d o m a i n is d e t e r m in e d a n d w ith in th is d o m a in
th e m o s t in fo rm a tiv e item.
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
236
J. W irth
Competencies
C o m p u t e r te c h n o lo g y a n d u se affects educational research in tw o ways. First, c o m
puters p ro vide the o p p o r tu n i ty to develop n e w ty p e s o f le a r n in g e n v iro n m e n ts and
n e w a s s e s sm e n t m e th o d s that can b e u s e d in research on traditional c o n c e p ts or on
n e w a s p e c ts o f th o se concepts. S econd , c o m p u te r u se and le a r n in g w ith n e w m e d ia
have estab lish ed n e w and in d e p e n d e n t fields o f ed u catio nal research le ad in g to the
definition o f n e w c o n s tru c ts a n d the d ev elo p m en t o f n e w tests a s s e s sin g th e se n e w
c o n s tru c ts. T h u s , c o m p u te r te c h n o lo g y pro v id es th e o p p o r tu n i ty to o p e r a te c o n v e n
tional c o n s tru c ts in n e w w ays or to develop n e w c o n s tru c ts, raising the pro sp ect o f
b e in g able to r e -c h a ra c te r iz e conventional c o n s tru c ts o r to define n e w c o n s tr u c ts
a n d n e w a re a s o f research (H a d w i n et al., 2005).
O n e o f the m o s t p r o m in e n t e x a m p le s o f r e - c h a r a c te r iz in g a conven tio nal c o n s tru c t
is d e m o n s tr a te d by th e w o rk on co m p lex pro b lem solving u n d e r t a k e n by D o r n e r and
his g r o u p in G e r m a n y (D o rn e r, K re u z ig , Reither, & Staudel, 1983; D o r n e r & Preufiler,
1990; D o rn e r, S chaub, & S troh schneider, 1999). D o r n e r created a c o m p u te r- s im u la t
ed to w n called L o h h a u s e n S ubjects w e re a p p o in te d m a y o r o f L o h h a u s e n and in
stru cted to g o v e r n th e tow n . T h e sim u latio n included ap p ro x im a te ly 2 0 0 0 variables
each o f w h ich w a s s o m e h o w c o n n e c te d to the others. Variables c h a n g e d th eir values
either as an effect o f a m a y o r's in terv en tio n a n d /o r a s a fu n c tio n o f time. D o r n e r and
his c o lle a g u e s w e re the first to u se the c o m p u te r to sim u late such a h ig h ly com plex
a n d d y n a m i c sy stem . T h e i r (and related) w ork had a s tro n g im p a c t on research on
pro b lem solving. C o m p le x pro b lem solving as the c o m p e te n c e re q u ire d to learn how
to control a co m p le x a n d d y n a m ic sy stem b e c a m e a n e w c o n s tr u c t in cognitive p sy
c h o lo g y (F re n s c h & F u n k e , 1995), and th e distin ctio n b e tw e e n k n o w le d g e a c q u is i
tion a n d k n o w le d g e application b e c a m e p r o m in e n t in definitio ns o f problem so lv in g
(F u n k e , 1985). B e c a u s e low o r even neg ativ e co rre la tio n s w e re found b e tw e e n c o m
plex p ro b le m -s o lv in g p e r f o r m a n c e a n d intelligence (e.g., P u tz -O s te rlo h , 1981), even
research on definitions and m e a s u r e s o f in telligence w e r e h igh ly influ en ced by this
a n d related w ork (e.g., K ro ner, 2001; Leutner, 2002; Sufi, 1996, 1999). N o w adays,
c o m p u te r-b a s e d tests are in d isp e n sa b le tools for th e a s s e s s m e n t o f p ro b lem -so lv
ing c o m p e te n c ie s even in la rge-scale a s s e s s m e n ts ( B a k e r & O Neil, 2002; K lie m e,
Leutner, & W irth , 2005; W i r t h & K liem e, 2003). T h e y are also used in th e a s s e s s
m e n t o f tacit k n o w le d g e a b o u t p ro c e d u r e s and strategies th a t c a n n o t easily be ver
balized and th e re fo re is v e ry difficult to assess u s in g convention al p a p er-b ased tests
( B e r r y & B ro ad b en t, 1995; B u c h n e r, F un ke, & Berry, 1995; K r a u s s et al., 2004).
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
240
J. W irth
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
246
J. W irth
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
250
J. W irth
L eu tn er, D , & P lass, J. L. (1998). M e a su rin g le arn in g sty les w ith q u estio n aire s v ersu s d irect
o b se rv a tio n o f p referen tial ch o ice b eh av io r in au th en tic le a rn in g situations: T he v isu a liz e r/
v e rb a liz e r b e h a v io r o b se rv a tio n scale (V V -B O S). C om pu ters in H um an B ehavior. 14. 543-557,
L o m p sch er, J. (1994). L e rn stra tc g ie n Z u g an g e a u f d e r R eflex io n s- un d H an d lu n g se b c n e (L earn in g
strateg ie s A ccesses on th e reflective and b eh a v io ral level], L L F -B erichle (Vol 9, pp 114-129).
P otsdam U n iv ersitat P o tsd am
L ord, 1' M . (1980). A p p lica tio n s o f item respon se th eo ry to p r a c tic a l problem s. H illsd ale, NJ:
L aw ren ce E rlb au m
L u ech t, R. M ., & C lauser. B. E. (2002). Test m o d els fo r co m p lex CBT. In C.N . M ills, M.T. P oten za,
J. J. F rem er. & W C. W ard (E ds.). C o m p u te r-b a sed testing. B uilding the foun dation fo r future
assessm en ts (pp. 6 7 -8 8 ). M ah w ah . NJ: L aw re n c e E rlbaum .
M ayer. R. E. (2001). M u ltim ed ia lea rn in g C am b rid g e: C a m b rid g e U niversity P ress.
M cA rdle, J. J.. & B e ll, R Q (2000) A n in tro d u ctio n to latent g ro w th m o d e ls for developm ental
d ata analysis. In T D L ittle. K U S ch n ab el. & J. B a u m e rt (E d s ). M o d elin g lon gitu din al a n d
m u ltilevel data. P ra c tic a l issues, a p p lie d a p p ro a ch es a n d specific exam ples (pp. 6 9 -1 0 7 ).
M ahw ah, NJ: L aw ren ce E rlbaum .
M ik elsk is, 11. F. (1997). D e r C o m p u ter ein m u ltim e d iale s W erk zeu g zu m L e rn e n vo n P hysik (The
c o m p u t e r - a m u ltim e d ia tool for le arn in g physics] P hysik in d e rS c h u le , 35, 3 94-398.
M islevy, R. J. (1996). Test th eo ry reco n ceiv ed J o u rn a l o f E d u ca tio n a l M easurem ent. 33. 379-416.
N erdel. C. (2003). D ie W irkung von A n im ation u n d Sim ulation a u f das C erstandm s von
stojfw ech selp h ysio lo g isch en P ro ze ssen [E ffects o f an im a tio n s an d sim u latio n s on u n d e rsta n d in g
m etab o lic processes]. U n p u b lish ed d isse rta tio n U n iv ersity K iel
N ieg em an n , H M , L eu tn er, D , & B rtln k e n , R (Eds.). (2004). In stru ction al d esig n f o r m u ltim edia
learn in g M unster: W axm ann
N o rm an n , M ., D eb u s, G., D o rre, P.. & L eutner, D (2004). T rain in g o f tram d riv e rs in w orkload
m an ag em en t - w o rk lo ad assessm en t in real life and in a d riv in g /tra ffic sim u la to r In T.
R o th en g atter & R.D . I lu g u e n in (E ds.). Traffic a n d tra n sp o rt p sy c h o lo g y th eo ry a n d a p p lication
(P ro c e e d in g s o f th e IC T T P 2 0 0 0 . pp. 113-121). A m ste rd a m E lsevier.
O lso n -B u eh an an . J B . D rasgow . F . M obcrg, P J , M ead , A. D .. K e e n a n , P. A , & D onovan, M.
(1998) C o n flict reso lu tio n sk ills assessm en t A m o d e l-b a sed , m u lti-m ed ia approach. P erso n ell
P sychology, 51, 1-24
Page, E B . & P etersen . N S (1995). T he c o m p u te r m oves in to essay g ra d in g P hi D elta K appan,
76, 561-565.
P in trich . P. R., S m ith , D. A. F., G arcia , T., & M cK each ie, W. J (1991). The m o tiv a te d stra te g ie s fo r
learn in g qu estio n a ire (M SLQ). A n n A rb o r. M I: N C R 1PT A L . T h e U niversity o f M ichigan.
P in trich , P. R.. W olters. C. A .. & B axter. G .P (2000). A sse ssin g m e ta c o g n itio n and self-reg u lated
learn in g . In G. S chraw & J. C Im p ara ( E d s ), Issues in the m easurem ent o f m etacogn ition (pp.
4 3 -9 7 ). L in co ln , N E B u ro s In stitu te o f M ental M easu rem en t
P lass. J L . C h u n . D M .. M ayer, R E .. & L eu tn er. D (1998) S u p p o rtin g v isu a l a n d verbal le a rn in g
p referen ces in a seco n d -lan g u ag e m u ltim e d ia le a rn in g en v iro n m en t J o u rn a l o f E du cation al
P sychology, 90. 2 5 -3 6
P lich a rt, P., Jad o u l. R ., V andenabeele, L., & I.ato u r. T. (2004, N ovem ber). TAG, a colla b o ra tive
d istrib u te d c o m p u te r-b a sed a ssessm en t fra m ew o rk built on sem an tic w eb stan dards. P aper
p rese n ted at th e In te rn a tio n a l C o n feren ce on A d v an ces in In tellig en t S ystem s - T heory and
A p p licatio n s A IS T A . L uxem bourg.
P rc n z e l, M ., von D avier. M . B lesch k c, M. G , S en k b cil, M .. & U rh a h n c, D (2000). D id a k tisc h
o p tim ic rte r E in sa tz N eu er M ed ien E ntvvicklung von c o m p u te rg c stu tz te n U n terrich tsk o n zcp ten
fu r die n a tu rw isse n sc h a ftlic h e n F ach er (D id actically o p tim iz e d use o f new m e d ia developm ent
o f co m p u ter-b ased te ach in g c o n c e p tio n s in scien ce teach in g ] In D. L e u tn e r & R B ru n k en
(E d s ), A'eue M edien in C n terricht, A u s- u n d W eiterbildung. A ktuelle E rgebn isse em pirisch er
p a d a g o g isc h e r F orschung (pp. 113-121). M u n ster: W axm ann.
2008
H o g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
255
2D08 H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
259
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
265
Q u e ry
V e cto r
qw!
t2
qw2
t3
qw3
t-m
qwm
Term-by-Document Matrix
qw,
qw2
qw3
ti
wn
w12
W,3
<2
W2 ,
w 22
w 23
t3
W31
W j2
W3 3
tm
Wm,
Wm 2
"m 3
w 1n
w 2n
wmn
mn
J i i
Com pute similarity between query vector
and each docum ent vector
tfn
doc*
Similarity Scores
doc3
doc,
Q.
O
Query
Vector
- Si
S2
s3
Sn
qwn
B e c a u s e the matrix o p e ra tio n s return a ve c to r of s c o re s , th e s e s c o r e s c a n be u s e d in a
variety of w a y s. F o r exam p le, e a c h u n s c o re d e s s a y cou ld be c o n s id e re d a q u e ry vector,
and the s c o r e of the e s s a y could be the m e a n of the 10 m o st sim ilar d o c u m e n ts from the
te rm -b y -d o c u m e n t matrix.
Figure 4. Cvm p u tin g similarity scores between the term-by-document m atrix and
query vector.
2008
H u g ic fc PubliJiing G m b H
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
271
B aker, E. L.. O 'N e il. 11 F., Jr., & L in n , R. L. (1993). Policy an d v alid ity p ro sp e c ts fo r p erfo rm a n ce based assessm en t. A m e n can P sych o lo g ist, 48. 1210-1218.
B arry. M .. & R u n y a n . G. B (1995). A review o f d ista n c e -le a rn in g stu d ie s in the U.S. m ilitary.
A m erican J o u rn a l o f D istan ce E ducation, 9. 3 7 -4 7
B en n ett. R E .. & B cjar, I 1 (1998). V alidity an d au to m ated sc o rin g It's not only the scoring.
E du ca tio n a l M easurem ent: Issues a n d P ractice. 17. 9 -17
B erry. M W., & Young. P. G. (1995). U sing latent se m an tic in d ex in g for m u lilan g u a g e info rm atio n
retrieval. C o m p u ters a n d the H um anities, 29, 413429.
B loom . B. S. (1984 ). T he 2 sig m a problem : T h e search for m e th o d s o f g ro u p in stru c tio n as effectiv e
as o n e-to -o n e tu to rin g . E d u ca tio n a l R esearcher. 13. 4 -1 6 .
B o n k . C J.. & W ish er, R A. (2000. S eptem ber). A p p lyin g co lla b o ra tive a n d e-lea rn in g tools to
m ilita ry ed u ca tio n d ista n ce learning: A resea rch fr a m e w o rk (In te rim T echnical R eport).
A le x a n d ria , VA U.S. A rm y R e sea rc h In stitu te fo r th e B eh av io ral and S ocial S ciences.
B reuer. K .. M o lk e n th in , R . & T ennyson, R D. (in press) R ole o f sim u latio n in w e b -b a se d le a rn in g
In 11 F O 'N eil & R Perez ( E d s ), W eb -b a sed learning; Theory, research, a n d p ra c tic e . M ahw ah.
NJ Law rence E rlbaum A ssociates.
B u rste in , J. (2003). T h e e -ra te r* sc o rin g en g in e: A u to m ated essay sco rin g w ith n atu ral lang u ag e
p ro cessin g In M. D. S h e rm is & J. B u rste in (E d s ),A u to m a te d e s sa y scorin g: A cro ss-d iscip lin a ry
p e r sp e c tiv e (pp. 113-121). H illsd ale, NJ: E rlbaum .
B u rste in . J., K ukich. K., W olff. S.. & L u , C. (1998. A pril). C o m p u ter an alysis o f e s s a y co n ten t f o r
a u to m a ted sco re p re d ic tio n P ap er p rese n ted at th e an n u al m e e tin g o f the N atio n al C ouncil on
M easu rem en t in E d u ca tio n S an D iego. CA
B u rste in , J., & M areu . I) (2003). A u to m ated ev alu atio n o f d isc o u rse s tru c tu re in stu d en t essays. In
M D S h erm is & J B u rste in (E ds.), A u to m a ted e s sa y scorin g: A c ro ss-d isc ip lin a ry p e rsp e c tiv e
(pp. 2 0 9 -2 2 9 ). M ahw ah. NJ L aw ren ce E rlbaum A ssociates.
C h an g , K .-E .. S ung, Y.-T.. & C h io u , S. K (2002). U se o f h ie ra rc h ica l h y p e r c o n c ep t m ap in w ebbased courses. J o u rn a l o f E d u ca tio n a l C om pu tin g R esearch , 27, 335-353.
C h an g . W .-C., H su. H -H .. S m ith, T K , & W ang, C -C (2004). E n h a n c in g S C O R M m e ta d a ta for
assessm en t a u th o rin g in c-L ea rn in g . J o u rn a l o f C o m p u ter A s s is te d L earning, 20. 305-316.
C huan g . S .- I I . & O 'N e il. H F. (in press). Role o f task -sp ecific a d a p ted feed b ack on a com puterbased co llab o rativ e p ro b lem -so lv in g task In II F O 'N eil & R Perez (E d s ), W eb-based learning;
Theory, research, a n d p ra c tic e . M ahw ah. NJ L aw ren ce E rlbaum A sso ciates
C hung , G K. W K., B aker, E. L.. B rill, I) G., S in h a , R S aad a t, F , & B ew ley, W L. (2003).
A u to m ated assessm en t o f d o m ain k n o w led g e w ith o n lin e k n o w led g e m apping. P ro ceed in g s o f
the I/ITSEC , 25, 1168-1179.
C hung . G. K. W. K ., de V ries. L. F.. C h e a k , A. C\. & Bew ley. W. (2005). P ro cess m easu res o f
pro b lem solving. M an u scrip t in p rep aratio n .
C hung . G. K W K . H arm o n . T. C ..& B aker. E. L (2001). T h e im p act o f a sim u latio n -b a sed le arn in g
d esig n project on stu d en t le a rn in g IEEE Transactions on E du cation. 44. 3 9 0 -3 9 8
C lark, R E. (1983) R e co n sid erin g research on le a rn in g fro m m edia. R eview o f E du cation al
R esearch, 53. 4 4 5 -4 5 9
C lark. R E. (1989). C u rre n t p ro g re ss and fu tu re d ire c tio n s for research in in stru c tio n a l technology.
E du ca tio n a l Technology R esearch & D evelopm en t. 37, 5 7 -6 6 .
C lark. R. E. (2005). M o tiv atio n strategies. In II F. O 'N e il (E d ). W hat w orks in distan ce learning:
G u idelin es (pp. 89-109). G reen w ich , CT: In fo rm atio n A g e P u b lish in g Inc.
C lauser, B E (2000). R e cu rre n t issu es and recen t ad v a n ce s in sco rin g p e rfo rm a n c e assessm ents.
A p p lie d P sych o lo g ica l M easurem ent. 24. 310-324
D eerw ester. S , D u m a is, S. T , F u rn a s, G W , L an d au er, T K , & H a rsh m a n , R (1990). Indexing by
latent se m an tic an aly sis J o u rn a l o f the A m erican S o c ie ty f o r Inform ation Science. 41, 391-407.
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g u b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
276
S inghal, A.. S alton. G .. M ilra. M .. & B uckley, C. (1996). D o cu m en t length n o rm alizatio n . Inform ation
P ro cessin g a n d M anagem ent, 32. 6 1 9 -6 3 3 .
S m ith . P. L . & D illon. C L. (1999) C o m p arin g d ista n c e le a rn in g and classro o m le a rn in g C o n cep tu al
co n sid eratio n s. A m erican J o u rn a l o f D istan ce E ducation, 13. 6 -3 6 .
T ennyson. R D (2005). L e a rn in g th e o rie s an d in stru c tio n a l d esig n A h isto ric a l p e rsp e c tiv e o f th e
lin k in g m odel In J. M S pector, C. O h ra z d a , A Van S ch aaek . & D A. W iley (E ds.), Innovations
in in stru ctio n a l technology. E ssays in h o n o r o f M. D a v id M errill (pp. 219 -2 3 5 ) M ah w ah , NJ:
L aw ren ce E rlb au m A ssociates.
U .S. D e p a rtm e n t o f D efense. (1996). D ep a rtm en t o f D efen se handbook: D evelo p m en t o f in teractive
m u ltim edia in stru ctio n (IMI) (p a rt 3 o f 4 parts). L ak eh u rsl. NJ: N aval A ir S y stem s C om m and.
W est. C. D.. Pom eroy. J. R ., P ark . J. K ., G ersten b erg er. E. A .. & S an d oval. J. (2000). C ritical th in k in g
in g ra d u a te m ed ical e d u c atio n A ro le fo r co n c ep t-m ap p in g assessm en t J o u rn a l o f the A m erican
M e d ic a l A ssociation , 284. 1105-1110.
W irt. J , R ooney. P . H ussar. B . C hoy, S . P ro v a sn ik , S . & H am p d en -T h o m p so n , C (2005) The
C on d itio n o f E du cation 2005 (N C E S 2 0 0 5 -0 9 4 ) W ashington, DC: U S D ep a rtm e n t o f E d u catio n ,
N ational C e n ter fo r E d u catio n Statistics.
W isher. R. A., C h am p ag n e, M. V., P aw lu k , J. L., E ato n . A ., T h o rn to n . D. M .. & C urnow . C. K
(1999). Training through d ista n ce learning: A n a ssessm en t o f research fin din gs (Tech. Rep. No.
1095). A lex a n d ria. VA: U .S. A rm y R esearch In stitu te for th e B eh av io ral and Social Sciences.
W ong. S. K M . Z iark o . W . R ag h av an , V V.. & W ong. P. C N (1987). O n m o d e lin g o f in fo rm atio n
retriev al c o n c e p ts in v e c to r sp a ces A C M Transactions on D a ta b a se System s. 12. 299-321
W u. C , B erry . M . S h iv ak u m ar, S., & M c L a rty , J. (1995) N eu ral n etw o rk s fo r fu ll-sc a le p ro tein
se q u en ce classificatio n S eq u en ce en c o d in g w ith s in g u la r v alu e d ecom position. M achine
Learning, 21, 177-193
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
2X1
Cross-sectional
One-shot
Trend
> 1
> 1
Simple panel
> 1
Complex panel
> 1
> 1
L o n g itu d in al
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e rla u b te W c itc rg a b c o d e r V c rv ic lfa tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
2X6
TIM SS
P IR L S
NAEP
NAALS
S a m p le
Random
Random
Random
Random
Random
Scope
International
International
International
National
National
Focus
a ssessm en t
Reading
Mathematics
Science
Mathematics
Science
Reading
Reading
English
Mathematics literacv
w
Science
Writing
Other subjects
periodically
B ack g rou n d
Student
background
Student
background
Student
background
Student
background
Adult
background
Target group
Students
Students
( 15-ycar-olds) (grades
4/8/12)
Students
(grades 3/4)
Students
(grades
4/8/12)
Adults
Design
Trend
Trend
Trend
Trend
Trend
(3-year cycle) (4-year cycle) (5-year cycle) ( I -year cycle) (10-year
cycle)
Initiator
OECD
IEA
IEA
NCES
NCES
K c in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
293
C o n te x ts
Authentic situations
where science and
technology are
important
C o m p e te n c ie s
Identifying scientific
questions
Explaining phenome
na scientifically by
applying scientific
knowledge
Using scientific evi
dence to make and
communicate deci
sions
K n o w le d g e
Knowledge of
science
(basic concepts)
Knowledge about
science
A ttitu d in a l re s p o n s e s
Attitudes towards
scientific and techno
logical issues
Figure 3. Framework for FISA 2006 Science Assessment (Australian Council for
Fducationa / Research . 2006, p. 13).
T h e test d ev e lo p m e n t follow s a strict technical p r o c e d u r e (O rg a n isatio n for E c o n o m ic
C o - o p e r a tio n and D e v elo p m en t, 2 0 0 2 a, 2002b, 2005a, 2005b, 2005c) w h ich in clud es
th e d ev e lo p m e n t o f a m anu al for item c o n s tru c tio n (item ty p e s, item form at, verbal
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r la u b te W c ite r g u b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
299
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
304
S havelson, R J. (19% ). S ta tistic a l reason in g f o r the b eh a vio ra l scien ces (3rd e d ) . N eed h am 1Ieights,
M A A lly n & B acon.
S havelso n . R J . & T ow ne, L. (Eds.). (2002). Scientific resea rch in edu cation W ash in g to n , DC:
N atio n al A cadem y P ress.
Stigler, J W . G o n za les, P . K a w a n a k a , T . K n o ll. S.. & S erran o , A (1999) The T1MSS V ideotape
C lassro o m Study. M eth ods a n d fin d in g s fr o m a n ex p lo ra to ry resea rch p ro je c t on eigh th -grade
m ath em atics in stru ction in G erm any, Japan , a n d the U nited S la tes W ash in g to n , D C. U.S.
D e p a rtm e n t o f E ducation,
van d e r L in d en , W. J., & G las. C A. (E d s ). (2000). C o m p u te rize d a d a p tiv e testing. T h eory a n d
p ra c tice . D ordrecht: Kluvver.
W ein ert. F. E. (2001). V erg leich en d e L e is tu n g sm e ssu n g in S chulen - ein e u m stritte n e
S elb stv erstan d lich k eit In F E W einert (Ed.), L eistun gsm essu ngen in Schulen (pp. 17-31).
W ein h cim B eltz V erlag
W iersm a, W (2000) R esearch m ethods in edu cation: A n in tro d u ction (7th ed.). N eed h am H eights,
M A P earson.
W irth , J. (2008). C o m p u ter B ased Tests: A lte rn a tiv e s for Test and Item D esign. In J. H artig, E.
K liem e. & D. L e u tn e r ( E d s ), A ssessm en t o f com p eten cies in ed u ca tio n a l con texts (pp. 235-252).
G ottingen: I Io g refe & I Iuber.
W u. M .. & A d am s, R (2004). P lau sible values: W hy th ey a re im portan t U npub lish ed paper.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
310
s ta n d a r d s in th e G e r m a n - s p e a k i n g co untries. W h i le different m e th o d s o f re p re s e n t
ing c o m p e te n c e levels in p s y c h o m e tric m o d e ls d o exist (R o st 2004), detailed m o d
els have only b een d ev elo ped for v e ry few subjects - for ex am p le, m a th em a tics.
C on seq u en tly , th e P IS A studies fo rm th e m a in point o f reference for c o m p e te n c e
m o d e ls in m a th e m a tic s , natural sciences a n d first la n g u a g e s , w h ile the C o m m o n
E u ro p e a n R e fe re n c e F ra m e w o rk (Council o f E u ro p e, 2001) plays th e s a m e role in
m o d e lin g levels o f c o m p e te n c e fo r foreign langu ages. So far, however, to o little ex
p e rie n c e has b e e n g a in e d to sp ecify general m o d e ls th a t will be con siste n t o ver all
su b g ro u p s o f the stu d en t population. F u r th e r m o re , w e can a s s u m e th a t th e system atic
descrip tio n o f c o m p e te n c e levels d iffers d e p e n d i n g on th e d o m a in . In g eneral, the
successiv e levels represen t c o m b in a tio n s o f th e facets listed a b o v e (kn o w led g e, skill,
u n d e rs ta n d in g , action, m otivation, etc.; ...). (.. .) It m ay be th e c a s e th a t th e levels o f
a c o m p e te n c y m odel can also be in te rp re te d as co n s e c u tiv e steps in th e acquisition
process. ( K lie m e et al., 200 4, pp. 68). A c o m p e te n c e model o f this k in d is capable
not only o f e x p re s s in g p a r tic u la r levels w ith in a specific age o r birth c o h o rt o f s tu
den ts but also o f d e s c rib in g h o w the differen t c o m p o n e n ts o f c o m p e te n c e evolve
in in teraction w ith one a n o th e r in th e le a r n in g b io g ra p h ie s o f c h ild ren and y o u n g
people, and h o w c o m p e te n c ie s a re f u n d a m e n ta ll y acq u ired . N evertheless, w e may
a s s u m e that d efining levels o f c o m p e te n c e fro m the p e rs p e c tiv e o f d ev elo pm ental
p sy cho lo gy w ould in c re ase th e c o m p le x ity o f th e m o d e ls dram atically. O verall, vast
room r e m a in s for f u r t h e r investigation on this asp ect o f im p le m e n tin g educational
standards.
O w i n g to their p ersistin g lack o f a n em pirical fo u ndatio n for c o m p e te n c e models,
G e r m a n ed u cation al s ta n d a rd s d o not s tr u c t u r e c o m p e te n c ie s into levels o f c o m p e
tence. Instead, they b re a k the s ta n d a r d s d o w n into core r e q u ire m e n t areas. T h e s e
areas differ f u n d a m e n ta lly fro m levels o f c o m p e te n c e (K u ltu s m in is te r k o n f e r e n z ,
200 4, p. 17), how ever, a n d in stead o f b e in g g r o u n d e d in em p irically v alid ate d test
ing p r o c e d u r e s they are b a s e d on t e a c h e r s professional e x p e rie n c e and task fo rm a ts
from traditional testin g m aterials. T h e core r e q u ir e m e n t a r e a s are therefore pro v i
sional. At th e In stitute for E d u ca tio n al P ro g re s s (Institut z u r Q u a lita ts e n tw ic k lu n g
im B ild u n g s w e s e n , IQB), w h ich w a s f o u n d e d specifically to develop te stin g m a te r i
als on a s o u n d scientific basis, e x p e r ts w o rk to d raft test item s th at c o n c re tiz e these
s ta n d a r d s a n d re n d e r th e m m e asu rab le. In 2 0 0 6 , the first s ta n d a r d iz e d ta s k s for the
subject o f m a th e m a tic s w e re d ev eloped for all o f th e G e r m a n I .cinder (federal states)
parallel to the 2 0 0 6 PISA study.
S w itz e rla n d has ta k e n a differen t app ro ach . W h i le in G e r m a n y educational
s ta n d a r d s w e re im p le m e n te d w ith o u t the ex isten ce o f any c o m p e te n c e m o d
els ( K u ltu r m in is te r k o n f e r e n z , 2004, p. 6), the S w iss C o n f e r e n c e o f th e C antonal
M in is t e r s o f E d u ca tio n ( E D K ) decid ed to first develop and e m p iric a lly test th e s ta n
d a r d s alon g w ith th e c o r r e s p o n d in g c o m p e te n c e m o d e ls in a m u lti-p h a s e process
(S ch w eizerisc h e K o n fe r e n z d e r k an to n alen E rz ie h u n g s d ir e k to r e n , 2004). In th is p r o
cess, differen t scientific c o n s o r tia have b een estab lish ed th at c o m p ris e specialists
fro m a r a n g e o f fields, te ach ers o f differen t subjects, and social scientists, w h o w ork
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
314
H am ilto n . L. S., Stecher. B. M . M arsh . J. A.. M cC o m b s, J. S., R o b y n, A .. R u ssell, J. L., N aftel, S., &
B arn ey , H. (2007). S ta n d a rd s-B a se d A cco u n ta b ility U nder N o C h ild L eft Behind. E xperien ces o f
Teachers a n d A d m in istra to rs in Three S ta tes S anta M o n ica. CA R A N D E ducation.
H am cycr. U.. Frey, K . & H aft. H (E ds) (1983). H andbuch d erC u rricu lu m forsch u n g: U bersich ten
z u r F orschung 1970 19SJ W ein h cim Beltz.
H e rm a n ,.! L (2004). T h e E ffec ts o f T esting o n In stru ctio n . In S H. F u h r m a n & R F. E lm ore (E d s ),
R ed esig n in g A cco u n ta b ility S ystem s fo r E ducation (pp. 141-166). N ew Y ork/L ondon T eachers
C ollege P ress.
H orn, R. A J. (2004). S tandards. N ew York: P eter Lang.
K lie m e, E., A v en ariu s, H.. B lu m , W.. D o b rich , P.. G ru b er. H ., P re n z e l, M . et al. (2004). The
D evelo p m en t o f N a tio n a l E d u ca tio n a l S tandards. An E xpertise. B erlin : B u n d e sm in iste riu m
fu r B ild u n g und F orschung. A v ailab le U R L : h ttp ://w w w .b m b f.d e/p u b /th e_ d ev elo p m en t_ o f_
nat ion a 1 educat io n cl_ st a nd ard s.p d f
K u ltu sm in iste rk o n fe re n z (2004). B i/du n gsstan dards d e r K ultusm inisterkonferenz. E r/auterungen
zu r K o n zep tio n u n d Entwicklung. M unchen: L u ch terh an d .
L u cv sh y n , .!. (undated). H an dreich u n g B ildun gss tan dards. Em Be it ra g zu r Q u a/itdtssich eru n g an
O sterreich s Schulen. S alzburg: o h n e Verlag.
M cL eod, D. B ., S take, R. E ., S ch ap p elle, B., M ellissin o s, M ., & G ierl. M. J. (1996). S ettin g the
S tan d ard s: N C T M 's role in th e refo rm o f m a th em atic s ed u catio n . In S. A R aizen & E. I). B ritto n
(E d s ). B o ld ventures: U.S. innovations in scien ce a n d m ath em atics edu cation. (Vol 3 C ases in
m ath em atics ed u catio n , p p 13-132). D o rd rech t: Kluvvcr
N ichols, S. L . & B erlin er. D. C. (2007). H o w high-stakes testin g co rru p ts A m erican ' schools.
C a m b rid g e H arv ard E d u ca tio n P ress.
O 'D ay. .1 A. (2004) C om plexity, A cco u n tab ility , an d School Im provem ent. In S H F u h rm a n & R.
F. E lm o re (E d s ), R edesign in g A cco u n ta b ility S ystem s f o r E ducation (pp. 15-43). N ew Y ork/
L ondon: T eachers C o lleg e P ress.
R avitch . D. (1995). N a tio n a l S ta n d a rd s in A m erica n E ducation. A C itizen's G uide. W ashington,
D C B ro o k in g s In stitu tio n P ress.
R ost, J. (2004) P sy ch o m c trisch c M odelle z u r U b erp rU fu n g von B ild u n g ssta n d ard s a n h a n d von
K o m p eten zm o d ellen . Z eitsch rift f u r P a d a g o g ik , 50 (5), 6 6 2 -6 7 8
R ychen. D S.. & S a lg a n ik . I. H . (E d s ). (2001). D efining a n d S electin g K ey C om peten cies. S eattle
H ogrefe & H u b e r P u b lish ers
R ychen. D. S., & S alg a n ik , L 11 (Eds.). (2003). K e y C o m p eten cies f o r a S uccessfu l Life a n d WellE unktioning S ociety. Seattle: I Iogrefe & I Iuber P ublishers.
S w iss C o n feren ce o f th e C an to n al M in is te rs o f E d u catio n (2004). H ARM O S. Z ielsetzun gen und
K on zeption. B ern: ED K
S e n k , S. L ., & T h o m p so n . D R (E d s ). (2003) S ta n d a rd s-B a sed S ch ool M ath em a tics C u rricu la.
W hat A re They? W hat D o S tu den ts Learn?. M a h w a h /L o n d o n L a w re n c e E rlbaum .
S techer. B M (2002). C o n seq u en c es o f larg e-scale, h ig h -sta k e s te stin g on sch o o l and classro o u m
p ractice In L S H am ilto n , B. M Stecher. & S. P. K lein (E d s ), T est-based a cco u n ta b ility in
edu ca tio n (p p 7 9 -1 0 0 ). S a n ta M onica: R A N D E ducation.
S w an so n , C. B , & S tev en so n , D. L. (2002). S tan d a rd s-b ased -re fo rm in practice: E v id en ce on state
policy an d classro o m in stru c tio n fro m the NAF.P sta te assessm en t. E du cation al E valu ation a n d
P o lic y A n alysis, 24, 1-27.
W einert. F E. (2001a) C o n cep t o f C o m p eten ce A C o n cep tu al C larification. In D. S R y ch en & L H.
S alg an ik (E d s ), D efining a n d S electin g K ey C o m p eten cies (pp. 4 5 -6 5 ). S eattle. B ern: H ogrefe
& H u b er P ublishers.
W ein ert, F. E. (2001b). S ch u lle istu n g en - L eistu n g en d e r S ch u le o d e r d e r S chuler? In F. E. W einert
(Ed.), L eistungsniessiingen in Schulen (p p 7 3 -8 6 ). W ein h eim B eltz.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
320
Student Characteristics
T h e r e are a lot o f well k n o w n c h arac teristics th a t d e t e r m i n e stu d en ts' a c h ie v e m e n t
a n d c o m p eten cies. T h e s tu d e n t's sex, age o r la n g u a g e are well k n o w n as im p o r
ta n t but will not b e d is c u s s e d b e c a u s e the im p o r ta n c e o f th o s e is o b v io u s and th e ir
m e a s u r e m e n t un pro b lem atic. H o w e v e r it is im p o r ta n t to notice, th a t s o m e o f the
v a ria b le s d e s c rib e d next are te m p o ra lly p re c e d in g to th e in stru c tio n at school, w hile
o th e rs m ay interact w ith past and c u r r e n t in s tru c tio n (e.g. s tu d e n t s 1 motivation).
O n e o f the m ost im p o r ta n t d e t e r m in a n t s o f c o m p e te n c ie s ta k en into a c c o u n t in
school effectiv en ess research is the social b a c k g r o u n d o f th e s tu d en ts w h ich can be
d iv id ed into SES and cultural capital ( B a u m e r t & S chu m er, 2001). In th e c u r r e n t
re s e a rc h th ere are different ind icato rs for th e SES such a s the g a in fu l e m p lo y m e n t o f
th e s tu d e n ts parents, th e ir level o f edu catio n o r th e relative p rosperity o f th e family.
P o ssessio n s in th is co n tex t include, for exam p le, electronic e q u ip m e n t, cars, b a t h
ro o m s, h o u s in g c o n d itio n s in general, and so on ( B a u m e r t & S chum er, 2001). T h e
te rm cu ltu ra l cap ital g o e s b ack to B o u rd ie u . In th e 1960s he pointed out that p ar
ticipation in the c u rre n t cu ltu re is an im p o rta n t d e t e r m in a n t o f school achiev em ent,
a n d co in ed the c o n c e p t o f cultural capital ( B o u r d ie u & P assero n, 1964). C u ltu ral
capital c o m p ris e s m a n y differen t a s p e c ts o f daily living such a s cultural assets and
resources. P IS A used differen t indicators to m e a s u r e the cultural capital such as the
national orig in o f stu d e n ts and their parents, h u m a n capital'1 or cultural practices
in the fam ily w h a t m e a n s the f a m i l y s closeness to its c o m m u n i t y s c o m m o n cu ltu re
( B a u m e r t & S chum er, 2001) w h ich is also called so cio -cu ltu ral m ilieu. T h e often
cited t e r m s c lo sen e ss to e d u c a tio n o r social class are m o stly c o m b in a tio n s o f
tw o or m o re o f th e ab o v e -m e n tio n e d indicators o f social b ac k g ro u n d .
T od ay's stu dies s h o w that B o u rd ie u w a s at least p artially right; d e t e r m in a n t s such
a s i m m i g r a n t sta tu s h av e a su b sta n tiv e influence, but only w h e n both paren ts are
from a foreign land is th e in flu en ce o f any practical relevance. Effect sizes o f social
b a c k g r o u n d on school a c h ie v e m e n t are small o r in term ed iate. For ex am ple, the study
Qualittitsuntersuchungan Schnlen zum Unterricht inM athem atik ( Q u a S U M ) sh ow s
an effect size o f /* = .31 for SES and c o m p e te n c e in m a th e m a tic s (H e lm k e , H osenfeld,
Schrader, & W agner, 2002a). T h e s a m e effect size w a s fo u n d by th e G e r m a n part
o f P I R L S , called Internationale Gritndschul-Lese-Unter.suchung (IG L U ), b e tw e e n
social b a c k g r o u n d and re a d in g c o m p e te n c e (S chw ip pert, Bos, & L a n k e s , 2003).
However, th e re are s o m e restrictio ns to th e se results. S o m e c o m p u ta tio n s i n previouslym e n tio n ed stu d ie s sh o w e d th at th e ind icato rs m e a s u r e partially r e d u n d a n t c o n s tru c ts
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
326
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
33 0
Hierarchical Structure
S tu d en ts are n ested in classes, classes a r e n ested in schools a n d scho ols in regions
a n d co u n tries. T h o u g h th e re h as b een an im p o r ta n t statistical a d v a n ta g e in d e a l
ing w ith hierarchical d a ta (e.g., B r y k & R a u d e n b u s h , 1992; R a u d e n b u s h , H ong, &
R ow an, in press), m a n y theoretical a n d practical q u e s tio n s r e m a in u n a n s w e r e d . O n
th e theoretical side, a p p ro a c h e s to causal m o d e llin g still lack a d a p ta tio n to su ch a h i
erarchical context. O n th e practical side, the effects o f d e t e r m in a n t s in f o r m e r s t u d
ies have b een m e a s u r e d quite heterogeneously, s o m e t im e s at th e stu d en t level and
s o m e tim e s at th e class o r school level. T h e fo llo w in g short e x a m p le should d e s c rib e
th e problem. M A R K U S e x a m in e d th e effect o f television and v id e o c o n s u m p tio n
on the class level and on th e single stu d en t level. In the first case the correlation
b e tw e e n th e p redictor and the c o m p e te n c y in m a th e m a tic s d iffers b e tw e e n r = - 0 . 0 2
and r = - . 2 0 , d e p e n d i n g on the school type. O n th e stu d en t level, a z e ro correla-
2 0 0 8 H ogrefe Publishing G m bH
K e in e u n e r la u b te W c itc r g a b c o d e r V c r v ic lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
Chapter 16
Monitoring and Assurance of School Quality:
Principles of Assessment and Internet-Based
Feedback of Test Results
Ingmar Hosenfeld
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
341
Information
about
level of
achievement
variation of
achievement
types of errors
aspects of
diagnostic
competency
Com parison
to
classrooms of
the same school
similar
classrooms
(contextwise)
Federal State
Educational
Standards
results of
previous tests
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
347
K e in e u n e r la u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.
354
I. Hosenfeld
2008
H u g ic fc PuMiJiing G m b H
K e in e u n e r lu u b te W c ite r g a b c o d e r V c r v ie lf a tig u n g .
You have either reached a page that is unavailable for viewing or reached your viewing limit for this
book.