Development of General Aptitude Test Battery (GATB) Forms E and F

Steven J. Mellon Jr., Mi !elle Da""ett, #in e Ma Manus and Brian Morits !

Su$mitted To% Division of S&ills Assessment and Analysis 'ffi e of (oli y and )esear ! Employment and Trainin" Administration *.S. Department of +a$or

Su$mitted By% (a ifi Assessment )esear ! and Development ,enter -.- +at!rop /ay, Suite A Sa ramento, ,A .01-0

-..2

Addendum
Please note that the General Aptitude Test Battery (Forms E & F) referred to within this report has been renamed the Ability Profiler (Forms 1 & 2) The name of the assessment was !han"ed to refle!t# 1) the fo!us on reportin" a profile of s!ore results from the instrument for !areer e$ploration purposes% 2) the te!hni!al impro&ements made to the assessment !ompared to pre&ious forms of the instrument% and ') the !apa!ity to use the Ability Profiler in !on(un!tion with other instruments to promote whole person assessment for !areer e$ploration

This material is in!luded as )hapter 2 in the unpublished report# * A +!)loy, T - *ussell, - .ise (Eds ), GATB improvement project final report .ashin"ton, /)# 0 1 /epartment of -abor

Table of Contents
CHAPTER 2 DEVELOPMENT OF GATB FORMS E AND F )han"es to 1pe!ifi!ations for Test -en"th and Format and to 1upportin" +aterials )on!erns and 2b(e!ti&es *edu!in" 1peededness 1!orin" Pro!edures 4nstru!tions to E$aminees *esear!h on Test Aestheti!s /e&elopment and *e&iew of 6ew 4tems 4tem .ritin" Editorial *e&iew and 1!reenin" 4tem Tryout and 1tatisti!al 1!reenin" 4tem Tryout Boo8let /esi"n 4tem Tryout 1ample /ata )olle!tion Pro!edures /ata Analysis )alibration and 1!reenin" of Power Test 4tems )alibration and 1!reenin" of 1peeded Test 4tems 1ele!tion of 4tems for Final <ersions of Forms E and F 1ele!tion of Power 4tems 1ele!tion of 4tems for the 1peeded Tests Forms E>uatin" 1tudy Forms E and F Test Tryout /ata )olle!tion Pro!edures /ata )olle!tion /esi"n 1ample )hara!teristi!s 1!orin" the GATB 1moothin" and E>uatin" )omposite E>uatin" 1ub"roup )omparisons *eliability Analysis <alidity Analysis 1ummary of Form /e&elopment A!ti&ities *eferen!es 1 2 2 2 3 3 5 11 11 12 17 17 13 19 1: 2; 22 2= 27 '2 '3 '3 '3 '5 =; =7 =9 =: 7; 7; 77 73

List of Tables
Table 2?1 *e!ommended )han"es in GATB Test 2rder, -en"th, and Time -imits ' Table 2?2 4tem Tryout Boo8let /esi"n 13 Table 2?' 4tem Tryout 1ample 1i@e for Ea!h Test Boo8let 15 Table 2?= Effe!t of )han"in" 4tem Position on 4tem /iffi!ulty 2; Table 2?7 A /es!ription of the 1amples by Ethni! and Gender 1ub"roups 21 Table 2?3 4tem 1!reenin" *esults for the Power Tests 22 Table 2?9 1ummary 4tem 1tatisti!s for )omputation 4tems Fla""ed by the /4F Analysis 2= Table 2?5 )omputation Test 4tem 1ele!tion# 1ummary 1tatisti!s for 1ele!ted Form E and Form F 4tems '7 Table 2?: 4ndependent?Groups 1ample 1i@es '9 Table 2?1; *epeated +easures /esi"n and 1ample 1i@es '9 Table 2?11 *epeated?+easures 1ample 1i@es by Test 1ite '9

Table 2?12 Table 2?1' Table 2?1= Table 2?17 Table 2?13 Table 2?19 Table 2?15 Table 2?1: Table 2?2; Table 2?21 Table 2?22 Table 2?2' Table 2?2=

Psy!homotor /ata )olle!tion /esi"n Group 1i@es for the Edited 4ndependent GroupsAE>uatin" 1ample Group 1i@es for the Edited *epeated?+easures 1ample /emo"raphi! )omposition of the 4ndependent?GroupsAE>uatin" 1ample /emo"raphi! )omposition of the *epeated?+easures 1ample /emo"raphi! )omposition of the Psy!homotor 1ample /emo"raphi! )omposition of the A""re"ate 1ample Aptitude 1!ore )omposition Alternate?Form *eliability Estimates, 6ormal /e&iates, and p?<alues /isattenuated )orrelations Between 6ew and 2ld GATB Forms )orrelations Amon" )o"niti&e, Per!eptual, and Psy!homotor )omposites Ad&erse 4mpa!t 1tatisti!s A!ross GATB Forms for 1ele!ted 1ub"roups GATB Forms E and F# Test -en"ths and Time -imits

'5 ': ': =; =1 =2 =' == 71 72 7' 7= 77

List of Figures
Fi"ure 2?1 Fi"ure 2?2 Fi"ure 2?' Fi"ure 2?= Fi"ure 2?7 Fi"ure 2?3 Test )hara!teristi! )ur&e (T))) /ifferen!es for Arithmeti! *easonin" Test )hara!teristi! (T))) /ifferen!es for <o!abulary Test )hara!teristi! )ur&e (T))) /ifferen!es for Three?/imensional 1pa!e Arithmeti! *easonin" Absolute 4nformation Graph <o!abulary Absolute 4nformation Graph Three?/imensional 1pa!e Absolute 4nformation Graph 29 25 2: '1 '1 '1

CHAPTER 2 DEVELOPMENT OF AT! FORM" E AND F 1te&en B

+ellon, Br , +i!helle /a""ett, <in!e +a!+anus, and Brian +orits!h Pa!ifi! Assessment *esear!h and /e&elopment )enter (PA*/)) The primary purpose of the GATB Forms E and F Development Project was to develop alternate forms of the cognitive portion of the GATB (Parts !"# following procedures that fulfill the highest professional standards$ The project was initiated prior to the %ational Academy of &ciences (%A&# review$ After the %A& review' the project e(panded to include other o)jectives e(plicitly recommended )y the %A& or otherwise implicit in its findings$ Those o)jectives included reducing test speededness and suscepti)ility to coaching' investigating scoring procedures' developing items free from )ias' assem)ling tests as parallel to each other as possi)le' improving the aesthetics of the tests' and revising answer sheets and other materials$ The A*DP met the e(panded o)jectives for the new GATB forms through a series of research steps$ First' to deal with the speededness and coaching issues' the A*DP made changes to specifications for test length and format$ +nce new items were written' the A*DP conducted item reviews' an item tryout' and statistical screening of items$ Based on the data' the A*DP developed new final forms$ An e,uating study lin-ed Forms E and F to )ase Form A$ 4n re&iewin" the General Aptitude Test Battery (GATB), the 6ational A!ademy of 1!ien!es (6A1) )ommittee identified se&eral problems relatin" to test se!urity and the speededness of the GATB tests (Carti"an & .i"dor, 1:5:) 1tated re!ommendations for alle&iatin" the problems in!luded the followin"# 1 There are !urrently two alternate forms of the GATB operationally a&ailable and two under de&elopment This is far too few for a nationwide testin" pro"ram Alternate forms need to be de&eloped with the same !are as the initial forms, and on a re"ular basis Form?to?form e>uatin" will be ne!essary This re>uires the attention to pro!edures and normati&e "roups des!ribed in the pre!edin" !hapter A!!ess to operational test forms must be se&erely limited to only those /epartment of -abor and Employment 1er&i!e personnel in&ol&ed in the testin" pro"ram and to those pro&idin" te!hni!al re&iew 1tri!t test a!!ess pro!edures must be implemented 1eparate but parallel forms of the GATB should be made a&ailable for !ounselin" and "uidan!e purposes A resear!h and de&elopment pro(e!t should be put in pla!e to redu!e the speededness of the GATB A hi"hly speeded test, one that no one !an hope to !omplete, is eminently !oa!hable For e$ample, s!ores !an be impro&ed by tea!hin" test ta8ers to fill in all remainin" blan8s in the last minute of the test period 4f this !hara!teristi! of the GATB is not altered, the test will not retain its &alidity when "i&en a widely re!o"ni@ed "ate8eepin" fun!tion (Carti"an & .i"dor, 1:5:, p 113)

' =

The primary purpose of the GATB Forms E and F /e&elopment Pro(e!t was to de&elop alternate forms of the !o"niti&e portion of the GATB (Parts 1?9) followin" pro!edures that fulfill the hi"hest professional standards The pro(e!t was initiated prior to the 6A1 re&iew and in!luded a re&iew of test len"ths and s!orin" pro!edures 1ubse>uent to the 6A1 re&iew, the fo!us of the pro(e!t was e$panded to in!lude other ob(e!ti&es e$pli!itly re!ommended by the 6A1 or otherwise impli!it in its findin"s

Chapter 2. Development of GATB Forms E and F

The e$panded ob(e!ti&es were D /e&elop new forms of the GATB that are less speeded and less sus!eptible to !oa!hin" by redu!in" the number of items and in&esti"atin" the feasibility of in!reasin" test time limits D 4n&esti"ate and in!orporate into the test the most appropriate s!orin" pro!edures, and de&elop instru!tions to e$aminees that !learly des!ribe those s!orin" pro!edures D /e&elop test items free from bias, in terms of both the ethni! and "ender sensiti&ity of the lan"ua"e and the statisti!al fun!tionin" of the items for different "roups D Assemble test forms as parallel to ea!h other as possible and lin8 s!ores on these forms to s!ores from earlier forms D 4mpro&e the aestheti!s of the test boo8lets and test items D *e&ise answer sheets and other related GATB materials to be !onsistent with !han"es in the test format and to pro&ide the opportunity for e$aminees to ma$imi@e their test s!ores The A*/P met the e$panded ob(e!ti&es for the new GATB forms throu"h a series of resear!h steps First, to address the speededness and !oa!hin" issues, the A*/P made !han"es to spe!ifi!ations for test len"th and format and to supportin" materials 2n!e new items were written, the A*/P !ondu!ted item re&iews, an item tryout, and statisti!al s!reenin" of items Based on the data, the A*/P de&eloped new final forms Finally, a study to lin8 new Forms E and F to base Form A was underta8en +ore detailed information about this resear!h is !ontained in the Te!hni!al *eport on the /e&elopment of GATB Forms E and F (+ellon, /a""ett, +a!+anus, & +orits!h, 1::3)

C#anges to "$e%ifi%ations for Test Lengt# and Format and to "u$$orting Materials
Con%erns and Ob&e%ti'es 4n its report, Fairness in Employment Testin", the 6A1 !on!luded that the se&en paper?and? pen!il tests of the GATB !ontained many more items than most e$aminees !ould possibly !omplete in the amount of time allotted for ea!h test 1!ores on items at the end of the test were more li8ely to be an indi!ation of whether the e$aminee was !oa!hed on a rapid respondin" strate"y than of the aptitude that the test is intended to measure )onse>uently, the in!lusion of items that few, if any, e$aminees rea!h if they seriously attempt to answer ea!h >uestion detra!ts from the &alidity of the test 2ptions for redu!in" the speededness of most of the GATB tests in!luded in!reasin" the time allotted andAor redu!in" the number of items for ea!h test An analysis of these options was the first step ta8en in re&isin" the test spe!ifi!ations Additional steps to redu!e the impa!t of test?ta8in" strate"y, in!ludin" !han"es to test instru!tions and the s!orin" pro!edures used with the tests that remain speeded, were also !onsidered The 6A1 )ommittee and GATB users also e$pressed !on!erns related to the testEs format, the o&erall aestheti! appeal of test items, and the format of the answer sheets A!tions ta8en to impro&e appearan!e and format were addressed in detail in the Test Aestheti!s Pro(e!t (/a""ett, 1::7), whi!h !onstituted the se!ond ma(or step in re&isin" the test spe!ifi!ations Redu%ing "$eededness The A*/P addressed issues pertainin" to the GATBEs speededness in three steps First, A*/P staff analy@ed s!ore distributions for the !urrent forms and de&eloped initial re!ommendations for redu!in" the number of items in ea!h test The se!ond step in&ol&ed new 2

resear!h on test speededness !ondu!ted by the Ameri!an 4nstitutes for *esear!h (A4*) under !ontra!t to /2- (1a"er, Peterson, & 2ppler, 1::=) The final step was a re&iew of the abo&e wor8 by an e$pert panel, who de&eloped re&ised re!ommendations !on!ernin" both the number of items and the time to be used for ea!h test Ea!h step is des!ribed briefly here, with more !omplete information pro&ided in the te!hni!al report (+ellon et al , 1::3) and in 1a"er et al (1::=) (nitial Re%ommendations) 1A*/) staff performed an initial analysis of possible test len"th redu!tion (1A*/), 1::2) They used normati&e data in the GATB de&elopment manual and s!ore !on&ersion tables in the administration manuals for Forms A throu"h / to determine the number of !orre!t responses re>uired to a!hie&e a ::th per!entile raw s!ore for ea!h test The intent was to appro$imate a pra!ti!al limit to the number of items used to differentiate amon" nearly all of the !urrent e$aminees )olumn four of Table 2?1 !ontains ran"es a!ross the four forms in the ::th per!entile s!ores
Table 2*+ Re%ommended C#anges in *e&ised Test 2rder ()urrent 2rder) Arithmeti! *easonin" (Part 3) <o!abulary (Part =) Three?/imensional 1pa!e (Part ') )omputation (Part 2) 6ame )omparison (Part 1) Tool +at!hin" (Part 7) Form +at!hin" H/roppedI (Part 9) Total, Tests 1?9 (Parts 1?9)
G

AT! Test Order, Lengt#, and Time Limits Forms A?/ -en"th Time 27 4tems 3; 4tems =; 4tems 7; 4tems 17; 4tems =: 4tems 3; 4tems ='= 4tems 9 +in 3 +in 3 +in 3 +in 3 +in 7 +in 3 +in =2 +in Form A?/ ::F ile *an"es 1:?21 4tems ='?72 4tems 2:?'2 4tems '9?=; 4tems 9;?52 4tems ':?== 4tems 2:?': 4tems 251?2:7 4tems 4nitial Proposal 2= 4tems 7; 4tems '7 4tems =; 4tems :; 4tems =2 4tems 7; 4tems ''1 4tems *e&ised Proposal -en"th Time 15 4tems 1=G 4tems 2; 4tems =; 4tems :; 4tems =2 4tems ; 4tems 22= 4tems 2; +in 3G +in 5 +in 3 +in 3 +in 7 +in ;GG+in 71 +in

1pe!ifi!ations for the final Forms E and F &ersions of the <o!abulary test were subse>uently !han"ed to 1: items with 5 minutes of testin" time GG Elimination of the Form +at!hin" test also sa&ed rou"hly 7 minutes in instru!tion and pra!ti!e (sample item) time

Be!ause the !al!ulations for estimatin" the number of items shown in !olumn = of Table 2? 1 do not ta8e into a!!ount the total number of items attempted (!orre!t plus in!orre!t), the test len"ths su""ested by these estimates !ould result in more than one per!ent of e$aminees !ompletin" all the items This, in turn, !ould artifi!ially !onstrain test s!ores PA*/) staff !onfirmed this in an e$amination of Form / operational data (6 J 15,;92) ()alifornia Test /e&elopment Field )enter, 1::2) For e$ample, on the Three?/imensional 1pa!e test, more than 1: per!ent of the 15,;92 e$aminees attempted 2: or more items (the number shown in !olumn =) PA*/) proposed new test len"ths based on the total number of items attempted by fewer than one per!ent of the e$aminees These results, !ombined with operational !onsiderations, led to a small in!rease in the number of items ori"inally re!ommended by the 1A*/) *e!ommendations based on the items attempted are shown in !olumn 7 of Table 2?1 A !omplete des!ription of the two studies is presented in 1outhern Test /e&elopment Field )enter (1::2) and )alifornia Test /e&elopment Field )enter (1::2)

'

Chapter 2. Development of GATB Forms E and F

Ne- Resear%# on Test Lengt#s) A4* re&iewed prior literature on issues of test speededness (Peterson, 1::') and desi"ned and e$e!uted a study to pro&ide further data on the most rele&ant issues found in the literature re&iew The re&iew !o&ered (1) methods of assessin" test speededness, (2) relati&e merits of power and speeded power tests, (') relationships between speeded and power tests of similar items, (=) differential effe!ts of speededness, and (7) ad&erse psy!holo"i!al rea!tions A 8ey findin" from this re&iew was that se&eral of the !onstru!ts measured by GATB tests (parti!ularly Arithmeti! *easonin", <o!abulary, Three?/imensional 1pa!e, and perhaps )omputation) were measured by mu!h less speeded instruments in other pro"rams Key !on!erns identified in!luded the e$tent to whi!h speeded and unspeeded tests in these areas measured the same !onstru!ts and whether there mi"ht be "reater s!ore differen!es for appli!ant "roups defined by ra!e, ethni!ity, a"e, and "ender on the speeded &ersus the non?speeded tests A!!ordin" to Peterson (1::'), La&ailable resear!h pro&ides mi$ed, at best, support for the e$pe!tation that redu!in" the speededness of the GATB power tests will redu!e ad&erse impa!t or ha&e other benefi!ial effe!ts for Afri!an Ameri!ans or females, but there are some potentially benefi!ial, pra!ti!al !onse>uen!es for redu!in" speededness (s!ores would be less sus!eptible to !han"es in administration !onditions, whether intended or not and adaptin" tests for disabled e$aminees is more easily a!!omplished)M (p =7) After !ompletin" the literature re&iew, A4* staff desi"ned a study to address the issues (ud"ed most salientNspe!ifi!ally, whether speeded and non?speeded &ersions of four of the GATB tests measured the same !onstru!ts% the e$tent to whi!h e$aminee sub"roups defined by ra!e, ethni!ity, a"e, or "ender showed "reater differen!es in one form or the other% and the e$tent to whi!h !orrelations amon" the tests in ea!h form were the same for different e$aminee "roups An additional issue addressed was whether !han"es in instru!tions, item formats, and answer sheets had a si"nifi!ant impa!t on test s!ores A non?speeded test battery in!ludin" )omputation, Three? /imensional 1pa!e, <o!abulary, and Arithmeti! *easonin" was !onstru!ted by redu!in" the number of items in Form / of ea!h of these tests and in!reasin" the time for Arithmeti! *easonin" from 9 to 11 minutes A speeded battery !omprisin" )omputation, Three?/imensional 1pa!e, <o!abulary, Arithmeti! *easonin", and 6ame )omparison was used for !omparison Parti!ipants for the study were re!ruited at si$ state employment offi!es in +aryland, Tennessee, and Te$as, yieldin" a total sample of 1,351 parti!ipants *ou"hly one half of the parti!ipants (539) !ompleted both the speeded and non?speeded batteries, usin" e$istin" instru!tions and format The other half (51=) !ompleted both batteries, usin" re&ised instru!tions and format 4n both !ases, the order of the speeded and non?speeded batteries was !ounter?balan!ed a!ross test sessions The sample in!luded 7:3 Afri!an Ameri!ans, 75; Cispani!s, ==7 parti!ipants who were at least =; years old, and 3;; females 4n "eneral, the nonspeeded &ersions of the GATB power tests measured !onstru!ts that were similar to the !onstru!ts measured by the speeded &ersions (i e , all estimated true s!ore !orrelations but one were "reater than 5;) Cowe&er, the !onstru!ts were not identi!al (i e , all true s!ore !orrelations were less than 1 ;;) The differen!e was that the speeded &ersions in!luded a speed fa!tor E&iden!e for the speed fa!tor was pro&ided by the hi"her !orrelations between 6ame )omparison and speeded &ersions of the GATB power tests than between 6ame )omparison and the unspeeded &ersions )onfirmatory fa!tor analyses also pro&ided e&iden!e of a speed fa!tor in the speeded &ersions Further, the !onfirmatory fa!tor analyses showed similar !onstru!ts a!ross the two instru!tionsAformat !onditions and a!ross the a"e, "ender, ra!e, and ethni!ity sub"roups 1a"er et al (1::=) drew si$ "eneral !on!lusions from this new resear!h on GATB test speededness issues#

D 6onspeeded &ersions of the Three?/imensional 1pa!e, Arithmeti! *easonin", and <o!abulary tests !an be de&eloped without lar"e in!reases in the operational time limits A nonspeeded &ersion of the )omputation test would re>uire an in!reased time limit D For the sub"roups parti!ipatin" in the study, nonspeeded &ersions of the GATB power tests would not "reatly in!rease mean sub"roup differen!es and !ould redu!e mean sub"roup differen!es between whites and Afri!an Ameri!ans on three power tests D 1peeded and nonspeeded &ersions of the GATB power tests measure the same !onstru!ts as the operational &ersions of these tests e$!ept for a speed fa!tor that appeared in the speeded &ersions The speed fa!tor also influen!es s!ores on the operational &ersions of the GATB power tests D The speeded and nonspeeded &ersions of the GATB power tests show the same relationships a!ross the sub"roups in!luded in the study D The study demonstrated that speededness does not lead to differential !onstru!t &alidity a!ross sub"roups There were, howe&er, somewhat smaller mean sub"roup differen!es between whites and Afri!an Ameri!ans on the power &ersions of three tests than on speeded &ersions of the same test These smaller differen!es !ould not be a!!ounted for entirely by differen!es in reliability between the speeded and power &ersions D The two instru!tionsAformat !onditions ha&e no effe!t on nonspeeded GATB power test s!ores and little or no effe!t on speeded GATB power test s!ores Furthermore, instru!tionsAformat does not affe!t mean sub"roup s!ore differen!es for speeded and nonspeeded GATB power tests Finally, the relationships between the speeded and nonspeeded GATB power tests do not !han"e a!ross the two instru!tionsAformat !onditions Re'ised Re%ommendations) After initial results from the speededness resear!h were a&ailable, /2- !on&ened a panel of e$perts to re&iew the results and de&elop re&ised re!ommendations on spe!ifi!ations for !ontent and len"th of the tests to be in!luded in the Form E and Form F batteries The panel, whi!h in!luded /r -loyd Cumphreys, /r *enee /awis, /r -auress .ise, and /r 6eal Kin"ston, met with A*/P representati&es and A4* staff responsible for the speededness study The panel !on!luded that the !urrent ran"e of &erbal, >uantitati&e, spatial, and per!eptual !onstru!ts should !ontinue to be measured but that three of the tests (i e , <o!abulary, Arithmeti! *easonin", and Three?/imensional 1pa!e) should not be speeded The relationship between test len"th and test reliability was !onsidered in de&elopin" the re!ommendations for re&ised test len"ths and times shown in the last two !olumns of Table 2?1 To the e$tent that it was not feasible to in!rease o&erall testin" time si"nifi!antly, the panel re!ommended droppin" the Form +at!hin" test to allow more time for other tests Tool +at!hin" was (ud"ed to be an ade>uate measure of the per!eptual speed and a!!ura!y !onstru!t, and this !onstru!t was (ud"ed to be somewhat less important than measures of &erbal, >uantitati&e, and spatial s8ills 2ne other re!ommendationNthat the power tests should be administered as a "roup before the speeded testsNled to the proposed 1 reorderin" of the remainin" tests shown in Table 2?1

Althou"h Form +at!hin" was e&entually dropped, the final de!ision !ame late in the pro(e!t )onse>uently, it was in!luded in many of the de&elopment steps des!ribed in this !hapter Also, Tool +at!hin" was e&entually renamed 2b(e!t +at!hin"

Chapter 2. Development of GATB Forms E and F

"%oring Pro%edures The A*/P altered s!orin" pro!edures to redu!e the effe!ts of test?ta8in" strate"ies Pre&ious GATB forms used number?!orre!t s!orin", in whi!h the final s!ore is simply the total number of >uestions answered !orre!tly, with no penalties for in!orre!t answers E$aminees who were willin" to "uess, e&en to the point of respondin" randomly (but rapidly) to items in the more speeded tests, were able to in!rease their total s!ores Efforts to redu!e the speededness of the power tests were desi"ned to redu!e the influen!e of this type of test?ta8in" strate"y After re&iewin" alternati&e approa!hes, the A*/P sele!ted a !on&entional formula s!orin" pro!edure (i e , one infli!tin" a penalty for ea!h in!orre!t response) for use with the three remainin" speeded tests The penalty for in!orre!t responses is based on the number of response alternati&es for ea!h item The !on!ept is that, if there are 8 alternati&es, an e$aminee who responds randomly will ha&e 8?1 in!orre!t responses for e&ery !orre!t response *espondin" randomly does not re>uire any 8nowled"e of the !onstru!t bein" measured The !on&entional formula introdu!es a penalty for in!orre!t responses that will !an!el out the number of !orre!t responses e$pe!ted by !han!e throu"h random respondin" The "eneral form of the formula is *?.A(8?1), where * is the number of !orre!t (ri"ht) responses, . is the number of in!orre!t (wron") responses, and 8 is the number of options for ea!h item The spe!ifi! s!orin" formulas for the three GATB speeded tests in Forms E and F are as follows# D )omputation# * ? .A= (a redu!tion of 1A=th point for in!orre!t responses)% D 2b(e!t +at!hin"# * ? .A' (a redu!tion of 1A'rd point for in!orre!t responses)% D 6ame )omparison# * ? . (a redu!tion of one point for in!orre!t responses) The A*/P !onsidered alternati&e approa!hes that in!luded a lar"er penalty for "uessin" +ost speeded tests !ontain items that all e$aminees should answer !orre!tly "i&en suffi!ient time +ore se&ere penalties for in!orre!t responses are sometimes introdu!ed in an attempt to for!e e$aminees to ta8e enou"h time with ea!h item to answer it !orre!tly This approa!h was not re!ommended for use with GATB Forms E and F for two reasons First, it introdu!es effe!ts of test?ta8in" strate"y, albeit in the opposite dire!tion Guessin" would lead to lower s!ores in !omparison to omittin" 1e!ond, pla!in" a "reater emphasis on a!!ura!y relati&e to speed !han"es the !onstru!t bein" measured to some e$tent, ma8in" "enerali@ations from prior &alidity studies more tenuous The A*/P de!ided that use of number?!orre!t s!orin" should be !ontinued for the power tests To the e$tent that e$aminees answer all items in a test, the number !orre!t and formula s!ores are linearly related E$aminees are ordered in e$a!tly the same way in both !ases The number? !orre!t s!ore is simpler to e$plain to e$aminees 4nstru!tions to e$aminees on how to ma$imi@e their s!oresNby attemptin" to answer e&ery itemNare also simpler when usin" number?!orre!t s!orin" (nstru%tions to E.aminees For GATB Forms A throu"h /, both the "eneral instru!tions and the test?spe!ifi! instru!tions pro&ide limited information re"ardin" test?ta8in" strate"ies, and neither dis!usses s!orin" pro!edures Test standards de&eloped by the AE*AAAPAA6)+E (oint !ommittee on test standards (1:57) re>uire that e$aminees be told how tests will be s!ored and "i&en spe!ifi! instru!tions that allow them to ma$imi@e their s!ores For this reason, !han"es to test s!orin" pro!edures were a!!ompanied by !onsideration and re&ision of e$aminee instru!tions

(tem Pretest) /urin" the item pretest, both "eneral and test?spe!ifi! instru!tions were modified to impro&e the information pro&ided to e$aminees The rele&ant portion of the "eneral instru!tions used durin" the pretest stated .ou pro)a)ly will not )e a)le to finish all the ,uestions in the first three parts$ Each part has so many ,uestions that very few people can finish in the time allowed$ /owever' answer as many as you can$ For ea!h of the speeded tests, the instru!tions "i&en after !ompletion of the pra!ti!e items in!luded the followin"# 0or- as FA&T and as 1A*EF233. as you can$ +n this e(ercise &PEED is very important$ 4f you have some idea of the answer to a ,uestion' even if you are not a)solutely positive' it is to your advantage to ta-e your BE&T G2E&&$ For e(ample' if you can eliminate one or more of the choices to a ,uestion' ta-e your BE&T G2E&&$ /owever' if you have no idea what the correct answer is' don5t spend time guessing$ 6ove on to the ne(t ,uestion$ For the power tests, e$aminees were simply instru!ted to Lwor8 as A))0*ATE-O and FA1T as you !an M Test Tr/out) /urin" the test tryout, instru!tions were modified to pro&ide information on s!orin" pro!edures as well as ad&i!e on test ta8in" strate"y At this time, the tests were reordered so that all of the power tests were administered first, followed by the three speeded tests 1eparate sets of "eneral instru!tions were pro&ided for the power and speeded tests The "eneral instru!tions for the power tests (Parts 1, 2, and ') stated +n the ne(t three parts wor- 1A*EF233.$ .ou should have enough time to answer each ,uestion$ 4t is to your advantage to A%&0E* E7E*. 82E&T4+%$ Even if you5re not sure of an answer' ma-e your BE&T G2E&&' fill in your answer' then go to the ne(t ,uestion$ .our score for each part will )e the num)er of ,uestions you answer correctly$ There is no penalty for answering incorrectly$ This information was repeated in the spe!ifi! instru!tions followin" the pra!ti!e items for ea!h of the power tests After the power tests were !ompleted, "eneral instru!tions for the speeded tests were pro&ided These instru!tions stated# The ne(t three parts are different from the parts you5ve already ta-en$ +n these parts' &PEED is 7E*. 46P+*TA%T$ .ou won5t have time to answer every ,uestion$ .ou must wor- as FA&T as you can )ut don5t )e careless$ 4f you have even the slightest idea of the answer' it is to your advantage to ma-e your BE&T G2E&&$ 4f you can eliminate one or more wrong choices to the ,uestion' then ma-e your BE&T G2E&&

Chapter 2. Development of GATB Forms E and F

from the remaining choices$ /owever' if you have no idea of the correct answer' don5t spend time guessing9 go to the ne(t ,uestion$ .ou will receive one point for each correct answer$ .ou5ll )e penali:ed for wrong answers$ Points will not )e su)tracted for ,uestions you don5t answer$ This information was also repeated in the spe!ifi! instru!tions followin" the pra!ti!e items for ea!h of the speeded tests At that point, e$aminees were told the spe!ifi! penalty for in!orre!t responses on that test For the )omputation test, for e$ample, e$aminees were told# .ou will receive one point for each correct answer$ .ou5ll lose one ,uarter ( ;<# of a point for each wrong answer$ Points will not )e su)tracted for pro)lems you don5t answer$ O$erational 0se) /2- determined that the instru!tions used in the test tryout should also be used operationally with Forms E and F 6o si"nifi!ant problems with these instru!tions were dis!o&ered in the test tryout Further, !han"es in test instru!tions mi"ht (eopardi@e the "enerali@ability of lin8in" results from the test tryout study Resear%# on Test Aest#eti%s The "oal of the Test Aestheti!s Pro(e!t was to impro&e the physi!al appearan!e and user friendliness of the GATB and supportin" administration materials The pro!edures used and resultin" impro&ements are summari@ed briefly here% a more !omplete des!ription of the pro(e!t is presented in /a""ett (1::7) The pro(e!t in&ol&ed three ma(or a!ti&ities# 1 inter&iews with testin" professionals to identify spe!ifi! areas to be addressed% literature sear!hes of re!ent aptitude tests, personality in&entories, &o!ational interest in&entories, and supportin" administration materials to define spe!ifi! aspe!ts of these areas% fo!us "roups with military testin" professionals, employment !ounselin" professionals, and representati&es from the fi&e Assessment *esear!h and /e&elopment )enters and the 6ational 2ffi!e to dis!uss user issues, editorial styles, and re!ommended pra!ti!es% a sur&ey of or"ani@ations that !ontra!t with the state of )alifornia to administer the GATB

'

The pro(e!t resulted in a number of re!ommendations for format re&ision whi!h were, in turn, in!orporated into the re&isions of the GATB test boo8lets, answer sheets, and administration manual +a(or format re&isions are presented below for ea!h do!ument Test !oo1lets and (nstru%tions) *e&isions made to the test boo8lets and a!!ompanyin" e$aminee instru!tions in!lude in!reasin" white spa!e, !han"in" the type font to !ontemporary 12?point Palatino, and in!reasin" the use of itali!s, bolded print, and underlinin" to emphasi@e spe!ifi! words or phrases 4n addition, !uein" "raphi!s were added to the bottom of pa"es when appropriate and hi"her "rade paper and print >uality were used *e&isions for the final &ersions of the test boo8lets in!luded (a) !ombinin" the two boo8lets used in pre&ious GATB forms into a sin"le boo8let, (b) re&isin" the instru!tions to redu!e redundan!y and the number of words in ea!h senten!e, and (!) ad(ustin" spa!e pro&ided for the

)omputation and Arithmeti! *easonin" items to !onform to the amount of spa!e needed by ea!h item Test (tems) 4n "eneral, a number of !han"es were made to impro&e the appearan!e and user friendliness of the test items themsel&es 1pe!ifi! !han"es for the indi&idual items are dis!ussed below for ea!h test Arithmetic *easoning P 4ndi&idual items were pla!ed in !ells !ontainin" one double &erti!al line (i e , two &erti!al lines ad(a!ent to ea!h other) and one sin"le hori@ontal line Althou"h the two?!olumn format was maintained, the number of items on ea!h pa"e was redu!ed 2nly Arabi! numerals (instead of words) were used to e$press numbers E$!ept for monetary &alues, a @ero was used as the first di"it for de!imal &alues less than one 1omputation P The same item format de&eloped for the Arithmeti! *easonin" items was used for the )omputation items Also, there were no more than ei"ht )omputation items on ea!h pa"e )onsisten!y in pun!tuation, response ali"nment, and use of monetary symbols was maintained throu"hout the test Finally, the arithmeti! operation symbol for ea!h item was pla!ed within the item (4n prior forms, the operation symbol was pla!ed abo&e the item ) Form 6atching P The se!ond item blo!8 was redu!ed from '7 to 27 items, and the number of response options was redu!ed from 1; to 7 %ame 1omparison P The number of items on ea!h pa"e was redu!ed from 7; to '; The hori@ontal line after e&ery fifth item was repla!ed by a blan8 line, and a blan8 spa!e pre!eded and followed the dash that separated the two names within ea!h item +)ject (Tool# 6atching P The print >uality was impro&ed, and the test title was !han"ed from Tool +at!hin" to 2b(e!t +at!hin" Three!Dimensional &pace P The print >uality and resolution were impro&ed throu"h the use of the )orel/*A.Q = "raphi!s pa!8a"e ()orel )orporation, 1::') to de&elop the indi&idual items 7oca)ulary P The item format was !han"ed from hori@ontal to &erti!al The number of items on ea!h pa"e was redu!ed from to '; to no more than 1; The 1: items were arran"ed in three !olumns of fi&e items and one four?item !olumn, ea!h separated by a double &erti!al line and one hori@ontal line Ans-er "#eet) 1e&eral !han"es were made to the answer sheets used in the Forms E and F de&elopment resear!h 4n "eneral, response formats were re&ised to !onform to those made in the test items *esponse bubbles were redu!ed in si@e by eliminatin" the top and bottom portions of ea!h bubble and addin" small hori@ontal lines to form o&als This !han"e was made in response to a !on!ern raised in the 6A1 re&iew that diffi!ulty in fillin" in lar"e !ir!les mi"ht impede performan!e on the speeded tests 1pe!ifi! !han"es in!orporated at ea!h sta"e in the de&elopment resear!h are des!ribed below 4tem Pretest Phase$ Four s!annable answer sheet formats were used durin" this phase for data !olle!tion The answer sheets were modified to# (1) !ontain four se!tions% (2) support the e$perimental test !onfi"uration% (') a!!ommodate format !han"es made to Form +at!hin" and <o!abulary tests% (=) present all demo"raphi! information on one pa"e% and (7) use more !ontemporary and le"ally appropriate wordin" 4n addition, ea!h answer sheet was further adapted to be used spe!ifi!ally with one of the four power tests The four answer sheets and 13 test boo8lets were !olor?!oded to fa!ilitate the appropriate test boo8let?answer sheet !ombination

Chapter 2. Development of GATB Forms E and F

Test Tryout Phase$ The test tryout answer sheet was used with both Forms E and F 4t was printed in purple to differentiate it &isually from Form A and Form B answer sheets The number of se!tions was in!reased from four to se&en, and they were rearran"ed to !oin!ide with the new test arran"ement (Part 9?Form +at!hin" was later eliminated) The shaded pra!ti!e bo$es were pla!ed at the top left of ea!h se!tion The !orre!t number of item responses was pla!ed in ea!h se!tion and reformatted into !olumns of e>ual numbers with the tops of ea!h !olumn startin" on the same line .ith the elimination of the wraparound !olumns, the phrase LBe"in CereM be!ame unne!essary and was eliminated The demo"raphi! se!tion was modified to !olle!t resear!h?spe!ifi! information The most notable modifi!ation, howe&er, was repla!in" the lar"e response !ir!les with o&als +perational Phase$ The operational answer sheet differs from the test tryout answer sheet in the followin" ways# D *esear!h?spe!ifi! demo"raphi! information was eliminated D The Form +at!hin" se!tion (Part 9) was eliminated D The response identifi!ation letter was pla!ed inside ea!h response o&al D Alternate blo!8s of fi&e item responses were shaded Administration Manual) The spe!ifi! re&isions made to date in!lude D reformattin" as a te!hni!al manual% D in!orporatin" a !on&ersational business tone (i e , employin" simple, !on!rete, !lear lan"ua"e)% D writin" in a manner to address readerEs needs first% D !onsolidatin" pro!edures and instru!tions into distin!t sub(e!t areas (i e , eliminatin" unne!essary lan"ua"e, in!onsisten!ies, repetition, and redundan!y)% D subtitlin" ea!h sub(e!t area% D updatin" and rewritin" information so that the readin" le&el and detail are appropriate for all users% D formattin" manual into a lo"i!al se>uen!e for better understandin" of administration pro!edures and instru!tions% D addin" !uein" "raphi!s and bullets% D usin" !olored paper as a !uein" te!hni>ue% D in!reasin" white spa!e and redu!in" line len"th% D !han"in" to 12?point serif typefa!e with in!reased use of headers% D usin" hi"her "rade paper with impro&ed print >uality% D de&elopin" a standardi@ed introdu!tion s!ript% and D bindin" the manual into an 5 1A2 in by 11 in hardba!8 binder (be!ause of on"oin" modifi!ations) The followin" !han"es ha&e been proposed for the final &ersion of the administration manual# D different !olors of print to ser&e as !uein" de&i!es for the test administrator (e " , phrases to be read aloud) at different lo!ations in the manual% D redu!in" the si@e of the pa"es to 9 in by 1; in with wire spiral bindin"% and D in!ludin" demonstration models for the Three?/imensional 1pa!e pra!ti!e items

The final administration manual for GATB Forms E and F will in!lude s!orin" pro!edures and !on&ersion tables The use of administration aids su!h as !olor, !he!8lists, and forms will be in!reased Additional aestheti! and format !han"es will be made to further enhan!e readability and usability, su!h as redu!in" the si@e of the manual, tabbin" the se!tions, addin" more !uein" "raphi!s, and in!orporatin" a two?!olumn format with shorter lines

De'elo$ment and Re'ie- of Ne- (tems

An e$tensi&e effort was underta8en to de&elop new items for GATB Forms E and F 6ote that many more items were de&eloped than used in the final forms Efforts to spe!ify !ontent and diffi!ulty !ate"ories, write items for ea!h of these !ate"ories, and then re&iew items for editorial and sensiti&ity !onsiderations are des!ribed here 4tem tryouts, statisti!al s!reenin", and !alibration pro!edures are des!ribed in the followin" se!tion (tem 2riting A brief des!ription of the de&elopment of e$perimental items for Parts 1?9 of Forms E and F is "i&en below For ea!h test, items from pre&ious forms were analy@ed and sorted into !ate"ories potentially related to item diffi!ulty 1our!es for item material also were identified +ore detailed information on spe!ifi!ations and item typesA!ontent !ate"ories for ea!h test are presented in the Forms E and F pro(e!t te!hni!al report (+ellon et al , 1::3) Brief summaries of the item de&elopment pro!edures used for ea!h test are presented here Name Com$arison) The =;; 6ame )omparison items were de&eloped to be parallel to Form A items and representati&e in terms of "ender and ethni!ity The number of items with names that were the same was e>ual to the number of items with different names 4tem sour!es in!luded dire!tories, di!tionaries, and item de&eloper !reati&ity Analyses were then performed to de&elop preliminary estimates of item diffi!ulty Based on these analyses, the number of !hara!ters in the left?hand !olumn of the two?!olumn format used for this test was sele!ted as the item diffi!ulty measure The 2;; items for ea!h form were di&ided into four 7;?item >uarters of appro$imately e>ual estimated o&erall diffi!ulty The item order was then randomi@ed within ea!h >uarter Com$utation) The 1'3 )omputation items were de&eloped to be parallel to Forms A?/ The ori"inal items were de&eloped and re&iewed to e&aluate diffi!ulty The number of di"its a!ross numbers within ea!h type of operation was used as the item diffi!ulty measure The 35 items for ea!h form were di&ided into four 19?item >uarters of e>ual estimated o&erall diffi!ulty Type of arithmeti! operation and response options were balan!ed within ea!h >uarter A low?diffi!ulty item was assi"ned to the first position within ea!h >uarter with the remainin" items ordered randomly T#ree*Dimensional "$a%e) The 1'; Three?/imensional 1pa!e items were de&eloped to be similar in !ontent to prior forms The number of folds was used as a measure of item diffi!ulty% it had si$ le&els 6ewly de&eloped items were "rouped a!!ordin" to the number of folds so that an e>ual number of items would be de&eloped for ea!h of the si$ diffi!ulty le&els 4tems were then drawn on a !omputer, usin" the )A//?' software pa!8a"e 4tems were !ontinually re&iewed for !larity and !orre!tness, and shadin" was added )ompleted items were transferred to +ylar paper and redu!ed in si@e photo"raphi!ally, then plates were made for printin" 4tems were re&iewed a"ain and re&ised when ne!essary 4tems were then assi"ned to forms on the basis of diffi!ulty, and response options were !he!8ed and tallied 2ption positions were !han"ed as ne!essary The items were rephoto"raphed and printed The A*/P used the )orel/*A.Q = "raphi!s pa!8a"e ()orel )orporation, 1::') to redraw all of the items to ma8e them !onsistent in appearan!e )amera?ready !opies of the reformatted items were prepared and sent to a "raphi! artist for proofin" 1ome of the items were later re&ised

Chapter 2. Development of GATB Forms E and F

to !orre!t the problems identified by the "raphi! artist Three diffi!ulty le&els were identified based on the number of folds andAor rolls made in ea!h item These diffi!ulty &alues were then used to form three 13?item >uartiles and one 19?item >uartile of appro$imately e>ual estimated o&erall diffi!ulty within ea!h form .ithin ea!h >uartile a low?diffi!ulty item was assi"ned to the first position with the order of the remainin" items randomi@ed The !orre!t response option fre>uen!ies were balan!ed within ea!h >uartile Vo%abular/) The 13; <o!abulary items were de&eloped to be parallel to Form B 4tem re&iew also fo!used on word diffi!ulty but used a different approa!h from pre&ious GATB de&elopment efforts 1pe!ifi!ally, The -i&in" .ord <o!abulary (/ale & 2E*our8e, 1:51) pro&ided estimates of item diffi!ulty This referen!e assi"ns a "rade le&el to ea!h word meanin" The assi"ned "rade le&el is based on the responses of students who !ompleted &o!abulary tests durin" the period of 1:7=?1:9: .hen multiple word meanin"s were reported for a "i&en word, the a&era"e "rade le&el was used Ci"her "rade le&els indi!ated "reater diffi!ulty The mean of the reported "rade le&els for the four words that made up ea!h item was used to estimate item diffi!ulty Four diffi!ulty le&el !ate"ories were formed These !ate"ories were used to prepare four 2;?item >uartiles of e>ual estimated o&erall diffi!ulty for ea!h form For ea!h >uartile, the two items with the lowest estimated diffi!ulty appeared in the first two positions with the order of the 15 remainin" items randomi@ed The !orre!t response option fre>uen!y distributions were balan!ed within >uartiles and forms Ob&e%t Mat%#ing) The 13' ori"inal 2b(e!t +at!hin" items were de&eloped to be parallel to Forms A?/ The A*/P used the number of shaded areas in the four response alternati&es for ea!h item to estimate diffi!ulty le&el /iffi!ulty le&el, !ontent !onsiderations, and lo!ation of the !orre!t response were used to form four 2;?item >uartiles of similar o&erall diffi!ulty for ea!h form (Three items were deleted ) The item order was randomi@ed within ea!h >uartile A surplus item was then added to ea!h >uartile to form three se&en?item pa"es that !ould be shifted to meet the re>uirements of the resear!h desi"n Arit#meti% Reasoning) The 33 Arithmeti! *easonin" items were de&eloped to be parallel to Form A 6ew situations, !ontemporary monetary &alues, "ender representation, e$!lusion of e$traneous information, and a si$th?"rade readin" le&el were additional !onsiderations in item de&elopment The A*/P re&iewed and re&ised the items so they !onformed more !losely to the "uidelines for de&elopment 4tem diffi!ulty was estimated by the number of operations needed to sol&e the problem, the type(s) of operations, and the number of di"its in!luded in the terms used in the operation(s) 2ne of the two least diffi!ult items was assi"ned to the first item position in Form E and the other item assi"ned to Form F The remainin" 3= items were then assi"ned to four ei"ht?item >uartiles for ea!h form on the basis of diffi!ulty, type (s) of operation(s), !orre!t response 8ey, and !ontent The items in ea!h >uartile were ordered from least to most diffi!ult with the item order then randomi@ed within ea!h >uartile Form Mat%#ing) The 2;; Form +at!hin" items were de&eloped to be parallel to Forms A?/ items in terms of !ontent and parallel to Form A item si@e and arran"ement Ei"ht 27?item blo!8s were de&eloped by modifyin" ea!h of the ei"ht blo!8s of items in Forms A?/ The number of response options for ea!h item was redu!ed from 1; to fi&e Editorial Re'ie- and "%reening Literature Re'ie-) The de&elopment of the item re&iew pro!edures be"an with a literature re&iew fo!usin" on the pro!ess for !ondu!tin" item re&iews and sele!tin" the parti!ipants in the re&iew pro!ess Two types of re&iew pro!edures were identified# (1) pro!edures used for resear!h purposes, and (2) pro!edures used in on"oin" testin" pro"rams

(tem Re'ie- (nstruments) Throu"h the literature re&iew, it was determined that (1) most pre&ious item re&iew pro!edures were desi"ned for use with edu!ational a!hie&ement tests, and (2) re&iew pro!edures used in most pre&ious studies were not hi"hly stru!tured and appeared to be de&eloped independently with limited "uidan!e from the edu!ational measurement literature Cowe&er, the literature re&iew did un!o&er ei"ht referen!es (Boldt, 1:5'% Cambleton & *o"ers, 1:55% Carms, 1:95% -o!8heed?Kat@, 1:9=% +adaus, Airasian, Cambleton, )onsal&o, & 2rlandi, 1:9:% 2lson & 1moyer, 1:55% 1!hrat@ & .ellens, 1:51% Tittle, 1:52) that pro&ided the foundation for the instruments and pro!edures that were used in the GATB Forms E and F item re&iew These ei"ht referen!es pro&ided information in three areas# bias "uidelines, pro!edural issues, and ratin" >uestions Preliminar/ Re'ie-) /raft &ersions of item sensiti&ity re&iew >uestions, instru!tions, and an answer form were sent to A*/ !enters for re&iew Based on the !omments, A*/P staff re&ised draft &ersions of the sensiti&ity re&iew materials and sent them to A*/)s for further re&iew The only re&ision was a minor !han"e in the answer form Pilot Test) A pilot test was !ondu!ted in?house with three )ooperati&e Personnel 1er&i!es ()P1) staff members, enablin" indi&iduals who were not in&ol&ed in the A*/P test resear!h pro"ram to pro&ide input to the re&iew pro!ess The results led to a number of modifi!ations in pro!edures, instru!tions, and do!uments that would be used for the item re&iew (tem Re'ie- Materials) 6ine do!uments were used in the item re&iew pro!ess# (1) a list of the !riteria to sele!t panel members, (2) a !onfidentiality a"reement, (') a des!ription of the GATB tests and aptitudes, (=) written instru!tions for panel members, (7) the administratorEs &ersion of the written instru!tions for panel members, (3) a list of !hara!teristi!s of unbiased test items, (9) a list of the re&iew >uestions with e$planations, (5) an answer form, and (:) an answer form supplement Panel Member C#ara%teristi%s) 1e&en panel members parti!ipated in the re&iew The panel in!luded two Afri!an Ameri!ans, three Cispani!s, and two whites Three members were male and four female Three members were personnel analysts, two were uni&ersity professors in !ounselor edu!ation, one was a personnel !onsultant, and one was a postdo!toral fellow in e!onomi!s Pro%edures) At an orientation meetin" held at ea!h of the three parti!ipatin" A*/)s, !onfidentiality a"reements were si"ned, GATB items and instru!tions were "i&en to panel members, and se&eral items in ea!h test were re&iewed and dis!ussed Panel members re&iewed the remainin" items at their !on&enien!e After all items were re&iewed, a follow?up meetin" was held at ea!h !enter to resol&e any problems and to dis!uss the re&iew pro!ess "ummar/ of Results) The answer forms of ea!h panel member were re&iewed 1ummaries of the !omments for ea!h test are presented below %ame 1omparison = )omments fo!used on ra!ial, ethni!, and "ender stereotypin" and representation 1pe!ifi! !on!erns in!luded the la!8 of female and minority businesses, and the need for more females in nontraditional professions, (obs, and businesses 1omputation = )omments primarily dealt with item !hara!teristi!s 1pe!ifi! !on!erns in!luded diffi!ult and time?!onsumin" problems that mi"ht be s8ipped by testwise appli!ants, poor distra!tors, and un!lear instru!tions Three!Dimensional &pace P )omments !on!erned possible "ender bias and item !hara!teristi!s )omments in!luded the presen!e of male?oriented items and abstra!t items

Chapter 2. Development of GATB Forms E and F

that mi"ht be unfamiliar to females, diffi!ult and time?!onsumin" items that !ould be s8ipped by testwise appli!ants, "ender?biased instru!tions, and o&erly !ompli!ated items 7oca)ulary P )omments !on!erned hi"h readin" "rade le&el, o&erly diffi!ult words% words with different meanin"s for different "roups% and in!lusion of forei"n?lan"ua"e words and te!hni!al, biolo"i!al, and s!ientifi! terms Tool 6atching P )omments fo!used mainly on possible "ender bias due to differen!es in familiarity and the presen!e of male?oriented items Cowe&er, !on!erns were also e$pressed that items with ele!tri!al and me!hani!al !omponents mi"ht !ause problems for minorities due to la!8 of familiarity and opportunity to learn 2ther !omments !on!erned !larity of instru!tions and positionin" of the response letters for the item alternati&es Arithmetic *easoning P +ost !omments were dire!ted toward two areas# (1) ra!ial, ethni!, and "ender representation, and (2) "ender o!!upational and a!ti&ity stereotypin" 2ther !omments !on!erned time?!onsumin" items that mi"ht be s8ipped by testwise appli!ants, !onfusin" and in!omplete instru!tions, the presen!e of items that were o&erly !ompli!ated or in&ol&ed too many steps, and some "roups not ha&in" the opportunity to learn how to perform the operations needed to answer the !omple$ items Form 6atching P )omments in!luded a possible pra!ti!e effe!t for the test and un!lear instru!tions be!ause of readin" le&el )omments that were dire!ted toward spe!ifi! items in!luded linear illustrations bein" per!ei&ed as Lhostile,M minute differen!es amon" shapes, and possible !onfusion due to shape similarity and lo!ation (tem Content Re'ision) Based on results from the panel e&aluation, the !ontent of spe!ifi! items was re&ised A summary of the types of !han"es introdu!ed for ea!h test is presented here %ame 1omparison P The re&isions addressed the ra!ial, ethni!, and "ender stereotypin" and representation !riti!isms Guidelines based on the 1::; 0 1 )ensus were used to in!rease ra!ialAethni! and "ender representation 1tereotypin" was addressed by in!ludin" items with minorities and females in nontraditional o!!upations and businesses% more professional o!!upations and businesses were in!luded Fewer items with Germani! names were used Format !han"es in!luded separatin" the items into blo!8s of fi&e, eliminatin" hori@ontal lines, and in!reasin" the hori@ontal and &erti!al spa!e within and between items Finally, the instru!tions were reworded to in!rease !larity% bold and itali!i@ed types were used for emphasis 1omputation P /istra!tors were re&ised to ma8e them more plausible based on fi&e error types +inor format !han"es in!luded addin" !ommas to numbers with at least four di"its and pla!in" the operation si"n within the item Finally, the instru!tions were reworded sli"htly to in!rease !larity, and bold and itali!i@ed types were used for emphasis Three!Dimensional &pace P 4ndi&idual items were re&ised when needed to in!rease !larity *e&isions were re&iewed by a "raphi!s e$pert familiar with the test format and the drawin" software to ensure that the items were free of errors 4nstru!tions were reworded sli"htly to in!rease !larity and eliminate possible "ender bias% bold and itali!i@ed types were used for emphasis 7oca)ulary P .ords were repla!ed on the basis of the item re&iew panel member !omments and on an analysis of word diffi!ulty in /ale and 2E*our8e (1:51) 4tems were modified as needed to ensure that ea!h itemEs le&el of word diffi!ulty was appropriate, word forms within items were identi!al, and the same type of !orre!t response (i e , synonym or

antonym) was maintained within ea!h item The item format was !han"ed from hori@ontal to &erti!al orderin" of words Finally, the instru!tions were rewor8ed (e " , bold and itali!i@ed types were used to emphasi@e important points, and a statement was added stressin" that all !hoi!es should be !onsidered before sele!tin" an answer) Tool 6atching P 4tem re&isions in!luded eliminatin" in!onse>uential differen!es amon" item responses, eliminatin" dupli!ate responses, and refinin" responses (e " , remo&in" e$traneous matter, drawin" sharper lines, eliminatin" bro8en lines) Finally, the instru!tions were reworded sli"htly to in!rease !larity% bold and itali!i@ed type were used for emphasis, and the test name was !han"ed from Tool +at!hin" to 2b(e!t +at!hin" (Future forms will in!lude more "eneri! items e&en thou"h the results from item analyses indi!ated that female s!ores are sli"htly hi"her than male s!ores on the !urrent items ) Arithmetic *easoning P *e&isions in&ol&ed four areas# ma8in" minor item format modifi!ations, eliminatin" "ender stereotypin", ma8in" the distra!tors more plausible, and in!reasin" ra!ial, ethni!, and "ender representation The instru!tions were reworded sli"htly to in!rease !larity% bold and itali!i@ed types were used for emphasis Form 6atching P )han"es in!luded enlar"in" fi"ures to in!rease !larity, repositionin" items to e>uali@e spa!e amon" items in the lower blo!8s, and re&isin" an item family to ma8e it less similar to another item family The number of response options was redu!ed from 1; to 7 Finally the instru!tions were reworded sli"htly to in!rease !larity% bold and itali!i@ed types were used for emphasis

(tem Tr/out and "tatisti%al "%reening

The A*/P !ondu!ted a tryout on!e new items were written and s!reened, administerin" the new items to a sample of e$aminees The statisti!al information "athered ser&ed two purposes# item s!reenin" and item !alibration 4tems were dropped from further !onsideration (s!reened out) if they were too easy or too diffi!ult, if they failed to dis!riminate between hi"her and lower ability e$aminees, or if they showed si"nifi!ant differential fun!tionin" by "ender or ra!e or ethni! "roup For the remainin" items, the tryout data were used to estimate item statisti!s for use in !onstru!tin" parallel forms The rest of this se!tion summari@es the results of the item pretest data !olle!tion, item analysis, and item sele!tion (tem Tr/out !oo1let Design There were many more items than any one e$aminee !ould reasonably be e$pe!ted to !omplete, so the A*/P or"ani@ed the items into separate boo8lets and assi"ned different boo8lets to different e$aminees The desi"n of the tryout boo8let addressed a number of issues First, to allow sele!tion of the best items, the A*/P needed to try out more items for ea!h test than would e&entually be used operationally Further, time?per?item was in!reased to impro&e the >uality and >uantity of the item data (e " , fewer omits and unrea!hed items) To"ether, these !onsiderations meant that it was not reasonable for ea!h e$aminee to !omplete all of the tests 4nstead, the tryout boo8lets were desi"ned so that e$aminees would !omplete some of the items for ea!h of the three speeded tests and all of the items for one of the four power tests (i e , Arithmeti! *easonin", Three? 2 /imensional 1pa!e, <o!abulary, and )omputation ) For the speeded tests, the A*/P "rouped new items within ea!h test into four sets and rotated the order of the four sets a!ross four different boo8let pairs for ea!h form 6ot all of the
The )omputation test was ori"inally in!luded with the power tests but was subse>uently treated as a speeded test (!f 1a"er et al , 1::=)
2

Chapter 2. Development of GATB Forms E and F

items were printed in ea!h boo8let 1pe!ifi!ally, for Form +at!hin", two sets of 27 items ea!h were printed in ea!h boo8let% for Tool (2b(e!t) +at!hin", two sets of 2; items ea!h were printed in ea!h boo8let% and for 6ame )omparison, three sets of 7; items ea!h were printed in ea!h boo8let The intention was that the items within ea!h blo!8 would be analy@ed for the boo8let where the blo!8 appeared in the first position 4n this way, enou"h e$aminees would !omplete (rea!h) the item to allow assessment of item diffi!ulty, item?total !orrelation, and sub"roup differen!es in item performan!e For the speeded tests, there was no attempt to e>uate a!ross boo8lets at the item le&el 4tem results were e$pe!ted to &ary widely a!!ordin" to ea!h itemEs position in the test, so e>uatin" would be handled at the test le&el in the form !alibration study For the power tests, an attempt was made to e>uate item diffi!ulties and 4*T parameter estimates not only between the Forms E and F item sets, but also with the item parameter estimates from an operational form, Form A )onse>uently, all items from the Form A &ersion of a "i&en power test were in!luded alon" with half of the new items for that test (i e , all items for either Form E or F) in Part = of a tryout boo8let The A*/P determined the order of items within ea!h power test by di&idin" Form A (an!hor) items into dis!rete blo!8s and then spa!in" these blo!8s throu"hout the tryout boo8let The remainin" item positions were filled with new items For power tests, an itemEs position within a form should not affe!t its diffi!ulty (or dis!rimination) The tryout boo8let desi"n in!luded pro&ision for testin" this assumption 1pe!ifi!ally, two &ersions of ea!h power test were !reated with the order of the new items re&ersed in the e&en?numbered tryout boo8lets relati&e to their positions in the odd?numbered boo8lets The (Form A) an!hor items were printed in the same position in ea!h of these boo8let pairs Table 2?2 summari@es the desi"n of the 13 boo8lets de&eloped for the item tryout study
Table 2*2 (tem Tr/out !oo1let Design Tryout Booklet (Booklet pair are i!e"ti#al e$#ept %or re&er e! or!er o% Part ' "e( ite) * For) E For) F 1,2 :,1; ',= 7,3 9,5 11,12 1',1= 17,13 Part + For) Mat#, -. ite) / +. )i"0 7te) 8 1?7; 23?97 71?1;; 93?1;;, 1?27 Part 2 Tool Mat#, '1 ite) / 2 )i"0 7te) 8 1?=: 21?3: =1?5;, 1?: 31?5;, 1?2: Part 3 Na)e Co)p0 +-. ite) / ++ Mi" 7te) 8 1?17; 71?2;; 1;1?2;;, 1?7; 171?2;;, 1?1;; Part ' O"e o% t,e Po(er Te t (For) A a"#,or ite) (ere 4roupe! i" %i$e! 5lo#k 6 "e( ite) %ille! re)ai"i"4 po itio" 0* Te t )omputation '/ 1pa!e <o!ab Arith *easonin" A"#,or 7; 4tems =; 4tems 3; 4tems 27 4tems Ne( 35 4tems 37 4tems 5; 4tems '' 4tems Ti)e 39 min 7; min 9; min 9' min

(tem Tr/out "am$le The primary tar"et sample was Employment 1er&i!e lo!al offi!e appli!ants -o!al offi!es used for data !olle!tion were representati&e of offi!es ser&in" the wor8in" population in terms of "ender, ethni!ity, a"e, and edu!ational le&el 1upplemental sour!es for study parti!ipants were also identified These sour!es in!luded employed wor8ers, !ommunity "roups or asso!iations, hi"h s!hool seniors and (unior !olle"e students, and &o!ational trainin" !enters The sample members were not to ha&e ta8en any form of the GATB within the 12?month period immediately prior to testin" 1tudy parti!ipants were reimbursed for tra&el e$penses

1ample si@e tar"ets were set to support a!!urate estimation of item statisti!s and analyses of differential item fun!tionin" (/4F) a!ross "ender and ra!eAethni! "roups An o&erall tar"et of 1,;;; e$aminees per item was set to support item response theory (4*T) and !lassi!al item analyses .ithin the o&erall tar"et, a rou"hly e>ual split of males and females, and a minimum of 2;; members from ea!h of the three ra!eAethni! "roups to be !ompared (.hites, Afri!an Ameri!ans, and Cispani!s) were desired 1amplin" at ea!h site was !ondu!ted to assure an e>ual "ender split within ea!h ra!eA"ender "roup insofar as possible These tar"et sample si@es applied to pairs of boo8lets that differed only in the orderin" of the Part = (power test) items E$!ept for analyses of item position effe!ts in Part =, the data for ea!h pair of boo8lets were pooled in the item analyses A tar"et sample si@e of 7;; e$aminees per boo8let was set to a!hie&e samples of 1,;;; for ea!h pair of boo8lets The sub(e!ts for the 4tem Pretest phase were appli!ants of Employment 1er&i!e lo!al offi!es in the fi&e A*/P "eo"raphi! re"ions Ea!h A*/) tested appro$imately the same number of e$aminees, and to the e$tent allowed by re"ional demo"raphi! !hara!teristi!s, ea!h !enter obtained the same minimum subsample si@es for the re>uired ethni! and "ender "roups as spe!ified in the resear!h desi"n The total number of indi&iduals tested for this study was :,2'9 The sample was appro$imately =7 per!ent female and 77 per!ent male 1ample ethni! !omposition in per!enta"es was as follows# Afri!an Ameri!an, =; per!ent% Asian, ' per!ent% Cispani!, 15 per!ent% white, '7 per!ent% 6ati&e Ameri!an, 2 per!ent% and sub(e!ts !hoosin" the LotherM !ate"ory, 2 per!ent The mean a"e and edu!ation of the total sample were '= :1 and 12 32 years, respe!ti&ely The standard de&iation (years) was 12 29 for a"e and 2 92 for edu!ation Table 2?' shows the sample !omposition by boo8let, "ender, and ra!e for sub(e!ts with usable data on the Form +at!hin" test (Part 1) Data Colle%tion Pro%edures Ea!h sub(e!t !ompleted 1 of 13 item tryout boo8lets Time limits were set to enable the sub(e!ts to !omplete all items in one power test, and 27 per!ent of the items in one form of the three speeded tests Althou"h the time limits for the speeded tests were in!reased sli"htly abo&e operational time limits, the speeded nature of these tests was preser&ed 2ne half of the sub(e!ts !ompleted Form E, and the other half !ompleted Form F All sub(e!ts !ompleted one of the three speeded tests (Parts 1, 7, and 9), one of the four power tests (Parts 2, ', =, or 3), and an an!hor test (GATB Form A) for the power test The o&erall time limit ran"ed from 99 to 1;; minutes for ea!h of the ei"ht test boo8lets for ea!h test form as shown in Table 2?2 Four answer sheets were prepared to !onform to the four tests within ea!h boo8let All data were re!orded on an opti!al s!anner and read to dis8ettes The answer sheets and dis8ettes were submitted to PA*/) for analysis PA*/) also prepared data !olle!tion and submission instru!tions

Chapter 2. Development of GATB Forms E and F

Table 2*3 (tem Tr/out "am$le "i4e for Ea%# Test !oo1let Booklet 1R2 1 2 'R= ' = 7R3 7 3 9R5 9 5 :R1; : 1; 11R12 11 12 1'R1= 1' 1= 17R13 17 13 Total 1,2'' 79= 37: 1,13: 3;= 737 1,2;1 777 3=3 1,1:' 3;= 75: 1,1;2 799 727 1,112 775 77= 1,;13 77' =3' 1,1=; 731 79: Male 333 2:: '39 3== '1: '27 335 '12 '73 331 '': '22 3;7 '23 29: 32' ';; '2' 793 '1= 232 3;2 ';2 ';; Fe)0 739 297 2:2 727 257 2=; 7'' 2=' 2:; 7'2 237 239 =:9 271 2=3 =5: 275 2'1 ==; 2': 2;1 7'5 27: 29: 9,ite 7=5 2== ';= '53 1:= 1:2 '1: 1=7 19= =9: 2== 2'7 =;' 22' 15; '97 1:9 195 299 173 121 =1' 211 2;2 Bla#k ':' 157 2;5 =9' 2=1 2'2 75: 275 ''1 ==3 215 225 =;' 1:7 2;5 =79 21: 2'5 =3= 2== 22; ='7 2;7 2'; Hi p0 211 11; 1;1 213 127 :1 2;; 111 5: 2;= 1;9 :9 2;9 112 :7 2;9 1;= 1;' 1:3 1;9 5: 21= 1;= 11;

Test Administrator Training) Trainin" for !olle!tin" item pretest data was a!!omplished in two sta"es First, A*/P staff !ondu!ted a two?day train?the?trainer session to pro&ide pro(e!t lead staff from ea!h of the fi&e A*/)s with a !omprehensi&e o&er&iew of the pro(e!t and detailed instru!tions for trainin" staff who would !olle!t the data The lead staff were then responsible for !ondu!tin" similar trainin" sessions in their respe!ti&e "eo"raphi!al re"ions Trainin" was di&ided into ei"ht modules# +odule 1 An 2&er&iew of the E and F /e&elopment Pro(e!t +odule 2 E and F Pretest /ata )olle!tion 2&er&iew +odule ' Administration 2&er&iew +odule = Administerin" the GATB +odule 7 Before, /urin", and After the Testin" 1ession +odule 3 1pe!ifi! 4nstru!tions for GATB Tests +odule 9 Administerin" the GATB ? A Pra!ti!al E$er!ise +odule 5 )he!8in" in with the /ata )olle!tion )oordinator

+aterials needed for !ompletin" the trainin" modules in!luded# Trainer5s Guide' GATB 4tem Pretest Administration 6anual, sample test boo8lets, and sample answer sheets The Administration +anual was desi"ned to !ontain all information needed by data !olle!tion staff for !ompletin" trainin" and a!tual data !olle!tion )opies of the Trainer5s Guide and the GATB 4tem Pretest Administration 6anual are in!luded in Appendi$ E of the Technical *eport on the Development of GATB Forms E and F (+ellon et al , 1::3) 1i$teen test boo8lets and four separate s!annable answer sheet formats were re>uired to !olle!t data for the 13 test &ersions 4nstru!tions for mat!hin" ea!h of the four answer sheets to the appropriate test boo8let were pro&ided in the GATB 4tem Pretest Administration +anual The Administration +anual instru!tions also in!luded a !he!8list of all supplies and materials needed for data !olle!tion PA*/) pro&ided all materials e$!ept s!rap paper, pen!ils, and stop wat!hes to ea!h A*/) 6A*/) de&eloped and distributed s!annin" software and instru!tions /ata were !olle!ted se>uentially for ea!h of the 13 test boo8lets, be"innin" with Boo8let 6o 1 .here possible, within ea!h re"ion, tar"eted sample si@es for the si$ sub"roups (Afri!an Ameri!an female, Afri!an Ameri!an male, Cispani! male, Cispani! female, white female, and white male) and the total sample si@e for ea!h boo8let were obtained before be"innin" data !olle!tion on ea!h su!!eedin" boo8let This was done to allow for preliminary re&iews of data before the entire data !olle!tion effort was !ompleted 1pe!ifi! instru!tions for !olle!tin", re!ordin", and submittin" data were pro&ided in the Administration +anual Data Anal/sis Analyses of the power and speeded test items pro!eeded somewhat independently due to differen!es in the nature of the data to be analy@ed and the information re>uirements for s!reenin" and form !onstru!tion The analysis of items for GATB power tests was done under separate !ontra!ts with +easured Pro"ress, C*1trate"ies, and Ooun" and Asso!iates .innie Ooun" performed preliminary edits and produ!ed !lassi!al item analysis statisti!s /r 6eal Kin"ston of +easured Pro"ress e&aluated the dimensionality of the power tests and performed !omputer analyses to estimate 4tem *esponse Theory (4*T) parameters, in addition to !ondu!tin" a preliminary sele!tion of items C*1trate"ies used this information, in !on(un!tion with their own analyses for assessin" differential item fun!tionin" (/4F), to sele!t the final items for the new forms (C*1trate"ies, 1::=) /r Frit@ /ras"ow pro&ided te!hni!al ad&i!e to C*1trate"ies staff The Cuman *esour!es *esear!h 2r"ani@ation (Cum**2) analy@ed speeded test data and sele!ted items for the four speeded tests (+!)loy, *ussell, Brown, /iFa@io, & Green, 1::=) /r Bert Green pro&ided te!hni!al ad&i!e to Cum**2 staff 6ote that the )omputation test items were in!luded in both sets of analyses As noted earlier, this test was ori"inally in!luded with the power tests but subse>uently treated as a speeded test The pro!edures used to estimate item statisti!s for (i e , !alibrate) the power test items and to s!reen them for fairness to different e$aminee "roups are des!ribed in the ne$t se!tion This is followed by a des!ription of the pro!edures used with the speeded test items and a dis!ussion of how item statisti!s were used to sele!t items for the final &ersion of ea!h Form E and Form F test

Chapter 2. Development of GATB Forms E and F

Calibration and "%reening of Po-er Test (tems After the raw data were edited, analysis of the tryout data for the power test items be"an with attention to two methodolo"i!al issues The first issue was whether there were si"nifi!ant differen!es in apparent item diffi!ulty (and to a lesser de"ree, item dis!rimination) as a fun!tion of the itemEs position in the tryout boo8let *e!all that ea!h new item was tried out in two boo8lets, with the orderin" of the new items in the e&en?numbered boo8lets bein" re&ersed from their orderin" in the ne$t lower odd?numbered boo8let The >uestion was whether it was reasonable to !ombine the data from the two boo8lets into a sin"le analysis of item statisti!s The se!ond issue was whether all of the items for a test measured a sin"le underlyin" !onstru!t Be!ause 4*T models (whi!h are !ommonly used in item s!reenin" and in sele!tin" items for in!lusion in a test) assume unidimensionality, it was important to !he!8 this assumption before pro!eedin" 2n!e these two issues were addressed, PA*/), +easured Pro"ress, and C*1trate"ies estimated item statisti!s and analy@ed /4F a!ross ethni! and "ender "roups At the !on!lusion of these steps, a few items had been dropped be!ause of /4F% the remainder of the items were !alibrated and ready for sele!tion into the final forms Ea!h of these steps is des!ribed briefly here and dis!ussed in more detail in +ellon et al (1::3) (tem Position Effe%ts) The effe!t of an itemEs position on its apparent diffi!ulty was assessed by !omparin" diffi!ulty estimates from the forward and re&ersed orderin" of the new items for ea!h test Table 2?= summari@es the distribution of differen!es in item diffi!ulty (proportion !orre!t) a!ross the two boo8lets for ea!h test Althou"h the results showed a si"nifi!ant !orrelation between item position and proportion passin", the si@e of the differen!es was "enerally modest The )omputation test showed the lar"est item position effe!ts, whi!h was !onsistent with !on!erns that the test was partially speeded )omputation was subse>uently treated as a speeded test +easured Pro"ress and PA*/) de!ided that !ombinin" data from the two item orderin"s was the best approa!h to minimi@e the effe!ts of item position The pooled data were used in the remainin" analyses Test Dimensionalit/) /r 6eal Kin"ston of +easured Pro"ress analy@ed the dimensionality of ea!h power test usin" the TE1TFA)T pro"ram (.ilson, .ood, & Gibbons, 1::1) Appli!ation of ordinary linear fa!tor analysis pro!edures to di!hotomously s!ored item &ariables typi!ally yields e$traneous diffi!ulty fa!tors TE1TFA)T lar"ely !orre!ts this problem by in!orporatin" a lo"isti! model of the relationship between the underlyin" fa!tors and the item responses
Table 2*5 Effe%t of C#anging (tem Position on (tem Diffi%ult/ Te t Arit,)eti# Co)putatio" 3:D Spa#e Vo#a5ulary Stati ti# Rea o"i"4 +inimum /ifferen!e :0+' :0+; :02+ :0+1st Suartile /ifferen!e :0.' :0.' :0.' :0.3 +edian /ifferen!e :0.+ 0.+ 0.. :0.+ 'rd Suartile /ifferen!e 0.0.< 0.3 0.3 +a$imum /ifferen!e 0+' 023 0+< 03< rpd :022 :0<2 :03+ :0-2 6otes The '9 ma$imum differen!e in diffi!ulties was found for <o!abulary item 1 (in the forward order), whi!h was an e$treme outlier The ne$t lar"est differen!e for <o!abulary was 11 rpd is the !orrelation between initial item position and !han"e in diffi!ulty

Form A items for ea!h test were analy@ed first *esults showed a stron" first fa!tor for ea!h test The )omputation test had the stron"est and most interpretable se!ond fa!tor, whi!h !ontrasted di&ision and other items Be!ause )omputation was subse>uently treated as a speeded test, 4*T analyses were not used with this test For the remainin" three tests, the first fa!tor a!!ounted for rou"hly half of the total &arian!e and the remainin" fa!tors were small and lar"ely uninterpretable 2ne of the '?/ 1pa!e fa!tors appeared to be related to diffi!ulty, but this was most li8ely an artifi!ial result not fully eliminated by the use of TE1TFA)T )onse>uently, the Arithmeti! *easonin", '?/ 1pa!e, and <o!abulary tests were (ud"ed suffi!iently unidimensional to support 4*T analyses (tem Calibration) Table 2?7 shows the number of e$aminees used in the analysis of ea!h of the power tests after editin" and poolin" a!ross boo8let pairs +easured Pro"ress used the B4-2G pro"ram (1!ientifi! 1oftware, 1::;) to estimate both !lassi!al item parameters (proportion passin" and item?total biserial !orrelation) and 4*T parameters Twel&e items were found to ha&e biserial !orrelations less then 17 and were eliminated from the 4*T analyses Parameter estimates (diffi!ulty, slope, and "uessin") for the three?parameter lo"isti! 4*T model were obtained for the remainin" items 4tem fit (in!ludin" item plots with fit probabilities less than ; ') was e$amined, but no other items were eliminated on the basis of item statisti!s 4n "eneral, the strate"y was not to s!reen out items alto"ether solely on the basis of their statisti!s, but rather to a&oid sele!tin" items for the final forms that were too easy or diffi!ult or had mar"inal dis!rimination unless no better items !ould be found
Table 2*6 A Des%ri$tion of t#e "am$les b/ Et#ni% and Te t Arithmeti! *easonin" <o!abulary Three?/imensional 1pa!e 9,ite 579 75' 3:: ender "ubgrou$s A%ri#a" A)eri#a" 9;; :19 9=1 Hi pa"i# '3= '32 '5: Male 1,1;2 1,127 1,1;7 Fe)ale :=3 :;; 59:

Differential (tem Fun%tioning 7D(F8) /4F refers to a situation where the probability of passin" an item differs for indi&iduals at a "i&en le&el of true ability who differ only on the basis of "roup membership (typi!ally defined by ethni! or "ender distin!tions) C*1trate"ies used two approa!hes to analy@in" the item tryout data for possible /4F 1eparate analyses were !ondu!ted !omparin" Afri!an Ameri!ans to .hites, Cispani!s to .hites, and females to males usin" 4*T? based approa!hes (*a(u, /ras"ow, & 1linde, 1::') and a +antel?Caens@el statisti! (Colland & Thayer, 1:55) that does not re>uire 4*T parameter estimation Ea!h of these approa!hes is des!ribed briefly here, followed by a summary of the s!reenin" de!ision that resulted from the /4F analyses +ellon et al (1::3) pro&ides more detail on both the appli!ation of the different pro!edures and the results 4*T approa!hes in&ol&e estimatin" separate item !hara!teristi! !ur&es (4))s) for the two "roups bein" !ompared 4))s "i&e the probability of a !orre!t answer as a fun!tion of true ability 4))s !ould not be estimated for some items in some ethni! "roups The lar"est diffi!ulty was with <o!abulary items in the Cispani! samples These items were eliminated from further /4F analyses (they were !onsidered, howe&er, for in!lusion on Forms E and F) For the remainin" items, the ES0ATE pro"ram (Ba8er, Al?Karni, & Al?/osary, 1::1) was used to ad(ust the underlyin" ability s!ales for true differen!es in the two samples bein" !ompared usin" the e>uatin" method de&eloped by 1to!8in" and -ord (1:5') /ifferen!es in the 4))s for the two "roups bein" !ompared were summari@ed in terms of (a) the area between these two !ur&es after the lower asymptote ("uessin" parameter) was !onstrained to be the same, and (b) a !hi?s>uare statisti! "i&en by -ord (1:5;) The two statisti!s a"reed !losely, and the !hi?s>uare statisti! was used in subse>uent item s!reenin"

Chapter 2. Development of GATB Forms E and F

The +antel?Caens@el estimates /4F as a ratio of the probability of passin" the item for members of the two "roups that is assumed to be !onstant a!ross different ability le&els 4tems ha&e /4F to the e$tent that this odds ratio differs from one (or the lo" of the ratio differs from @ero) 2bser&ed number !orre!t s!ores are typi!ally used to e>uate ability le&els for indi&iduals within ea!h ethni! or "ender "roup, and proportion passin" statisti!s are analy@ed within ea!h s!ore le&el 1oftware written by Cambleton and *o"ers (1::') was used to "enerate +antel?Caens@el statisti!s for ea!h of the power test items The primary "oal of the /4F analyses was to identify those items ha&in" !learly aberrant /4F 4tems were fla""ed if /4F statisti!s were si"nifi!ant at pT ;1 usin" either of the two pro!edures After inspe!tion of the fla""ed items, 17 items were dropped for e$treme /4F &alues 2ne of these items, a Form A &o!abulary item, had been fla""ed for /4F in the Cispani! sample by the 4*T pro!edure but not the +antel?Caens@el pro!edure All other items had been fla""ed by both pro!edures Table 2?3 shows the number of items s!reened out by the /4F analyses as well as by other fa!tors and the number of items that remained for possible use in the final forms That only items with e$treme /4F were dropped refle!ted a strate"y of !ontrollin" /4F at the test le&el by balan!in" items with positi&e and ne"ati&e /4F, rather than !ontrollin" ti"htly at the item le&el Eliminatin" items with mar"inal /4F !ould inad&ertently narrow the !ontent of the test, perhaps redu!in" its &alidity
Table 2*9 (tem "%reening Results for t#e Po-er Tests Te t S#ree" Total E & F items tried out U dropped for biserial T 17 U e$!luded from 4*T /4F analyses U dropped for /4F (Afri!an Ameri!an?.hite) U dropped for /4F (Cispani!?.hite) U dropped for /4F (Female?+ale) Total new items remainin" Arit,)eti# Rea o"i"4 33 1 1 ; ; 1 3' 3:D Spa#e Vo#a5ulary 13; ' 1; = ; ' 1=; 1'; 5 9 ; ; ; 122

Calibration and "%reening of "$eeded Test (tems 4tem analysis pro!edures for speeded tests are !onsiderably less standardi@ed relati&e to power test item analysis pro!edures 4n some models, where all of the emphasis is on assessin" speed rather than a!!ura!y, traditional !on!epts of diffi!ulty and dis!rimination do not apply .ithout !omputer administration, howe&er, it is not possible to re!ord response laten!ies on an item?by?item basis and so analyses of speed at the item le&el are not feasible For the speeded tests on the GATB, there is some e&iden!e that items do &ary in terms of diffi!ulty, with response errors bein" more !ommon for some items than for others 4t was therefore essential that measures of item diffi!ulty and dis!rimination be analy@ed so the new forms !ould be balan!ed with respe!t to these statisti!s Gi&en the importan!e pla!ed on test fairness, analyses of /4F a!ross "ender and ethni! "roups were also deemed essential A spe!ial item analysis pro"ram was de&eloped to !ompute the proportion passin" (diffi!ulty) and point biserial !orrelation (dis!rimination) for ea!h item usin" only those e$aminees who rea!hed the item 4f the proportion passin" an item were based on all e$aminees, items toward the end of a speeded test would automati!ally ha&e low &alues, be!ause &ery few e$aminees rea!h that point in the test and attempt to answer the item Althou"h only items in the first blo!8 of ea!h

speeded test were analy@ed, the relationship of item position to the number passin" the item was substantial )omputin" the proportion passin" only for those e$aminees who rea!hed the item "reatly redu!ed the artifi!ial !onfoundin" of an itemEs diffi!ulty estimate and its position in the tryout boo8let The relationship between item position and point biserials !omputed on all e$aminees was e>ually substantial For e$ample, for the 2b(e!t +at!hin" test, the !orrelation between item position and point biserials a&era"ed :1 a!ross the ei"ht item blo!8s when all e$aminees were in!luded in the !omputation of the point biserials .hen e$aminees who did not rea!h the item in >uestion were e$!luded, howe&er, the a&era"e !orrelation de!reased to 37 The spe!ifi! item and test statisti!s !omputed for the speeded tests in!luded# D 6ean and standard deviation of the total score on the analy:ed items$ D 6ean and standard deviation of the item!corrected total test scores$ The item?!orre!ted total test s!ore is the total test s!ore e3 ludin" the item in >uestion D 4tem difficulty The diffi!ulty of an item is assessed in !lassi!al test theory by the proportion of e$aminees who answered the item !orre!tly Be!ause this !omputation was based only on those e$aminees who rea!hed the item in >uestion, the results are referred to as L!orre!ted diffi!ulty &alues M D Point )iserial correlations Point biserial !orrelations are an inde$ of item dis!rimination in !lassi!al test theory Point biserials were !omputed based on all e$aminees and also based only on the e$aminees rea!hin" the item in >uestion (!orre!ted point biserials) 2nly the !orre!ted point biserials were used in the item s!reenin" and form !onstru!tion analyses D *esponse!alternative information For ea!h item response alternati&e, the proportion of e$aminees who endorsed that alternati&e, the mean !orre!ted total test s!ore for those e$aminees, and the point biserial !orrelation between the response alternati&e and the !orre!ted total test s!ore were !omputed The statisti!s were also !omputed separately for separate e$aminee "roups, in!ludin" females, males, Afri!an Ameri!ans, Cispani!s, and whites Be!ause total s!ores are based more on speed than on a!!ura!y, the models for /4F that "roup e$aminees on the basis of total s!ore and loo8 for differen!es in item passin" rates within total s!ore le&els do not appear appli!able 4n the present analyses, /4F was analy@ed in terms of effe!t si@es !omputed as the standardi@ed differen!e in !orre!ted passin" rates (i e , proportion !orre!t !al!ulated only on those indi&iduals who rea!hed the item) between ea!h fo!al "roup (females, Afri!an Ameri!ans, or Cispani!s) and the !orrespondin" referen!e "roup (males or whites) Althou"h this strate"y !an !ause problems when applied to power tests (e " , lower reliability and &alidity of the resultin" test), the approa!h is reasonable when sele!tin" items for hi"hly speeded tests, be!ause few if any substantial "roup differen!es are typi!ally en!ountered for su!h tests 4tem sele!tion for the )omputation subtest was supplemented by e$aminin" +antel?Caens@el /4F statisti!s !omputed by C*1trate"ies 4tem response alternati&e data were also e$amined, primarily to identify &ery poor items (e " , those with positi&e point biserials for the distra!tor% a distra!tor endorsed by e$aminees whose mean test s!ore e$!ludin" that item was hi"her than the mean s!ore for those who !orre!tly responded to the item) 6o items were e$!luded for this reason E$perimental items for the )omputation test were tried out in two item orders# forward and re&ersed )orrelations between item diffi!ulty estimates from the odd numbered boo8lets (forward dire!tion) were !orrelated with the diffi!ulty estimates obtained from the !orrespondin" e&en numbered boo8lets (re&erse dire!tion) The !orrelations were 9: (for boo8lets 1 and 2) and 51 (for boo8lets : and 1;), su""estin" that item diffi!ulties are hi"hly similar, re"ardless of order of

Chapter 2. Development of GATB Forms E and F

presentation )onse>uently, item data were pooled a!ross the !orrespondin" boo8lets in the !omputation of item statisti!s To minimi@e the ne"ati&e impa!t on sub"roups, items e&iden!in" a sub"roup effe!t si@e "reater than ; 27 (that is, one?>uarter standard de&iation) were remo&ed from further !onsideration 4tem response alternati&e data were also e$amined for the presen!e of distra!tors with positi&e point biserials, or mean item?!orre!ted total test s!ores that were (a) based upon a sample si@e of at least ten or more, and (b) hi"her than the mean item?!orre!ted total test s!ores for those who responded !orre!tly to the item 6o items were remo&ed from the 2b(e!t +at!hin" and 6ame )omparison tests for these reasons Ei"hteen e$perimental )omputation items were fla""ed by the /4F analysis 2f these, ei"ht were in!luded in the proposed Forms E and F (si$ on Form E, two on Form F) 4n no instan!e did the sele!ted items possess effe!t si@es of "reater than one?half standard de&iation Table 2?9 details the !hara!teristi!s of the items fla""ed by the /4F analysis

"ele%tion of (tems for Final Versions of Forms E and F

4tems for Forms E and F power tests were sele!ted by first !reatin" information "raphs for the base form (A) and draft new Forms E and F 4nformation "raphs were !al!ulated for the thetas between ?2 and 2 4nformation for Form A was !al!ulated by summin" the information for all items, and then s!alin" down to the len"th of the new forms For e$ample, Form A <o!abulary had =; items Forms E and F will ha&e 1: items Thus, at e&ery theta, information for <o!abulary Form A was multiplied by 1:A=; 4n desi"nin" the new Forms E and F, an attempt was made to impro&e measurement si"nifi!antly o&er Form A between thetas of R1, where appro$imately 9; per!ent of e$aminees s!ore, and where most de!isions are li8ely to be made
Table 2*: "ummar/ (tem "tatisti%s for Com$utation (tems Flagged b/ t#e D(F Anal/sis Su5: 4roup p:&alue ,o(i"4 D7F E 12 Add .?C 5;= E 13 +ult +?F 9:9 E 21 /i& +?F 592 E 2' +ult .?B 5== G E 23 Add +?F ?? E '; Add .?C :;3 E '3 1ub .?B 33: E =; Add .?C :;5 E 3; /i& +?F 59' E 37 /i& +?F 93' E 39 /i& .?B ''3 F 9 /i& .?B '79 F 5 /i& +?F 5'9 F 1: 1ub .?C 9=3 F 2: Add .?C 92= F '1 1ub .?B 39; F =2 /i& .?C 922 F =9 1ub +?F 3'5 G Form E item 23 was misprinted in one test boo8let 4t was omitted Ori4i"al GATB For) Ori4i"al 7te) 8 7te) Type Corre#te! Poi"t Bi erial '25 ''' '32 '19 ?? 2'; ==: 2=3 231 '11 22; ':= ';9 =2= ==' =5' =2' ''9 E%%e#t Si=e ? 1;= ? 2=' ;=; ? ;;2 ?? ? 131 =37 ? ;': 1;7 ? '11 ;== 773 ;'1 ? ;=5 ? 1;; 729 2:5 ? ;33 Ne( For) E 6A E F ?? E E E 6A 6A 6A 6A F 6A 6A 6A 6A E Ne( 7te) 8 21 6A = 11 ?? : '= 7 6A 6A 6A 6A 5 6A 6A 6A 6A '=

"ele%tion of Po-er (tems The analyses and report of power test item sele!tion were !ondu!ted by C*1trate"ies E$!erpts from their report of pro!edures and results (C*1trate"ies, 1::=) are in!luded here Pre&ious wor8 !ondu!ted by +easured Pro"ress had produ!ed draft forms for all three power tests These forms were preliminary, howe&er, in that /4F analyses had not been !ompleted These draft forms were used as a startin" point in draftin" the new Forms E and F of the power tests usin" the full pool of items that passed the s!reenin" for /4F 4tems were sele!ted in a!!ordan!e with professionally a!!epted pra!ti!es (see Cambleton & 1waminathan, 1:57% -ord, 1:99) to meet the followin" ob(e!ti&es# D The new Forms E and F power tests must be fair for !andidates of all ba!8"rounds D The new Forms E and F power tests must be as parallel as possible D The new Forms E and F power tests should impro&e upon the measurement properties of Form A D To the e$tent possible, the new Forms E and F power tests should pro&ide measurement that is uniformly pre!ise o&er a broad ran"e of abilities 4tem sele!tion pro!eeded as follows# D The draft forms !reated by +easured Pro"ress ser&ed as a startin" point These initial forms were based on a desire to ha&e informati&e (i e , dis!riminatin") items that spanned the ran"e of item diffi!ulties 4n a!tuality, there were many informati&e, diffi!ult items and few informati&e, easy items Therefore, item sele!tion fo!used on the need to pro&ide ade>uate pre!ision below theta J ?1 D Any item dis!arded durin" the /4F s!reenin" was deleted from the form D 0sin" software written in Pas!al, item information fun!tions were !omputed at 1' theta points for all items in the pool, separately by "ender and ethni! "roup The number of theta points used appeared more than ade>uate for !omparin" the smooth information !ur&es obser&ed in the present resear!h D 4nformati&e items were sou"ht to repla!e those deleted as a result of the /4F analyses D The test information fun!tions were then !omputed for Forms A, E, and F as the sum of the item information fun!tions The use of a spreadsheet fa!ilitated !omputation and "raphin" of the test information fun!tions for &arious !hoi!es of items The abo&e steps were iterated when !omparisons between forms su""ested a la!8 of !orresponden!e between E and F, or when the measurement properties of A were superior to those of E andAor F for some ran"e of abilities 1ome of the items in!luded in the forms may ha&e been asso!iated with small to moderately lar"e /4F statisti!s durin" s!reenin" Therefore as a final !he!8, measurement e>ui&alen!e for ethni! and "ender "roups was !he!8ed at the test le&el Assessment of Test*Le'el E;ui'alen%e) The final step in forms de&elopment was to assess the e>ui&alen!e of the forms on a total test?s!ore le&el to ensure that e$pe!ted s!ores of e>ually able persons were essentially the same 4t was !ru!ial to ma8e this last, test?le&el !omparison be!ause de!isions will be made on the basis of the total test s!ore The pro!edure for this !omparison was to
'

'

Fair, as it is used here, means that indi&iduals with the same le&el of ability ha&e the same e$pe!ted s!ore

Chapter 2. Development of GATB Forms E and F

D 0se the final linear lin8in" parameters !omputed pre&iously to mat!h ea!h fo!al "roup ability metri! to the referen!e "roup metri! (1to!8in" & -ord, 1:5') usin" ES0ATE (Ba8er, Al?Karni, & Al?/osary, 1::1)% and D Plot the T))s for ea!h "roup on a !ommon metri! The plots of the e$pe!ted number !orre!t differen!es for the final forms are presented in Fi"ures 2?1 throu"h 2?' for the final Arithmeti! *easonin", <o!abulary, and Three?/imensional 1pa!e tests, respe!ti&ely These differen!e plots subtra!t the referen!e "roup e$pe!ted s!ore from the fo!al "roup e$pe!ted s!ore so that re"ions where the lines lie abo&e @ero indi!ate that the fo!al "roup is ad&anta"ed% the distan!e abo&e or below @ero indi!ates the de"ree of ad&anta"e or disad&anta"e for the fo!al "roup, respe!ti&ely, in raw s!ore points 4n "eneral, the "raphs show that de&iations from perfe!t fairness are small 4n no !ase was the differen!e in e$pe!ted s!ores "reater than half a raw s!ore point Further, differen!es did not !onsistently fa&or either the referen!e or fo!al "roups% the lines lie abo&e and below @ero for all !omparisons As a result, it was !on!luded from these analyses that the two forms of ea!h of the three power tests are fair for Afri!an Ameri!ans, Cispani!s, and whites, and for males and females The draft forms were modified three times to !reate parallel, pre!ise, and fair instruments Ea!h modifi!ation was e&aluated by de&elopin" test information fun!tion "raphs whi!h were e$amined and dis!ussed /is!repan!ies were obser&ed between the e$pe!ted s!ores of some referen!e and fo!al "roups for some forms Therefore, the ob(e!ti&e of the first set of modifi!ations was to ensure that the forms were fair This was a!!omplished by deletin" items e$hibitin" /4F in undesirable dire!tions and addin" items showin" either no /4F or /4F in the desired dire!tion For e$ample, if one form showed ad&anta"e for males near thetaJ;, then one or more items with /4F fa&orin" males near @ero would be deleted and repla!ed with items showin" no /4F or showin" /4F fa&orin" females in that ran"e

Di%%ere"#e 5et(ee" t,e e$pe#te! "u)5er #orre#t #ore o% %o#al a"! re%ere"#e 4roup )e)5er 0 Poi"t a5o&e =ero i"!i#ate a!&a"ta4e %or t,e %o#al 4roup0

Figure 2*+) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for Arit#meti% Reasoning

Di%%ere"#e 5et(ee" t,e e$pe#te! "u)5er #orre#t #ore o% %o#al a"! re%ere"#e 4roup )e)5er 0 Poi"t a5o&e =ero i"!i#ate a!&a"ta4e %or t,e %o#al 4roup0

Figure 2*2) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for Vo%abular/

Di%%ere"#e 5et(ee" t,e e$pe#te! "u)5er #orre#t #ore o% %o#al a"! re%ere"#e 4roup )e)5er 0 Poi"t a5o&e =ero i"!i#ate a!&a"ta4e %or t,e %o#al 4roup0

Figure 2*3) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for T#ree*Dimensional "$a%e

2:

The resultin" forms pro&ided less test information than (i e , were not as pre!ise as) the ori"inal forms and in some re"ions were somewhat less pre!ise than Form A Thus, a se!ond round of modifi!ations was made to ma8e the forms more pre!ise without sa!rifi!in" fairness This re>uired addin" a small number of Form A items to the final &ersions of Forms E and F For Arithmeti! *easonin", the new Forms E and F !ontain two and one items from an!hor Form A, respe!ti&ely For Three?/imensional 1pa!e, Forms E and F !ontain two items ea!h from Form A 6o Form A items were added to the <o!abulary tests The third set of modifi!ations was performed to erase a moderate disparity between the e$pe!ted s!ores of Afri!an Ameri!ans and whites on one of the forms of Arithmeti! *easonin" The resultin" forms were fair within the limits imposed by the pool of items 1ome !ompromise had been made in terms of pre!ision to ha&e e>uitable forms )ompared to the "ains in e>uity, howe&er, the loss of pre!ision was small, and the new forms are "enerally !omparable to Form A The de"ree of differen!e between referen!e and fo!al "roups is rou"hly !omparable to that found by /ras"ow (1:59) for the A)T En"lish 0sa"e and +athemati!s 0sa"e tests Measurement Pro$erties of t#e Final Forms) The measurement properties of ea!h iteration of Forms E and F, as well as the an!hor Form A, were e$amined from se&eral perspe!ti&es, in!ludin" inspe!tion of three types of "raphs based on parameters estimated in the total sample First, !ur&es displayin" test information at ea!h of 1' theta le&els were e&aluated 1e!ond, the test information !ur&es of Forms E and F were "raphed alon" with "raphs showin" the ratio of Form E or F information relati&e to Form A information at ea!h of these points 4n addition, be!ause test information may not be easily interpretable to all readers, a third "raphi! based on a heuristi! pro!edure was used to pro&ide estimates on a reliability s!ale For the absolute and relati&e information "raphs, the information for a hypotheti!al, shorter Form A was estimated by multiplyin" by the number of items on Forms E and F di&ided by the number of items on Form A For e$ample, Form A of the Three?/imensional 1pa!e test had =; items, whereas Forms E and F had 2; To ma8e the forms !omparable, the test information plotted for Form A was multiplied by 2;A=; J ; 7; at ea!h theta point prior to plottin" The absolute information "raphs for the final forms are pro&ided here as Fi"ures 2?= throu"h 2?3 The "raphs "enerally show that the new forms are about as informati&e as Form A, e$!ept for the <o!abulary test, where Forms E and F ha&e mu!h more information than Form A at low to moderate ability ran"es These differen!es are ma"nified in the *elati&e 4nformation "raphs presented in +ellon et al (1::3) 2n the *elati&e 4nformation "raphs, it is important not to o&er? interpret lar"e relati&e differen!es between !ur&es of small absolute information For e$ample, on pa"e 1;, the "raph shows that the new forms are more than 3; per!ent less informati&e than Form A at theta J R' The absolute amount of information for any form at theta J R', howe&er, is &ery low

';

Figure 2*5) Arit#meti% Reasoning Absolute (nformation

ra$#

Figure 2*6) Vo%abular/ Absolute (nformation

ra$#

Figure 2*9) T#ree*Dimensional "$a%e Absolute (nformation

ra$#

4n sum, the information "raphs demonstrate that the new Forms E and F are about as informati&e as Form A For Arithmeti! *easonin", the new forms are not >uite as pre!ise as Form A, parti!ularly o&er the ran"e ; T theta T 2 7

'1

Chapter 2. Development of GATB Forms E and F

"ele%tion of (tems for t#e "$eeded Tests

4tem Sele tion Strate"y. The statisti!s for items !onstitutin" the e$tant GATB forms su""est that the speeded test items assess somethin" other than simply a speed !omponent A test that assesses speed alone should !ontain only items with &ery hi"h p &alues The !urrent GATB forms !omprise items that demonstrate a ran"e of p &alues Therefore, pursuant to re!ommendations from the A*/P, items were sele!ted that were similar statisti!ally to !urrent GATB forms 1pe!ifi!ally, items were sele!ted that appro$imated the item diffi!ulty data !al!ulated from operational administrations of GATB Forms B and / These data were not used as the primary !riteria for item sele!tion% rather, the data from Forms B and / were used to pro&ide rou"h "uidelines for the item diffi!ulty distributions of the sele!ted items +ore strin"ent attempts to mat!h the item statisti!s of the sele!ted e$perimental items to those of the items from Forms B and / were deemed unwise, "i&en that the data from Forms B and / (a) were obtained se&eral years a"o, (b) refle!t a different sample of e$aminees, (!) were !olle!ted under different !ir!umstan!es (operational !onditions, as opposed to the !urrent e$perimental !onditions), and (d) appear to treat omitted items as missin" obser&ations rather than in!orre!t responses Be!ause of the differen!es between the data for the e$perimental items and those for Forms B and /, e$perimental item diffi!ulty distributions for the Form +at!hin", 2b(e!t +at!hin", and 6ame )omparison tests were e$amined and !ompared to the same distributions for the first ni items (ni J the number of e$perimental items that were analy@ed for test i) from the operational forms 4n all instan!es, the ran"es of diffi!ulties were appro$imately e>ui&alent For the )omputation test, !omparison of the item statisti!s of the e$perimental items to item statisti!s from a pre&ious form was a!!omplished by embeddin" an!hor items from GATB Form A within the e$perimental )omputation test The se!tion on item sele!tion for )omputation Forms E and F details a !omparison of the item statisti!s for the e$perimental items with the statisti!s for the Form A items These analyses pro&ide a more useful !omparison of the !hara!teristi!s of the sele!ted items to pre&ious GATB forms, be!ause the data des!ribe item responses from the same indi&iduals, under the same administration !onditions, at the same testin" time 4n "eneral, the approa!h used to sele!t items for ea!h of the speeded tests was to ma$imi@e the ability of the tests to dis!riminate between those hi"h on the attribute and those low on the attribute, while minimi@in" differential impa!t on &arious sub"roups For the )omputation test, /4F information (a list of items found si"nifi!ant throu"h +antel?Caens@el tests) was !omputed and re&iewed The se!tion des!ribin" the sele!tion of )omputation items details the use of these data The followin" se!tions in!lude a des!ription of the item sele!tion strate"ies adopted for ea!h of the four tests +ellon et al (1::3) presents tables that !ontain item analysis information for the items that were sele!ted, as well as all other e$perimental items (tem "ele%tion for t#e Ob&e%t Mat%#ing Test) 0sin" the statisti!s from the item analyses as a "uide, =2 e$perimental 2b(e!t +at!hin" items were sele!ted for ea!h new form 4tems were !hosen whi!h ma$imi@e dis!riminability and minimi@e differential impa!t on sub"roups Cen!e, items with lar"e point biserial !orrelations and small effe!t si@es were tar"eted To bolster the effe!t si@e data, "raphs of !orre!ted item diffi!ulty were produ!ed for the "ender and ethni! "roups 4n addition to these psy!hometri! !on!erns, e$perimental items were e$amined to ensure they represent the &arious !ontent spe!ifi!ations (e " , distribution of 8eyed responses,
=

Form +at!hin" is not dis!ussed in this se!tion, be!ause the de!ision to drop it was made before the analyses were !ondu!ted 7 +u!h of this se!tion was e$!erpted from +!)loy et al (1::=)

'2

number of shaded areas in the stimulus) as spe!ified in Test &pecifications for GATB Forms E and F (PA*/), 1::7) 4tems were sorted by item dis!rimination &alues Be!ause the !orrelation between item order and point biserial !orrelation did not &anish, items were sele!ted only if the ori"inal tar"et of :7 per!ent of the e$aminees rea!hed the item Cen!e, those items at the end of the test boo8lets often were not !onsidered, e&en thou"h they demonstrated si@able point biserials The possible influen!e of !areless responders on the point biserial &alues for these items !ould ha&e !ontributed to these lar"e &alues Therefore, the additional sele!tion !riterion was used C#ara%teristi%s of t#e Pro$osed Ob&e%t Mat%#ing Test) The proposed Forms E and F for the 2b(e!t +at!hin" test and their su""ested administration order are "i&en in Appendi$ B of +ellon et al (1::3) The forms are &irtually identi!al in their mean diffi!ulty ( 5:= and 5:', respe!ti&ely) and dis!rimination ( 299 and 23', respe!ti&ely) 4n addition, the effe!t si@e statisti!s indi!ated that the sele!ted items pro&ide little a""re"ate differential impa!t a!ross sub"roups These &alues were all low and similar a!ross test forms# For maleAfemale, whiteAAfri!an Ameri!an, and whiteACispani! !omparisons, the &alues were ?; ;32, ; 1;:, ; ;;5, for Form E% and ?; ;25, ; 1;1, and ; ;17 for Form F% respe!ti&ely (tem "ele%tion for t#e Name Com$arison Test) 0sin" the item statisti!s as a "uide, :; items were sele!ted for ea!h new form 4tems were !hosen to ma$imi@e dis!riminability and minimi@e differential impa!t on sub"roups Cen!e, items with lar"e point biserial !orrelations and small effe!t si@es were tar"eted 4n addition to these psy!hometri! !on!erns, the e$perimental items were e$amined to ensure they represent the &arious !ontent !onsiderations (e " , male and female names, Cispani! names, number of !hara!ters in the stimuli) as spe!ified in Test &pecifications for GATB Forms E and F (PA*/), 1::7) 2n!e sele!ted, the items were sorted by 8eyed response and !orre!ted diffi!ulty &alues, and the !ontent spe!ifi!ations of the items were tallied Althou"h the psy!hometri! >ualities of the initial proposed Forms E and F were satisfa!tory, the !ontent spe!ifi!ation tally indi!ated 2' male items appeared on Form E, whereas Form F !omprised only 1= Thus, items were shifted between forms, repla!in" the male items from Form F with indeterminant items from Form E 2n!e sorted into the final proposed forms, the items were a"ain sorted for ea!h form by 8eyed response and !orre!ted diffi!ulty 4tems were sele!ted to obtain =7 L1ameM items and =7 L/ifferentM items To determine the order of administration within ea!h proposed form, a table of random units was used to pro&ide a 8eyed response pattern for ea!h form +erely orderin" items from easiest to most diffi!ult was infeasible for the 6ame )omparison test, be!ause the 1ame items were far easier than the /ifferent items 2rderin" by diffi!ulty alone would ha&e resulted in all the 1ame items appearin" at the front of the test )onse>uently, the se>uen!in" of 1ame and /ifferent items was randomi@ed After "eneratin" the response pattern, the items were ordered so that the easiest item ha&in" the re>uired 8eyed response was sele!ted Thus, the item position represents an orderin" by item diffi!ulty (from easiest to most diffi!ult), !onditional upon the randomly "enerated 8eyed response pattern After e$amination of the test mo!8ups, howe&er, it be!ame apparent that some ad(ustments to item orderin" were re>uired (e " , four !onse!uti&e Form F items !ontained the abbre&iation L)o M) Two items on Form E and three items on Form F were e$!han"ed to eliminate this type of !ontent repetition *e"ardin" the !ontent !hara!teristi!s of the sele!ted 6ame )omparison items, !onsideration was "i&en to the number of !hara!ters in the stimulus name (the name on the left side of the pair) ran"in" from 3 to 27, the ethni! "roup the name represents (A J Asian, B J Afri!an Ameri!an, C J ''

Chapter 2. Development of GATB Forms E and F

Cispani!, 2 J 2ther), the "ender of the name (+ J male, F J female), whether it was a business (B) or personal (P) name, and the type of differen!e that o!!urred for dissimilar items (1 J in&erted letters, 2 J same soundAdifferent meanin", ' J different sound and letter, = J addition or deletion of a sin"le letter, and 7 J different word forms but similar meanin") +!)loy et al (1::=) and +ellon et al (1::3) in!lude more detailed information on the distribution of items a!ross these !ate"ories in the final forms An attempt was made to sele!t items that represent a fair distribution of the rele&ant !ontent !hara!teristi!s a!ross forms Althou"h there is less balan!e for !ertain !hara!teristi!s, "reater emphasis was pla!ed upon assurin" balan!e of the "enderAethni! name representation and the BusinessAPersonal !hara!teristi!s, in parti!ular The Type of /ifferen!e !hara!teristi! is less balan!ed a!ross proposed test forms, but the items fun!tion e>ui&alently from a psy!hometri! perspe!ti&e 6o items with LType of /ifferen!eM e>ual to L7M (i e , different word forms but similar meanin") were !hosen for the new test forms These items demonstrated lar"e differential impa!t to Asians 1pe!ifi!ally, Asians were disproportionately li8ely to miss these items that, althou"h stru!turally &ery different, mean the same thin" For e$ample, item =; from boo8s 9 and 5 has a whiteAAsian effe!t si@e of 1 125 The item is Nat<l Pane Com$an/ *** National Pane Com$an/ )learly, the elements of the item pair differ stru!turally% yet, they ha&e the same meanin" )onfusion of stru!tural similarity with !ontent similarity !reated numerous problems that are apparent in the data Be!ause of the substantial differential impa!t these items !ause, they do not appear on GATB Forms E and F The proposed Forms E and F are &ery similar, with mean !orre!ted diffi!ulty &alues of :21 and :'=, respe!ti&ely +ean dis!rimination &alues are &irtually identi!al ( ''9 and ''5, respe!ti&ely) 4n addition, the effe!t si@e statisti!s indi!ate that the sele!ted items pro&ide little a""re"ate differential impa!t a!ross sub"roups These &alues are all low and similar a!ross test forms# For the maleAfemale, whiteAAfri!an Ameri!an, whiteACispani!, and whiteAAsian !omparisons, the &alues are ? ;31, ;97, ;;:, and ;53 for Form E% and ? ;7=, ;:;, ;12, and ;95 for Form F, respe!ti&ely (tem "ele%tion for t#e Com$utation Test) For the )omputation test, the primary "oal was to sele!t 5; items with desirable psy!hometri! and !ontent properties for in!lusion in GATB Forms E and F (i e , =; items for ea!h form) The e$perimental &ersions of GATB Forms E and F in!luded )omputation test items in four boo8lets# Boo8lets 1 and 2 (Form E), and Boo8lets : and 1; (Form F) Form E and Form F ea!h had 35 e$perimental items .ithin ea!h form, items were presented in one order in one boo8 and in re&erse order in the se!ond boo8 of the same form (e " , Boo8let 1 item 1 was the last item in Boo8let 2) 4n addition, 7; items from operational Form A were embedded in all four boo8lets as an!hor items 4t is also important to note that the )omputation test in!ludes four ma(or types of items?? addition, subtra!tion, multipli!ation, and di&ision .ithin ea!h type, !ontent !hara!teristi!s of the items &ary, ma8in" some more diffi!ult than others For e$ample, some addition items in&ol&e simply addin" two sin"le?di"it numbers, whereas others re>uire addin" fi&e or si$ multi?di"it numbers The final &ersion of both Form E and Form F was to in!lude =; items, 1; of ea!h type of operation, and within ea!h type of operation, items should represent the ran"e of !ontent !hara!teristi!s as spe!ified in Test &pecifications for GATB Forms E and F (PA*/), 1::7) 4tems were !hosen to ma$imi@e dis!riminability and minimi@e differential impa!t on sub"roups Cen!e, items with lar"e point biserial !orrelations and small effe!t si@es were tar"eted

'=

Anal/sis of Form A An%#or (tems) The data for Form A items (whi!h were an!hor items on all forms) were !ompared with the e$perimental item data to see how !omparable they were in terms of item diffi!ulty and dis!riminability 4n "eneral, the entire pool of e$perimental items was sli"htly more diffi!ult than the pool of Form A items +ean diffi!ulties for Form A and e$perimental items respe!ti&ely were 5; and 95 for addition, 57 and 9: for subtra!tion, 9: and 97 for multipli!ation, and 91 and 33 for di&ision (+ellon et al , 1::3, Appendi$ -) The e$perimental items were somewhat less dis!riminatin" than the Form A items +ean !orre!ted point biserials for Form A and e$perimental items respe!ti&ely were# '1 and 2' for addition, '9 and '' for subtra!tion, =2 and '7 for multipli!ation, and =' and '5 for di&ision 4tems were sele!ted based on their psy!hometri! properties, and ra!e and se$ differen!es 4tems were pooled a!ross forms and or"ani@ed a!!ordin" to the type of arithmeti! operation 4tems with point biserials one or more standard de&iations below the mean point biserial for that type of operation were identified As mentioned earlier, +antel?Caen@el !hi?s>uare analyses !ondu!ted by C*1trate"ies (1::=) were also used to e$amine /4F 4tems that (a) were fla""ed by the /4F analysis and (b) showed more than half a standard de&iation differen!e between sub"roups, were not sele!ted Ei"hteen e$perimental items were fla""ed by the /4F analysis 2f these, ei"ht were retained (si$ on Form E, two on Form F) 4n no instan!e did the sele!ted items possess effe!t si@es of "reater than one half standard de&iation The pool of remainin" "ood items was sorted a!!ordin" to !ontent !hara!teristi!s, and the items were or"ani@ed a!ross forms After ma8in" an initial !ut a!ross forms, the mean item statisti!s were !omputed for ea!h new form To ma8e the forms as similar as possible (statisti!ally as well as in !ontent), some items were shifted from one form to the other, yieldin" the final set of proposed items for Forms E and F Table 2?5 summari@es the statisti!s for the sele!ted items of ea!h of the proposed forms 4tems were ordered as follows# (1) an addition item, (2) a subtra!tion item, (') a multipli!ation item, and (=) a di&ision item This se>uen!e of item types was repeated throu"hout the test 4tems were ordered a!!ordin" to diffi!ulty (within type of operation) For e$ample# 4tem U1 !ontained the easiest addition item, 4tem U7 !ontained the ne$t easiest addition item, , and 4tem U'9 !ontained the most diffi!ult addition item This pro!ess was repeated for ea!h type of item until all items were sele!ted
Table 2*= Com$utation Test (tem "ele%tion> "ummar/ "tatisti%s for "ele%ted Form E and Form F (tems
Form E (tem T/$e Addition 1ubtra!tion +ultipli!ation /i&ision Grand +ean F Addition 1ubtra!tion +ultipli!ation /i&ision Grand +ean Corre%ted Diffi%ult/ Mean "D ; 9: ; 9: ; 93 ; 3: ; 93 ; 5; ; 99 ; 97 ; 3: ; 97 ; '5 ; ': ; =; ; == ; =1 ; '5 ; ': ; =1 ; =' ; =; Point !iserials All (tems 2?O (tem '1 '7 =; =1 '9 '2 '5 =; =3 ': 2: '' '9 '5 '= 2: '3 '9 =' '3 M*F ?; ;5 ?; 1= ?; ;: ?; ;9 ?; ;: ?; ;9 ?; 11 ?; ;: ?; ;: ?; ;: Effe%t "i4e 2*! 2*H ; 21 ; '; ; 2= ; 25 ; 23 ; 2' ; 29 ; 2= ; '7 ; 29 ; ;2 ; 12 ; 17 ; 2; ; 12 ; ;: ; 12 ; 17 ; 1: ; 1=

'7

Chapter 2. Development of GATB Forms E and F

Forms E;uating "tud/

Forms E and F Test Tr/out Data Colle%tion Pro%edures The Test Tryout data !olle!tion in&ol&ed a nationwide sample of 5,:97 indi&iduals that was representati&e of the appli!ant populations of lo!al Employment 1er&i!e offi!es Parti!ipants were reimbursed for their tra&el e$penses The amount of reimbursement &aried by !ondition and "eo"raphi!al re"ion, but it was not dependent on test performan!e These sample data were used for e>uatin" new Forms E and F to base Form A The te!hni!al re>uirements and rationale for the pro!edures dis!ussed below are presented in detail in 1e"all and +on@on (1::7) Data Colle%tion Design The data !olle!tion desi"n is presented in three se!tions, ea!h se!tion !orrespondin" to one of the three primary samples in!luded in the GATB e>uatin" study The first se!tion des!ribes the data !olle!tion desi"n for the independent?"roups (4G) sample This sample was used to e>uate the new and old GATB forms The se!ond se!tion des!ribes the data !olle!tion desi"n for the repeated?measures (*+) sample This sample was used primarily for !omparin" the reliability and !onstru!t &alidity of the new and old forms Cowe&er, a portion of this sample was used as supplemental data for the e>uatin" analysis The third se!tion des!ribes the data !olle!tion desi"n for the psy!homotor (P+) sample This sample was used to e$amine the need for !omposite e>uatin"s, and to e$amine !onstru!t &alidity issues in&ol&in" the psy!homotor tests 4ndependent?Groups 1ample E$aminees in the independent?"roups sample were randomly assi"ned to one of three forms (i e , A, E, and F) As indi!ated in Table 2?:, a total of 7,5:2 e$aminees were tested Appro$imately e>ual numbers of e$aminees were tested on ea!h of the three forms (6 V 1,:3=) Table 2?: also displays the numbers of e$aminees tested on ea!h form at ea!h of the fi&e A*/)s A!ross ea!h of the fi&e A*/)s, there were appro$imately =; testin" sites At ea!h site, e$aminees were randomly assi"ned to test form (A, E, or F) The old (A) and new (E and F) forms of the GATB possess different test orderin", time limits, and instru!tions, thus !ompli!atin" the assi"nment of test forms to e$aminees )onse>uently, these &ersions !annot be administered to a sin"le "roup simultaneously They must be administered in different testin" sessions, where the sessions are separated physi!ally either by lo!ation (testin" room) or by time Cen!e, at a "i&en testin" site, one of two methods of assi"nment was used, dependin" on whether one or two testin" rooms were a&ailable# D At two?room sites, e$aminees were randomly assi"ned to Forms A, E, and F upon arri&al E$aminees assi"ned to Form A were tested in one room% e$aminees assi"ned to either of Forms E or F were tested in a se!ond room D At one?room sites, some sessions were dedi!ated to Form A, and other sessions were dedi!ated to the new Forms E and F All e$aminees at one?room sites were s!heduled for testin" prior to their arri&al at the test site At the time of s!hedulin", ea!h e$aminee was randomly assi"ned to one of the three forms (A, E, or F) 2n!e assi"ned to a spe!ifi! form, the e$aminee was "i&en a !hoi!e of se&eral test dates that had been dedi!ated to the assi"ned form

+u!h of this se!tion e$!erpted from 1e"all and +on@on (1::7)

'3

Table 2*@ (nde$endent* rou$s "am$le "i4es ARDC EA*/) 6A*/) 1A*/) PA*/) .A*/) Total Form A ==9 ='3 ';1 =;2 =77 2,;=1 Form E '9; '9; ''; '92 =73 1,5:5 Form F '5: =;1 ''= ':2 ='9 1,:7' Total 1,2;3 1,2;9 :37 1,133 1,'=5 7,5:2

Re$eated*Measures "am$le) E$aminees in the repeated?measures sample were administered two forms of the GATB These data were used primarily for e$aminin" the reliability and !onstru!t &alidity of the GATB Cowe&er, a portion of these data were also used to supplement the e>uatin" data These data were used to perform a detailed !omparison of measurement properties between the old and new forms Ea!h e$aminee parti!ipatin" in the repeated?measures portion of the study was randomly assi"ned to one of ei"ht !onditions These !onditions and the numbers of e$aminees in ea!h !ondition are presented in Table 2?1; 6ote that !onditions 1, 2, 9, and 5 !onsist of samples of appro$imately e>ual si@e (6 V =';) The remainin" !onditions listed in Table 2?1; also !onsist of appro$imately e>ual sample si@es (6 V 215) The rationale for the sample si@e re>uirement is outlined in the data analysis se!tion The numbers of e$aminees tested in ea!h !ondition at ea!h site are pro&ided in Table 2?11
Table 2*+A Re$eated Measures Design and "am$le "i4es "e%ond Test First Test A B E F A 2 (='2) 3 (217) =(213) 5 (==3) ! 1 (=11) E 7 (2;:) 9 (==3) F ' (2'3)

Table 2*++ Re$eated*Measures "am$le "i4es b/ Test "ite Condition ARDC EA*/) 6A*/) 1A*/) PA*/) .A*/) Total + =' 51 := 1;9 53 =11 2 =3 52 1;; :5 1;3 ='2 3 29 =1 72 3' 7' 2'3 5 22 =7 71 =: =: 213 6 2= =; =9 7' =7 2;: 9 27 =; =7 =: 73 217 : 7' 55 93 1;' 123 ==3 = =5 :' 99 1;5 12; ==3 Total 255 71; 7=2 3'; 3=1 2,311

At two?room sites, e$aminees were randomly assi"ned to the ei"ht !onditions upon arri&al At one?room sites, all e$aminees were s!heduled for testin" prior to arri&al This pro!edure ensured random assi"nment of e$aminees to !ondition

'9

Chapter 2. Development of GATB Forms E and F

Ps/%#omotor "am$le) This sample of 7'5 e$aminees re!ei&ed the fi&e psy!homotor tests alon" with the non?psy!homotor portions of Forms A and F The desi"n is presented in Table 2?12 E$aminees were randomly assi"ned to one of two "roups Ea!h "roup re!ei&ed three se!tions# (1) Form A (non?psy!homotor), (2) Form A (psy!homotor), and (') Form F (non?psy!homotor), with the order of presentation !ounterbalan!ed a!ross the two "roups As indi!ated in Table 2?12, Group 1 re!ei&ed Form A (non?psy!homotor) and Form A (psy!homotor) portions in the mornin" session, and Form F (non?psy!homotor) in the afternoon Group 2 re!ei&ed the same battery of tests with the order of the non?psy!homotor se!tions of Forms A and F re&ersed At two?room sites, e$aminees were randomly assi"ned to the two !onditions upon arri&al At one?room sites, all e$aminees were s!heduled for testin" prior to arri&al This pro!edure ensured random assi"nment of e$aminee to !ondition "am$le C#ara%teristi%s This se!tion pro&ides an e&aluation of the demo"raphi! !hara!teristi!s of ea!h of the three samples in!luded in this study# (1) the independent?"roups (4G) sample, (2) the repeated?measures (*+) sample, and (') the psy!homotor (P+) sample An e&aluation of the random e>ui&alen!e of sele!ted "roups within ea!h of the three samples is also pro&ided, be!ause the random e>ui&alen!e of these "roups is a 8ey assumption made in the e>uatin", reliability, and &alidity analyses
Table 2*+2 Ps/%#omotor Data Colle%tion Design rou$ + 7N B 2968 +ornin" 1 Form A (non?pmotor) 2 Form A (pmotor) Afternoon ' Form F (non?pmotor) ' Form A (non?pmotor) 1 Form F (non?pmotor) 2 Form A (pmotor) rou$ 2 7N B 2:38

This se!tion also summari@es the data editin" pro!edure used to remo&e unmoti&ated e$aminees and other hi"hly influential !ases from ea!h of the three samples The numbers of !ases remo&ed from ea!h sample are reported Demogra$#i%s and rou$ E;ui'alen%e) For ea!h of the three primary samples, statisti!al tests of the differen!es amon" randomly e>ui&alent "roups were !ondu!ted by "ender, ra!e, a"e, and edu!ation 6on?si"nifi!ant results are !onsistent with the e$pe!tation based on random assi"nment of e$aminees to !ondition and support the assumption of e>ui&alent "roups made in the e>uatin", reliability, and &alidity analyses All si"nifi!an!e tests of differen!es a!ross "roups by form (for the 4G sample) and by !ondition (*+ and P+ samples) yielded non?si"nifi!ant results with W J ;7 (!f 1e"all & +on@on, 1::7, for detailed results) The results of these analyses indi!ate di&erse samples with respe!t to "ender, ra!e, a"e, and edu!ation Furthermore, the si"nifi!an!e tests performed on the three samples (independent?"roups, repeated?measures, and psy!homotor) pro&ide reassuran!e that the assi"nment pro!edures wor8ed as intended, produ!in" "roups that are randomly e>ui&alent with respe!t to demo"raphi! !hara!teristi!s Althou"h the e>ui&alen!e of the "roups on !o"niti&e and psy!homotor abilities !annot be tested with e$istin" data, the results based on the demo"raphi! &ariables pro&ide additional !onfiden!e in this assumption, sin!e in some instan!es demo"raphi! and !o"niti&eApsy!homotor &ariables tend to be !orrelated Outlier Anal/sis) Prior to data analysis, a small number of !ases with unli8ely s!ores were deleted from the database )ases for deletion were identified usin" a pro!edure su""ested by '5

Cotellin" (1:'1), whi!h identifies !ases that are unli8ely "i&en that the obser&ations are sampled from multi&ariate ellipti!al?shaped distribution 1eparate outlier analyses were performed for the three samples (independent?"roups, repeated?measures, and psy!homotor) Furthermore, separate analyses were performed for ea!h "roup, within ea!h sample 6ote that it is possible for two types of patterns to be fla""ed and deleted by the !hosen pro!edure 2ne type o!!urs when an e$aminee re!ei&es e$treme s!ores on many tests (e " , all low s!ores) Another type of unli8ely pattern o!!urs when an e$aminee s!ores hi"h on one test and low on a se!ond that is hi"hly !orrelated with the first (i e , alternate forms of the same test) A small number of !ases with @ero number?ri"ht s!ores on one or more tests were also deleted, be!ause /2- poli!y di!tates that s!ores should not be pro&ided to su!h e$aminees (nde$endent* rou$s "am$le) Table 2?1' pro&ides the editin" results for the independent?"roups sample Cere, sele!ted !ases from the *+ sample were !ombined with the 4G sample 1pe!ifi!ally, data from the first test administered (Forms A, E, or F) of )onditions 1, ', =, 3, 9, and 5 were !ombined with the 4G data to in!rease the sample si@es for the e>uatin" study /ata editin" was performed in ea!h of the three 4GA*+ "roups# LA,M LE,M and LF M A few !ases were remo&ed from ea!h "roup, ran"in" from 1= to '= e$aminees The final "roup si@es used in the e>uatin" analysis are listed in the last row of the table
Table 2*+3 rou$ "i4es for t#e Edited (nde$endent "am$le 4G *+?1 (AAB) *+?' (AAF) *+?3 (EAB) *+?9 (EAF) *+?= (FAA) *+?5 (FAE) Total % 6umber /eletes Final % A 2,;=1 =11 2'3 rou$s?E;uating "am$le Form E 1,5:5 217 ==3 213 ==3 2,355 '= 2,37= 2,77: 1= 2,7=7 2,317 15 2,7:9 F 1,:7'

Re$eated*Measures "am$le) Table 2?1= displays the editin" results for the repeated?measures sample Between ' and 1' !ases were deleted from ea!h of the ei"ht "roups The final "roup si@es after editin" are displayed on the bottom row of the table
Table 2*+5 rou$ "i4es for t#e Edited Re$eated*Measures "am$le Fre>ue"#y Total % 6umber /eletes Final % + =11 1' ':5 2 ='2 1; =22 3 2'3 = 2'2 Sa)ple ' 213 2;: ' ' 21' 2;3 2 217 7 21; < ==3 12 ='= ; ==3 1; ='3

Ps/%#omotor "am$le) For the P+ sample, four !ases were deleted from Group 1, and @ero !ases were deleted from Group 2, yieldin" final "roup si@es after editin" of 231 e$aminees in Group 1 and 29' e$aminees in Group 2

':

Chapter 2. Development of GATB Forms E and F

"ummar/ Demogra$#i% Data) 1ummary demo"raphi! data for the three samples are presented in Tables 2?17 throu"h 2?19, and for the a""re"ate sample in Table 2?15 "%oring t#e AT!

A number of GATB s!ores are routinely produ!ed and used 1!orin" for the new Forms E and F is !ompli!ated by the absen!e of the Form +at!hin" test, and by the use of formula s!ores for the speeded tests The method of s!ore !omputation for the old and new GATB forms is detailed below This des!ription in!ludes the !omputation of both test and !omposite s!ores
Table 2*+6 Demogra$#i% Com$osition of t#e (nde$endent* rou$s?E;uating "am$le Group Total Group Gender Female +ale 6ot *eported *a!eAEthni! Group Afri!an Ameri!an Asian Cispani! .hite 6ati&e Ameri!an 2ther 6ot *eported A"e Group 17 13 ? =; Oears 2&er =; Oears 6ot *eported Edu!ation -ess than 3 3 ? 11 Oears 12 Oears 1' ? 17 Oears 13 Oears and 2&er 6ot *eported Sa)ple Si=e 9,9:3 ',752 =,2;: 7 2,:'1 2;7 :91 ',==7 117 112 19 1 7,22; 2,77; 27 2 1,=;1 ','31 1,:5= 1,;'; 15 Per#e"t o% Total Sa)ple 1;; =3 7= T1 '5 ' 12 == 2 1 T1 T1 39 '' T1 T1 15 =' 27 1' T1

=;

Table 2*+9 Demogra$#i% Com$osition of t#e Re$eated*Measures "am$le rou$ Total Group Gender Female +ale *a!eAEthni! Group Afri!an Ameri!an Asian Cispani! .hite 6ati&e Ameri!an 2ther 6ot *eported A"e Group 13 ? =; Oears 2&er =; Oears 6ot *eported Edu!ation -ess than 3 3 ? 11 Oears 12 Oears 1' ? 17 Oears 13 Oears and 2&er 6ot *eported "am$le "i4e 2,75: 1,;5= 1,7;2 1,15= 9: '77 593 =5 =' = 1,992 5;5 : 1 =73 1,15= 327 '17 5 Per%ent of Total "am$le 1;; =2 75 =3 ' 1= '= 2 2 T1 35 '1 T1 T1 15 =3 2= 12 T1

=1

Chapter 2. Development of GATB Forms E and F

Table 2*+: Demogra$#i% Com$osition of t#e Ps/%#omotor "am$le rou$ Total Group Gender Female +ale *a!eAEthni! Group Afri!an Ameri!an Asian Cispani! .hite 6ati&e Ameri!an 2ther A"e Group 13 ? =; Oears 2&er =; Oears Edu!ation 3 ? 11 Oears 12 Oears 1' ? 17 Oears 13 Oears and 2&er "am$le "i4e 7'5 22; '15 227 = '9 271 12 : '=9 1:1 95 2=2 17= 3= Per%ent of Total "am$le 1;; =1 7: =2 1 9 =9 2 2 3= '3 1= =7 2: 12

=2

Table 2*+= Demogra$#i% Com$osition of t#e Aggregate "am$le rou$ Total Group Gender Female +ale 6ot *eported *a!eAEthni! Group Afri!an Ameri!an Asian Cispani! .hite 6ati&e Ameri!an 2ther 6ot *eported A"e Group 17 13 ? =; Oears 2&er =; Oears 6ot *eported Edu!ation -ess than 3 3 ? 11 Oears 12 Oears 1' ? 17 Oears 13 Oears and 2&er 6ot *eported "am$le "i4e 5,:97 =,;3= =,:;7 3 ',==9 2'; 1,;5: ',:2= 1=; 129 15 1 7,::5 2,:=: 29 2 1,3;2 ',5:1 2,2:2 1,13: 1: Per%ent of Total "am$le 1;; =7 77 T1 '5 ' 12 == 2 1 T1 T1 39 '' T1 T1 15 =' 23 1' T1

Test "%oring) For the purpose of this study, all s!ores for Forms A and B were !omputed a!!ordin" to !on&entions spe!ified in the L+anual for the 01E1 General Aptitude Test BatteryM (1e!tion 4) Forms A and B The set of raw s!ores for Forms A and B, denoted by X>ar(A), >vo(A), , >di(A)% >ar(B), >vo(B), , >di(B)Y ,

are ta8en as the simple sum of the number of !orre!t responses for the power and speeded tests The raw s!ores for the fi&e psy!homotor tests are obtained a!!ordin" to the operational pro!edures 1tandard?s!ores 1 are obtained from a loo8?up table These standard s!ores are summed in &arious !ombinations to form the nine aptitude s!ores XAg Av' An' As' Ap' A,' A-' Af' AmY displayed in Table 2? 1: The !on&ersion of raw s!ore to standard s!ore depends on the aptitude for whi!h the test s!ores will be used For tests that enter into two different aptitude

='

Chapter 2. Development of GATB Forms E and F

Table 2*+@ A$titude "%ore Com$osition AT! Test Ag Arithmeti! *easonin" <o!abulary '/ 1pa!e )omputation 6ame )omparison 2b(e!t +at!hin" Form +at!hin" +ar8 +a8in" Pla!e Turn Assemble /isassemble &as &di
(f#

A$titude "%ore Av
(v#

An &ar
(n#

As

Ap

A,

A-

Af

Am

&ar(g# &vo(g# &?d

(g#

&vo

&?d &co
(n#

(s#

&nc &om &fm

(p#

(,#

(p#

&mm

(-#

&pl &tu
(f#

(m#

(m#

s!ores, there are two !on&ersion tables For tests that enter into only one aptitude s!ore, there is a sin"le !on&ersion table Table 2?1: pro&ides the notation for the test standard s!ore and the aptitude s!ore !omposition Aptitude s!ores are formed from the simple sum (down ea!h !olumn of Table 2? 1:) of test standard s!ores Forms E and F) The set of raw s!ores for forms E and F, denoted by X>ar(E), >vo(E), , >om(E)% >ar(F), >vo(F), , >om(F)Y ,

are ta8en as either (a) the simple sum of the number of !orre!t responses (for the three power tests A*, <2, and '/), or (b) the !han!e?!orre!ted formula s!ore (for the three speeded tests )2, 6), and 2+) The formula s!ores for ea!h of the speeded tests are "i&en by the "eneral formula > @ %1 ! 0 ( A! # ' (-)

where 6) is the number of !orre!t responses, . is the number of wron" answers (items answered in!orre!tlyNdoes not in!lude omits or not?rea!hed), and A is the number of response options asso!iated with the test items For the three speeded tests, the pre!edin" e>uation simplifies to > co (f# @ %1 ! 0 < (5)

> nc (f# @ %1 ! 0 0 ' > om(f# @ %1 ! ? where f e>uals LEM or LF M

==

1tandard s!ores 1 for Forms E and F are obtained from loo8?up tables produ!ed from the e>uatin" des!ribed in the followin" se!tion Thus, the pro!ess of !omputin" aptitude s!ores is identi!al to that des!ribed abo&e for Forms A and B, with one e$!eption Be!ause Form +at!hin" was dropped from the new forms, the aptitude s!ore Ap is set e>ui&alent to 1om(p), rather than (p) !omputed as Ap J 1om R 1fm(p) (as in Forms A and B) 6ote that the distributions of Ap a!ross the new and old forms are ensured to be e>ual throu"h the appropriate spe!ifi!ation of the e>uatin" transformation This point is des!ribed in more detail in a later se!tion Com$osite "%oring) 4n addition to the aptitude s!ore !omposites des!ribed abo&e, two other sets of !omposites were studied# three component !omposites and fi&e jo)!family !omposites Ea!h of the three !omponent !omposites was !omputed from the sum of sele!ted aptitude s!ores# 1ognitive A 1 gvn @ Ag B Av B An Perceptual A 1 sp, As B A p B A, Psychomotor A @ 1 -fm A- B A f B Am $ @ (6)

1!ores on the !o"niti&e and per!eptual !omposites were !al!ulated for all e$aminees in the e>uatin", reliability, and &alidity analyses These !omposites were in!luded in all 8ey analyses Cowe&er, s!ores on the psy!homotor !omposites were !al!ulated only for e$aminees in the psy!homotor sample, whi!h was used to address sele!ted !omposite e>uatin" and &alidity issues 4n addition, to address these same !omposite e>uatin" and &alidity issues, fi&e (ob?family !omposites were !omputed from the wei"hted linear !ombinations# C @ $DE ( 1 gvn B $?F ( 1 sp, B $ C G @ $ ? ( 1 B $H" ( 1
gvn -fm

( 1 -fm (7)

C<

C ? @ 1 gvn @ $"? ( 1 gvn B $G" ( 1

-fm

C D @ $<< ( 1 gvn B $DI ( 1 -fm $ Be!ause all !omposites but one (whi!h is redundant with )"&n) are a fun!tion of one or more psy!homotor tests, these !omposites were !omputed only for the psy!homotor sample "moot#ing and E;uating The ob(e!ti&e of e>uiper!entile e>uatin" is to pro&ide a transformation that will mat!h s!ore distributions of the new forms with the distribution of s!ores from the referen!e Form A This transformation, whi!h will be applied to the new Forms E and F, will allow s!ores on the new &ersions to be interpreted relati&e to the old s!ale represented by Form A 2ne primary ob(e!ti&e of the method of e>uatin" proposed here was to use smoothin" pro!edures that pro&ide an a!!eptable tradeoff between random and systemati! error 4n this study, smoothin" (spe!ifi!ally, polynomial lo"?linear smoothin") was performed on ea!h distribution (of Forms A, E, and F) separately These smoothed distributions were then used to spe!ify the e>uiper!entile transformation (see 1e"all & +on@on, 1::7, for a more detailed dis!ussion of issues pertinent to the e>uatin" transformation and the smoothin" pro!edures) The data used in this analysis were pro&ided from two samples# the independent?"roups =7

sample, and the repeated?measures sample /ata !olle!ted on the first?administered test from sele!ted "roups of the repeated?measures sample were !ombined with same?form data of the

=3

Chapter 2. Development of GATB Forms E and F

independent "roups sample The sample si@es used to estimate the Forms A, E, and F distributions appear in Table 2?1' Cero Cells) The lo"?linear smoothin" pro!edure is undefined for s!ore le&els that ha&e a fre>uen!y of @ero For these analyses, a two?step pro!edure was used for spe!ifyin" @ero?!ell entries First, the ma$imum s!ore was identified for ea!h test Then the distribution smoothin" was performed for the raw?s!ore ran"e Ll ? ma$,M where Lma$M is the ma$imum s!ore for the test Any s!ore le&els ha&in" no obser&ations fallin" within the Ll ? ma$M ran"e were !han"ed to ; 7 for lo"?linear estimation For the speeded?test distributions of Forms E and F, the low ran"e was set e>ual to the obser&ed minimum in the sample, be!ause the minimum !an fall below 1 for formula s!ores P*A$titude 7Ap8) The new Forms E and F do not !ontain the Form +at!hin" test 4n the old Forms A?/ this test was used in the Ap 1!ore (see Table 2?1:) The Ap s!ore (Forms A?/) was !omputed from the sum of Form +at!hin" and Tool (2b(e!t) +at!hin" *ather than e>uatin" 2b(e!t +at!hin" a!ross the new and old forms, 2b(e!t +at!hin" (Forms E and F) was e>uated dire!tly to the Ap distribution (Form A) This dire!t e>uatin" will allow s!ores on 2b(e!t +at!hin" (Forms E and F) to be transformed to an Ap s!ore ha&in" the same distribution as Form A, e&en thou"h the Form +at!hin" test has been omitted from the new forms This mat!hin" in&ol&ed se&eral steps First, the distribution of Ap was !omputed from Form A data by summin" the standard s!ores of Form +at!hin" and Tool +at!hin" 6e$t, the distribution of Ap s!ores was smoothed by applyin" lo"?linear smoothin" The final e>uatin" was obtained by mat!hin" this smoothed distribution with the smoothed 2b(e!t +at!hin" distributions of Forms E and F, usin" the e>uiper!entile pro!edure Pol/nomial E.tra$olation) 1e&eral of the new and old tests differ substantially in their len"th, whi!h leads to a si"nifi!ant differen!e in their ma$imum attainable s!ores For e$ample, <o!abulary had 3; items in Form A and 1: items in Forms E and F A ma$imum s!ore on the Form A &ersion represented se&eral standard de&iations abo&e the mean, while a ma$imum s!ore on Form EAF represents less than two standard de&iations abo&e the mean )onse>uently, mat!hin" the ma$imum s!ores on the new and old &ersions (a !onse>uen!e of e>uiper!entile e>uatin") does not appear to be appropriate Althou"h the three speeded tests were shortened, the test !ontent and time limits remained &irtually un!han"ed a!ross the new and old GATB &ersions Be!ause shortenin" these tests did not !ause a noti!eable !eilin" effe!t, the ma$imum s!ore on the new &ersions was set e>ual to the e>uated s!ores on Form A For the three power tests (A*, <2, and '/), a polynomial e$trapolation was used to spe!ify the ma$imum e>uated s!ore The hi"hest fi&e points of the e>uiper!entile transformation were appro$imated by a se!ond?order polynomial fitted with least?s>uares The e>uated &alue for the ma$imum s!ore le&el of Forms E and F was set e>ual to the predi!ted &alue of the resultin" polynomial E8uatin" Transformations. A total of 15 smoothin"s were performedN3 smoothin"s for ea!h of the three GATB forms These smoothed distributions were used to !ompute 12 e>uatin" transformationsN3 transformations e>uatin" Form A and E tests, and another 3 transformations e>uatin" Forms A and F

=3

Tables "i&en in 1e"all and +on@on (1::7) pro&ide a translation between Form EAF raw s!ores $ and Form A standard s!ores -inear interpolation was used in !on(un!tion with e>uiper!entile e>uatin" to spe!ify appropriate standard s!ores The e>uated standard s!ores were obtained from ((# ! 1 &((# @ & 3 B F E;F J &2 ! & 3 K '
3

(0)

12 ! 1
3

where FEAF($) is the !umulati&e distribution fun!tion on the new form e&aluated at raw?s!ore le&el $, )- and )0 are the lower and upper &alues of the Form A !umulati&e distribution inter&al whi!h !ontains FEAF($)# 1 3 Z F E;F Z 12 ' (2)

and 1- and 10 are the Form A standard?s!ores !orrespondin" to the inter&al defined by ()-,)y) The formula for 1($) was used to spe!ify raw to standard s!ore !on&ersions for ea!h raw?s!ore le&el of Forms E and F Com$osite E;uating E>uatin" the new and old paper?and?pen!il (P&P) GATB forms in&ol&es mat!hin" test distributions usin" an e>uiper!entile method This distribution mat!hin" pro&ides a transformation of Forms E and F to standard?s!ore e>ui&alents on the referen!e form (Form A) s!ale 2n!e this transformation is spe!ified for ea!h test, standard?s!ore e>ui&alents !an be !omputed These standard?s!ore e>ui&alents pro&ided the basis for the !omputation of GATB !omposites The same formulas used to !ompute !omposites from standard s!ores on Form A !an be used to !ompute !omposite s!ores from standard?s!ore e>ui&alents on the new forms 2ne !on!ern is that the distribution of !omposite s!ores from the new forms will differ systemati!ally from the !orrespondin" distributions of the old forms This differen!e !ould result from differen!es in test inter!orrelations between the old and new forms /ifferent test inter!orrelations may result from one or more re&isions made to the new P&P?GATB (!han"es in test len"ths, time limits, instru!tions, et! ) Be!ause the &arian!e of a !omposite is a fun!tion of the !orrelations amon" its tests, differen!es in !omposite &arian!es !ould result as a !onse>uen!e Ci"her order moments of the !omposite distributions !ould be affe!ted in a similar manner Non*Ps/%#omotor Com$osites) The data used in this analysis were the same data used to estimate the e>uatin" transformations These data were pro&ided from the independent?"roups sample and the repeated?measures sample /ata !olle!ted on the first?administered test from sele!ted "roups of the repeated?measures sample were !ombined with same?form data of the independent?"roups sample The sample si@es used to e$amine !omposite distributions a!ross Forms A, E, and F are pro&ided in Table 2?1' The distributions of se&eral !omposites were e$amined )omparison of !omposite s!ore distributions in&ol&ed a number of steps First, s!ores on Forms E and F were transformed to standard?s!ore e>ui&alents usin" the transformation estimated from the e>uatin" 6e$t, for ea!h !omposite, s!ores were !omputed for three "roups# 1 )omposite s!ores were obtained for those e$aminees ta8in" Form A by applyin" the !omposite formulas to the standard s!ores Four !omposites were e$amined# Ag and An (aptitude s!ores), and 1gvn and 1sp, ()o"niti&e and Per!eptual !omposites) All other aptitude s!ores are a fun!tion of a sin"le test% thus, the a"reement of their =9

distributions a!ross the new and old forms will be "uaranteed by the e>uiper!entile

=5

Chapter 2. Development of GATB Forms E and F

transformation Be!ause the Ap aptitude s!ore distribution of Form A was e>uated dire!tly to the Ap s!ore distribution of Forms E and F (whi!h is a nonmonotoni! fun!tion of 2b(e!t +at!hin"), these distributions are also mat!hed throu"h the e>uatin" transformation, and thus no !onfirmation is ne!essary )omposites !ontainin" tests were analy@ed separately (see se!tion below) 2 )omposite s!ores were obtained for those e$aminees ta8in" Form E )omposite s!ores were obtained for Ag' An' 1gvn, and 1sp, by applyin" the !omposite formulas to the standard s!ore e>ui&alent s!ores )omposite s!ores were obtained for those e$aminees ta8in" Form F )omposite s!ores were obtained for Ag' An' 1gvn' and 1sp, by applyin" the !omposite formulas to the standard s!ore e>ui&alent s!ores

'

The distributions of s!ores for ea!h of the new forms (Forms E and F) were !ompared to the !orrespondin" !omposite distribution of the referen!e form (Form A) Four !ut points were used to di&ide the distribution into fi&e "roups )ut s!ores were based on the area under a normal density fun!tion The : &alues (!omputed from Form A means and standard de&iations) that di&ided the distribution into "roups ha&in" the e$pe!ted proportions X 1;, 27, ';, 27, 1;Y were applied to the !omposite distributions to produ!e the obser&ed proportions displayed in Tables 9 1?9 = of 1e"all and +on@on (1::7) The proportion of e$aminees fallin" in ea!h "roup was !ompared a!ross the two new &ersions (E and F) and the sin"le old &ersion (A) The si"nifi!an!e of the differen!e in these proportions was e$amined usin" a ' $ 7 !ontin"en!y table analysis The 2 Pearson [ statisti! was used to test the null hypothesis of no differen!e amon" distributions Althou"h two !omposites (Ag and 1sp,) ha&e mar"inally si"nifi!ant differen!es, an e$amination of the distributions indi!ated the four !omposites are &ery similar a!ross the new and old forms 6one of the !omposites was si"nifi!ant at the ;1 le&el These results su""est that the standard GATB !omposite formulas !an be applied to the e>uated standard s!ores of the new forms, and that these !omposite s!ores will ha&e similar distributions a!ross the new and old &ersions Therefore, separate !omposite e>uatin" tables for the non?psy!homotor !omposites are unne!essary for the new Forms E and F Ps/%#omotor Com$osites) 4n this study, the e&aluation of !omposite e>uatin" is !ompli!ated by the absen!e of the psy!homotor tests from the new Forms E and F 6ew &ersions of the psy!homotor tests are bein" de&eloped under a separate data !olle!tion and analysis effort Be!ause some of the !omposites !omputed under the (ob?family system in!lude a !ombination of psy!homotor and non?psy!homotor tests, a !omplete e&aluation should e$amine the similarity of these distributions (a!ross old and new forms) /ata for this analysis were !olle!ted from the psy!homotor sample This sample was administered one old Form (A), one new Form (F), and one form of the psy!homotor portion of the battery (from Form A) /istributions of !omposite s!ores were !ompared a!ross the two "roups usin" data !olle!ted in the mornin" session only (Table 2?12) For Group 1 (whi!h re!ei&ed Form A psy!homotor and non?psy!homotor tests), four !omposites (C ' CG' C<' CD) were !omputed a!!ordin" to the pro!edures des!ribed earlier Group 2 s!ores for the !orrespondin" !omposites were !omputed in a similar manner from standard?s!ore e>ui&alents usin" the Form F (non?psy!homotor) and Form A (psy!homotor) portions of the battery

=5

Four !ut points were used to di&ide the distributions into fi&e "roups as in the analysis abo&e )ut s!ores were based on the area under a normal density fun!tion The : &alues (!omputed from Group 1 means and standard de&iations) that di&ided the distribution into "roups ha&in" the e$pe!ted proportions X 1;, 27, ';, 27, 1;Y were applied to the !omposite distributions The proportion of e$aminees fallin" in ea!h "roup was !ompared a!ross the new (Form F) and old (Form A) &ersions The si"nifi!an!e of the differen!e in these proportions was e$amined usin" a 2 2 $ 7 !ontin"en!y table analysis The Pearson [ statisti! was used to test the null hypothesis of no differen!e amon" distributions The results indi!ated that these proportions are &ery similar a!ross new and old GATB forms, and do not differ si"nifi!antly from what would be e$pe!ted from samplin" error (see 1e"all & +on@on, 1::7, for detailed results) These results imply that the (ob?family !omposite formulas !an be applied to the e>uated standard s!ores of the new formsNand that these !omposite s!ores will ha&e similar distributions a!ross the new and old &ersions These results su""est that separate !omposite e>uatin" tables for the psy!homotor !omposites are unne!essary for the new Forms E and F 2ne 8ey assumption of this analysis is that the new forms of the psy!homotor tests (!urrently under de&elopment for Forms E and F) will be parallel to the form used in this study 4f they are not parallel, then the !o&arian!es between these tests and the non?psy!homotor tests may be poorly represented by those obtained in the !urrent study That is, if the new and old forms of the psy!homotor tests are not parallel, then the results obtained in this study may not "enerali@e to the new psy!homotor tests "ubgrou$ Com$arisons Althou"h e>uiper!entile e>uatin" mat!hes test distributions for the total sample, it does not ne!essarily "uarantee a mat!h for distributions of sub"roups !ontained in the sample This result follows from the fa!t that the new and old &ersions of the P&P?GATB are not stri!tly parallel Althou"h we mi"ht e$pe!t to obser&e small differen!es in sub"roup performan!e a!ross the new and old &ersions as a result of differen!es in measurement pre!ision, many of the other re&isions made to the new forms !ould also !ause "roup differen!es 4t is therefore instru!ti&e to e$amine the performan!e of sub"roups to determine whether any are pla!ed at a substantial disad&anta"e by the new forms, relati&e to their le&el of performan!e on the old GATB forms 4n the analyses des!ribed below, four sub"roups were e$amined# (1) Afri!an Ameri!ans, (2) Cispani!s, (') females, and (=) e$aminees =1 years of a"e or older The e>uatin" transformation based on the total sample was applied to sub"roup members who had ta8en the new GATB forms (E and F) For ea!h sub"roup, mean performan!e le&els were !ompared a!ross new and old forms 1i$ test &ariables were e$amined# &ar(g# ' &vo(g#' &?d(g#' &co(n#' &nc(,#, and Ap These &ariables are monotoni! fun!tions of the si$ test s!ores (For Form A, the &ariable Ap is a fun!tion of both Form +at!hin" and 2b(e!t +at!hin" ) The si"nifi!an!e of the differen!es amon" means was e$amined for Forms A, E, and F usin" A62<A Amon" the 2= !omparisons, only one was si"nifi!ant at the ;1 le&el 1pe!ifi!ally, a si"nifi!ant differen!e was obser&ed a!ross A, E, and F for Afri!an Ameri!ans on the <o!abulary testNAfri!an Ameri!ans administered Forms E and F tended to s!ore sli"htly hi"her than those administered Form A 4n "eneral, the results indi!ate similar a&era"e performan!e le&els a!ross new and old &ersions for ea!h of the four sub"roups e$amined (!f 1e"all & +on@on, 1::7, for detailed results)

=:

Chapter 2. Development of GATB Forms E and F

Reliabilit/ Anal/sis A primary issue in the in&esti"ation of new GATB forms is that of pre!ision 1e&eral of the new test &ersions ha&e fewer items than their ori"inal !ounterparts Althou"h fewer items may be offset by an in!rease in testin" time, it is important to show that the new forms ha&e suffi!iently hi"h le&els of reliability relati&e to the old GATB forms -ower reliability would lead to lower le&els of &alidity Met#od) Four "roups of the repeated?measures sample were used in this analysis (see Table 2?1;) /ata from Groups 1 and 2 were !ombined to form a sample of 52; e$aminees whose data were used to !ompute the alternate form !orrelations between the old Forms A and B These !orrelations are displayed in Table 2?2; for test, aptitude?s!ore, and !omposite &ariables Groups 9 and 5 were !ombined to form a sample of 59; e$aminees whose data were used to !ompute the alternate form !orrelations between the two new Forms E and F These alternate form !orrelations are also displayed in the table FisherEs @ transformation was used to test the si"nifi!an!e of the differen!e between the alternate form !orrelations of the new and old GATB forms As des!ribed in )ohen and )ohen (1:5', p 7=), the si"nifi!an!e of the differen!e between two !orrelation !oeffi!ients obtained from two different random samples !an be e&aluated from the normal !ur&e de&iate :@ nn ! :\ J
n

: \n ! : \o ? B no ! ?

'

(9)

where 1 1

Hln(1 R r ) ] ln(1 ] )I , nn \ r nn \ 2 Hln(1 R r ) ] ln(1 ] )I , oo r oo \ 2

(1)

:\ J
o

and rnnL is the alternate form !orrelation for the new test &ariable (based on Forms E and F), rooL is the alternate form !orrelation for the old test &ariable (based on Forms A and B), and nn and no are the sample si@es for the "roups used to !ompute the alternate form !orrelations (nn J 59;, no J 52;) 6ormal de&iates : were !omputed for ea!h test, aptitude s!ore, and !omposite &ariable The results are displayed in Table 2?2; Also displayed are the probability &alues asso!iated with these normal de&iates, 1P :(^:^), where : is the normal !umulati&e distribution fun!tion Results) The alternate form reliabilities of the new GATB forms are "enerally as hi"h as, or hi"her than, those of the old GATB Forms A and BNan en!oura"in" findin", be!ause the len"th of the three power tests was de!reased The in!rease in testin" time, howe&er, may ha&e added to the reliability of these power tests, offsettin" the detrimental effe!ts of shortenin" test len"ths 2nly one !omparison displayed a si"nifi!antly lower alternate?form !orrelation for the new form# &co(n# ()omputation) The ma"nitude of the differen!e is small, howe&er, and none of the !omposites that use )omputation displays a si"nifi!antly lower new?form reliability estimate Validit/ Anal/sis This se!tion addresses the third primary issue in the e&aluation of the new formsN !onstru!t &alidity 4t is hi"hly desirable for the new and old GATB forms to measure identi!al or hi"hly !orrelated !onstru!ts The measurement of similar !onstru!ts would enable the &alidity of 7;

the new forms to be inferred from the lar"e body of e$istin" &alidity resear!h !ondu!ted on the old forms of the GATB
Table 2*2A Alternate*Form Reliabilit/ Estimates, Normal De'iates, and $*Values ,orrelations rnnL 5;; &ar(g# 5;; (n# &ar 5=3 (g# &vo 57; (v# & vo 52: (g# &?d 5'2 (s# &?d 515 & (n# co 995 (,# &nc 52' (p# &om :;5 Ag 57; Av 593 An 5'2 As 52' Ap 995 A, 1gvn :1: 1sp, 5:' 6ote#rnnL J new form reliability% rooL J old form reliability% p @ 1 ? :(^:^) <ariable rooL 5;' 5;2 57: 575 5;7 5;7 5=3 977 99; 553 575 55= 5;7 52= 977 :1' 5=: Si"nifi an e Test : ? 19: ? 129 ? :27 ? 75; 1 72: 1 3=5 ?1 59' 1 1=7 2 ::3 2 25= ? 75; ? 9;= 1 3=5 ? ;=: 1 1=7 992 ' 3:= p =2: =7; 199 251 ;3' ;7; ;'1 123 ;;1 ;11 251 2=1 ;7; =5; 123 22; ;;;

The !onstru!t &alidity analysis is presented in two parts The first part des!ribes an analysis of the non?psy!homotor test and !omposite &ariables based on the repeated?measures data The se!ond part addresses the !onstru!t &alidity of &ariables that enter into the (ob?family !omposites (i e , the !o"niti&e, per!eptual, and psy!homotor !omposites) These analyses are based on the psy!homotor sample Non*Ps/%#omotor Constru%t Validit/) For this analysis, all ei"ht "roups of the repeated?measures sample were used (see Table 2?1;) /ata from Groups ', =, 7, and 3 were !ombined to form a sample of 531 e$aminees For the purpose of these analyses, s!ores on Forms A and B were treated as an L2ld TestM &ariable% s!ores on Forms E and F were treated as a L6ew TestM &ariable )orrelations between the old and new batteries (denoted by rn'o) were obtained for test, aptitude, and !omposite &ariables These !orrelations are "i&en in Table 2?21 The alternate?form reliability estimates !omputed from Groups 1 and 2 (old forms) and Groups ' and = (new forms) were used to obtain the disattenuated !orrelations between the new and old forms These alternate?form !orrelations were !omputed as des!ribed in the pre&ious se!tion The disattenuated !orrelations were !omputed from the )lassi!al Test Theory e$pression

_(`n '`o # @

r n 'o r n n\ ( r o o\

(.)

These &alues also appear in Table 2?21 (Asymptoti! standard errors of the disattenuated !orrelations were obtained usin" a stru!tural e>uation modelin" approa!h% see 1e"all & +on@on, 1::7, Appendi$ 4 )

71

Chapter 2. Development of GATB Forms E and F

Table 2*2+ Disattenuated Correlations !et-een Ne- and Old <ariable

(g# &ar (n# &ar (g# &vo (v# &vo (g# &?d (s# &?d (n# &co (,# &nc (p# &o)

AT! Forms M(Nn'No# :71 :77 :=; :'5 :1; :;5 :3: :72 :;7 :77 :'5 :52 :;5 :;9 :72 :35 :'5

rn'o 932 937 5;2 5;1 9=' 9=' 5;3 9'; 921 579 5;1 53= 9=' 9=5 9'; 559 519

Ag Av An As Ap A, 1gvn 1sp,

As indi!ated, all disattenuated !orrelations between old and new forms were e$tremely hi"h, ran"in" from :;7 to :52 Althou"h numerous !han"es were made to the test battery (e " , test format, time limits, test len"ths, deletion of the Form +at!hin" Test, !han"e in s!orin" formulas), these !han"es do not appear to ha&e si"nifi!antly altered the dimensionality of the battery Be!ause of the hi"h !orrelations between the dimensions measured by the new and old forms, the lar"e number of &alidity studies !ondu!ted on the old GATB forms !an !ontinue to pro&ide useful data for inferrin" the &alidity of the new GATB forms Ps/%#omotor Constru%t Validit/) Fi&e (ob family !omposites were used for (ob !ounselin" and referral with the <G?GATB pro"ram There is a !han!e that these !omposites !ould !ontinue to be used on a limited basis for (ob !ounselin" with GATB Forms E and F Be!ause four of the fi&es !omposites are !omputed, in part, on the basis of psy!homotor tests, a !omplete e&aluation of the new GATB forms should in!lude the psy!homotor tests 4f, for e$ample, the !o&arian!e between psy!homotor and non?psy!homotor tests differed a!ross the old and new forms, then the &alidity of the fi&e (ob?family !omposites mi"ht be affe!ted That is, these fi&e !omposites !omputed from the old forms mi"ht measure different traits than those !omputed from the new forms This analysis used 7'= sub(e!ts from the psy!homotor?sample As indi!ated in Table 2?12, ea!h e$aminee was administered Form A (non?psy!homotor), Form A (psy!homotor), and Form F (non?psy!homotor) portions of the GATB For ea!h e$aminee, fi&e s!ores were !omputed# 1gvn(A#' 1sp,(A#' 1gvn(F#' 1sp,(F#' and 1-fm(A#, where 1gvn(A# denotes the !o"niti&e !omposite !omputed from Form A tests% 1gvn(F# denotes the same !o"niti&e !omposite !omputed from Form F tests, et! To !ompute the Form F !omposites, s!ores were transformed to Form A Lstandard?s!ore e>ui&alentsM usin" the e>uatin" transformation presented in e>uation 7 The formulae "i&en in e>uation ' were applied to the aptitude s!ores to !ompute the fi&e !omposites Table 2?22 displays the !orrelations amon" these fi&e &ariables As indi!ated in the last row, the patterns of !orrelations between the psy!homotor !omposite 1-fm and the !o"niti&e and

72

per!eptual !omposites (1gvn and 1sp,) appear to be similar9 a!ross Forms A and F ( 29 &s 2', == &s =2) These similar patterns pro&ide some assuran!e that the same relations amon" !o"niti&e, per!eptual, and psy!homotor !omposites hold for both the new and old GATB forms These results ta8en in !on(un!tion with the hi"h disattenuated !orrelations between the new and old !o"niti&e and per!eptual !omposites ( :9 and :=, respe!ti&ely% !f Table 2?21) su""est that the dimensions measured by the new and old (ob?family !omposites (whi!h are linear !ombinations of 1-fm' 1gvn, and 1sp,) will also be &ery hi"hly !orrelated
Table 2*22 Correlations Among Cogniti'e, Per%e$tual, and Ps/%#omotor Com$osites Com$osite 1gvn(A) 1sp,(A) 1gvn(F) 1sp,(F) 1-fm(A) Cgvn7A8 1; 9' 5: 91 29 Cspq7A8 1; 31 5= == Cgvn7F8 Cspq7F8 Ckfm7A8

1; 3: 2'

1; =2

1;

"ubgrou$ Com$arisons Althou"h e>uiper!entile e>uatin" mat!hes subtest distributions for the total sample, it does not "uarantee a mat!h for distributions of sub"roups within the sample This is be!ause the new and old &ersions of the paper?and?pen!il GATB are not stri!tly parallel /ifferen!es between the &ersions in measurement pre!ision, as well as the &arious re&isions made to the new forms, !ould !ause "roup differen!es To e$amine the impa!t of the re&isions on sub"roup distributions, two sets of analyses were performed# (a) analyses e$aminin" the le&el of performan!e of ea!h sub"roup a!ross the old and new forms, and (b) analyses e$aminin" ad&erse impa!t for sele!ted sub"roups (!ondu!ted separately for the old and new forms) "ubgrou$ Performan%e A%ross Forms) For this first set of analyses, four sub"roups were e$amined# (a) Afri!an Ameri!ans, (b) Cispani!s, (!) Females, and (d) e$aminees =1 years of a"e or older The e>uatin" transformation based on the total sample was applied to sub"roup members who had ta8en the new GATB forms (E and F) For ea!h sub"roup, mean performan!e le&els were !ompared a!ross new and old forms 1i$ subtest &ariables were e$amined# (g# , &vo(g#, &?d(g#, &ar &co(n#, &nc(,#, and Ap These &ariables are monotoni! fun!tions of the si$ subtest s!ores 6ote that Ap is a fun!tion of both Form +at!hin" and 2b(e!t +at!hin" for GATB Form A ) The analyses in&ol&ed !ondu!tin" an A62<A a!ross the si$ subtest &ariables for ea!h of the four sub"roups For ea!h A62<A, the si"nifi!an!e of the differen!e amon" means was e$amined for Forms A, E, and F Amon" the 2= !omparisons, only one was si"nifi!ant at the ;1 le&el# means a!ross forms for Afri!an Ameri!ans on the <o!abulary subtest (1&o(")), with hi"her s!ores e&iden!ed on Forms E and F than on A 4n "eneral, the results indi!ate similar a&era"e performan!e le&els a!ross new and old &ersions for ea!h of the four sub"roups e$amined Adverse 4mpa t. Ad&erse impa!t analyses were !ondu!ted for three sub"roups# (a) Afri!an Ameri!ans, (b) Cispani!s, and (!) females The results of these analyses are "i&en in Table 2?2' /ifferen!es in mean le&els between ma(ority and minority "roups are reported for old and new GATB forms on the si$ subtest &ariables
The hypothesis that these pairs of !orrelations were si"nifi!antly different was tested usin" a !onfirmatory fa!tor model where a was set e>ual to the !orrelation matri$ of the obser&ed &ariables, and by !onstrainin" b71 J b7'% b72 J b7= Based on a !hi?s>uare differen!e test, these pairs of !orrelations did not differ si"nifi!antly ([2 J = ;9, df J 2, p J 1')
9

7'

Chapter 2. Development of GATB Forms E and F

Ea!h numeri!al entry in Table 2?2' is an effe!t si@e, !al!ulated as c@ (m ! 6# sT ' (-;)

where m is the minority "roup mean, 6 is the ma(ority "roup mean, and sT is the total "roup standard de&iation The indi&idual sample statisti!s used to !ompute the ad&erse impa!t &alues are "i&en in Appendi$ A of 1e"all and +on@on (1::7) The first &alue in the table (?; ::) indi!ates that Afri!an Ameri!ans s!ored ; :: standard de&iation unit lower than whites on the Arithmeti! *easonin" subtest from Form A
Table 2*23 Ad'erse (m$a%t "tatisti%s A%ross AT! Forms for "ele%ted "ubgrou$s Effe%t "i4e 7D8 "ubgrou$ Afri!an Ameri!an "%ore 1 ar
(g# (g# (g#

Form A ?; :: ?1 ;2 ?; 5= ?; 9; ?; 9; ?; 7: ?; 37 ?; 52 ?; 2: ?; =: ?; 77 ?; 21 ?; ;3 ; 19 ?; 1: ; 17 ; =' ; 23

Form E ?; :2 ?; 59 ?; 99 ?; 37 ?; 72 ?; 7: ?; 33 ?; 93 ?; '' ?; 7; ?; '7 ?; 22 ?; ;9 ?; ;1 ?; 19 ; 1: ; == ; ';

Form F ?; 5: ?; 5: ?; 9: ?; 31 ?; 72 ?; 79 ?; 71 ?; 33 ?; 2' ?; '5 ?; '1 ?; 1' ?; ;' ; ;; ?; 17 ; 1: ; =7 ; ''

1vo

1?d 1co 1nc Ap Cispani!s 1 ar

(n# (,#

(g# (g# (g#

1vo

1?d 1co 1nc Ap Females 1

(n# (,#

(g# (g# (g#

1vo

1?d 1co 1nc Ap

(n# (,#

4n "eneral, le&els of ad&erse impa!t tend to be similar a!ross the old and new formsN althou"h some minor trends are e&ident For e$ample, GATB Forms E and F tend to display sli"htly lower le&els of ad&erse impa!t for Afri!an Ameri!ans The ad&erse impa!t statisti!s for Cispani!s tend to possess "reater &ariability than for other minority sub"roups, a result li8ely attributable to the small samples (% V '27) for this sub"roup

7=

"ummar/ of Form De'elo$ment A%ti'ities This portion of the !hapter des!ribed the de&elopment of GATB Forms E and F (see Table 2?2=) The ma(or issues were fairness, speededness, and test se!urity To address these issues, fi&e steps or phases were implemented This !hapter des!ribed ea!h phase# item de&elopment pro!edures, item redu!tion and test format modifi!ations, item pretest and analysis, test tryout and analysis, and final forms preparation
Table 2*25 AT! Forms E and F> Test Lengt#s and Time Limits AT! Test Arithmeti! *easonin" <o!abulary Three?/imensional 1pa!e )omputation 6ame )omparison 2b(e!t +at!hin" Number of Test (tems 15 1: 2; =; :; =2 Time Limit 7minutes8 2; 5 5 3 3 7

De'elo$ment and Re'ie- of Ne- (tems) This phase in!luded the de&elopment, re&iew, and re&ision of test items The A*/P de&eloped measures of item diffi!ulty for all tests e$!ept Form +at!hin", whi!h was !omprised of two blo!8s of items, to arran"e items within ea!h test A stru!tured item re&iew pro!edure was de&eloped and implemented in three A*/)s with se&en panel members to e&aluate item bias 4tems were then re&ised based on the !omments of the panel members and further analyses C#anges to "$e%ifi%ations for Test Lengt# and Format and to "u$$orting Materials) 4tem redu!tion was addressed in two ways# (1) A*/P staff e$amined te!hni!al and operational issues, and (2) A4* !ondu!ted an empiri!al study that addressed issues related to item redu!tion *e!ommendations for item redu!tion based primarily on te!hni!al issues were made by A*/P staff The results of the A4* study led to re!ommendations for item redu!tion, test administration time limits, whether a test is speeded or nonspeeded, and test order The impa!t of speededness was further redu!ed by introdu!tion of formula s!orin" for the speeded tests 4nstru!tions to the e$aminees were !han"ed to refle!t differen!es in s!orin" methods and made mu!h more e$pli!it Test format modifi!ations were addressed by the Test Aestheti!s Pro(e!t, whi!h in!luded inter&iews, literature sear!hes, fo!us "roups, and a sur&ey 1pe!ifi! re!ommendations were made for the format and !ontent of the GATB administration manual, test boo8lets and instru!tions, test items, and answer sheet (tem Tr/out and "tatisti%al "%reening) The item pretest and analysis had two "oals# (1) !ondu!tin" an item analysis to obtain preliminary diffi!ulty and dis!rimination indi!es, and (2) obtainin" a >uantitati&e estimate of ethni! and "ender performan!e differen!es for ea!h item The sample !omprised :,'29 appli!ants from 01E1 lo!al offi!es in the fi&e "eo"raphi! re"ions represented within the A*/P /ata were obtained by administerin" to the sample members 13 test boo8lets !omprisin" three speeded tests and one power test Ea!h sample member !ompleted one test boo8let )lassi!al test theory item analyses were performed for the speeded test items 4tem sele!tion !riteria in!luded diffi!ulty, dis!rimination, and !ontent !onsiderations 4*T pro!edures were used for the power test items The analyses in!luded dimensionality, position effe!ts, item and test fairness, and test information "raphs 4tem /4F analyses were also performed with +antel? Caens@el pro!edures 4*T pro!edures were used for test?le&el /4F analyses 77

Chapter 2. Development of GATB Forms E and F

Constru%tion of t#e Final Version of Forms E and F) After items were s!reened and !alibrated, a final set of items was sele!ted for ea!h Form E and Form F test 4tems were sele!ted to yield forms as parallel to ea!h other as possible with respe!t to !ontent !o&era"e, diffi!ulty, and test information The forms were also balan!ed on sub"roup differen!e statisti!s so that no one form pro&ided any relati&e disad&anta"e to females, Afri!an Ameri!ans, or Cispani!s 4nsofar as possible, the power tests were also desi"ned to be similar with respe!t to diffi!ulty and information to Form A, after ad(ustin" for differen!es in test len"ths Form Lin1ing "tud/) E>uatin" of the new GATB forms to the old forms pro&ed su!!essful /espite the !han"es made in the new test forms, the e&iden!e su""ests that there is suffi!ient similarity to ob&iate the need for separate !omposite e>uatin" tables for the non? psy!homotor !omposites A&era"e sub"roup performan!e le&els are similar a!ross the old and new forms, and reliabilities of the new GATB forms are "enerally as hi"h as or hi"her than those of the old forms )onstru!t &alidity analyses of the old and new forms su""est that the GATB &alidity data !an !ontinue to be used for the new forms

Referen%es
Ameri!an Edu!ational *esear!h Asso!iation, Ameri!an Psy!holo"i!al Asso!iation, & 6ational )oun!il on +easurement in Edu!ation (1:57) &tandards for educational and psychological testing$ .ashin"ton, /)# Ameri!an Psy!holo"i!al Asso!iation Ba8er, F B , Al?Karni, A , and Al?/osary, 4 + (1::1) ES0ATE# A !omputer pro"ram for the test !hara!teristi! !ur&e method of 4*T e>uatin" Applied Psychological 6easurement' D' 95 Ber8, * A (1:52) /and)oo- of methods for detecting test )ias Baltimore, +/# The Bohns Cop8ins 0ni&ersity Press Boldt, * F (1:5') *eview for perceived )ias on A&7AB Forms *+?5'?=) Prin!eton, 6B# Edu!ational Testin" 1er&i!e ' G' and ? (*eport 6o ET1?

)alifornia Test /e&elopment Field )enter (1::1) Proposal to reduce the num)er of alternatives in GATB Form 6atching items 0npublished manus!ript )alifornia Test /e&elopment Field )enter (1::2) +perational considerations for reducing the num)er of items in the General Aptitude Test Battery 0npublished manus!ript )ohen, B , & )ohen, P (1:5') Applied multiple regression;correlation analysis for the )ehavioral sciences (Gnd Ed$# Cilllsdale, 6ew Bersey# Erlbaum )orel )orporation (1::') )orel/*A.Q userEs manual P &ersion = ; 2ntario, )anada# Author /a""ett, + - (1::7) Test aesthetics improvement project 0npublished manus!ript /ale, E , & 2E*our8e, B (1:51) The living word voca)ulary )hi!a"o# .orldboo8?)hild!raft 4nternational, 4n! /ras"ow, F (1:59) 1tudy of the measurement bias of two standardi@ed psy!holo"i!al tests Cournal of Applied Psychology' "G' 1:?2: Cambleton, * , & *o"ers, B (1::') 6/A A F+*T*A% "" program to compute the 6antel!/aens:el statistic for detecting differential item functioning Author# 0ni&ersity of +assa!husetts?Amherst Cambleton, * K , & *o"ers, C B (1:55) Design of an item )ias review formA 4ssues and ,uestions Albany, 6O# 6ew Oor8 1tate Edu!ation /epartment

73

Cambleton, * K , & 1waminathan, C (1:57) 4tem response theoryA Principles and applications 6orwell, +A# Kluwer A!ademi! Publishers Group Carms, * A (1:95, +ar!h) The development' validation' and application of an e(ternal criterion measure of achievement test )ias Paper presented at the Annual +eetin" of the 6ational )oun!il on +easurement in Edu!ation, Toronto, 2ntario, )anada Carti"an, B A & .i"dor, A K (Eds ) (1:5:) Fairness in employment testingA 7alidity generali:ation' minority issues' and the General Aptitude Test Battery .ashin"ton, /)# 6ational A!ademy Press Colland, P & Thayer, / T (1:55) /ifferential item performan!e and the +antel?Caens@el pro!edure 4n C .ainer & C 4 Braun (Eds ), Test 7alidity Cillsdale, 6B# Erlbaum Asso!iates Cotellin", C (1:'1) The "enerali@ation of 1tudentEs ratio Annals of 6athematical &tatistics' G' '3;?'95 C*1trate"ies (1::=) *evised technical documentation for the General Aptitude Test Battery Grosse Pointe, +4# Author Culin, ) - , /ras"ow, F , & Parsons, ) K (1:5') 4tem response theoryA Application to psychological measurement Comewood, 4-# /ow Bones?4rwin -o!8heed?Kat@, + (1:9=) &e( )ias in educational testingA A sociologist5s perspective (*eport 6o ET1?*+?9=?1') Prin!eton, 6B# Edu!ational Testin" 1eri&!e -ord, F + (1:99) Pra!ti!al appli!ations of item !hara!teristi! !ur&e theory Cournal of Educational 6easurement' <, 119?1'5 -ord, F + (1:5;) Applications of item response theory to practical testing pro)lems Cillsdale, 6B# Erlbaum +adaus, G , Airasian, P , Cambleton, * , )onsal&o, * . , & 2rlandi, - * (1:9:) Development and application of criteria for screening commercial standardi:ed tests for the 6assachusetts Basic &-ills 4mprovement Policy Boston# +assa!husetts 1tate /epartment of Edu!ation +!)loy, * A , *ussell, T * , Brown, K + , /iFa@io, A 1 , & Green, B F (1::=) 4tem selection for the speeded su)tests of the General Aptitude Test Battery (GATB#' Forms E and F (Cum**2 Final *eport F*?P*/?:=?1;) Ale$andria, <A# Cuman *esour!es *esear!h 2r"ani@ation +ellon, 1 B , /a""ett, + , +a!+anus, < , & +orits!h, B (1::3) Technical report for the development of GATB Forms E and F 1a!ramento, )A# Author 2lson, A , & 1moyer, 1 (1:55, April) Developing ,uality science programs Paper presented at the Annual +eetin" of the Ameri!an Edu!ational *esear!h Asso!iation, 6ew 2rleans, -A PA*/) (February, 1::7) Test specifications for GATB Forms E and F$ 0npublished manus!ript 1a!ramento, )A# Author Peterson, 6 G (1::') *eview of issues associated with speededness of GATB tests .ashin"ton, /)# Ameri!an 4nstitutes for *esear!h *a(u, 6 1 , /ras"ow, F / , & 1linde, B A (1::') An empiri!al !omparison of the area methods, -ordEs !hi?s>uare test, and the +antel?Caens@el te!hni>ue for assessin" differential item fun!tionin" Educational and Psychological 6easurement' D?, ';1?'1= 79

Chapter 2. Development of GATB Forms E and F

1a"er, ) E , Peterson, 6 G , & 2ppler, 1 C (1::=) An e(amination of the speededness of the General Aptitude Test Battery power tests .ashin"ton, /)# Ameri!an 4nstitutes for *esear!h 1!hrat@, + K , & .ellens, B (1:51, Au"ust) 6inority panel review in the development of an achievement test Paper presented at the Annual +eetin" of the Ameri!an Psy!holo"i!al Asso!iation, -os An"eles, )A 1!ientifi! 1oftware (1::;) B43+G ?A 4tem analysis and test scoring with )inary logistic models )hi!a"o, 4-# Author 1e"all, / 2 , & +on@on, * 4 (1::7) Draft reportA E,uating Forms E and F of the POP!GATB 1an /ie"o, )A# 6a&y Personnel *esear!h and /e&elopment )enter 1outhern Test /e&elopment Field )enter (1::2) *educe the num)er of items on the General Aptitude Test BatteryA *ecommended num)er of items for GATB forms A'B'1' and D 0npublished manus!ript 1to!8in", + - & -ord, F + (1:5') /e&elopin" a !ommon metri! in item response theory Applied Psychological 6easurement, 9, 2;1?21; Tittle, ) K (1:52) 0se of (ud"mental methods in item bias studies 4n * A Ber8 (Ed ), /and)ooof methods for detecting test )ias Baltimore, +/# The Bohns Cop8ins 0ni&ersity Press .ilson, / T , .ood, * , & Gibbons, * (1::1) TE&TFA1TA Test scoring' item statistics' and item factor analysis )hi!a"o, 4-# 1!ientifi! 1oftware, 4n!

75

7:

3;

31

32

3'

3=

37

33

39

35

3:

9;

91

92

9'

9=

97

93

99

95

9:

5;

51

52

5'

5=

57

53

59

55

5:

:;

:1

ar

:2