General Specifications for Listening Test

OVERALL DESIGN
Purpose
To create a measure of foreign language proficiency in perceptive
skills via the media of listening at the level corresponding to C1 of
the CEF (Common European Framework). To assess candidates
aility to understand the gist and the overall message of a spoken
te!t" to e!tract specific information and to follow the significant
points in an oral utterance (listening comprehension) in order to
cope with social and academic environment.
Intended population Candidates who take their final foreign language e!amination at
tertiary level (which is also a graduation re#uirement)
$ge% 1&'
(e!% oth male and female
)on*native speakers of the target language
Intended
decisions/stakes
To make statements regarding the candidates aility to listen at
level C1 of CEF
The results (e!pressed in the accepted grading system of each
university) will contriute +, - towards the final foreign language
e!amination grade.
Response format .ultiple Choice /uestions (.C/) comprising 0 options
(ingle $nswer Format (($F) limited to a ma!imum of 0 words
1ap Filling Task (1FT)
Number of items 02
Order of items
3ogical se#uence%
(ection 1. 3istening for gist% + .C/ tasks of 0 options
comprising 1+ (4'4) items.
(ection +. 3istening for specifics% + ($F tasks (gap filling)
comprising 1& (5'5) items.
Weighting of items $ll items are e#ually weighted.
Accommodations $ccording to e!isting official policies formulated y each institution
Reporting type (cores translated into grade according to institutional grading
system
Section 1: Tasks 1 and 2
TAS DE!ANDS
Purpose
3istening for gist
Response format .C/ (.ultiple Choice /uestions) comprising 0 options
Known criteria
Candidates are informed of the procedure and assessment criteria
in advance (information availale on the wesite)
Weighting E#ual
Number of items 4 in each task
Order of items
3ogical se#uence (i.e. the items will occur in the same order as
they appear in the listening test)
ime constraints , min for each task% 2., min for pre*listening" 6 min for listening"
2., min for post*listening
Intended operations 7ecogni8ing the message of a longer audio te!t
TEST DE!ANDS
Ling"istic inp"t
!hannel
$udio recording with #uestions and answer options written on
paper. $ll listening tests are prepared y computer9 all sound files
are digital.
"iscourse mode 3ecture:presentation
#ength 6 min each
Nature of information
)ot known to candidates
The recorded te!ts are ased on authentic materials of general
nature or relevant to the candidates field of study.
The acoustic input is presented as recorded te!t chunks of 6 min.
The written input is presented on paper as a task ased on
multiple choice.
!ontent knowledge
;nternal knowledge related to language (level C1 of CEF) and
topic (related to own e!perience)
E!ternal knowledge% topic may not e familiar and may contain
some unknown language elements (up to ,-)
#e$ical range 1eneral" general*specific spoken language at level C1 of CEF
%tructural range $t level C1 of CEF
&unctional range ;nformational
arget domain 1eneral or academic language
Speaker
%peech rate
$ppro!imately 1,2 words per minute
'ariety of accent (tandard spoken language
Ac(uaintanceship (peaker not known to candidates
Number of speakers 1
)ender .ale or female9 a range of speakers throughout the test
Section 2: Tasks 1 and 2
TAS DE!ANDS
Purpose
3istening for specific information
Response format 1ap Filling Task (1FT)
Known criteria
(tudents are informed of the procedure and assessment criteria in
advance (information availale on the wesite).
(pelling mistakes will not e taken into account if the meaning is
clear.
Weighting E#ual
Number of items 5 in each task
Order of items
3ogical se#uence (i.e. the items will occur in the same order as
they appear in the listening te!t)
ime constraints 6., min for each task% 1 min for pre*listening" 0 min for listening"
2., min for post*listening
$ppro!imately 1 item per 02 seconds
Intended operations E!tracting specific information
TEST DE!ANDS
Ling"istic inp"t
!hannel $udio recording with written summary. $ll listening tests are
prepared on computer9 all sound files are digital.
"iscourse mode ;nterview (+*0 speakers)
#ength 0 min
Nature of information
)ot known to candidates
The recorded te!ts are ased on authentic materials of general
nature or relevant to the candidates field of study.
The acoustic input is presented as recorded te!t chunks of 0 min.
The written input is presented on paper as a summary of the
recorded te!t containing gaps every 02*,2 seconds throughout the
recording (items deleted selectively).
!ontent knowledge
;nternal knowledge related to language (level C1 of CEF) and
topic (related to own e!perience)
E!ternal knowledge% topic may not e familiar and may contain
some unknown language elements (up to ,-)
#e$ical range 1eneral" general*specific spoken language at level C1 of CEF
%tructural range $t level C1 of CEF
&unctional range ;nformational
arget domain 1eneral or academic language
Speakers
%peech rate
$ppro!imately 1,2 words per minute
'ariety of accent (tandard spoken language
Ac(uaintanceship (peakers not known to candidates
Number of speakers +
)ender <ne male and one female speaker
S#ORING
)eneral %tatement .anually marked answer sheets
%coring plan Each item is awarded a separate score.
Answer key Full answer key to e produced
Item analysis
;tem facility
;tem discrimination
Internal consistency =uder*7ichardson formula (=*7 +2)
*rror of measurement (tandard Error of .easurement
+arker reliability ;nitially" all data to e doule*inputted y trained secretarial staff
7eliaility of inputting to e estimated (minimum acceptale level
55-)
AD!INISTRATION
Physical conditions $dministered in the classroom
<ne person per desk" no more than 62 students in a
classroom
Accommodations $s re#uired y each institution
,niformity of
administration
;nstructions are oth recorded and provided on paper at level >+ of
CEF. The spoken te!ts are recorded and scripted eforehand.
3istening tests with the instructions on paper are placed on the
desks efore candidates enter the classroom. Candidates will also
hear the recorded instructions on the tape efore each task. There
is some e!tra space at the ottom of each page for making notes.
Common acceptale printing standards should e applied.
The invigilators (+ teachers)" who have een trained in advance"
will handle the technical e#uipment" conduct and oserve the
whole process of the e!amination.
There should e no noise in the classroom. The suggested
administration time is 6*4 minutes% +*0 minutes at the eginning of
the test for getting seated and ready for writing the test" and +*0
minutes for collecting the test papers and leaving the classroom.
$dministration time is not included in the test time. Candidates are
not allowed to ask any #uestions after the test has started.
The recording is played to candidates once.
The listening test takes 15 min and contains + sections.
The first section lasts 12 min and consists of + tasks (, min ' ,
min). Each task has pre*listening time of 2., min" a 6*min recorded
te!t" post*listening time of 2., min for performing the task.
The second section comprises + tasks" which last 5 min (6., min '
6., min). Each task has pre*listening time of 1 min" a 0*min
recorded te!t" post*listening time of 2., minutes for performing the
task.
The listening test is administered twice a year and the same
e!amination format is used. The agreed assessment criteria and
standards are applied.
%ecurity .ultiple versions of each paper are to e created. $ll the tests
should e numered" etc. The tests should e counted efore and
after the e!amination and kept in a safe place. (ecurity should e
guaranteed to e!clude cheating.
Researc$%&ased 'alidation
S#ORE VAL(E )#o*parisons+
with different
-ersions of the same
test
7esearch pro?ects to e set up to investigate e#uivalence of
different versions of the test in terms of listening skills tested as
well as scores achieved.
This will e achieved y administering two versions of the test with
the same group of students. (tatistical analyses as well as @e!pert
panel #ualitative analysis of test content and standards (i.e.
passing or reporting grades) will e performed on resulting data to
estalish e#uivalence.
(uggested #uantitative analyses to include%
Correlation analysis
T*test
with the same test
administered on
different occasions
This will e done y administering the same listening paper to a
group of candidates () A 122 minimum) within a period of
appro!imately + weeks. (hort #uestionnaire to investigate the
degree to which they rememer the listening te!ts" test items and
their original responses to these items should e included.
(imilar #uantitative analyses to aove suggested. /ualitative
analysis to e ased on the responses to the @test memory items.
with other
tests/measurements
Bro?ect to e set in place to link the test to the CEF ased on
recommendations in CofE .anual. $s suggested in the manual"
this will have a numer of phases%
1. $c#uaint personnel with levels on CEF
+. (et up panel to make #ualitative links etween CEF levels
and the test content
0. Cse $3TE @C$) Do statements as a asis for an appro!imate
#uantitative link etween the test and the CEF
with other
tests/measurements
Bro?ect to e set in place to link the test to the CEF ased on
recommendations in CofE .anual. $s suggested in the manual"
this will have a numer of phases%
6. $c#uaint personnel with levels on CEF
,. (et up panel to make #ualitative links etween CEF levels
and the test content
4. Cse $3TE @C$) Do statements as a asis for an appro!imate
#uantitative link etween the test and the CEF
with future
performance
3ongitudinal tracking study to e set in place with stratified random
sample of test takers for up to one year post test.
S#ORE VAL(E )I*pact+
Washback
Eashack studies to e set up to look at effect of introduction of
test on language classrooms at each participating institution
;nterview data F with learners" teachers and administrators
(prior to and post operationalisation)
<servational data (language G other classrooms) F set in
place when previous system is still operational" for
aseline data.
.onitoring of candidate performance on test over time
(ideally using ;tem 7esponse Theory ased statistical
analysis)
%ocial concerns
.a?or impact study to investigate
(ocial" financial" educational effects
Halue of test within and eyond institutions