Second

Language
Assessment :
Meeting 11
 Why test
grammar?
 Why test grammar? (Hughes, 2003)
• Probably, most proficiency tests that are administered on a
large scale still retain a grammar section.

• One reason for this must be the ease with which large number
of items can be administered and scored within a short period
of time.

• The question of content validity: if we decide to test writing

ability directly, then we are severely limited in the number of
topics, style of writing and the grammatical elements that we
can cover in any version of the test.
continued,
• Wherever the teaching of grammar is thought
necessary, then consideration should be given to the
advisability of including a grammar component in
achievement area.
 Testing grammar
• Grammar as an object of testing (and teaching) has a long
history
• Generally seen as knowledge of language structure at the word
level (morphology, e.g., inflections) or sentence level (syntax,
e.g., inversion)
• De-emphasized in the communicative language teaching (and
testing) tradition: more emphasis on ability for use than just
knowledge
• Not tested separately in the “big three” tests but common in
achievement tests and placement test, e.g., DIALANG
(Alderson, 2005)
 How should we test grammar?

◼ Separated from skills?

◼ Discrete point or integrative items?
◼ Focus on grammar or meaning?
A developmental
approach
Chapelle, Chung, Hegelheimer,
Pendar & Xu (2010)

What grammar at
what level?
 Designing test development according to
Bachman and Palmer (1996, p. 88)
1. a description of the purpose(s) of the test,
2. a description of the TLU domains and task types,
3. a description of the test-takers,
4. a definition of the construct(s) to be measured,
5. a plan for evaluating test usefulness, and
6. a plan for dealing with resources.
 Writing specifications, sampling,
and writing items based on Hughes
(2003)
 Writing specifications
•Specification of content for achievement tests:
▫ From the syllabus, which list the grammatical structures to
be taught.
▫ Direct from textbooks and other teaching materials
•For placement test:
▫ Inferring from the structures identified above
▫ Also perhaps possible structures for the lowest classes
 Sampling

– Reflect an attempt to give the test content validity by

selecting widely from the structures specified.
– It should also take into account of what are regarded as
the most important structures.
– It should not deliberately concentrate on the structure
that happen to be easiest to test.
 Writing items
• Whichever techniques are chosen, it is important for the text
of the item to be written grammatically correct and in natural
language.
▫ E.g. We can’t work in this class because there isn’t enough
silence.
• To avoid unnatural language, it is recommended to use
corpus based examples.
• The 4 techniques are gap filling, paraphrase, completion and
multiple choice. The first 3 require the candidate’s
production, the last calls only for recognition.
Grammar test task types
 RECOGNITION

– Multiple choice items

– Error-recognition items
– True/False
– Pairing and matching items
– Cloze/Modified Cloze
 Production

– Completion items
– Transformation items
– Paraphrase
– Rearrangement
– Editing
– Combination and addition items
– Items involving the changing of words
Reasons for using grammar recognition task

–◼ More material needs to be covered.

–◼ Test different levels of learning.
–◼ Have little time for scoring.
–◼ Not interested in evaluating how well a test
taker can formulate correct answer.
–◼ Have a large number of test takers.
Reasons for using grammar production task

– To evaluate a person’s ability to formulate a correct answer.

– Having more time to score the items
– To test a persons ability to apply concepts and information to a
new situation.
– Having a clear idea of the aspects and concepts that should be
tested.
DIALANG grammar sample question:
MC
 Multiple Choice (MC)

– They can be tricky or too picky

– Difficult to test attitudes towards learning
– Knowledge is limited to options provided
– Difficult to construct at higher levels
– Encourages guessing (25% chance)
– More than one option may be possible
– All options must be grammatically possible
But it can be used to test continuous elements.

A: Poor man, he ________at that for days now.

B: Why doesn’t he give up?

a) was working
b) has been working
c) is working
d) had worked
 Gap filling:
John usually ______(work) from Monday to Friday

– They must be part of a broader context

– More than one option may be possible
– First gap may be offered as a sample
– Difficult to test attitudes towards learning
– Knowledge is limited to options provided
– Limitations result in negative backwash
– Focus should be on the aspect assessed.
Cloze/C-test:
John ______ works ______Monday ______Friday

– ◼ More than one option may be possible

– ◼ First letter may be offered as a sample
– ◼ Difficult to test attitudes towards learning
– ◼ Some words may be impossible to guess
– ◼ Can test student’s IQ rather than language proficiency
– ◼ Difficult for some students (bias)
Sentence Building:
Friday/usually/to/John/Monday/work

– Tests recognition and/or production

– More than one option may be possible
– Difficult to test attitudes towards learning
– Some context may be added
– Can test student’s IQ rather than language proficiency
Sentence Transformation:
John is a doctor. He ______________ (work)

– More than one option may be possible

– Clues will limit the answers
– Difficult to test attitudes towards learning
– Some context may be added
– Instructions are essential
– Few aspects might be assessed
 Correction/Editing:
John usually working from Monday to Friday

– Tests recognition of mistakes

– Can also test production through corrections
– More than one correction may be possible
– Mistakes must be grammatically possible
– Good backwash
– Context is essential
 Paraphrase

• Require the student to write a sentence equivalent in

meaning to one that is given.
• It is helpful to give part of the paraphrase in order to
restrict the students to the grammatical structure being
tested.
▫ When I came, my mom was setting up the dinner table.
▫ When I came, the dinner table____________
What is your best preference of
grammar task type? And why?
Scoring Production Grammar Tests
• The important thing when scoring is to be clear about what each
item is testing, and to award points for that only.

• Nothing should be deducted for non-grammatical errors, or for

errors in elements of grammar which are not being tested by the
item.
• If two elements are being tested in an item, then points may be
assigned to each of them.
• Alternatively, it can be predetermined that both elements have to
be correct for any points to be awarded.
• To ensure scoring is valid and reliable, careful preparation of the
scoring key is necessary.
(Weigle, 2002)
DIALANG scoring
Why test vocabulary?
• If there is little teaching of vocabulary, it may be
argued that there is little call for achievement tests of
vocabulary.

• For those who believe that systematic teaching of

vocabulary is desirable, vocabulary achievement
tests are appreciated for their backwash effect.
• The usefulness and feasibility of a general diagnostic
test of vocabulary is not readily apparent.
• In placement tests, we would be looking for some
general indication of the adequacy of the student’s
vocabulary.
 Testing vocabulary

• Not a separate section in any of the “big three”

• Common in achievement and placement tests as well as
assessment instruments for SLA research
• Beglar & Nation (2014) distinguish three constructs:
– Receptive vocabulary size
– Productive vocabulary size
– Depth of lexical knowledge
Writing items

–Testing recognition
–Testing production
Recognition: MC

– Synonym
– Definition
– Appropriate words for context

– Distractors:
Should not be very too common
Should be about the same length
Testing production

–Pictures
–Definition
–Gap filling
DIALANG vocabulary test sample question