See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/257483772

Examining Writing: Research and Practice in Assessing Second Language

Writing. Studies in Language Testing No. 26, S.D. Shaw, C.J. Weir. University of
Cambridge ESOL Examinations/C...

Article  in  Assessing Writing · January 2012

DOI: 10.1016/j.asw.2011.11.001

CITATIONS READS
0 891

1 author:

Liz Hamp-Lyons
University of Bedfordshire
113 PUBLICATIONS   1,998 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Editorship of Journal of English for Academic Purposes 2002-2015 View project

School-based assessment projects for HKEAA 2005-12 View project

All content following this page was uploaded by Liz Hamp-Lyons on 07 October 2017.

The user has requested enhancement of the downloaded file.

Our reference: ASW 191 P-authorquery-v9

AUTHOR QUERY FORM

Journal: ASW Please e-mail or fax your responses and any corrections to:

E-mail: corrections.eseo@elsevier.thomsondigital.com

Article Number: 191 Fax: +353 6170 9272

Dear Author,

Please check your proof carefully and mark all corrections at the appropriate place in the proof (e.g., by using on-screen
annotation in the PDF file) or compile them in a separate list. Note: if you opt to annotate the file with software other than
Adobe Reader then please also highlight the appropriate place in the PDF file. To ensure fast publication of your paper please
return your corrections within 48 hours.

For correction or revision of any artwork, please consult http://www.elsevier.com/artworkinstructions.

Any queries or remarks that have arisen during the processing of your manuscript are listed below and highlighted by flags in
the proof. Click on the ‘Q’ link to go to the location in the proof.

Location in Query / Remark: click on the Q link to go

article Please insert your reply or correction at the corresponding line in the proof
Q1 Please confirm that given name and surname have been identified correctly.

Thank you for your assistance.

 G Model
ASW 191 1–4 ARTICLE IN PRESS
Assessing Writing xxx (2011) xxx–xxx

Contents lists available at SciVerse ScienceDirect

Assessing Writing

Book review

1
-1 Examining Writing: Research and Practice in Assessing Second Language Writing. Studies in Lan-
0 guage Testing No. 26, S.D. Shaw, C.J. Weir. University of Cambridge ESOL Examinations/Cambridge
1 University Press (2007). 344 pp., ISBN: 978-0-521-69293-9

2 Examining Writing aims to inform its readers of Cambridge ESOL’s ongoing work to validate its
3 portfolio of writing tests. Hence, the focus is almost exclusively on the Cambridge Main Suite of
4 exams with only occasional reference to other assessment systems. This concept, of overtly link-
5 ing theory and research beyond language testing with a specific examinations body’s own practices,
6 enables the agency concerned to lay out for a scholarly audience their practices and also their ratio-
7 nale and larger mission. Alderson, Clapham, and Wall (1995) had paved the way for such in-depth
8 internal analyses in their book-length report and critical analysis of the practices of a range of
9 examining bodies, including what is now known as Cambridge ESOL. This book, and also Chapelle,
10 Enright, and Jamieson (2008), who developed a book-length “validity argument” for the design
11 and development of the ‘new’ TOEFL, the iBT TOEFL, have taken that even further. As Milanovic
12 and Weir explain in the Series Editors’ Note, Cambridge ESOL has been developing, administer-
13 ing and scoring writing tests since the introduction of the Cambridge Certificate of Proficiency
14 in English in 1913. Its exams became ‘large-scale’ at the national level quickly, and internation-
15 ally from the 1930s. The sociocultural focus that is nowadays taken for granted among North
16 American writing and writing assessment researchers is absent from this book; but that set of
17 concerns springs from far more recent social and political events and pressures, and discussion of
18 them has rarely been found emanating from large-scale testing organisations. Differences in his-
19 tory, geography, size and perhaps above all, purposes can explain a great deal about differences in
20 foci.
21 In this book, Shaw and Weir have taken a ‘sociocognitive framework’ or model for conceptualizing
22 test performance developed by Weir (2005) and adapted it specifically to the assessment of writing (in
23 other volumes, Weir and colleagues at Cambridge ESOL have done the same with other language areas,
24 looking closely into one examination body’s principles and practices in assessing a single language
25 ‘skill’.). This narrow focus permits not only an in-depth description and reporting of the processes
26 of writing test development at Cambridge ESOL; it also allows for a valuable detailed self-reflection
27 and gives other language testers, scholars and students as well as practitioners in other agencies, the
28 opportunity to learn from good practice and critique any weaknesses they discover. The structure of
29 the book hangs around Weir’s (2005) model(s)/framework: that book is outstanding: well-organised,
30 excellently signposted, clearly and ‘simply’ written in the best way. Although the model as a whole
31 didn’t completely convince me due to the low level of importance attached to context/culture (in
32 that socio-cognitive framework, ‘context’ is equated with content) I was happy with it in that book
33 because Weir demonstrated his ability to use it coherently and in very practical, down-to-earth terms.
34 Limiting the book away from the complexities of cultural, social and linguistic backgrounds, beliefs
35 and opportunities was a sound way to teach the whole technical structure of how language tests may
36 be developed and validated. But when we come to the assessment of writing, in the sense that we

1075-2935/$ – see front matter

doi:10.1016/j.asw.2011.11.001

Please cite this article in press as: Hamp-Lyons, L, Book review, Assessing Writing (2011),
doi:10.1016/j.asw.2011.11.001
 G Model
ASW 191 1–4 ARTICLE IN PRESS
2 Book review

37 define it in this journal – that is, multi-dimensionally, encompassing multiple levels of education,
38 viewed from multiple perspectives only one of which is second-language learners, and multi-vocally
39 (where a goal, not yet achieved, is to give learners a real voice) – I feel the absence of a sociocultural
40 perspective strongly.
41 Shaw and Weir have slightly updated Weir’s (2005) framework as it relates to writing, mainly by
42 reducing what Weir had called ‘theory-based validity’ with three sub-areas to one, called ‘cognitive
43 validity’ with just one area. What had been ‘Internal Processes’ with two macro-components, ‘Exec-
44 utive processes’ on one side and ‘Executive resources” (comprising language knowledge aspects and
45 content knowledge aspects) on the other, with a monitoring link between them, is now just one list
46 which is headed ‘Cognitive validity’ and which contains items that look very much like the familiar
47 ones of composing processes (macro-planning, organisation, revising, etc, but not including ‘writing’).
48 I found the more elaborated view of composing in Weir (2005) more in line with my experience
49 and my knowledge of the literature on writing development and writing assessment than this newer
50 version.
51 The book is fairly technical and its audience is necessarily limited to those with more than a single
52 MA class in language assessment. It also deals only with second language writing assessment. However,
53 there is a significant dearth of books focussing on L2 writing assessment compared to the recent
54 growth in the area of first language (or ‘unmarked’, non-language specific) writing assessment. The
55 book is valuable for that reason, and also for the unifying thread that runs through it of attention to
56 validity, with careful elaboration of the Cambridge exam body’s theorization and application of its
57 view of validity. It is also particularly valuable for its frequent use of examples from its own writing
58 exams/tests to illustrate good practice, but also problems and dilemmas.
59 It is encouraging that the first chapter after the Introduction (i.e. Chapter 2) is dedicated to test-
60 taker characteristics. However, this turns out to be a short (17 page) chapter and instead of what
61 readers might expect, is mostly concerned with the need to provide for “individual variations” in test-
62 takers which essentially means test-takers with special needs such as dyslexia. After the briefest of
63 nods towards the issues of (1) whether or not to cater to linguistic varieties, as argued by Chew (2009)
64 and Rubdy and Saraceni (2006); and (2) differences of cultural preferences between L2 and L1 writers
65 and among writers from different L1s, Shaw and Weir correctly point to the pragmatic imperative that
66 must drive all large-scale examinations:

67 Cambridge ESOL addresses directly and proactively many of the physical and physiological (sic:
68 but may mean ‘psychological’ in accordance with the heading-levels in the Chapter) features
69 that we have defined in our framework but for the present, given the heterogeneous nature of
70 the candidature and the general nature of many of the examinations, it is limited to avoiding
71 bias arising from these features against individuals. . . (p.33)

72 Chapter 3 on cognitive validity is stronger. The model for cognitive validity hangs around work by
73 Field, a British psycholinguist: this chapter raises some important aspects of linguistic processing not
74 present in earlier models of writing (or other aspects of language acquisition) such as attention and
75 avoidance; but we are not shown how these contribute to the model here, although this information
76 may be in documents not seen, such as test specifications for the various Cambridge ESOL exams, or the
77 reference levels of the Common European Framework of Reference. It would have enriched this chapter
78 if it could have referred the interested reader to empirical work done on these newly emerging aspects
79 of the cognitive processes in composing. The chapter usefully gives examples, drawn from Cambridge
80 tests which, for each test level, describes a writing task, gives an ‘examination report’ (comments from
81 a principal examiner on how the task performed in live use), and then provides a specification of the
82 writing process(es) that might be predicted of writers who are competent to write at the targeted
83 level. This is a resource easily overlooked because it takes some careful processing by the reader: but
84 that care will be repaid with many ideas that arise. A modest little Table on page 61 summarises all
85 this work very handily and is worth spending time on.
86 Chapter 4 deals with context validity, for which one of the aims stated by Shaw and Weir is to
87 “approximate to ‘the performance conditions’ of the authentic real-life context.”(p.63): but we are
88 quickly reminded of the limited definition that is being applied here:

Please cite this article in press as: Hamp-Lyons, L, Book review, Assessing Writing (2011),
doi:10.1016/j.asw.2011.11.001
 G Model
ASW 191 1–4 ARTICLE IN PRESS
Book review 3

89 Context validity relates to the linguistic and content demands that must be met for successful
90 task realisation and to features of the task setting that serve to describe the performance required
91 (p.63)

92 Within this partial view, substantial work is reported on the ‘setting’ (task), on time constraints,
93 physical conditions and other elements that are of vital importance to an exam board, but that only
94 begin to seem important to researchers in writing assessment once they become involved in large-
95 scale testing programmes. Yet all these elements are important, as anyone who has tried to ensure
96 the parellelness of two or more writing tasks within the same setting has discovered. We vary these
97 context conditions at our peril; and though this book reports the work the Cambridge exams have
98 done in a more pedestrian way than we will find in research articles in Assessing Writing, it is clear
99 that considerable work has been put into ensuring that the writing tests at the various levels are
100 as fair as is possible in large-scale conditions. In the section on writer–reader relationships, Shaw
101 and Weir broaden their lens to consider writing from the various perspectives more familiar to writ-
102 ing/composition researchers in North America: social constructivist, expressivist, social-interactional
103 and cognitivist: this section needed to be a good deal fuller for it to have an immediate value in appli-
104 cation to the examples from Cambridge practice on how writer-reader relationship is treated. This
105 chapter also includes in-depth reporting of the handling of lexical resources, particularly vocabulary;
106 brief overviews of discoursal and functional resources; and some consideration of the role of content
107 knowledge. Each of these aspects could easily be an entire chapter in itself, and so the reader is left
108 feeling that they have begun to see that there is a question, but not to see the answer: this is inevitable
109 in a book that endeavours to cover so much territory. Tables 4.21 and 4.22 on pages 126–7 will be
110 of great interest to anyone thinking about how to develop a progression for their own examination
111 system: but it’s unfortunate that these appear after the section on functional resources and in the
112 middle of the section on content knowledge. Since the book has a minimalist Table of Contents there
113 is no way to find these Tables other than working all the way through the book.
114 Chapter 5, on scoring validity, is for me the core of the book. Cambridge ESOL have been scoring
115 formal written examinations since 1913 without break, perhaps a record only beaten by the Chinese
116 imperial examination system two millennia ago. It was a surprise, therefore, to fail to find a section in
117 the chapter on multiple rating. In Britain multiple rating has been happening since before Wiseman
118 reported on it in a scholarly journal in 1949; in North America and many countries multiple marking
119 has been the ‘gold standard’ for 50 years. It is not discussed in this book, but careful exploration of
120 information about the various tests produced and delivered by Cambridge ESOL reveals that even in
121 2011 not all of them use multiple rating. British applied linguists have been criticizing the lack of
122 multiple scoring of the IELTS writing test since it was first introduced in 1980, and yet this is still
123 not happening. Although Shaw and Weir are certainly not responsible for this major omission, this
124 knowledge detracts from my general positive view of the careful treatment of rating scales and rater
125 training in this chapter.
126 Chapters 6 and 7 are short chapters, on consequential validity and criterion-related validity respec-
127 tively. I valued the presence of this focus on consequential validity, as this often left out of the public
128 face of testing agencies. Cambridge ESOL have definitely done a great deal of work in this area. Shaw
129 and Weir refer to Weir’s (2005) argument that for tests to have beneficial effects on teaching and
130 learning, it is necessary to put substantial effort into training teachers for the new content and meth-
131 ods to be used in preparing students in the skills and knowledge needed for success on the test. The
132 chapter reports on the IELTS Impact Study (see Hawkey, 2006) and provides sufficient detail to help
133 others wishing to carry out a smaller-scale study of their own.
134 Chapter 7 discusses a pedestrian yet important set of issues: how do we “compare” tests? This is,
135 of course, really a political question, since the comparisons are needed for educational portability, the
136 movement of scholars from one level to another, one country to another. Yet this is a vital purpose,
137 probably the most vital other than to discover (and inform the learner) whether they have learned
138 and if not, what they still need to learn. The chapter appropriately focuses on the Cambridge Common
139 Scale for Writing project, which has placed all the organisation’s writing tests on a common metric,
140 using techniques explained in the chapter. Shaw and Weir use this opportunity to call other tests and
141 agencies to task for taking easier routes to claiming that their tests relate reliably and validly to the

Please cite this article in press as: Hamp-Lyons, L, Book review, Assessing Writing (2011),
doi:10.1016/j.asw.2011.11.001
 G Model
ASW 191 1–4 ARTICLE IN PRESS
4 Book review

142 Common European Framework. This is an important issue, and the example set by the Common Scale
143 project is one that should be held up to others aiming to similarly give their tests the market edge
144 that: being benchmarked” to the CEFR offers.
145 In the conclusion, Shaw and Weir state that “This volume marks the first comprehensive attempt
146 by any examination body to expose the totality of its practice to scrutiny in the public arena. (p.256)
147 Cambridge ESOL has since gone on to produce volumes on Examining Reading and Examining Speaking.
148 This book and its siblings offer a relatively small audience a great deal of benefit; all of us who care
149 about good practice in writing assessment can find something to praise and something to criticize:
150 we can all learn, perhaps in some unexpected ways, from this book.

151 References

152 Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
153 Chapelle, C., Enright, M., & Jamieson, J. (2008). Building a validity argument for the Test of English as a Foreign Language. London,
154 UK: Routledge.
155 Chew, P. G. L. (2009). Emergent lingua francas and world orders: the politics and place of English as a world language. London, UK:
156 Taylor & Francis.
157 Hawkey, R. (2006). Impact theory and practice: Studies of the IELTS test and Progetto Lingue 2000. Studies in Language Testing 24.
158 Cambridge: UCLES/Cambridge University Press.
159 Rubdy, R., & Saraceni, M. (Eds.). (2006). English in the world: Global rules, global roles. New York: Continuum.
160 Weir, C. (2005). Language testing and validation. Basingstoke: Palgrave Macmillan.
161 Wiseman, S. (1949). The marking of English composition in grammar school selection. British Journal of Education Psychology,
162 19 (3), 200–209.

163 Q1 Liz Hamp-Lyons

164 University of Bedfordshire, Centre for Research in English Language Learning and
165 Assessment, Putteridge Bury, United Kingdom

166 E-mail address: lizhl@hkucc.hku.hk

167 Available online xxx

Please cite this article in press as: Hamp-Lyons, L, Book review, Assessing Writing (2011),
doi:10.1016/j.asw.2011.11.001

View publication stats