Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

ESL Reading & Writing Placement Test Intensive English Program, MIIS Winter, 2011
Based on Language Assessment with Professor Jean Turner (EDUC 8540, Spring 2011)

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

Table of Contents Overview Report Appendix A: Articulation Document (AD)1 Appendix B: Self-assessment Component Appendix C: the RWPT Test (Reading and Writing) Appendix D: Answer Key and Scoring Instructions

11. This

document was produced by the MIIS ESL program. Attached as Appendix A in the hard copy, it has a separate page range (1-9) from the rest of the report due to a conflict in Microsoft Word page format.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

The Reading and Writing Placement Test (RWPT) was developed for the purpose of revamping the existing placement exam for reading and writing (R&W) used by the English as Second Language (ESL) Center at the Monterey Institute of International Studies (MIIS). The development of the RWPT integrates rationales from multiple sources: assessment theories, research findings, empirical data from the ESL program, test results from the previous piloted RWPT, and consultation from the field experts. Design and Development of the RWPT In this section I will summarize the shortcomings of the existing ESL placement exam, and then elaborate on how these issues can be ameliorated through the implementation of the new RWPT. The Existing Placement Exam for Reading and Writing The Intensive English as Second Language Program, also known as IEP2 within the MIIS community, attracts foreign students in their early 20s from all over the world. More than half of these students have the goal of matriculating into a U.S. college program. The IEP program has been using an R&W placement exam3 and a 1-8 point scale to decide the integrated proficiency level of the students R&W skills. Following the test, the students new to the IEP are normally placed into one of three classes: A (1-2), B (3-4), and C (5-6)4. The R&W level descriptors on the 1-8 scale are included in the ESL Articulation Document (AD) which is attached in Appendix A. The levels are as follows: 1-3 (novice), 4-6 (intermediate), and 7-8 (advanced).

22. 33.

In this report, the terms IEP and ESL interchangeably refer to the MIIS ESL program. The current ESL placement test comes in two forms, namely lower (less advanced) and higher (more advanced). The lower is what this project is focused on, targeting learners at levels 1-6. The higher form is used to diagnose levels 6-8. 44. A large majority of test takers are placed into levels 3-6, with most of the remaining students being placed into level 2.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

The existing test has several weaknesses. First, as the IEP program administrators have acknowledged, the number of reading subtests needs to be increased for a better judgment of the learners reading proficiency (personal communication, July 2011). Secondly, the test lacks a clear definition of constructs beyond the AD level descriptors. Without clearly defined test constructs, the interpretations of the test scores are open to question (Bachman & Palmer, 1996). Thirdly, the authenticity of the test needs to be improved. Bachman and Palmer (1996) contend that all assessed features of the language should match the test takers domain of target language use (TLU). The existing test prompts are not entirely composed of authentic materials derived from the students TLU. Nor do the tasks justify a high degree of authenticity. Lastly, the IEP graders rely mainly on their intuitive judgment rather than a standard rubric to determine each students writing placement level. The New RWPT (Lower Form) The design of the new RWPT attempts to address the issues illustrated above, and highlights a number of new features to further enhance the effectiveness of placement. The discussion regarding the test design delves into the following topics: test usefulness, the guiding principles from the AD, the interconnection of R&W skills, the test constructs and subskills, and the component of self-assessment (Appendix B). Test usefulness. Test developers should always keep in mind test usefulness to ensure the quality of a given test (Bachman & Palmer, 1996). Bachman and Palmers (1996) model of test usefulness serves as a theoretical framework to provide systematic directives for developing this new test. This model consists of six test qualities to measure a tests overall utility and soundness. In Table 1 below, the requirement refers to the theoretical standard for each quality. The implementation plan introduces actions taken in order to meet the standard in question.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

Although the six qualities are listed individually in Table 1, they should not merely be evaluated independently, but in terms of their combined effect (Bachman & Palmer, 1996, p. 18). Table 1 Test Usefulness Qualities Reliability Requirement (Bachman & Palmer, 1996) Variations in the characteristics of assessment tasks corresponding to variations in TLU tasks Implementation Plan (c.f., Bachman & Palmer, 1996) Avoid a wide range of construct components to maximize reliability (highstake test) Use task types reported with high reliability in the literature Use analytic scoring rubric Match test tasks with language tasks in the TLU (MIIS ESL language community) Define test constructs integrating the current research findings on R&W Build a strong link between R&W skills through the writing subtest Use authentic texts from the TLU as test input Confer the AD regarding the types of text and task associated with level descriptors Select texts and design tasks in line with the language abilities measured by the test constructs as well as test takers background features Use the test result as an

Construct validity Test score interpretations correspond to the language abilities measured

Authenticity

The degree of correspondence of the characteristics of a test task to the features of a TLU task (content validity)

Interactiveness

The degree of engaging a test takers language ability, topical knowledge, and schemata via test tasks (construct validity)

Impact

The tests impact on the test takers

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL and the IEP R&W curriculum

inference of class placement Introduce skill-integrated tasks into the classroom (washback) Collect and screen test prompts Design assessment tasks Hand-grade the tests by 2-3 in-house ESL instructors Run tests with no extra need for technology or cost

Practicality

Resources required for developing and implementing the RWPT

The guiding role of the AD. Apart from above, other practical considerations have been integrated into the test design. Maintaining the integration of R&W skills is compatible with the ESL curriculum design. In addition, the AD scale descriptors provide referential guidelines in terms of the selection of test stimulus and task functions. One reason for doing so is that the program staff is already familiar with the AD scale. Another reason is that the AD scale resembles multi-level learning objectives of the IEP program. To base the test content on course objectives is one common design scheme for a placement test (Bachman, 1990). Furthermore, most of the new students would most likely fall within the range 1-6 (personal communication, July 2011), which provides a rationale to base the test design around AD levels 1-6. The inter-connections between R&W. The new RWPT aims to promote and reinforce the links between R&W, and there are theoretical reasons for this beyond the IEPs curriculum design. L2 R&W research literature reveals several rationales for merging R&W. Conventionally, language literacy is associated with learning how to read and write (Hedgcock & Ferris, 2009). The New Literacy Studies (NLS) argues that literacy cannot be reduced to cognitive skills. Literacy means more than just acquiring knowledge. It includes what R&W do in the context of social interaction (Barton, 2007). The authentic texts used in this test simulate the milieu of the social interaction of the test takers. The writing task of this test (Appendix C) was based on

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

authentic materials so that the test takers must decode the passages (knowledge) in such a way as to produce a writing sample (what to do with the knowledge) so as to reflect how the language is used socially (literacy). Furthermore, a students R&W abilities themselves are interdependently related. The development of one modality is indispensable with the cultivation of another. Writing proficiency is heavily determined by what one gains at the end of the reading process (Just & Carpenter, 1987). Unsurprisingly, a proficient writer normally possesses a sound reading ability (Ferris & Hedgcock, 2005). Likewise, an experienced reader's ability to identify critical information in the written text can partially be traced to the schematic knowledge of his/her writing experience (Hirvela, 2004). Hayes (1996) argues for three types of reading that are critical to writers. A writer needs to read his/her own writing in order to detect errors, measure coherence, build organization, and make possible improvements. This process is known as reading to evaluate. Secondly, the reader must comprehend source texts, and thirdly, their written instructions, both from which the need to write may follow. Hayes (1996) justifies using reading texts as a stimulus to generate writing tasks, and integrates reading ability into the list of writing skills (below). Reading construct. The current research on reading reveals an integrative view of bottom-up and top-down approaches of information processing (Hedgcock & Ferris, 2009). How multiple processes interact is enhanced by the readers schematic knowledge, such as cultural expectations, genre knowledge, and linguistic knowledge (Hedgcock & Ferris, 2009). The reading construct of the RWPT assesses the readers ability to extract useful information from both top-down and bottom-up processes via the aid of schematic knowledge.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

Grabe and Stoller (2011) use a car-related metaphor to illustrate the interplay of bottom up and top-down processes along with schemata. Arriving at reading comprehension is comparable to driving a car to a destination. The fuel is comprehension at word-level (starting point of bottom-up), and the engine is constructing meaning out of linguistic units at a higher level, such as sentences and paragraphs (bottom-up moving up). Finally, it is the higher order of skills and meta-strategies, such as use of schematic knowledge, inferencing and predicting abilities as well as other top-down controls, which push the vehicle towards its destination, to meet reading comprehension goals from the text (top-down). Successful reading therefore requires a smooth coordination of multiple skills and sources of knowledge. Proficient readers are more likely to be able to do so than novice readers (Hedgcock & Ferris, 2009). This sheds light on how an integrative approach of test design can differentiate between proficient and novice readers. To define the reading subskills measured by the RWPT, I extracted the subskills repeatedly featured in the AD descriptors for levels 1-6, which signifies the emphasis of reading abilities required of the learner. Due to the fact that the AD lacks evaluation of any interactive skills, I integrated several interactive skills from Rosenshines (1980) categories of reading attributes. These interactive skills taken from Rosenshine (1980) and the subskills from the AD form the following list of reading subskills measured by the RWPT. Information sequence recognition (interactive) Recognition of word(s) in context (bottom-up) Identification of main idea(s) (top-down) Decoding of detail(s) (bottom-up) Inferencing ( interactive)

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Comparing and contrasting (interactive)

Writing construct. As discussed above, writing represents a form of production one performs in literacy events. The nature of writing has both cognitive and social dimensions. Undoubtedly, writing requires knowledge of linguistic features that are stored in the writers working memory (Hayes, 1996). Writing occurs in a social context where the writer needs to have the target audience in mind (Hayes, 1996). To address a particular audience requires an understanding of discourse and sociolinguistic knowledge, such as social register, functional uses of language, and other aspects of audience consideration, as extracted from Grabe and Kaplans (1996) taxonomy of language knowledge. Writing based on text sources and their instructions requires the ability to interpret the input provided (Hayes, 1996). Hayes (1996) maintains that the writers reflection is a process deeply influenced by text interpretation. Reflection is how new ideas are formed out of previously existing internal representations. The writer uses these to form his/her schemata in linguistic production (Ferris & Hedgcock, 2005). Individual schemata have to do with ones expectations of the texts content and organization (Carrell, 1987). Thus the writer relies on his/her prior experience (in addition to the information in the text source) to generate ideas, and to strategize how to present and organize them. To summarize what is written above, the writing construct is defined as the ability to read a text source, identify the audience, utilize schematic notions and discourse knowledge, and to organize them into a coherent body of written text. The writing subskills measured by this construct take several factors into consideration. First of all, one must measure all the major subskills through a rubric. Weigle (2002) suggests the use of analytic scoring rubrics to ensure placement and diagnostic accuracy, which the new RWPT does accordingly. Due to time

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

10

constraints for grading, in this case (two hours assigned to 2-3 faculty members), the design of the rubric has to be efficient and easy to use. A lengthy list of subskills on the rubric would have been counter-productive. Based on my interview with an IEP R&W instructor (personal communication, October 2011), I discovered that the IEPs writing instruction places its major emphasis on grammatical construction, choice of vocabulary, and topical organization (including coherence). The AD level descriptors 1-6 do not evaluate for the language at the level of academic texts or tasks. Only from AD level 7 and above do academic R&W skills begin to be evaluated. Therefore, rhetorical knowledge and other higher order knowledge, typically associated with academic writing, is not evaluated in the rubric of the new RWPT. On the other hand, sociolinguistic (e.g., audience consideration) and discourse knowledge (e.g., genre structure) are important to writing assessment of all sorts (Weigle, 2002). Therefore, audience consideration and discourse knowledge are also included in the list of subskills below. Text interpretation was regarded as a separate skill area by the RWPT rubric (Appendix D) as a result of integrating R&W skills. To avoid a lengthy list I grouped distinct but interrelated subskills into the following three skill categories. For language use and organization I have consulted the ESL Composition Profile compiled by Jacobs, Zinkgraf, Wormuth, Hartfiel, and Hughey (1981). Language use (tense, agreement of subject and verb, articles, prepositions, syntactic structures, and effective word choice, etc.) Organization (cohesion, coherence, and genre structure) Text interpretation (topical relevance, understanding of source text and instructions, and consideration of audience) Self-assessment. A self-assessment survey (Appendix B) was introduced for the new RWPT for two main purposes. First, the IEP is considering installing a self-evaluation tool to

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

11

preliminarily identify applicants who may not be suitable for the ESL program at MIIS (e.g., low-novice learners at English alphabet level or high-advanced learners ready for university courses). Secondly, the survey can serve as additional evidence of test validity when compared with the test score as well as the learners current ESL placement level. The rationale of the surveys design was to distinguish at which point test takers feel less comfortable with certain R&W tasks. The survey consists of ten statements, half of which are targeted to the novice level, and the other half to the intermediate. The statements were crafted around the task attributes reflected in the AD. Each statement appears in the format of I can perform certain tasks in reading, or writing, or both of them combined (e.g., I can read and write simple class notes in English). The response is marked by four options (not at all, not really, sometimes, or most of the time) that describe at which level the respondent feels his/her ability is matched with the task. I have devised a method that converts the students responses into a rough proficiency level in congruence with the AD scale. The conversion formula is based on the largest observed concentration of answers, namely sometimes and most of the time, among the statements (see Appendix B for more details). The formula helps identify at which level the test taker feels most confident in his/her performance, which will be used as the self-reported proficiency level. The Test Components Prompt Attributes As mentioned above, AD levels 1-6 provide major guidelines in the selection of test stimulus. All test prompts were chosen based on the ESL students life-familiar topics, which were derived from the AD level descriptors 1-6. To ensure test authenticity the test prompts were all exacted from authentic materials in the MIIS language community (Table 2).

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Response Attributes

12

The decision regarding the formats of task rests on several factors. First, despite the small size of the program, time-efficient and easy-to-use scoring is preferred by the IEP, mainly due to the teaching staffs tight working schedule upon the arrival of new students (Personal Communication, July, 2011). Normally 2-3 instructors need to grade an average of twelve tests within two hours. The MC format allows only one correct answer, which ensures the efficiency of scoring (Bachman & Palmer, 1996). Plus, as Table 1 shows, test reliability is a key factor to the RWPT, which can be fulfilled by the format of MC (Bachman & Palmer, 1996). Due to the reasons above, MC items take up two-thirds of the RWPT (Appendix C). Secondly, constructed response items are used as an alternative format to MC to elicit more solid evidence of reading comprehension (Hedgcock & Ferris, 2009). Also compared to MC, short-response items minimize the chances for guessing (Bachman & Palmer, 1996). The answers for Items 14 and 15 are relatively controlled by the facts embedded in the prompts. Item 13, particularly targeting readers use of schemata, is open to more free production and variations of answer. Thirdly, as defined in the test constructs, the RWPT assesses the readers ability to use interactive reading subskills along with R&W integrated literacy skills. Text reconstruction tasks have a good potential to meet this assessment goal, where readers have to activate their schemata and integrate them with their strategic skills (Hedgcock & Ferris, 2009). For both items 16 and 17, test takers need to reassemble the order of four sentence strips into a coherent short passage. The authenticity of these two tasks lies in Hayes (1996) contention of a writers necessary reading ability to examine their own writing for organizational coherence (integration of R&W).

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

13

Lastly, the writing task (item 18), an email response based on the reading of two rental advertisements, incorporates the writers text interpretation skills of the test prompts and task directions (Hayes, 1996). The minimum word limit is 150 words to ensure a ratable sample (Hamp-Lyons, 1991). Besides the use of authentic sources for the prompts, the task also models itself on authentic social interaction: the writer, in need of a new residence, receives two advertisements from a friend, and writes the friend back to explain his/her choice via comparing the two places and giving supporting facts, such as the needs of his/her personality and lifestyle. To link the test tasks with the test constructs defined above, Table 2 below offers a quick glimpse into the subskill(s) measured by individual test task. Table 2 Distribution of Text Types, Task Types, and Subskill(s) Measured cross Test Items Item No. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. Type of Text Public sign on campus Public sign on campus Looking-for-roommate advertisement Same as Item 3 Same as Item 3 Social event invitation Apartment tenant notice Same as Item 7 Movie synopsis Campus-related news story Same as Item 10 Same as Item 10 Email invitation to a workshop Poster of a campus photo contest Same as Item 14 Description of a potential roommates life routine Type of Task MC MC MC MC MC MC MC MC MC MC MC MC SR* SR SR TR* Focal Subskill(s) Identification of main idea(s) Identification of main idea(s) Recognition of word(s) in context Decoding of detail(s) Decoding of detail(s) Identification of main idea(s) Identification of main idea(s) Recognition of word(s) in context Identification of main idea(s) & inferencing Identification of main idea(s) Recognition of word(s) in context Decoding of detail(s) Identification of main idea(s) & inferencing Decoding of detail(s) Decoding of detail(s) Information sequence recognition

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL 17. 18. Campus-related news story Sublet rental advertisements (2 pieces)
Note. SR*: short-response; TR*: text-reconstruction; W*: writing

14

TR W*

Information sequence recognition Language use, organization, and text interpretation (incl. the reading subskill comparing and contrasting)

Answer Key and Scoring Rubrics One way of weighing test components is to take into account pedagogical implications (Alderson, Clapham & Wall, 1995). To match the ESL curriculum, the test will give an equal weight to reading and writing. To meet this goal as well as to comply with the AD 1-6 level design, I have assigned the reading and writing sections an equal value of 6 points. The average of the individual R&W results combined determines the final placement level (1-6). The MC items in the reading subtest are objectively measured by a standard key (Appendix D). The answers to the short-response items may involve graders subjectivity. One of the text-reconstruction tasks is open to partially correct answers. Suggested correct answers and solutions to partial credit are also discussed in the answer key (Appendix D). The tallied raw score of the reading subtest is then converted to a 1-6 point scale for level assignment (Appendix D). The cut-off points suggested by the scale merely represent a tentative proposal. The results from future pilot tests should be conducive in fine-tuning the scales converting accuracy in relation to the score band. Weigle (2002) contends that analytic scoring, which measures different writing attributes and provides diagnostic information separately, has a potential to assess the writers ability more accurately than holistic scoring. Due to the high-stakes nature of a placement test, an analytic scoring rubric is used to calculate the test takers writing level (Appendix D). As indicated in the writing construct above, three writing skill areas formulate the focus of the rubric. Each writing skill area (worth 2 points) is evaluated with four possible results: poor, fair, good, and excellent.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

15

For the final writing score the results of the three independent areas are added up. The entire writing task is worth 6 points. Test Administration and Pilot Testing Owing to the pilot testing of the older RWPT (see my previous C6 attached in C11 with readers comments) I have maintained contact with some of the test takers. During the winter break of the ESL program I sent out a group email seeking a volunteer to take the new RWPT. One respondent asked if he could finish the test electronically. On January 2, 2012, I emailed the volunteer the self-assessment form, the RWPT test, and test instructions. These instructions included directions such as finishing the test in one sitting, and timing oneself. The following are my suggested testing procedures and resources for a future pilot test, some of which were integrated into my email instructions to the volunteer test taker. The pilot test should take place in a quiet and comfortable classroom. Prior to the test, the proctor told the test takers to bring their own pencils and dictionaries. During the self-assessment stage test takers can use dictionaries to aid their reading of the evaluative statements. Upon taking the test no dictionaries or electronic devices should be allowed. All cell phones must be turned off. Although the test seems pretty straightforward, the proctor should prompt test takers to record their concerns regarding specific items as well as their overall impression of the test in the feedback section. He/she should also be prepared to answer questions from the test takers regarding the task instructions. Meanwhile, the proctor can consider holding a post-pilot-test talk to elicit more informal feedback from the test takers. This action provides an additional measure to enhance the tests face validity (below). I received the completed test from the volunteer on January 7, 2012. He reported that it took him three minutes to finish the self-assessment, and eighty-five minutes to complete the

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

16

entire test. This small fact can be used as a preliminary measure of the required time length for the RWPT. Test Validity and Reliability Validity The RWPT, a criterion-referenced test, was developed for a program-specific diagnostic purpose. There is a lack of an established placement test used for the same purpose to compare the results with. Therefore, criterion validity is not a relevant concern. The major evidence of validity for the RWPT lies in the degree of correspondence of the test tasks to the test construct (construct validity), the degree of the tests content representative of the content measured (content validity), the comparison with the test takers current placement level (a weak form of concurrent validity), and the participants opinions of the test experience (face validity). I obtained valuable feedback from two experienced ESL and testing specialists (native speakers of English) who helped me improve the test items and evaluated the validity of the construct along with its content at their face value. By measuring individual test items against the subskills illustrated in Table 2, they both affirmed an observable corresponding relationship between the subtests and the test constructs. Also, they endorsed the overall test authenticity and the integration of R&W skills assessed by the writing task. Lastly, they both believed that the tests content (choice of test stimulus and item types) seemed to match the R&W needs of the ESL students at MIIS. The first person is an ESL teacher from University of Oregon. He streamlined the instructions for Item 13 to make the meaning more clear. He also questioned whether I should use the idiomatic expression pave the way as a part of the correct answer for MC Item 10, which may test additional linguistic knowledge from what the item intends to measure. In light

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

17

of this concern I reworded the correct answer. Lastly, he suggested I remix the order of sentence strips from the sequencing tasks, Items 16 and 17, to increase their overall difficulty due to the short length of each text. The second specialist is my original class project advisor Prof. Jean Turner. She also suggested I simplify the wording of some of the task instructions to take into account low-proficiency readers. She suggested ways to rephrase the question of Item 14 to further align the task to the elicited responses indicated in the answer key. One important way to measure face validity is to gather feedback from test takers regarding the tests relevance to their learning (Gronlund, 1998). I looked into the volunteers responses to the feedback questions. Also, in my email I encouraged the test taker to give additional comments on the test, such as whether he found the task instructions straightforward and readable enough. The volunteer did not raise any questions regarding any test items. He found the test useful to measure his R&W skills, and believed the language use featured in this test was relevant to his R&W needs in real life (test authenticity). In the future the pilot test administrators should continue to elicit written and oral feedback from all test takers for a more thorough estimate of face validity. To gather a preliminary gauge of the concurrent validity, I calculated the volunteers test score and compared the result to his current ESL placement level as well as self-rated proficiency level. The volunteer was placed into level 5 by the IEP program (personal communication, December 2011). He scored level 6 for reading and 4.5 for writing in the RWPT. The average of the two, 5.25, marks his RWPT placement level. This result matches his current IEP R&W placement level. His self-assessment falls in the range of level 4-5, which also seems to correspond well with his RWPT test result.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL For the future pilot test, Spearman's rank order correlation coefficient should be

18

calculated between the RWPT placement level and the test takers current R&W class level. Due to the fact that the self-assessed proficiency is an estimated range rather than a discrete number, Spearmans correlation coefficient is not applicable. However, the test administrator can nevertheless empirically compare it against the other two placement levels for additional evidence of concurrent validity. Reliability Inter-rater reliability should be considered as the main evidence of test reliability for the writing subtest. Two raters, preferably experienced in teaching and grading ESL writing, should receive a brief training on using the scoring rubric before the pilot test. After having graded the test, they should provide feedback on the scoring effectiveness of the rubric as well as problems detected. The input will lend a basis for further revision of the rubric. For the reading subtests I propose using the Kuder-Richardson formula 20 (KR-20) to calculate the RWPTs internal consistency. The proposal is based on two main reasons. First, KR-20 is ranked as a conservative estimate of reliability (Brown, 2005). It seems to be compatible with the high-stake nature of the RWPT. Secondly, Bachman (1990) argues that KR20 tackles the shortcomings of various split-half methods by taking the average of all the possible split-half coefficients on the basis of the statistical characteristics of the test items (p. 176). Conclusion The careful planning around test usefulness of the RWPT is meant to achieve a fair, justifiable, and accurate placement decision. The test design, particularly in terms of test authenticity, may generate positive washback to the instruction and curriculum design of the IEP.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

19

The initial pilot test with one volunteer test taker suggested a positive potential of the RWPTs placement ability. The RWPT project can be tremendously enhanced through full-scale pilot testing with test-takers representing a wide-range of proficiency levels on the AD scale 1-6. The inter-rater grading process and statistical item analysis (e.g., item facility and discrimination) will generate more evidences for test reliability. The item analysis result will also come out as an informative ground for individual item revision, possible removal of certain items, as well as adding new tasks. I have gained a tremendous amount of knowledge and experience via conducting this assessment project. Although the test is not in Chinese, the target language I choose for my MA track, it has offered me ample opportunities to (a) glean in-depth insights and theoretical grounds from the contemporary literature on R&W skills, (b) gain hands-on experience in designing a R&W integrated test that promote test usefulness, and (c) enhance my awareness of ensuring construct validity and test reliability through design of assessment tasks and scoring tools. I believe the skill and knowledge components listed above will better serve me in handling language assessment projects in Chinese when the future need arises. (Word count: 4971, not including page 1 and 2)

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL References

20

Alderson, J.C, Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge, UK: Cambridge University Press. Bachman, L. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press. Bachman, L., & Palmer, A. (1996). Language testing in practice. Oxford, UK: Oxford University Press. Barton, D. (2007). Literacy: An introduction to the ecology of written language (2nd ed.). Malden, MA: Blackwell. Brown, J.D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York, NY: McGraw-Hill. Carrell, P. L. (1987). Text as interaction: Some implications of text analysis and reading research for ESL composition. In U. Connor & R. Kaplan (Eds.), Writing across languages: Analysis of L2 text (pp. 47-56). Reading, MA: Addison-Wesley. Ferris, D. R. & Hedgcock, J. S. (2005). Teaching ESL composition: Purpose, process, and practice (2nd ed.). Mahwah, NJ: Lawrence Erlbaum. Grabe, W., & Kaplan, R. B. (1996). Theory and practice of writing. New York, NY: Longman. Grabe, W., & Stoller, F. (2011). Teaching and researching reading (2nd ed.). Harlow, England: Longman/Pearson Education. Gronlund, N. (1998). Assessment of student achievement (6th ed.). Boston, MA: Allyn & Bacon. Hamp-Lyons, L. (Ed.). (1991). Assessing second language writing in academic contexts. Norwood, NJ: Ablex.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

21

Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In C.M. Levy and S. Ransdell (Eds.), The science of writing. Mahwah, NJ: Lawrence Erlbaum Associates. Hedgcock, J. S., & Ferris, D. R. (2009). Teaching readers of English: Students, texts, and contexts. New York, NY: Routledge. Hirvela, A. (2004). Connecting reading and writing in second language writing instruction. Ann Arbor, MI: University of Michigan Press. Jacobs, H., Zinkgraf, S., Wormuth, D., Hartfiel, V, & Hughey, J. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury. House. Just, M., & Carpenter, P. (1987). The psychology of reading and language comprehension. Boston, MA: Allyn & Bacon. Rosenshine, B.V. (1980). Skill hierarchies in reading comprehension. In R. J. Spiro, B. C. Bruce, & W.F. Brew (Eds.), Theoretical issues in reading comprehension: Perspectives from cognitive psychology, linguistics, artificial intelligence, and education (pp. 535-554). Hillsdale, NJ: Lawrence Erlbaum. Weigle, S. C. (2002). Assessing writing. Cambridge, England: Cambridge University Press.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Appendix B Self-Assessment Component

22

Since this RWPT project involves only the lower form, the self-assessment survey is mainly used as an additional piece of evidence of test validity rather than a tool to help the ESL students choose which form of test to take. The self-reported proficiency levels will be compared next to the students test score and his/her current ESL class placement. Statements numbered 1-5 represent R&W skills at novice level (placement levels 1-3). Statements numbered 6-10 represent R&W skills in an intermediate proficiency class (placement levels 4-6). Statements 4 and 5 are intended for upper-novice level (level 3), or as a transition between novice and intermediate levels. Statements 8 and 9 measure upper-intermediate (level-6) reading skills; statement 10 measures upper-intermediate writing skills. Students are asked to circle one answer out of four options on a descriptive Likert scale indicating how confident and comfortable they are with a task. The four options are: 1. Not at all. 2. Not really. 3. Yes, sometimes. 4. Yes, most of the time.

The self-assessment result will only be used as an empirical estimate of the students proficiency level. The key approach is to find out how the answers numbered 3 and 4 are distributed (confidence in task performance). If they are concentrated among the statements 1-5 but not 6-10, the examinee is a self-reported novice learner (levels 1-3). If the 3s and 4s are also concentrated among statements 6-10, the examinee is in the intermediate range (levels 4-6). If he/she circles answers 3 or 4 for statements 8-10, the examinee is likely a level 5-6 reader and writer. If the concentration of answers 3 and 4 is among statements that border between the novice and intermediate levels, as in statements 4-6, the student is likely a level 3 or 4 learner.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL The following statements appear on the self-assessment survey. 1. I can read and understand most public signs (e.g., street & store signs) in English. 2. I can write simple questions based on an English text I understand. 3. I can read and write simple class notes in English. 4. I can read and write a simple email in English to a friend. 5. I can write simple paragraphs in English. 6. I can communicate with my teacher over complicated issues in an email. 7. I can read business correspondences from banks and schools. 8. I can read news events and write a brief summary in English. 9. I can read a news article that voices an opinion. 10. I can write a 2-3 page essay in English.

23

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Appendix C

24

MIIS ESL Reading and Writing Placement Test

Winter, 2011 Name: _________________ Country:______________ Completed Time:_____________

Multiple Choice: Please read each item carefully and choose the best answer. Item 1: Look at the sign in a public bathroom.

1) What does it tell you about toilet paper? A) It is a daily necessity. B) Use all you want. C) It needs to be saved. D) It is bad for our environment.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

25

Item 2: You see a sign posted on the window near a library.

2) This sign means that smoking is not allowed A) within 20 feet of the library. B) anywhere around the library. C) within 20 feet inside the library. D) anyplace in the library.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Items 3-5: You are looking for a roommate and find this advertisement on the internet.

26

I'm a 22 year old student living in Salinas with my boyfriend. We have a smaller dog (not teeny tiny, about 16 lbs) and are open to more dogs. We both like to drink socially or have a drink with dinner. We would sometimes have a few people over, but not an insane amount of partying. Cigarette smoking is fine, but it must be done outside. 3) Based on the context of the advertisement, insane means: A) unexpected B) unspecified C) uncontrolled D) unannounced 4) The writer does not want______ in the house. A) more dogs B) visitors C) smoking D) partying 5) Choose the sentence that best describes how much the writer drinks. A) She does not drink at all. B) She only drinks sometimes. C) She only drinks with her boyfriend. D) She often drinks a lot. Item 6: You received a flier in your MIIS email. CROSS-CULTURAL LUNCHES!!! A GREAT MIIS TRADITION!!! 12:15 Thursday, October 20th Holland Center courtyard Free dessert and refreshments! Six students will sit at each table arranged by nationality and academic program. Faculty members will be there to facilitate a great cross-cultural discussion! To participate, please sign up in advance! Email Professor Peter Grothe at pgrothe@miis.edu. Tell me your name, nationality and academic program. ALL MIIS STUDENTS WELCOME!!!

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL 6) What kind of event is being held on Oct. 20th? A) a light-hearted lecture B) a study group C) a local food sampling D) a cultural exchange experience Items 7-8: Read the note that was left on your apartment door. Dear Resident:

27

Unfortunately, we have to shut the water off on Tuesday, 4/5/11, from the hour of 10:00 AM to 2:00 PM. This is necessary to complete the plumbing work in progress. We regret any inconvenience this may cause you. 7) What do you expect to happen on 4/5/11? A) There will be some repair work done. B) There will be no electricity. C) The office will be closed. D) Some people will move out. 8) What does the word regret mean in the last sentence? A) inform you of B) apologize for C) avoid D) warn you of Item 9: Here is a summary of a movie that is playing downtown. The Music Never Stopped Runtime: 105min. Synopsis An engineer is unable to communicate with his long lost 35-year-old son, a former hippie, who has a brain tumor preventing him from creating new memories until they discover that they can communicate through music, including Bob Dylan and the Grateful Dead.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL 9) If you chose to watch this film, which type of movie could you expect to see? A) romantic comedy B) serious drama C) action thriller D) science fiction Items 10-12: This is a news story going around at MIIS.

28

Thirty-seven Fulbright scholars from countries ranging from Madagascar to Poland and Panama to Bangladesh have arrived for a three-week intensive course at the Monterey Institute designed to prepare them for graduate studies in the United States. The focus of their studies at MIIS is on academic English, public speaking and other graduate school skills. After their three weeks in Monterey, the students will matriculate into masters or doctorate programs at universities across the United States, where they will be studying everything from engineering to tourism, public health to animal breeding and genetics. Although the program is intensive, there are also a number of exciting extracurricular opportunities. The students have already toured both San Francisco and Carmel and have other excursions planned for next weekend. They also visited the Monterey Farmers Market and were hosted for dinner by members of the Monterey Community. 10) What is the Fulbright scholars main purpose for being in Monterey? A) to enrich their cultural experience B) to exchange academic expertise C) to participate in special tours D) to get ready for school programs 11) The word extracurricular means A) extraordinary B) exceptional C) after-school D) recreational 12) Which of the following activity was arranged? A) visiting a farm nearby B) going to restaurants C) eating at a locals home D) hiking in a scenic area

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

29

Item 13: This is an email regarding a MIIS workshop. Please read it and answer the following question. Dear new international students, Please note the email that you received today from my wonderful colleague, Kathy Sparaco. There is an extremely important workshop for all new international students about dealing with the stresses and strains of difficult graduate work and about adapting to American culture and the MIIS academic culture. You will probably find it very, very helpful! There will be many helpful hints and pieces of advice from professionals that you will probably find very useful! Kindest regards, Peter Professor Peter Grothe, Ph.D. Director of International Student Programs Emeritus Monterey Institute of International Studies

13) Faculty members will hold a question-and-answer session at the end of the workshop. You will attend the workshop. Write two questions you will ask, or that might come from the audience. Be as specific as you can.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

30

Items 14-15: You saw a poster on the campus. Please provide a short response to each question below.

14) Based on the poster above, name two kinds of MIIS students who would most likely participate the contest.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

31

15) What will happen to the top-selected work? (2 details)

Items 16-17: Below you find the titles of two passages. Use the passages title as a clue to put the sentences of each passage into a coherent order. Write the numbers in order above the broken lines. For passage 1, there is more than one full-credit answer. However, for passage 2, there is only one full-credit answer. 16) Passage1: a self-description My life-routine 1. I am usually home by 6:00 PM, spending my evenings making dinner, watching the news, going for a walk, reading or watching a movie. 2. I usually spend Mondays at home cooking, doing laundry, cleaning and getting ready for my workweek ahead. 3. I have Sundays and Mondays off. 4. I am an early riser, up and out by five to swim before heading off to work. Answer: ___ ___ ___ ___ 17) Passage 2: a news story Tatiana Ivanova: First Translated Book Published Weeks after Graduation 1. Tatiana has already started translating her next book for the publishing house, Because of Mr. Terupt by Rob Buyea. Not a bad start to what is sure to be a wonderful career! 2. Tatiana Ivanova (MATI 11) graduated in May from the very demanding Translation and Interpretation degree program at the Monterey Institute. 3. Her translation was published by the Russian Childrens publishing house Pink Giraffe in the beginning of June, less than a month after she received her masters degree.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

32

4. As if that was not enough of a challenge, she somehow found time in between classes and assignments to work on a translation of the 1989 award-winning novel Holes by Louis Sachar into Russian. Answer: ___ ___ ___ ___ Item 18: Write an email response You are thinking about moving to a new location in Monterey. Your friend at MIIS sent you two rental advertisements. Compare the two advertisements carefully, and write your friend back about which place suits you better. Consider the various features of each place, and give specific reasons why your choice appeals to you. Discuss aspects relating to your life-style and personality. You must write at least 150 words. For convenience you can use the terms place 1 and place 2 in your writing. Place 1 $750 beautiful house in the heart of Monterey Rooms for rent in beautiful house, 750 a room, available now. looking for tenants who are clean and self-sufficient. Need to share bath with the other tenant. Close to MIIS, DLI, and downtown. Beautiful private location with nice front and backyards. new paint and carpet, plenty of parking, and lots of room for your own garden if interested ,big outdoor grill with sink on back deck, safe neighborhood with minimal street traffic. Available now call 831 XXX XXX. Place 2 $670 room for rent in Monterey Room for rent in fun, Monterey home. The house is located very close to MIIS and offers washer/dryer, garage, garbage service (included) open fireplace, huge kitchen and living room and sunny patio area for BBQ or sunbathing during summer. Deposit (300) and rent required (670). Current residents are 20's 30's male/female mix who like to party and have fun but also expect residents to be responsible and clean up after themselves in the public areas of the house. We are seeking a candidate who shares similar interests and has an understanding of our expectations of responsibility. Please txt 831 XXX XXX.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

33

----------------------------------------------End of Test------------------------------------------------------Feedback 1. Overall, do you find this test overall useful to measure your reading and writing skills?

2. Do you find all the situations relevant to your English communication needs in Monterey? If no, which one situation(s) were not useful?

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Appendix D Scoring Key and Rubrics

34

General Instructions The rating of this test will retain the existing 8-point scale adopted by the IEP. The scale divides test-takers into four R&W classes, namely A (1-2), B (3-4), C (5-6), and D (7-8). Due to the fact that the test in question is the lower form, only levels 1-6 will be tested. In other words, the score range for the reading and writing section each is 1-6 based on the 8-point scale. The reading and writing subtests will be calculated separately, each of which accounts for 6 points. The averaged score of the sum from both sections will be used as the basis for placing individual the student. The Reading Section Items 1-12: Multiple Choice (each item=1 point, total=12 points) 1. C 2. A 3. C 4. C 5. B 6. D 7. A 8. B 9. B 10. D 11. C 12. C Items 13-15: Short Response (each item=2 points)

13. The two written questions should pertain to either graduate-study or culturally related issues (each question=1 point, total=2 points). Below are a few examples of possible answers. Please note that the first two examples are based on the answers given by the volunteer test taker mentioned in the report.

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL finish?

35

How do I deal with multiple class assignments that I do not seem to have enough time to

When I talk to my American classmates, they seem to understand me very well despite

my errors. But how would I know when I made an embarrassing mistake in my speech? 14. Does MIIS have free tutors who can help me with my academic writing? Can MIIS students give their professors thank-you gifts? How can I rapidly increase my vocabulary in academic reading? 1) A MIIS student who likes photography/taking photos (1 point). 2) A MIIS student who has traveled to exotic and scenic places (1 point). 15. Any two of the three details below (each detail=1 point, total=2 points). 1) The photos will be put on display (at the International Bazaar) 2) The photos will become MIIS property. 3) The photos will be used in the MIIS calendar. Items 16-17: Text reconstruction (each item/passage=2 points) Table 1 Scoring Rubric for Items 16 and 17 Item No. 16 2 points 4 1 3 2 or 3241 1 point The answer contains only one pair of cohesive sentences (as in 4 1 and 3 2). N/A 0 point The answer contains no cohesive order of sentence pairs (as in 4 1 and 3 2). Any other order

17

2431

Once the rater has tallied the total reading points from Items 1-17, he/she needs to refer to the converting scale in Table 2 in order to translate the score into a proficiency level. Table 2

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Converting Test Takers Total Reading Score into Levels 1-6 Number of points Converted level (AD 1-6) 1-4 1 5-8 2 9-12 3 13-16 4 17-19 5

36

20-22 6

The Writing Section (Item 18) An analytic scale facilitates scoring when an individual has uneven strengths or weaknesses among the skill areas (Bachman & Palmer, 1996). It is thus known to work more accurately and reliably than a holistic scale for a placement test. Therefore an analytic scoring scheme was proposed for assessing the writers samples for the RWPT. To comply with the AD levels 1-6, each writing skill area is assigned a value of two points. There are four levels in each area, marked by poor (=0.5), fair (=1.0), good (=1.5) and excellent (=2.0). The rater needs to decide which level represents each skill area from the students writing sample, and add all three up to reach a final score. A writing sample that successfully demonstrates proficiency in all three subskill areas equals 6 points. The list below shows the categories of subskills measured by the rubric. Some of the rubric descriptors are modeled on the analytic scale for ESL composition compiled by Jacobs, Zinkgraf, Wormuth, Hartfiel, and Hughey (1981). Language use (tense, agreement of subject and verb, articles, prepositions, syntactic structures, and effective word choice, etc.) Organization (cohesion, coherence, and genre structure) Text interpretation (topical relevance, understanding of source text and instructions, and consideration of audience) Table 3

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL Scoring Rubric for the Writing Task Language Use Excellent (=2.0) - Very few to no grammar errors - Effective use of varied sentence constructions - Meaning clearly expressed with effective choice of words Organization - Coherently and logically organized text - Ideas clearly presented and fully developed - Effective use of cohesive devices

37

Text Interpretation - Thorough understanding of the text prompts and task instructions - Relevant ideas highly organized - Addresses the target audience properly - Very minor misdirection from following the task instructions - Slight deviation from the topic - Slight inconsistencies in addressing the target audience - Major lack of understanding of the prompt and or the directions - Significant deviation from the topic - Major deviation from addressing the target audience - Little/no understanding of the prompt and or task directions - Ignores the target audience or not enough text (incomplete) to evaluate

Good (=1.5) - Limited grammar errors - Minor mistakes with sentence structure and word choice - Meaning seldom obscured

- Some loosely organized sentences - Main ideas clearly stated but need further development - Minor errors in using cohesive devices

Fair (=1.0) - Frequent grammar errors - Major problems with sentence structure and word choice - Meaning frequently obscured

- Disconnected and logically confusing sentences - Ideas not well-developed and choppy

Poor (=0.5) - Dominated by grammar errors - Meaning confusing or incomprehensible or not enough text (incomplete) to evaluate

- Little/no organization - Little/no logical order or topical cohesion or not enough text (incomplete) to evaluate

References

Component 6: Overview Report of the Revised RWPT Zhengzheng Wu MATFL

38

Bachman, L., & Palmer, A. (1996). Language testing in practice. Oxford, UK: Oxford University Press. Jacobs, H., Zinkgraf, S., Wormuth, D., Hartfiel, V, & Hughey, J. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.