CHAPTER 4 An Example of Contstructing A Performance Assesment

Introduction Statewide writing assessments, in place since the 1980s, illustrate large scale performance assessments that often have high stakes, such as graduation or promotion, attached to the outcome. Both the stakes and the size of the constituency impose a degree of rigor in task construction especially relevant to large-scale testing programs, whether for state education departments or national credentialing agencies. These steps also are informative in the development of performance assessments for program evaluation and research studies. In education, development of a writing assessments include : 1. A review of language arts content standards to identify the standards to be assessed 2. Alignment of the writing prompts, scoring rubrics, and administration conditions with the standards 3. Design features and review criteria for prompt development 4. Design features for scoring rubrics 5. Field testing tryouts 6. Benchmarking/range finding 7. Scoring/rater training and monitoring 8. Sensitivity and bias review 9. Standard setting 10. Staff development for teachers

Like the writing process, the phases of test development are recursive. For example, the rubric is developed early in the process; however, it is refined as it is applied to papers in the initial selection of student essays that will serve as examples for each score level of the rubric, a process referred to as range finding or benchmarking. These phases vary somewhat depending on whether a new test is being developed or an existing one revised. In new development, all steps apply. Revision of an existing program may be limited to prompt development and tryouts of new prompts because the ones in the item bank have been used or the curriculum

standards have changed. In the latter case, scoring rubrics would be developed and standard setting conducted in addition to prompt development. The scope of the this book precludes detailed discussion of all the phases listed above in order to fully describe prompt task construction, which addresses the review of the standards; alignment of the assessment with the standards; prompt design and review, rater training, scoring and standard setting are addressed in subsequent chapters in this text. The topics of equating and staff development are left to other texts.

REVIEW OF LANGUAGE ARTS CONTENT STANDARDS

Writing assessment on a large scale begins with a review of the language arts content standards and the identification of the standards to be assessed. Advisory committees consisting of classroom teacher, language arts superviors, and teacher-educators conduct this review, typically led by assessment and curriculum experts from state departments of education. Committee members are selected to ensure content knowledge and pedagogical expertise and to fully reflect the demographic characteristics of the examinee population. It is helpful to have members who are knowledgeable about special needs populations and students who typically perform poorly on writing tests so that their concerns can be raised and addressed before substantive problems are discovered in the prompt and bias review process. Including teachers who represent the grade levels on either side of the grade being assessed provides a reality check of what students know and can do and the knowledge and skills they will need to succed at the next level. Teachers also provide ideas about how to use assessment as a tool for improving instruction and learning and play an important role in creating the staff development of the assessment package. The language arts content or curriculum standards include some writing skills that cannot appropriately or feasible be assessed in a standardized context. The standards also vary in importance to the language arts community. Writing standards typically include (1) genres or types of writing and the distinguishing characteristics of each; (2) writing for a variety of purposes and audiences; (3) conventions-the rules of written, edited English; (4) the writing process; and (5) use of resources and tools (e.g., the Internet, word processors, dictionaries). Standards to be tested are selected

based on their value for instruction; their appropriateness for coverage in a performance- based, standardized context; and the constrains imposed by budget, scoring requirements, and time. Advisory committees debate how they can use the reality of testing to promote sound instruction (recall our conversation about consequences). Although the conventions of written language are included in the standards, on demand writing assessments are rough drafts typically produced without access to the tools that writers use to check the rules. To create writing tasks and scoring rubrics that value conventions over the development and organization of ideas could lead teachers to emphasize grammar worksheets you get what you assess (Resnick & Resnick, 1992). Furthermore, conventions can be efficiently assessed in selected response tests, so it may not be necessary or desirable to emphasize them in a writing test. In contrast, testing student capabilities to incorporate stylistic elements of multiple types of writing or genres can have positive instructional impact (i.e., consequences). Genre offers an illustration of one of the issues considered by an advisory committee. Personal experience narratives, long considered the easiest genre, are not easy for students who, for reasons of culture or gender, are reluctant to share their realities. Imaginative narratives challenge those students who by temperament are inclined to stick to the facts or who are linear thinkers. More important, the nonfiction narrative, the form of narrative found outside of literature and language arts classrooms, is rarely taught or tested. If the writing test includes a range of genres, such as exposition and persuasion, those writers who find narrative more difficult can be taught general skills about development (e.g., how much detail is enough, what is relevant) and organization (e.g., the purpose of the parts of a text introduction, body, and conclusion the logical relationships of comparison-contrast or cause effect) in the context of informational writing. Then teachers can guide students to transfer these skills to the study of other types of writing.

ALIGNMENT OF WRITING PROMPTS, SCORING RUBRICS, AND ADMINISTRATION CONDITIONS WITH THE STANDARDS

The standards serve as background for constructing all aspects of the testing materials, not just the writing prompts. Figure 4.1 shows the alignment of a prompt with the Georgia English language arts standards by connecting wording cues in the prompt to elements in the standard. At the bottom of the figure the persuasive writing topic is linked to the eighth grade standards (i.e., ELA8w2 b,1 ELA8W2 d, ELA8W2 f, and ELA8W2 g) that are assessed by the prompt elements Decide how you would change your schools dress code and Write an essay to convince your principal to change your schools dress code. Thus in responding to the prompt, students will state a clear position or perspective in support of a proposition or proposal (ELA8W2 b). other standards are of course assessed, but not contained in this example. This particular standard on persuasive writing is introduced and taught at several grades before it is tested, providing students with the opportunity to learn. If the test is to be used to encourage instruction in the writing process, administration condition and test materials can approximate the phases in writing, providing the students with cues to plan, draft, edit, and revise. In addition to the scan able response booklet, test materials can include forms for prewriting and drafting and a writing checklist that guides students trough these phases of composing. Directions to students can include familiar process writing terms. Figure 4.2 shows the alignment of this same standard to a section of the analytic rubric. The Georgia writing rubric for eight-grade contains four domains: Ideas, Organization, Style, and Conventions. In the current version of the rubric, each domain is scored on a 5-point scale. In Figure 4.2, the top performance level (i.e., a 5) is shown for each domain. A more complete version of the rubric is shown in Chapter 6 in Figure 6.2. As shown in the top center section of Figure 4.2, the domain of Ideas includes a consistent focus on the assigned topic and purpose.

ASSESSING PERFORMANCE

ELA6W2

The

Georgia Performance Standards for Grade 6-8* student ELA7W2 The student ELA8W2

The

student

demonstrates competence in a demonstrate competence In demonstrated competence in variety of genres. The student produces a variety of genres. a The student a variety of genres student produces a

produces a The

multiparagraph essay that : a. Engages the a

persuasive multiparagraph essay that: reader

persuasive multiparagraph essay that :

persuasive

by a. Engages the reader by a. Engages the reader by establishing a context, creating a speakers establishing a context,

establishing

context,

creating a speakers voice, and otherwise developing reader interest. b. States a clear position of a proposition or proposal c. Supports the position with organized evidence d. Excludes information and arguments irrelevant e. Creates an organizing that are and relevant

creating a speakers voice, and otherwise developing reader interest. b. States a clear position or perspective in support of a proposition or proposal c. Creates structure an organizing that is

voice, and other wise developing interest. b. States a clear position or perspective in support of a proposition or proposal c. Describes the points in support proposition, of the reader

appropriate to the needs, value, and interests of a specified audience, and arranges details, reason,

employing

well-articulated, relevant evidence information

structure appropriate to a d. Excludes specific purpose, audience, and context.

and examples appropriate

and arguments that are d. Includes irrelevant an organizing

relevant information and arguments.

f. Anticipates and addresses e. Creates readers concerms and

structure appropriate to a e. Excludes information and specific purpose, arguments irrelevant that are

counter arguments g. Provides a sense of closure to the writing

audience, and context

f. Articipates and addresses f. Provides details, reasons, readersconcems counter arguments. g. Provides a sense of and and examples, arranging them effectively by

anticipating and answering reader concerns and

closure to the writing.

counter-arguments. g. Supports arguments with detalled evidence, citing sources of information as appropriate h. Anticipates and addresses reader concems and

counter arguments i. Provides a sense of

closure to the writing. *Repetition at grades 6-8 provides opportunity to leam prior to testing in grade 8.
Persuasive Writing Topic

Writing situation
ELA6W2-b

School rules help keep students safe and able to concentrate on their school work. One rule that affects students is the dress code. Your principal is considering changing the schools dress code and wants some information from the students. (Decide how you would change your schools dress code. Directions for writing Write an essay to convince your principal to change your schools dress code [Be sure to include detailed reasons and examples]

FIGURE 4.1. a sample of alignment of a writing prompt with language arts standards for Georgias Grade 8 Writing Test. Reproduced with permission from the Georgia Department of Education (www.georgiastandards.org).

Form the Georgia Performance Standards 8 Language Arts. ELA7W2 The student demonstrate competence In a variety of genres. The student produces a multiparagraph persuasive essay that: a. Engages the reader by establishing a context, creating a speakers voice, and other wise developing reader interest. b. States a clear position or perspective in support of a proposition or proposal c. Describes the points in support of the proposition, employing well-articulated, relevant evidence d. Excludes information and arguments that are irrelevant e. Creates an organizing structure appropriate to a specific purpose, audience, and context f. Articipates and addresses readersconcems and counter arguments. g. Provides a sense of closure to the writing.

Domain 1 : IDEAS. The degree to which the writer establishes a controlling idea and elaborates the main points with examples, illustrations, facts, or details that are appropriate to the assigned genre. Components Controlling Depth of Idea/Focus Development Supporting Sense of Ideas Completeness Relevance of Awareness of detail Genber

Domain 2 : ORGANIZATION. The degree to which the writers ideas are arranged in a clear order and the overalt structure of the response is consistent with the assigned genre. Components Overall Plan Grouping of Ideas Introductio/Body within paragraphs Conclusion Genre specific Sequence of ideas Strategies Transitions

Full command of the components of Ideas. The writing is characterized by most or all of the following:
 Consistent focus on the assigned topic purpose. (b)  Fully developed controlling idea that addresses all aspects of the assigned writing task. (b)  Supporting ideas and elaboration are relevant to the writers topic, assigned writing task. (b)  Supporting ideas are fully elaborated throughout the paper with logical examples and details. (d)  Response contains information that fully addresses reader concems and perspectives. (f)  Uses genre-appropriate strategies to develop the writers ideas. (a, b, d, e, f, g, h) 

Full command of the components of Organization. The writing is characterized by most or all of the following:
 Organizing strategy is appropriate to the writers topic and the assigned genre of writing. The overall strategy facilitates the writers communication of idea. (c)  Logical and appropriate sequencing of ideas within paragraphs and across parts of the paper (c, f)  Introduction engages and sets the stage, and conclusion provides a sense of resolution or closure. (a, i)  Both introduction and conclusion fit the writers ideas and the purpose of the genre. (a, i)  Related ideas are grouped in a logical manner within paragraphs. (c, f)  Uses effective and varied transitional elements to link all elements of the response: parts of the paper, ideas, paragraphs, and sentences. Transitions extend beyond the use of transitional word and phrases. (a) 

Domain 4 : CONVENTION. The Degree to which the writer demonstrates control of sentence formation, usage, and mechanics. Note : In general sentence formation and usage are weighted more heavlly than mechanics in determining the overall conventions score. Components Sentence formation Usage Mechanics correctness subject-verb internal punctuation Clarity of meaning agreement spelling Complexity standard word forms paragraph breaks End punctuation possessives capitalization contractions

Domain 3: STYLE. The degree to which the writer controls language to engage the reader. Components. Word Choice Sentence Variety Audience Strategeis Awareness Appropriate Voice to the Genre

Full command of the components of Conventions. The writing is characterized by most or all of the following : Components Sentence formation Usage Mechanics correctness subject-verb internal punctuation Clarity of meaning agreement spelling Complexity standard word forms paragraph breaks End punctuation possessives capitalization contractions

Full command of the components of Style. The writing is characterized by most or all of the following: Carefully crafted phrases and sentences create a sustained tone and advance the writers purpose with respect to the intended audience. (a) Varied, precise, and engaging language that is appropriate to the assigned genre. Word choice reflects an understanding of the denotative and connotative meaning of language Figurative or technical language may be used for rhetorical effect Sustained attention to the audience. (a) Evocative or authoritative voice that is sustained throughout the response. An extensive variety of sentence lengths, structures, and beginnings. A variety of genre-appropriate strategies to engage the reader. (a, f)

Full command of the components of Convertions. The writing is characterized by most or all of the following: Clear and correct simple, complex, and compound sentences with correct end punctuation. Variety of subordination and coordination stategles. Correct usage in a variety of context: subject, verb agreement, word forms (nouns, adjectives, adverbs), pronoun-antecedent agreement. Correct mechanics in a variety of contexts: punctuation within sentences, spelling, capitalization, and paragraph indentation Infrequent, if any, errors.

Linked to other standards

FIGURE 4.2. A. Sample of alignment of section of Georgias Grade 8 Writing test scoring rubrics with the Georgia Performance Standards. Reproduced with permission of the Georgia Department of Education (www.georgiastandards.org)

(ELA8W2-b), fully developed controlling idea that addresses all aspects of the assigned writing task (ELA8W2-b), supporting ideas and elaboration are relevant to the writers topic, assigned genre of writing , and audience (ELA8W2-d). Similarly, the domains of Organization, Style, and Conventions are linked to the ELA standards, thus supporting the content validity (and construct representativeness) of the writing assessment.

DESIGN FEATURES AND REVIEW CRITERIA FOR PROMPT DEVELOPMENT

Although Ruth and Murphy (1988) have identified the characteristics of prompts consistent with rhetorical demands, prompt task construction for large scale assessment is more complex than considerations of topic, purpose, and audience (Weigle, 2002). State assessment tasks are created in a political climate with competing constituencies. Writing teachers and experts firmly believe that student do their best writing when they choose and are engaged in their topic. Allowing students to do so, however, can penalize those who are accustomed to teacher assigned topics or it can result in sensitive, even inappropriate, subjects. Furthermore difficulties in the consistent application of scoring rubrics can result from too broad a range of responses. High stakes writing assessment tasks are constructed to maximize the examinees content or topic familiarity and to minimize emotional interference. Prompt construction often begins with the identification of prompt kernels or lists of familiar and appropriate subjects. The process also includes identifying and eliminating overused topics, which requires a review of adopted commercial materials or knowledgeable teachers, serving as prompt writers, who recognize and delete overused topics.

Writing Prompt Development Considerations

in preparing to develop a prompt, potential topics are compiled for advisory committee review. Sources include current secure test bank; widely used textbooks and other instructional materials; and public item banks from commercial materials, state testing websites, and online prompt banks. At this stage, the topics are screened for grade level appropriateness, required background knowledge familiarity, and avoidance of controversy. Sensitive or overused topics are flagged for prompt- writer training, to demonstrate what not to do. Topic kernels are evaluated by the committee

to determine whether they are within the realm of student experiences either experiential or academic whichever source of knowledge is consistent with the standards being assessed. Once potential topics have been compiled into a list of topic kernels that are appropriate for the target population, decisions about genre and the use of prompt specifications are applied. If genre specific standards are to be assessed, they are kept in mind as the prompt is drafted. A topic that lends it self to the particular genre of writing must be selected. If persuasive writing is the target, the topic must be an issue with differing and defensible view points. If examines are expected to respond to audience concern about the issue or to produce a counterargument, they must be familiar with both the issue and the audience. A topic for expository writing must be common knowledge, one that students know in depth, or the test materials themselves must include documents or text providing sufficient background information for writers to complete the writing task. Recall the earlier example of the legal materials provided for examinees for the Multistate Performance Test (National Conference of Bar Examiners, 2001). A document-based test increases the testing time but reflects what is expected of student in school settings. Narrative topics need to lend themselves to storytelling rather than plot summaries of retellings or lists of everything the writer did in a specified time period. Figure 4.3 presents topic concerns that are addressed before full-length prompts are drafted. Although a formal sensitivity review takes place once student impact data are available from field testing, topics that fail to meet sensitivity and bias baseline requirements are eliminated from the topic pool prior to field testing. Modifications or suggested ways to approach the topic can expand the topic pool at this point. For example, while appearance could be problematic for overweight teens, it could be developed as a prompt about what teens can do to be healthy. Allowances or but could evolve into a prompt about school fund-raising. Screening criteria vary across testing programs but generally include considerations of appropriateness, diversity, values, and privacy. It is helpful to work from specifications that contain the agreed-upon elements (Weigle, 2002). A consistent prompt format presented to students in their writing classroom prior to formal testing reduces what the examinees have to think about at the actual time of testing so that they can concentrate on the writing process. Promptwriting rules should be

Compile List of Potential Topic Kernels y y y What topics are available for this grade level from a variety of sources? Which topics are inappropriate? Why? Which topics are overused? Review Topics for Familiarity to Examinee Population y Can students at this grade level write an effective response based on knowledge of the topic that they have acquired trough first-or secondhand experience? Or, can students at this grade level who have prior knowledge of an academic nature write an effective response that is either primarily academic or a blend of personal experience and academic knowledge? Are students who typically perform poorly on writing tests familiar with the topic? What, if any, are the key concepts that need to be defined or clarified through example in the full-length prompt? What, if any, are keyword that would be used if this topic is turned into a fulllength prompt? Is this vocabulary familiar to the students? Can it be simplified or clarified? Is the topic better suited to a different age level? Review Topics for Sensitivity and Bias Concerns y Is the writing task appropriate with respect to gender, ethnicity/race, age of students, disability, geographical region, socioeconomic status, and English as a second language? Can student who differ with respect to experience based on gender, ethnic group, socioeconomic status, and geographical region respond to the topic? Is the topic of a sensitive nature or likely to be interpreted as such? Does it address or would it elicit large numbers of responses focusing on sex, drugs, violence, death, body image, or religion? Will it lead to the revelation of private values, family matters, or confessions of unreported crimes? If the topic itself has sensitivity problems, can it be modified or developed in an appropriate way? How? If inappropriate for the target examinee population, is the topic appropriate for use with a different grade level? What grade? FIGURE 4.3. Topic selection questions.

y y y

y y

y y

Created to determine the types of information (such as audience and form of the response) and cues that will be provided either in the general test directions to the students or in writing-revising checklists. The prompt itself may contain all the necessary directions and cues. Figure 4.4 lists the design features that an item writes should consider when developing writing prompts. The features appropriate for particular standards and age groups can be selected and turned into prompt specifications so that prompt writers include all the necessary elements.

Prompt-Writing Steps Direction for prompt writing are presented below. They assume prompt writers have been trained in the process. Training include an overview of the relevant performance standards, explanation of the prompt specification with examples of well-written prompt, and reminders about language demands and bias issues. One of the most beneficial training activities requires the prompt writers to edit and improve prompt that contain typical flaws. The following steps guide the development of a prompt : 1. Select one of the appropriate topic kernels 2. Follow the prompt template, item specification, or sample prompt format 3. Brainstorm about 10 different ways students at this grade level could develop a response to the topic in the required genre. Make sure that students of high, middle and low writing ability can generate a variety of responses. Make sure that most approaches could be developed into multi paragraph responses (unless the standard calls for a shorter writing sample) 4. Identify key terms and/or vocabulary and decide how to define or clarify them with examples, synonyms, or paraphrases. Use examples that let the lower-ability writer quickly know he/she does understand the topic. Avoid examples that prescribe the response or that elicit formulaic writing 5. Decide on an audience that is appropriate for the topic and genre and that is known to the writers. Consider the impact of that audience on the level of language, primary supporting points, and elaboration students would have to include to score within the effective range

The Topic or General Subject y Is familiar to all examinees y Satisfies sensitivity and bias concerns y Allows student to develop a response from either an experiential or academic knowledge base y Elicits diverse responses A Context or Setup to Help the Writer Understand the Topic y Introduces the topic y Leads in to the writers task The Writers Purpose y Matches the required type of writing (exposition, narration, persuasion) y Fits the topic The Reader Audience y Is consistent with writers purpose y Is familiar to all examinees Genre/Mode y Identifies genre or mode y Cues expected method of development y Cues appropriate forms of support or reasoning

Format of the Writers Response y Fits the writers purpose y Conforms to scoring criteria Format of the Prompt Itself y Is consistent across prompt y Cues the writer Hints of Cues Confirms that the writer understands the topic Guides the writer about what to include Guides the writer about how to structure Indicates how the response will be evaluated

y y y y

Writing Checklist y Reinforces the use of the writing process in the test context y Conveys the scoring criteria to the writer y Assumes prior and repeated use in instructional setting prior to test administration y May be genre-specific Content and/or Performance Standards y To ensure alignment with the curriculum, appropriate portions of the prompt are linked to specific standards (Figure 4.1)

6. Provide a clear purpose for the writer that is appropriate for the genre and topic. Make sure the audience and the purpose fit together 7. Determine if genre cues are needed and what they will be 8. Think of a way to begin the prompt (opening sentence[s] or stimulus material) that quickly present the topic, engages the writers interest, and sets the topic in a meaningful context 9. Include concise directions to the writers about what to write and for whom. These directions will contain cues about genre and methods of development. If the prompt is lengthy, remind the writer of the topic 10. Once the entire prompt is drafted, repeat step 3 for eliciting a variety of responses.