E-rater, ETSs proprietary scoring engine, is used in
Criterion to automatically evaluate essay quality. The
system targets aspects of writing fluency, including genre-independent features (such as grammar, usage, mechanics, spelling error detection, transition words), and genre-dependent features (such as discourse structure elements). In Criterion, e-rater is used to provide a holistic score and diagnostic feedback related to aspects in writing. The genre-independent features are applicable to any genre of writing; the genredependent features, on the other hand, were designed for expository and persuasive genres of writing that have specific discourse structure (including thesis statement, main points, supporting ideas, and conclusions). As Criterion has been designed for expository and persuasive writing, teachers may choose to write a prompt other than expository or persuasive (such as narrative). In this case, the genre-independent feedback (including, grammar and spelling errors) will be relevant and accurate, and the teacher should plan to provide feedback for the content and discourse aspects. Genredependent discourse structure features are not relevant for narrative (story) writing such as the BamBam submission, and should be disregarded for this genre of writing. The discourse structure of a story is very different from that of an expository or persuasive writing sample. Figure 1 illustrates how Criterions discourse structure analysis applies to a persuasive essay. Figure 1, below, illustrates an example output of a persuasive essay where the discourse elements are highlighted. [To illustrate how e-rater can work well for discourse structure, insert an image of an acceptable discourse structure output of the kind of essay that e-rater is
intended to evaluate to show them how it is intended to
work. If there is a non-narrative example from 5 th grade, try that, and if not, just use a 6 th grade example. There should be examples of 6th grade/middle school expository writing] Figure 1. Persuasive essay labeled with discourse structure elements Below in Figures 2 and 3 are relevant, genreindependent analysis and score that e-rater produced. As illustrated in Figure 2, below, usage errors detected were accurately detected. [insert usage feedback screenshot to show what was found] Figure 2. Usage error feedback [insert transition words screenshot] Figure 3. Transition words showing organizational development Criterion was designed to be a support resource to teachers, intended to offer writing practice and facilitate student-teacher dialogue. The Criterion best practice philosophy was developed around the notion that teachers would be present and working with students as they write using Criterion. When teachers take the time to write their own prompts, such as in the case of the BamBamsubmission example where genre-dependent features are unavailable, teachers will need to engage in dialogue with students about aspects of writing, such as discourse structure.
E-rater uses special methods for assigning scores to
Instructor Topics. The essay written by Tess A. titled BamBam is a story (or narrative) genre. It was submitted to Criterion as a Grade 5 Scored Instructor Topic. Instructor topics allow teachers to write their own prompts. E-rater uses its full feature set to assign a score; the full feature set includes many features, some of which are used as feedback, and others which are used only for scoring purposes, and address aspects of an essay such as vocabulary usage. In the case of the BamBam essay, e-rater assigned a score of 5 out of 6 which indicates that for a 5th grade writing sample, this is a pretty good piece of writing for the following reasons: (1) it has few grammar and spelling errors; (2) it has good use of vocabulary, including some words and terms that support the coherence of the essay; (3) general vocabulary choice is reasonable. There are many different genres of writing, and this is more recently emphasized by the demands of the Common Core. E-rater was developed to serve expository and persuasive modes of writing, though the system is being continuously updated to expand the genres and range of content that can be evaluated by the system.