Part 4: Teaching Materials Reflection on an Assessment Project: Assessing Vocabulary Recognition

Have you ever wondered what goes into creating a test? In the class AL 6730: Assessment in TESOL, I was able to have this experience. One of the major requirements of the course was to design, develop, administer and evaluate an assessment tool appropriate for a group of ESL students. This requirement was a group project, and I got to collaborate with three other classmates. We were assigned to create a test for a class called International Education at Hawaii Pacific University. The class was part of a HPUs International Bridge Program, which aims to help international students improve their English and academic skills so they can achieve success at an American university. Together, we collaborated with the class professor to design a vocabulary test for his students. Designing, developing, administrating and evaluating the test was a very comprehensive process. In the beginning of the project, we had a meeting with the class professor. He gave us the objectives of the test and the target vocabulary. He also requested that we only assess vocabulary recognition. This led us to use multiple choice as our assessment technique. The vocabulary consisted of twenty words taken from the Academic Word List. We split the vocabulary into four different parts of the test. Part 1 had five items, and each item had a context sentence with an underlined word. The students task was to choose the option closest in meaning to the underlined word. In part 2, the students had to match the ten vocabulary items with their definitions,. Part 3 had five items, and the students had to choose the synonym of the word presented. Finally, Part 4 had a passage

with ten gaps. The students had a word bank with 15 words, five extra to function as distractors. Creating the test was challenging, and we had to do several revisions.

I was in charge of creating Part 1 of the test. Designing the distractors was challenging, since there were many considerations to take. For instance, the distractors had to have similar length as the target word, to avoid it standing out. Also, if the target word was an adjective, then all the distractors had to be adjectives. Considerations like these made it especially difficult to find appropriate distractors. It was an extensive process that required a lot of editing. When the test was finished, we administered it to the host class. The students had 30 minutes to finish the test, and they all managed to complete it within the time limit. The students results revealed that the most frequent score was 26.5 out of 30 points, and only one student received a lower score than 25. Therefore, the majority of the students did well on the test. After evaluating the results, we did an item analysis. Unfortunately, it revealed that many of our distractors were not chosen by anyone. We concluded that they must have been too easy. After some reflection, I believed the test would have suited the students more advanced skill level better if it had assessed their productive ability. Many of the distractors were too weak, making it easy for them to choose the correct answer.

This led me to do a literature review on techniques to use when assessing vocabulary production. I also discussed how some of these techniques could have been applied to our test. The literature review convinced me of the benefits of vocabulary production vs. vocabulary recognition. The multiple choice technique might make it easy

to score a test, since it allows you to be completely objective. However, finding good distractors can be difficult and very time consuming. This knowledge led me to believe that vocabulary production is a more reliable approach. Though, assessing the production ability also has its challenges. For instance, if you create tests items using the gap-filling technique, then you have to make sure that only one word fits the gap. Also, eliminating alternative answers can be difficult, and it might lead you to use cues, like providing the initial letter of the target word. Nevertheless, I think that the test we created would have been better if we had assessed the students production ability, and used techniques like gap filling. The reason for this is that too many of our distractors were not chosen by anyone, causing us to believe our choices were too easy. The challenge to create good and unambiguous distractors makes the multiple choice format difficult, especially for inexperienced test creators.

All in all, after this whole experience, I feel very prepared and capable of creating tests in the future. The experience also humbled me, as I got to see how much work, preparation, revision and analysis went into one vocabulary test.