Effects of Assessment Literacy on ESL’ Speaking Performance

EFFECTS OF ASSESSMENT LITERACY

ON ESL SPEAKING PERFORMANCE

EMY HARLINA BINTI IBRAHIM

Kolej Matrikulasi Selangor
emyharlina@kms.matrik.edu.my

ABSTRACT

This study was conducted to examine the effects of assessment literacy on

ESL speaking performance. It intends to help students to get better insights of
how they are being evaluated in an oral presentation. Thus, it is hoped to
improve their speaking performance and make them feel more confident in their
own ability. This study involves twenty-two ESL matriculation students from
Kolej Matrikulasi Selangor. The samples were given a speaking task which they
have to present after a duration of three weeks. Subjects were exposed to
assessment literacy after their first oral presentation. The presentation scores
of before and after the intervention was recorded. Interview sessions were also
conducted on the subjects to obtain their opinion regarding the approach. The
effects of assessment literacy provided to the subjects on their speaking
performance have revealed the result favoring the use of the approach. The
outcome suggests that it could be integrated into traditional teaching sequence
and instructional strategies.

INTRODUCTION

Selangor Matriculation College (SeMaC) is the 13 th college opened by the Matriculation Division
under the Ministry of Education. Announced as the Centre of Excellence among all Ministry of
Education matriculation colleges, SeMaC formed a Research Unit at the college level to improve
the teaching quality through involvement in action research. Members of the academic staffs are
encouraged to participate in writing and presenting their research papers in upgrading the quality
of teaching and learning in their respective content areas. The concept of teacher as a researcher
was taken up with the adoption of action research to improve teaching. It was argued that the
practitioners should be involved in their own problem instead of becoming the subjects to be
studied by an external researcher. Malaysia has adopted this idea and action research culture is
being propagated in the education institutions nationwide.

Zuber Skerrite (1982) proposed; through systematic, controlled action research, higher education
teachers can become more professional, more interested in pedagogical aspects of higher
education and more motivated to integrate their research and teaching interests in a holistic way.
This, in turn, can lead to greater job satisfaction, better academic programmes development of
new educational theories, improvement of student learning and practitioner’s insight and
contributions to the advancement of knowledge in higher education.

1
 Effects of Assessment Literacy on ESL’ Speaking Performance

In the Matriculation colleges, the English Unit is in charge of teaching two English syllabuses. The
first syllabus is the Malaysian University English Test (MUET) for students to prepare for the
MUET examination which is the English entry requirement for our local universities. The other
syllabus is the Matriculation English Program for students of the One-Year or Two-Year
Matriculation Program prescribed by the Matriculation Division of the Ministry of Education. In
both syllabuses, each component of reading, writing, listening and speaking, is tested separately
unlike the tests that they are used to in primary or secondary school which comprise of more
written tests.

Students at the matriculation level face a transition from traditionally assessment method of paper
and pencil type (Gronlund, 1998), terminal end year type examination to alternative assessment
which looks at students’ performance in a continuous manner throughout the whole learning
process. This includes oral presentations, portfolios, projects, article review, role-play etc. Even
though they were taught all the four major skills; reading, writing, listening and speaking, but all
throughout their primary and secondary education, the skills were not tested separately, like how
it is done at the matriculation level. Reading, writing, listening and speaking is treated as an
individual component which students are not very well-found of. Thus, this study aims to see if
providing the matriculation students with assessment literacy will help improve their speaking
performance.

DEFINITION OF TERMS

The notion of "literacy" has been interpreted in many ways (Wagner and Kozma, 2003). For
example, not so long time ago, only a hand full of people regard computer skills as indispensable
for assessment literacy (Séror, 2005). Those who are hoping for a single, cogent definition of
"assessment literacy" that can fit in and works for all groups will be disappointed because the
construct represents a wide matrix of skills which vary significantly from population to population.
"Assessment literacy" from the viewpoint of a university student, a high school teacher, and a
professional test developer probably involve vastly different skills and varies in its definition.

Assessment literacy: What it means for teachers

The importance of assessment literacy for different groups also ranges extensively. Newfields
(2006) suggested that there are three compelling reasons for teachers to be assessment literate.
First, assessment is a widespread (if not intrinsic) feature of most educational systems. Teachers
are estimated to spend from 10% - 50% of their work time on assessment-related activities
(MacBeath & Galton, 2004, p. 31; Gunn, cited in Brindley, 1997) In many schools in the United
States, a good portion of the budget also goes into formal testing. With so much time and money
devoted to assessment, it's worth critically understanding how assessment decisions are made.

He further explained the second reason assessment literacy is essential because it's necessary
to understand much of the educational literature. A familiarity with basic statistical terms is
needed not only to critically read specialized journals, but even many general articles in academic
publications. A final reason assessment literacy is needed is that it allows teachers to
communicate their own classroom results with others. As Hopkins (1985, p. 58-60) suggests,
teachers should share their research with peers and develop a community that fosters learning.
Mastering the conventions of qualitative and quantitative inquiry are essential to make classroom

2
 Effects of Assessment Literacy on ESL’ Speaking Performance

research comprehensible to a wide audience. Newfields (2006) also mentioned that assessment
literate researchers should be committed to an ongoing self-critique their own research and
sharing the findings in ways that are technically convincing. All too often articles with interesting
insights lack sufficient analysis and/or evidence to allow readers to critically interpret the ideas.

Assessment literacy: What it means for university undergraduates

Instead of describing assessment literacy as a single phenomena with some sort of unitary
meaning, this paper adopts a grounded theory perspective by suggesting it means different things
to different populations. For students, it largely means knowing how to perform well in classroom
evaluations, graded presentations and in exams. For teachers, it is associated with the ability to
grade students ethically and accurately. And for professional test developers, every aspect of their
work revolves around assessment literacy.

A study conducted by Ohata (2005) suggests that test anxiety is far from rare in Japan. Many
Japanese have what Baker (2006) describes as a victim mentality when it comes to testing. At the
same time, Japan can be described as an edumetric society in which test results significantly
impact life outcomes.

One way of operationalizing assessment literacy in terms of what it might mean for university
undergraduates can be list down as in Table 1:

Table 1: A suggested operationalization of "assessment literacy" for one first year college student
population (Newfields, 2006)

1 An ability to interpret statistical raw data in terms of common measures of

centrality (mean, mode, median) and deviation (SD, quartiles).
2 A basic understanding of the concept of measurement error and confidence intervals.
3 An ability to discern whether or not the difference between two or more data sets is
significant.
4 A capacity to logically distinguish between correlation and causation.
5 An understanding of what constitutes ethical assessment – and what should be done if
encountering unethical testing practices

These points represent many of the competencies outlined in the American Association for the
Advancement of Science's Benchmarks for Science Literacy (1993). In other words, they denote
what some educators think that university students should know. Chances are, this is far from
what most students themselves actually want to know. Often, the main dispute in promoting
assessment literacy seems to be convincing end-users that the topic is actually worth learning:
when many people encounter the esoteric jargon and complex statistical formulas sometimes
used in assessment, perplexity is a frequent response. (Newfields, 2006) There's so much to
know and most learners (and perhaps teachers as well) are frankly unprepared. Generally
speaking, little time is given in promoting assessment literacy skills in most curriculum in Japan:
people take tests without considering why such tests are being conducted or whether they
ethically assess a given skill. This I believe is a common scenario in many other countries as well,
Malaysia unexceptional.

3
 Effects of Assessment Literacy on ESL’ Speaking Performance

PROBLEM DESCRIPTION

Assessment is a fundamental part of the teaching and learning process. According to Belk and
Calais (1993) assessment enables the teacher to gather information about the students’ progress,
program goals and objectives as well as the extent to which methods of instruction deployed in
the classroom are helping the students achieve these goals. In accordance to that, being an
examiner at national level examinations has improved my teaching skills tremendously. My fellow
friends who are assessment literate have shown a higher level of confidence in their teaching too.
Their selection of methods and materials are tailored to meet students’ need for the assessment.
Knowing how a task will be assessed guides our organization skills, time management skills, task
fulfillment skills, creative thinking skills etc. It has given me a clearer guide on what is needed for
my students to excel in their assessments. Assessment literacy helps to improve both teaching
and learning.

Personal Reflection

After teaching for some time I noticed that my students’ written assignments or oral presentations
do not meet the assessment criteria of the course. Their response was either too shallow, too
general or somewhat superficial leaving me to wonder whether they actually have mastered the
skills for each component.

I reflect back on my own experience during examination coordination meetings, every examiner
was clear with the standard set, with the evaluation criteria that was to be met. If a nationwide
group of examiners can be standardize to evaluating the same sample script through assessment
literacy, by having assessment literacy examiners have a strong content mastery for that
particular test. I anticipated that it might work on my students too. If students have to develop
tools and procedures for monitoring, recording, and assessing their friends’ progress and
achievement in the classroom on a more systematic and formal basis (Brindley, 1997),in other
words providing them with ‘assessment literacy’ (Fullan, 1999), maybe it could give them some
useful insights on what they are expected to do to improve their own performance. I decided to
embark on action research to find out if providing ‘assessment literacy’ will help my students
improve their speaking performance.

RESEARCH OBJECTIVES

The intention of providing ‘assessment literacy’ is to help the students to meet the assessment
criteria of each test better. Thus, this study aims to see if providing the matriculation students with
‘assessment literacy’ will help improve their understanding of the different skills needed to score
for each component.

RESEARCH METHODOLOGY

I believe that action research approach can contribute to our understanding of teaching and
learning problems at tertiary level and it can contribute in finding solutions to some of the teaching
problems. This methodology also offers a systematic approach of introducing innovations in the
teaching and learning in the classroom. It would serve or encourage our lecturer to play a dual

4
 Effects of Assessment Literacy on ESL’ Speaking Performance

role of being a producer of educational theory as well as a user of that theory. (Like what I am
attempting now) The approach provides one way of increasing our understanding about learning
and teaching practices. Thus the approach is adopted for this research.

RESEARCH PROCEDURE

The sample for this study is a small number of students from the same class. On average we
have 10-22 students in a class at the matriculation college. The Two-Year Matriculation Program
was selected. This is because they have presentations in their syllabus.

The instrument used in this study is the presentation assignment for the course. It is prescribed in
the Matriculation syllabus. As a tool to elicit the results for ‘assessment literacy’, the scoring guide
will be used.

Another vital instrument in this study is the researcher herself. The researcher will be involved in
the administrating of the study. The researcher will observe the subjects during the process to
write down her reflection as a form of triangulation.

A semi structured interview questions will also be used illicit further understanding about the
effect of the ‘assessment literacy’ treatment on the subjects. The interviews were conducted on
the subjects and it is very practical to obtain useful insights and obtain accurate information.
(Mohd Majid Konting, 1998) Some of the interview questions involve gathering information on
subjects’ perception and experiences, it has an entity of a survey research method as well.
Kerlinger (1986) suggest that survey research has strength in terms of realism, significance,
theory-oriented and have a high quality of heuristic value.

The qualitative method of data collection is used in this study to further support the quantitative
findings. It is hoped the combination of these two methods will result in a strong outcome of the
research.

The students were informed at the beginning of the semester the format of the assessment. The
students were to prepare a 10 minute Story Telling presentation after a duration of 3 weeks. The
presentations were video recorded so that it can be use later in the ‘coordination meeting’. They
did the presentation in pairs but be graded individually. This was to reduce the anxiety and give
them more confidence in doing the task. After 3 weeks they presented their Story Telling. During
this time the researcher assessed them using the scoring guide. The researcher took down notes
about each presentation and justification for the marks. The scores were recorded as Score 1.

After all the subjects have presented, the researcher asked the subjects to make a scoring guide.
The subjects were put in groups to discuss

 What are the criteria that they think an examiner should assess for the presentation.
 How many marks should be allocated for each criteria.
 Should all criteria carry equal mark.

All the response of the subjects will be written on the board for them to see. After the discussion
the researcher will give the real scoring guides to the students. Then the students viewed their
presentation recording. Each student wais to give their own marks for each of the presenters
using the scoring guide. After that, the researcher compared the students marks (Score 2) and

5
 Effects of Assessment Literacy on ESL’ Speaking Performance

marks given by the researcher. (Score 1).

Here is where the ‘coordination meeting’ between the researcher and the students take place.
Justifications of marks awarded and scoring points was discussed. Each student gave their own
justification for the marks that they give for each presentation. The researcher provided
‘assessment literacy’ to the subjects. The subjects learned how they were assessed.

After the ‘coordination meeting’ the subjects are given 1 week to do the presentation again. This
time the scores (Score 3) was given again by the researcher and the subjects (Score 4). The
scores between Score 1 and Score 3 was compared. This is to see if there is a significant
improvement in their performance. Score 2 and Score 4 will be compared to see if the subjects
have indeed gain ‘assessment literacy’ from the treatment.

RESEARCH FINDINGS

Table 1:
To find out if the subjects find the approach effective in developing their speaking performance.

Response YES NO
n % n %
22 100 0 0

Table 1 indicates that the approach was perceived as effective in developing their speaking
performance.

Table 2: Reasons for Finding It Effective

Reasons n %

It makes them think like an examiner 16 72.7

It gives them more confidence 19 86.4

It gives them a clear guideline of what is 9 40.9

expected of them during a presentation.

It makes them see their mistakes 19 86.4

It develops their speaking skills 17 77.2

It is easier to see when others make 13 59

mistakes

6
 Effects of Assessment Literacy on ESL’ Speaking Performance

From Table 2, it was indicated that majority of the respondents feels that the approach helps them
to see their mistakes and make them look at the assignment from a different perspective. Almost
all of the respondents admitted to feeling more confident to do the presentation again after the
treatment.

Table 3 : Main Problem When Doing Oral Presentation

No Interview Items
n %
1 Not fluent with the language 6 27.3
2 Do not know the best way to present 14 63.6
3 Nervous in front of an audience 5 22.7
4 Lack of self confidence 9 40.9
5 Not enough time to practice 4 18.1

From the interview conducted, it can be seen that majority of the respondents admitted that not
knowing the best way to present is the main problem when doing oral presentation. Others
admitted that they lack of confidence and experience nervousness when standing in front of an
audience.

Table 4: Perceptions Towards Assessment Literacy in Developing Students’ Speaking

Performance
5= totally agree 4=agree 3=partially agree 2=disagree 1=totally disagree

Interview Items 5 4 3 2 1
n % n % n % n % n %
1. The approach helps me a
lot in understanding the 16 72.7 5 22.7 1 4.5
requirement of the task.
2. The approach increases
my confidence in doing 9 40.9 11 50 2 9.1
the 2nd presentation.
3. The approach helps me
to identify my 12 54.5 8 36.4 1 4.5 1 4.5
weaknesses.
4. I can help to teach my
friends with this 8 36.4 14 63.3
approach
5. I can see other people’s
strength in their 8 36.4 8 36.4 6 27.3
presentation
6. The approach gives me a
sample of a good
16 72.7 6 27.3
presenter for me to
follow

Majority of the respondents agreed that the approach used was very helpful in giving them a
sample of a good presenter among themselves for them to follow. Through an understanding of
what they were being assessed of during a presentation makes them feel more confident to do

7
 Effects of Assessment Literacy on ESL’ Speaking Performance

the 2nd presentation better. More than 70 percent respondents strongly agreed that the approach
used helped them understand the requirement of the task better.

TESTS RESULT
Table 5 : Scores on Presentation

Score 1 & Score 3DIFFERENCE

Presentation 1 Presentation 2

Score 1 & Score 2DIFFERENCE

Score 3 & Score 4DIFFERENCE

(Total marks = 30) (Total marks =30)
Subjects
Score 1 Score 2 Score 3 Score 4

1 19 25 +6 25 25 - +6
2 17 26 +9 23 24 +1 +6
3 17 26 +9 20 18 -2 +3
4 20 26 +6 25 23 -2 +5
5 25 27 +2 25 23 -2 -
6 26 28 +2 26 24 -2 -
7 22 27 +5 25 25 - +3
8 22 27 +5 22 22 - -
9 17 26 +9 20 22 +2 +3
10 24 26 +2 26 25 +1 +2
11 24 26 +2 24 25 +1 -
12 24 30 +6 24 24 - -
13 24 30 +6 24 26 +2 -
14 24 27 +3 26 24 -2 +2
15 16 26 +10 22 25 +3 +8
16 16 26 +10 20 24 +4 +4
17 17 25 +8 22 25 +3 +5
18 20 27 +7 25 24 -1 +5
19 16 27 +11 20 20 - +4
20 20 30 +10 26 26 - +6
21 23 30 +7 26 25 -1 +3
22 23 26 +3 23 27 +4 -
Total Difference 138 Total Difference 21-12=9

STATISTIC OF SUBJECT PERFORMANCE

To see if there are any significant changes in the scores of the first and second presentation after
the provision of Assessment Literacy on the subjects, the marks are compared in table 6.

8
 Effects of Assessment Literacy on ESL’ Speaking Performance

Table 6 : Difference of Score 1 & Score 3

Total INCREASE(+) DECREASE(-) UNCHANGE

15 ((68.2 %) 0 (0 %) 7 (31.8 %)

REFLECTION OF RESEARCH

Discussion of Findings

Classrooms assessment that are effectively planned and systematically carried out can promote
active learning and self-confidence amongst students. The treatment of ‘assessment literacy’
given to the students has a positive significance and would open new possibilities and new
approaches towards improving teaching and learning. The study has increased our understanding
about learning and teaching practices. Thus the significance of this study is very much
highlighted.

My findings show a marked difference between score 2 and score 4 from the subjects. The
subjects’ speaking performance has shown an increase of 68.2% from their 1st presentation while
the difference between Score 3 and Score 4 had shown a significant similarity where the subjects’
scores and the lecturer’s score have a smaller gap. The results of this study support the notion
that the exposure of assessment literacy on the students will indeed have a positive effect on their
speaking performance. This study not only increased my students’ performance for their
presentation, but it also rekindled my excitement for teaching because now my students are
seeing what I am seeing.

CONCLUSION

Although some may not agree with the use and methods in action research as compared to the
real traditional empirical researches, I do believe that somewhere along the line there are some
significant if not useful outcomes that contribute to the development of teaching and learning in
the classroom. The next thing that interests me would be to see if Assessment Literacy will also
have a similar impact on other samples of population, like lecturers and exam paper setters.
Hopefully it too will produce the same if not more positive values that will help advance education
in Matriculation colleges nationwide.

9
 Effects of Assessment Literacy on ESL’ Speaking Performance

Bibliography

Baehr, M. (2004) Overview of Assessment. In S.W. Beyerlein and Daniel K. Apple

‘A Comprehensive Tool For Improving Faculty Performance’. Lisle, IL: Pacific Crest.

Bailey, K.M. (1998) Learning about Language Assessment. Dilemmas, decision and
directions. London: Hainie and Hainie.

Baker, B. (n.d.) Victim mentality. Accessed December 12, 2006 from

http://www.selfgrowth.com/articles/Baker4.html.

Belk, J.A and Calais, G.J (1993) ‘Portfolio Assessment in Reading and Writing: Linking
Assessment and Instruction to Learning’. Paper presented at the annual meeting of
Mid South Educational Research Association, New Orleans, November 10-12
(ERIC Document Reproduction Service No ED 365732)

Berry, R. (2003) ‘Alternative Assessment and Assessment For Learning’. Paper

presented at the 29th International Association for Educational Assessment (IAEA)
Annual Conference, Manchester, United Kingdom.

Chan SweeHeng, Sharon GohSeng Pang, Lee GeokIm, Ong Wei Siang, UPM, (2006)
‘ Exploring the Social Responsibility of Tests: Views From Univerity Lecturers and
Students’. Kuala Lumpur International Conference on Assessment.

Gunn, M. (1995). Criterion-based assessment: A classroom teacher's perspective. In G. Brindley

(Ed.), Language assessment in action. Sydney: National Centre for English Language
Teaching and Research, Macquarie University. Cited in Brindley, G. (1997, September).
Assessment and the Language Teacher: Trends and Transitions. The Language Teacher
Online. 21 (9). Accessed on January 28, 2007 from http://jalt-
publications.org/tlt/files/97/sep/brindley.html.

Hopkins, D. (1985). A teacher's guide to classroom research. Philadelphia: Open. University

Press

Newfields Tims (2006) (Toyo University) Authentic Communication: Proceedings of the 5th
Annual JALT Pan-SIG Conference.May 13-14, 2006. Shizuoka, Japan: Tokai University
College of Marine Science. (p. 48 - 73).

Ohata, K. (2005, December). Potential Sources of Anxiety for Japanese Learners of English:
Preliminary Case Interviews with Five Japanese College Students in the U.S. TESL-EJ, 9
(3) 1-21. Accessed on Dec 31, 2006 from http://www.kyoto-su.ac.jp/information/tesl-
ej/ej35/a3.pdf.

W. James Popham Is Assessment Literacy the "Magic Bullet"? June 4, 2009

http://www.hepg.org/blog/19

10
 Effects of Assessment Literacy on ESL’ Speaking Performance

Séror, J. (2005). ‘Computers and qualitative data analysis: Papers, pens, and highlighters vs.
screen, mouse, and keyboard’. TESOL Quarterly, 39 (2), 321- 328.

Shohamy, E. (1998) ‘Critical language testing and beyond’. Studies in Educational

Evaluation, 24 (4), 331-45.

Shohamy, E. (2001) The power of tests: A critical perspective on the uses of language
tests. London: Longman.

Subahan T. Mohd Meerah. (2006) An Action Research Approach to Alternative

Assessment. Kuala Lumpur International Conference on Assessment.

Suzieleez Syrene Abdul Rahim, Tajularipin bin Sulaiman (2006) Classroom Assessment
: Paper-Pencil Vs Alternative. Kuala Lumpur International Conference on
Assessment.

Wagner, D. A., & Kozma, R. (2003, Oct. 1). New technologies for literacy and adult education: A
global perspective. University of Pennsylvania International Literacy Institute. Accessed on
June 12, 2006 from http://ncal.literacy.upenn.edu/products/wagner_kozma.pdf.

White, Eddy ‘Assessment literacy for effective classroom-based assessment’ Jan 11 2009
http://www.slideshare.net/ewhite/assessment-literacy-effective-classroombased-assessment-
presentation-907377

11