Running head: VALIDITY OF TOEIC IN JAPAN

Validity of TOEIC in Japanese Context

Manami Fujino
University of Hawaii at Manoa

VALIDITY OF TOEIC

Introduction
TOEIC is known as one of the most famous English proficiency test in Japan. Many
people take the test with different purposes: placement test, measurement of English learning
progress, requirement for earning credits, voluntary, and job-hunting (EST, 2013). Due to the
high demand of TOEIC, more and more TOEIC preparation courses are provided across Japan.
As a Japanese student, I recently took TOEIC for job-hunting because many of my teachers and
friends in Japan suggested me to take the test before I start job-hunting. Although TOEIC has
been widely used and known as a valid English proficiency test, I became to be curious about its
validity after my study in SLS 490 and own experience of taking TOEIC. This paper will analyze
TOEICs face validity, content validity, construct validity as well as its wash back especially in
Japan after introducing what TOEIC actually is and its current usage.

What is TOEIC?
The original name of TOEIC is Test of English for International Communication, and it is
deigned to measure L2 English learners English ability to communicate in a business setting. In
1989, the Ministry of International Trade and Industry of Japan requested the Educational
Testing Service (ETS) to develop a new English language proficiency test that can be given to a
large number people, especially to Japanese business people. I was made to measure their
speaking, writing, listening and reading comprehension skills in English. ETS was chosen
because ETS had developed world-famous tests such as GMAT: Graduate Management
Admission Test, and TOEFL: Test Of English as a Foreign Language, and SAT: Scholastic
Aptitude Test (Maher & Toda, 2006). When it comes to the format of TOEIC, it is a NormReferenced Test (NRT) and comprised of reading and listening sections. Each section contains

VALIDITY OF TOEIC

100 questions, and the reading section has three parts and the listening section has four parts. All
200 questions are multiple choice and the test is in English. Each sections has the same range of
possible scores which is between 5 to 495 points, and the total score range is between10 to 990.
The results are divided into five different levels such as 10 to 400 points as beginner, 400 to 600
points as an intermediate level student, 600 to 700 points as high intermediate, 700 to 800 points
as low advanced, 800 to 990 as advanced (Lawson, 2008). The following chart provides general
description of each part.

Part 1

Part 2
Part 3
Part 4

LISTENING COMPREHENSION
Choose the correct sentence that describes the
picture
4 Sentences will not be printed on question sheet
Sentences will be read only once
Choose the proper response to a question
Three possible responses
Question and possible responses will be read only
once
Choose the best answers for three questions about a
short conversation
Four Printed possible answers
Conversation, question, and possible answers will be
read only once
Same style as Part 3 except announcement or
narration will be provided instead of a conversation
TOTAL

20 items

30 items

20 items
20 items
100 items

READING COMPREHENSION
Part 1

Part 2
Part 3

Complete a sentence (fill in the blank)

Complete sentences in a long passage
(fill in the blank)
Reading comprehension passages and answer
questions

40 items
20 items
40 items

TOTAL 100 items

(Morikawa, 2014)

VALIDITY OF TOEIC

Current usage
Although TOEIC was taken only by 2,773 people when the test is first introduced in
1979, it is now developed as one of the most famous English language tests in the word. In 2014,
2.4 million people took TOEIC in over 90 countries. It has been used by more than 5300
companies in the world, and over 2700 Japanese organizations, companies and organizations are
using TOEIC as an administration or placement tool. Moreover, There are 480 high schools and
higher education institutions are using TOEIC as their part of entrance exams (Maher & Toda,
2006). In addition, Japanese companies expect applicants to have the score of 465 to 670 for new
graduates, 605 to 815 for potential oversea employees, and 660 to 840 for employees of
international operations. High TOEIC score is believed to be one of the most important and
valuable possessions for job-hunting in Japan (Maher & Toda, 2006). ETC official website
provided a chart that indicates what test takers can do with their scores based on their own
research:
TOEIC Score

Things you can do

Understand treatise with terminals in ones field
Understand discussions among native speakers

900-900

Gather information online

Understand discussions about problems at work

Explain their own routine work

Understand internal documents at work

Understand details in short and slow conversation with

intermediate-level vocabularies
Understand short and simple notes about daily work

800-895

700-795

600-695

VALIDITY OF TOEIC

Understand timetables in stations and airports

Understand easy questions asked by friends such as How are
you?, How do you feel?, Where do you live?

Understand signboards at stores

500-595
400-495
(Ito, Kawaguchi & Ohta 2011)
One interesting fact I found from this chart is that productive skills are not provided in this chart
Although TOEIC is developed to judge test takers both receptive and productive English skills.
ETS also provided a list of appropriate uses of TOEIC scores: hiring of applicants for an open
position within a corporation or organization where workplace/everyday English is a required job
skill, placement of applicants or test takers within a corporation or organization where
workplace/everyday English is a required job skill, promotion of test takers within a corporation
or organization where workplace/everyday English is a required job skill, measurement of
workplace/everyday English proficiency levels of students in secondary schools and
universities , and measurement of individuals progress in workplace/ everyday English
proficiency levels over time(Lawson, 2008, p.13). ETS seems to know and approve the fact that
TOEIC is used as a general English proficiency test.
Face Validity
According to Lawson (2008), TOEIC has remarkably high face validity among
individuals, academic institutions and companies particularly in Japan since TOEIC is officially
approved by the Ministry of International Trade and Industry of Japan, and created by ETS. Like
previous section mentioned, TOEIC is developed under the request of MITI if Japan, and MITI
declared TOEIC and other tests which had received wide recognition by society in 1999
(Rebuck, 2003, p. 30). Consequentially, TOEIC became to be one of the most famous English

VALIDITY OF TOEIC
proficiency tests, and about 65% of test takers are Japanese(Maher & Toda, 2006).
Construct Validity
Although TOEIC has a very high face validity and it is used by a huge number of
individuals, institutions, and companies all over the world, TOEIC has some problems in its

construct validity. The biggest problem is that the TOEIC test only contains listening and reading
sections. The purpose of the test is to measure L2 English speakers communicative English
abilities, which includes reading, writing, listening and speaking. The test producer, ETS claims
that TOEIC is deigned upon the theory that individuals productive language abilities such as
writing and speaking can be measured by testing the individuals receptive abilities (Lawson,
2008). According to ETSs study, the significant correlation of 0.83 between TOEIC reading
score and a writing test, and the correlation of 0.75 between TOEIC listening and speaking test
were found. However, it is impossible to take those estimated productive abilities for granted
when those abilities are measured indirectly. Moreover, another study done by Hokkaido
university showed lower correlation between TOEIC scores and productive abilities. In this
study, the researcher collected the test scores from 475 students enrolled in a business English
program, and the students were divided into intermediate and advanced classes. Students took
TOEIC as well as a company-internal interview test that tests their oral comprehension,
vocabulary, pronunciation, grammatical accuracy, and fluency in addition to a writing test that
measure their ability to write well-organized business oriented notes, emails, and letters. The
results showed only 0.59 correlation coefficient points between TOEICs reading test and the
writing test, which is much lower than the correlation reported by ETS. 0.59 correlation
coefficient points are not enough to prove that TOEICs reading score is reflected on the test
takers writing ability. When it comes to the correlation between TOEIC listening and speaking

VALIDITY OF TOEIC
score, the study found 0.73 correlation coefficient points, which is fairly close to the correlation
reported by ETS. However, the correlation coefficient points dropped when the scores of

intermediate and advanced students were examined individually. There are only 0.49 correlation
coefficient points for the scores of intermediate students and 0.65 points for the scores of
advanced students. The author concluded that ETSs claim and the notion widely held among
Japanese companies and institutions that TOEIC is valid to test English communicative abilities
are not reliable because there is not significant correlation between TOEIC scores and test takers
productive abilities (Chapman, 2006).

Content Validity
Since ETS claims that TOEIC scores indicate how well people can communicate in
English with others in the global workplace (ETS, 2007, p.4), two problems were appeared in
terms of its content validity. The first problem is that only about 40 % out of the 200 items were
found to be the direct measurement of English abilities in the international business setting.
Providing only 40% of items to measure communicative English skills in business contexts can
be a problematic when the purpose of the test is purely to testing peoples workplace English
proficiency (Lawson, 2008). However, ETS also motioned that the TOEIC can be used as the
measurement of peoples English proficiency level in everyday life. If that is the case, having
40% of business related items can be also a problem since language skills used in business
context and daily life are different, and including business related items makes it impossible to
measure true daily life English competence. The other problem is that the listening section does
not provide enough characteristics of oral conversation such as fast speech, hesitation,
phonological modification, negotiation etc (Lawson, 2008). Although TOEIC has some problem

VALIDITY OF TOEIC
in content validity, there are also some items that fit with the purpose of the test. For example,
TOEIC includes wide range levels of vocabularies so that the test can be given to the large

number f people at the same time, which was one of the original purposes of the test provided by
the Ministry of International Trade and Industry of Japan (Maher & Toda, 2006). In addition,
TOEIC adapted different kinds of English accents in listening section such as American,
Scottish, Irish, Australian an British accents in order to assess learners ability to understand
conversations in a global setting (Morikawa, 2014).

Wash Back Validity

According to Wood (2011), TOEIC has a positive wash back that TOEIC can motivate
students to study in order to attain higher scores. However according to several researchers, there
are more negative wash backs compared to positive wash backs (Wood, 2011). Lawson (2008)
claimed that many higher educations in Japan started to replace general English classes with
TOEIC preparation courses due to TOEICs high face validity and demand, although TOEIC is
testing only a part of general English competence. Therefore, studying for TOEIC does not
directly increase general English proficiency. In addition, Brown (2006) pointed out the negative
effect of test specific preparation courses that students started to ignore the meaning of the
reading, and see the tests as puzzles solving by just finding grammatical errors or key words.

Strategies taught in TOIEC preparation courses

As we have seen in the previous section, both negative and positive wash backs were
related TOEIC preparation courses. This section introduces and analyzes the strategies used in
those preparation courses.

VALIDITY OF TOEIC

Listening

Strategies
1. Understand the directions for each section before the test
starts so that you can use the time provided for reading
directions/examples to read actual questions on the test
2. Do not try to understand everything
3. Do not take time to select answers (Time is limited)
4. Select B or C if you do not know the answer
5. Try to understand a word in a context if you do not know the
word
6. Ungrammatical choices can be the answer

All of the Parts

Part 1 (10sec to choose

the correct answer)

1. Pay attention to who, when, what, where, how

2. Pay attention to similar pronounced words and homonyms
- understand the contexts that the words are used in
3. Answer with in 6 to 8 sec and use the reminding time to read
the question ahead
1. Recognize the category of the question
a. Questions that ask for specific information
- If the question asks about place, time explanation or a
person, YES or NO answers and vague or general
responses can be eliminated
b. Questions that provide a choice between 2 answers
e.g. Is he planning to travel by train or bus?
- Answers that includes neither of the choices can be
eliminated
c. Questions that ask you to deny or confirm the question
- Answers with specific information can be eliminated
- Answers for request, offer, suggestion questions are
usually YES or NO
d. Questions ask opinion of a subject
- Answers often begin with in my opinion, I think, I
feel etc.

Part 2
Part 3

1. Spend 10 to 12 sec to select answers

2. Read the possible answers and get clues for what to listen to
3. Recognize the questions ask for the subject of the
conversation, speakers occupation, relationship or when the
conversation is held
a. Questions ask for speakers occupation
- Pay attention to where speakers are at, what speakers have

10

VALIDITY OF TOEIC
and what they are doing
- Answer contains -ect,or,er,ist,cer,ary,ian,ent, or ant
b. Question ask for the main subject of the conversation
- Requires the summary for the conversation
- Recognize when, what, where, how

Part 4

Reading

1. The first questions for each conversation is usually about

subject or the main idea
2. Recognize the problem, purpose, and main idea of the
conversations

Strategies
1. Pay attention to time (45 sec per question!)

All of the Parts

1. Try to understand meaning of a word in a context if you do
not know the word
Part 1
1. Ungrammatical sentences/words can be eliminated
a. Pay attention to word order, part of speech, tense
2. Learn segments: roots, prefix, and suffix of vocabularies
Part 2
1. Read passages only once
2. Read possible answers before reading passages
3. Select the answer as you read (questions were written in
order)
4. Do not spend time on guessing uncertain answer
5. Pay attention to time
6. Do not regret insufficient preparation during the test
7. Recognize three types of factual questions
a. Questions with basic facts (usually begin with Wh questions,
of the following, according to the author etc)
- The answer is often contains synonyms
b. Questions with excluded facts
- Answer is something that is not written in the passage
c. Questions with computational facts
- Requires basic calculation
d. Questions ask for the main idea
- Too specific or vague answers can be eliminated
e. Inference questions (usually starts with it can be inferred
from the passage that)
- Grasp the result of the passage
- Eliminate illogical answers
- The correct answer often contains different forms of the
words that are used in the passage
Part 3

11

VALIDITY OF TOEIC
(Douglas & Richard, 1998)

After looking at strategies that are taught in TOEIC preparation courses, it is obvious that
most of the strategies are useful only to increase TOEIC score. 17 out of 20 strategies for
listening section were found to be TOEIC specific. Strategy 2 for all of the parts in listening
section: Do not try to understand everything, Strategy 1 for part 1: Pay attention to who, when,
what, where, how, and Strategy 2 for part 4: Recognize the problem, purpose, and main idea of
the conversations can be effective techniques for general listening practice. In addition, there was
no strategy that would directly increase peaking competence. When it comes to strategies for
reading section, only two strategies were found to be effective to improve general reading skills.
The two strategies were Strategy 1 for part 1: Try to understand meaning a word in a context if
you do not know the word, and Strategy 2 in part 2: Learn segments: roots, prefix, and suffix of
vocabularies. Strategy 1- a in section: Pay attention to word order, part of speech, and tense to
check grammar of sentences can be an effective technique for writing practice. Overall, most of
the strategies were test specific, and a few strategies were useful to increase general reading and
listening skills. Only one strategy was found to be effective to increase productive language
competence.
This result shows the negative wash backs of TOEIC and TOEIC preparation courses.
Although coachable do not always relate to lack of validity, the test is not constructed well
when the coaching of the test leads negative wash back such as increasing the score without
improving the language competence that the test aim to measure. In fact, Shin and Seong (2011)
proved the effectiveness of TOEIC specific preparation courses. The study concluded that 166
(90.7%) students increased their TOEIC score through the preparation courses, and the courses
showed a significant influence on each part of the test. When the scores are increased by using

12

VALIDITY OF TOEIC
those test specific strategies in preparation courses, that would lead to negative wash back as
well as the lack of TOEIC validity. Students are not improving much of their actual English
communicative skills while studying for a test, and that means TOEIC is failed to judge their
communication skills in a global business context.
Conclusion/Reflection

In conclusion, TOEIC has very high face validity in Japan due to the bedrock fact that the
test was developed by EST under the request of Ministry of International Trade and Industry of
Japan. As a TOEIC test taker, I feel TOEIC has a high face validity among individual test takers
not only because of the bedrock fact but also because of its high demand. In fact, I strongly and
naturally believed in TOEICs validity although I did not know the bedrock fact that created
TOEICs high face validity. Despite its remarkably high face validity and demand, the studies
found out some problems in TOEICs content and construct validities. Content has a crucial
problem that TOEIC is not measuring test takers accurate communicative English competence.
The study done by Hokkaido University showed that listening and reading skills measured by
TOEIC do not correlated with test takers productive skills. Some concerns appeared in terms of
construct validity that not as crucial than the genuine concerned in content validity. Since EST
conducts many surveys and studies every year and revised test content twice in the past, future
improvements on the problems mentioned in this paper is expected. Lastly, negative wash back
was found in terms of how TOEIC can be coached. Although it has a positive wash back that
motivate students to study to earn higher scores, the way the students increase their scores in
TOEIC preparation courses are not developing their actual English communicative competence.
Some improvements to overcome those problems are needed to determine TOEICs validity.

13

VALIDITY OF TOEIC

References
Chapman, M. (2006, April). An over-reliance on discrete item testing in the Japanese business
context. In International Conference on English Instruction and Assessment.
Educational Testing Service (2007) TOEIC User Guide Listening & Reading. Educational
Testing Service.
Educational Testing Service (2013) TOEIC Test World Wide Report 2013. Educational
Testing Service.
Ito, T., Kawaguchi, E., & Ota, R. (2011). A Study of the Relationship between TOEIC Scores
and Functional Job Performance: Self-assesment of Foreign Language Proficiency.
Educational Testing Service.
Lawson, A. J. (2008). Testing the TOEIC: Practicality, Reliability and Validity in the Test of
English for International Communication. Unpublished Paper.
Maher, J., & Toda, K. (2006). TOEIC What it isWhat it's used forWhat you should
know. , 40(1), 79-85.
Morikawa, M. (2014). The First TOEIC Preparation. Tokyo: .
Rebuck, M. (2003) National Standards. The Use of TOEIC by Companies in Japan. NUCB
Journal of Language, Culture and Communication, Vol. 5, No.1, 23-32. Accessed online
8th September 2008.
Shin, Y. H., & Seong, M. H. (2011). Effectiveness of college TOEIC courses based on teaching
strategies. , (21), 207-228.
Wood, D.J. (2001). How effective are TOEIC preparation guiltiness?.

, 2011, 6:15-25.

VALIDITY OF TOEIC

14