Quantitative Methods

HOSSEIN FARHADY
Introduction
There has been a plethora of ongoing debates in the field of applied linguistics on many
aspects of quantitative research method (QRM) in the last few decades. Despite the
validity and significance of the debates and their contribution to sharpening our
understanding of the concepts, QRM remains as one of the most widely practiced
approaches in social sciences in general, and in applied linguistics in particular
(Lazaraton, 2005; Mackey & Gass, 2016). Although such debates are important and
necessary, this entry will focus on explaining the major features of QRM in as
nontechnical terms as possible.
Most scholars would agree with the classic definition of the term “research” as a
systematic approach to answering questions (Hatch & Farhady, 1982, p. 1). There are
three key terms in this definition: a question, a systematic approach, and an answer. The
controversies in the field are not on the definition of the term “research” but on different
interpretations of the key terms in the definition. For instance, the nature of the question
and the way it should be formulated, the approach to be taken to answer the question, and
the quality of the answer to the question are all issues upon which no wide agreement
exists among scholars. The differences in the interpretations of these concepts stem from
different theoretical orientations that scholars adhere to because advocates of each theory
believe in a number of assumptions that justify their interpretations (Kinn & Curzio,
2005). Therefore, to put the issue in an appropriate context, first an explanation of the
theoretical assumptions of QRM is given and a description of how the three key terms in
the definition function in practice is discussed.

Theoretical Assumptions
Generally, an inquiry is performed within a particular school of thought commonly
referred to as a paradigm. This paradigm is believed to have four cornerstones: ontology,
epistemology, methodology, and axiology (Creswell, 2007; Lochmiller & Lester, 2017;
Onwuegbuzie & Leech, 2003). There are several popular paradigms such as positivism,
constructivism, and relativism each with some axiomatic principles that scholars follow,
or should follow, in practice. Since QRM is rooted in the positivistic paradigm, a brief
definition of the key concepts and their treatment in this paradigm will help clarify the
characteristics of QRM within this framework.
Ontology raises questions about the nature of reality and the nature of human beings in
the world. According to positivism, there is a real world, the reality of which is expressed
in terms of the relationships among variables, and the extent of these relationships can be
measured in a reliable and valid manner using a priori operational definitions. Positivists
believe in the verifiability principle, which states that something is meaningful if and
only if it can be verified by direct observation. Thus, ontologically, positivism is a school
of thought that focuses on observable, and measurable phenomena and their
interrelationships.

The Encyclopedia of Applied Linguistics, Edited by Carol A. Chapelle.

© 2013 Blackwell Publishing Ltd. Published 2013 by Blackwell Publishing Ltd.
DOI: 10.1002/9781405198431.wbeal0989
 Epistemology asks how we know the world and what the relationship between the
inquirer and the known is. Epistemologically, positivism places a premium on objective
observation of the “real world” out there. Positivists contend that the researcher (i.e.,
knower) and the object of the study (i.e., known) are independent. As such, researchers
should remain objective and impartial in studying phenomena (Cohen, Manion, &
Morrison, 2011).
Methodology focuses on the best means of acquiring knowledge about the world.
Methodologically, positivists use deductive reasoning which is a system for organizing
known facts in order to reach a conclusion. Conclusions are true only if the premises
upon which they are based are true. That is, positivists emphasize the importance of a
priori hypotheses and theories. The researcher manipulates at least one variable and then
measures its effect on another variable, controlling potentially extraneous variables.
Through such a procedure, valid cause–effect relationships are believed to be established
and generalized as laws.
Axiology deals with the ethics and asks how moral a person, a researcher should be in
the world. Every ontology or epistemology has its own ethical stance toward the world
and the self of the researcher. Positivists believe that inquiries should be value free. In
other words, the researcher’s values, interpretations, feelings, and musings have no place
in the positivist’s view of scientific inquiry. Positivists prefer rhetorical neutrality,
involving an exclusively formal writing style using impersonal voice and specific
terminology.
Putting the above-mentioned principles together, a working definition for QRM can be
formulated as an inquiry where an impartial researcher investigates cause–effect
relationships between the variables in a world of real objects. This derived definition is
conceptually matching textbook definitions where scholars contend that in QRM
numerical data are collected on the variables and analyzed using mathematical methods
(Aliaga & Gunderson, 2006). Following this definition, the three key concepts of
“question,” systematic approach, and “answer” in QRM will be detailed below.

Formulating the Research Question

QRM starts with a question which is formulated about the relationship between at least
two variables. A variable is any attribute that changes from person to person (height),
place to place (size), or time to time (temperature). Variables can be concrete, that is,
directly observable and measurable such as height and weight; abstract, that is, not
directly observable or measureable but inferred from observations and measurements
such as intelligence and language ability; discrete or categorical, that is, either/or type
such as left or right handedness, male or female; and continuous, that is, can take any
value such as weight and language ability. Research questions can be formulated about
the relationship between the variables to indicate either a cause–effect relationship or just
go togetherness between them.
Example 1: What is the effect of using multimedia tasks on the learners’ listening
comprehension (LC) ability?
Example 2: What is the relationship between the students’ LC and reading
comprehension abilities?
The first question is of a cause–effect type where the researcher sets out to investigate
whether using multimedia tasks will have any effect on the learners’ LC ability. The
second question, however, is of correlational type where the researcher intends to find an
existing relationship between the two variables. No instruction or manipulation is planned
to find a cause–effect relationship since the researcher appears on the scene after the fact
and has no role in changing any one of the variables.
An important point in formulating a research question is defining the variables under
investigation as clearly as possible in both theoretical and operational forms. To provide
a reasonable theoretical definition, an in-depth review of literature needs to be performed.
Operational definition, on the other hand, is to suggest a clearly measurable procedure for
the variable in question. For example, LC ability may be defined theoretically as the
ability to understand oral messages in real-life situations, and operationally as one’s
performance on a test which is designed to measure one’s ability to understand oral
messages.
Theoretical and operational definitions of a variable given in a particular research
context may not be acceptable to all readers because they are specific to that research
design. For instance, there may be different definitions of LC ability in theory and
different measures of LC in practice. However, a working definition should be provided
by the researcher that is most reasonable in the context of research. Despite the potential
disagreement among scholars, it is an advantage of QRM that the researcher and readers
share common definitions of the variables.
In addition to defining the variables, the researcher should determine the functions that
variables are designed to serve in the process of research. Most common functions that a
variable can serve in QRM are: independent, dependent, moderator, intervening, control,
and extraneous. An independent variable is the main or the cause variable which is under
the control of the researcher to be manipulated, while the dependent variable is the
variable that depends on, or changes as the result of, the manipulation of the independent
variable. For instance, in the example above, using multimedia tasks is the independent
variable and performance on a measure of LC ability is the dependent variable because
performance on this measure depends on the quality, quantity, and effectiveness of the
multimedia tasks designed by the researcher.
A moderator variable is the variable that may influence the outcome of the dependent
variable without being necessarily manipulated. For instance, within the context of this
example, the researcher may hypothesize that males and females may benefit
differentially from the multimedia instruction. In such a case, gender, as a moderator
variable, may change the outcome of research. A control variable, on the other hand, is a
variable whose effect is controlled or eliminated by the researcher so that the variation of
this variable does not influence the outcome of research. Suppose the researcher believes
that familiarity with computer may have an effect on the achievement and performance of
the learners. Therefore, to control the effect of familiarity with computer, the researcher
selects participants who are absolutely unfamiliar, or equally familiar for that matter, with
computer technology. In this case, any effect due to computer familiarity is removed from
the research design, that is, controlled.
An intervening variable is the variable that intervenes between the independent and
the dependent variables. It is often left unobserved and unmeasured because it is very
often an abstract variable. In our example, due to the effect of the independent variable,
that is, multimedia instruction, the participants learn and improve their LC ability.
Although this learning process can neither be observed nor measured, its effect is
manifested in the measurement of the performance on an LC test. In other words, the
learning process is situated between the independent and the dependent variables and
intervenes with the outcome of the research without being observed or measured. And
finally, an extraneous variable is a variable whose presence is neither observed nor
accounted for in the research design. Such variables may influence and contaminate the
outcome of research without the researcher’s awareness. In the example above, many
variables such as age, educational background, learning style, motivation, and so forth
may have some effect on the students’ learning. In short, any variable that is not
controlled by the researcher and may influence the outcome of research is called an
extraneous variable.
When the research question is formulated with well-defined variables, it is converted
into a research hypothesis to be tested. A hypothesis is a tentative statement about the
outcome of research and can take two forms: null and alternative. A null hypothesis,
symbolized as H0, is generally stated in the form that the manipulation of the
independent variable will not have an effect upon the dependent variable. The alternative
hypothesis, on the other hand, symbolized as H1, stipulates an effect, either positive or
negative, of the independent variable on the dependent variable. For instance, null and
alternative hypotheses for the example above would be:
Null H0: There is no relationship between using multimedia tasks and
improvement in LC ability, or
Using multimedia tasks will not have any effect on the improvement
of LC ability.
Alternative H1: There is a positive/negative relationship between using multimedia
tasks and improvement in LC ability, or,
Using multimedia instruction will have a positive or a negative
effect on the LC ability of the participants.
Selecting a null or an alternative hypothesis is often based on the review of the past
research findings. If there is strong evidence about the relationship between the two
variables, the researcher might decide to choose the alternative hypothesis to confirm or
reject the previous findings. However, when there are controversial reports on the
relationship between the variables under investigation, researchers usually go with null
hypothesis. Possible forms of hypotheses are illustrated in Figure 1.

Figure 1 Forms of hypothesis

Following a Systematic Approach

When the research hypothesis is formulated, the researcher attempts to choose a
systematic approach, technically called a research design, to test the research hypothesis.
Many factors including the nature of the research question, the type and number of
variables, the number and groups of participants in research, and the type of collected
data interact to form an efficient design that will optimize the outcome of research. In
fact, there is no single best way to determine the design of research (Clark-Carter, 2018).
However, research designs can be broadly divided into two major groups of experimental
and nonexperimental where cause–effect relationships are investigated in the former and
correlations in the latter. Depending on the factors mentioned above, experimental
designs are classified into three categories of true, quasi-, and pre-experimental. A true
experimental design is the most rigorous of all and has the following principles:
1. Subjects are selected randomly.
2. Two groups of experimental and control are formed.
3. A pretest is administered to both groups.
4. A treatment is given to the experimental group and a placebo to the control group.
5. A posttest is administered to both groups.
In a true experimental research, subjects are selected randomly. In random selection,
every individual is given an equal chance to be included in the research and no bias is
exercised since the selected sample should represent, as closely as possible, the
population to which the findings of the research are intended to be generalized. Then the
participants are divided into a control group and an experimental group. In the example
above, the researcher should make certain, as much as possible, that the two groups have
the same LC ability before starting the experiment. Therefore, even after random
selection which minimizes prior differences among the selected groups of participants, a
pretest is given to both groups and their performance is compared before the experiment.
When the equivalence of the groups is statistically established, the experimental group
receives the treatment, in this case multimedia tasks on LC, and the control group
receives a placebo, or an irrelevant treatment, to make sure that both groups are treated
similarly. After a predetermined length and intensity of giving the treatment, a posttest is
given to both groups to see the effect of treatment on the participants’ LC ability. A
schematic representation of a true experimental design is given in Figure 2.

Figure 2 A schematic representation of true experimental design

It should be mentioned that not all QRM designs meet all these requirements. It is
often very difficult to set up a true experimental design, especially in an educational
context. If any one of the requirements of the true experimental design is not met, the
researchers use pre-experimental designs. Further, quasi-experimental methods are
designed to compensate for the weakness of pre-experimental designs. Of course, the
farther away a researcher moves from meeting the requirements of the true experimental
design, the weaker the claims about the cause–effect relationship between the variables
will be. In the following section, some of the most common types of pre- and quasi-
experimental designs will be presented. For the purpose of illustration, the symbol G will
be used to refer to a “group,” R to “random,” T for “treatment,” and X for a score.
The simplest design includes only one group of subjects, one treatment, and one test
given to the subjects. This design, referred to as one shot case study, is represented in
Figure 3.
Figure 3 A schematic representation of one-shot case study
This design can be used as a base to build up more complex designs. For instance, we can
improve this design by adding one more group to make it a posttest only design as
illustrated in Figure 4.

Figure 4 A schematic representation of a posttest only design

To make this design more rigorous, pretests are given before the treatment and the
resulting design is a two intact group pretest–posttest design as illustrated in Figure 5.

Figure 5 A schematic representation of a two intact group pretest–posttest design

Pre-experimental designs are frequently used in educational contexts in general, and in
applied linguistics in particular because conducting true experimental designs is
extremely impractical in educational contexts. For instance, two intact group pretest–
posttest design, where a particular treatment, for example, multimedia tasks, is given to
one group and no treatment to the other group, and both groups receive pretest and
posttest, is very close to a true experimental design. The only missing parameter is the
random selection of the subjects. It is not simply practical to select a group of subjects
randomly from among the subjects who have already been placed in different classes on
the basis of certain criteria.
To compensate for this shortcoming, quasi-experimental designs are developed where
one group of participants functions as both the treatment and control group. A common
example of such a design is time series design. In time series design one intact group of
subjects is selected and exposed to the treatment on several occasions and tested after
each treatment. In one form of time series design, the subjects receive a pretest before the
experiment. Then they receive a few sessions of treatment in a row as if they were in the
experimental group and a posttest. Then they follow their regular instruction without
receiving the treatment as if they were in the control group and a posttest. This design,
called repeated time series design, is schematically represented in Figure 6.
Figure 6 A schematic representation of repeated time series design
There are two likely outcomes for this design. If the average score of the subjects
increases from the pretest to the first posttest, it indicates that the treatment has had a
positive effect on the performance of the subjects (black line) provided that there is no
considerable increase from posttest 2 to posttest 3 indicating that the increase in the first
part of the experiment was due to the treatment. However, if there is a consistent
increment in the scores from posttest 1 to posttest 2 (dotted line), it implies that the
treatment does not have a specific effect and the trend of increase is the natural result of
education.
Another version of this design is to present the treatment and test in alternating
sessions. This design, called alternate time series design is depicted in Figure 7.

Figure 7 A schematic representation of alternate time series design

If the treatment is effective, the scores on the tests would show a rising and falling pattern
as shown in the graph implying that the rise of the scores is due to the treatment. If the
treatment does not have any effect, the score pattern will not show systematic rise or fall.

Factorial Designs
The designs mentioned above all involved one independent variable and one dependent
variable. The test scores are obtained at different points in time either prior to or after the
treatment. More often than not, the research project involves more than one variable. It is
quite a common practice to include a moderator variable or allow more than one level of
observation for the dependent variable. In such cases, factorial designs are employed. In
fact, any simple design can be expanded into a factorial design by adding more levels to
the independent variable or adding a moderator variable. For instance, in the example
provided before, suppose the researcher speculates that boys and girls will respond
differently to receiving multimedia tasks regarding LC ability. In such a case, a moderator
variable is added to form a posttest only factorial design as shown in Figure 8.

Figure 8 A schematic representation of posttest–only factorial design

The design can be more rigorous by adding a pretest before the treatment to form a
pretest–posttest factorial design shown in Figure 9.
Figure 9 A schematic representation pretest–posttest factorial design
This is very similar to intact group pretest–posttest design with the difference that in the
factorial design the groups are male and female. Another example of factorial design
would be to introduce two levels of the independent variable. For example, giving
multimedia tasks in class versus offering them through the Internet will form two groups
for the same independent variable. Each group will receive multimedia tasks in a
different form, but the dependent variable is the same, that is, improving LC ability.
A few points are worth mentioning here. First, all the above-mentioned research
designs, whether one way or factorial, involve only one dependent variable, that is, one
measurable variable. Therefore, they are called univariate designs. When there are two or
more dependent variables in the study, the design will be more complex and is called
multivariate design (Meyers, Gamst, & Guarino, 2013). Second, not all QRM falls into
the category of experimental designs. QRM could also be descriptive or correlational.
When the focus of research is on investigating the existing relationship between the
variables without any intervention or manipulation, it is referred to as correlational
research and the design is called bivariate since two sets of scores are obtained from each
subject. In correlational research the selection of independent and dependent variables is
arbitrary because none of the variables is manipulated. For instance, to investigate the
relationship between students’ language ability and their academic success, a
correlational design is used by correlating scores on a measure of language ability and a
measure of academic success obtained from each subject.
Following a version of experimental or nonexperimental designs, the researcher
collects data on the variables. Data collection is an important stage in QRM because the
validity of the findings of research will depend very much on the quality of the collected
data. Therefore, great care should be exercised in selecting appropriate instruments for
data collection.

Answering the Research Question

In order to answer the research question, the collected data should be analyzed. Data
analysis in QRM is performed by utilizing a body of techniques called statistics. There
are good guidebooks on how to apply statistical techniques using computers. Probably
SPSS is the most commonly used statistical package with good guidebooks specifically
written for data analysis in applied linguistics and SLA research (Larson-Hall, 2016).
The final stage in quantitative research is to interpret the findings and discuss the
implications of the findings for improving the theory and their applications to practice.
The validity of the findings depends on the validity of research. Despite the objectivity in
QRM, researchers should observe some reservations in making conclusions about the
findings because statistical significance does not necessarily lead to a firm law. There is
always human wisdom that should be applied to the interpretation of the findings. Strong
statements should be avoided because the findings may not be perfectly valid.
There are two types of interrelated validity in QRM: internal and external. Internal
validity refers to the extent to which the findings are due to the process of research and
not due to other variables, and external validity refers to the extent to which the findings
are generalizable to the population from which the samples are drawn. To maximize the
internal validity, researchers focus on the unique effect of the treatment by controlling as
many variables as possible that might have an effect on the dependent variable. However,
the more controlled the context of research, the farther away it will be from the real world
and the findings will lose generalizability to the population because the context of
population will be different from the context of experiment. That is, the higher the
internal validity, the lower the external validity or generalization of the findings to the
population. On the other hand, to maximize the external validity, the context of research
should approximate the context of the real world. In such a case, the effect of the
independent variable will be blurred due to the intervention of so many uncontrolled
extraneous variables, thus decreasing the internal validity. Therefore, depending on the
context of research and the importance of the findings, researchers are recommended to
keep a balance between the internal and the external validity of the findings.
Further, some factors such as history, that is, what is happening to the participants
outside the context of experiment; mortality, that is, the attrition in the number of
participants during the experiment; test effect, that is, the effect of pretest on the posttest,
just to name a few, are considered threats to the validity of the findings of research. As
Kinn and Joan Curzio (2005) state, in many of the 450 research articles they examined,
the methodological and statistical procedures used in the quantitative research to inform
recommended practices were not convincing. This implies that only in rare cases can
researchers meet all the rigorous and vigorous requirements of QRM, especially in the
contexts of human and social sciences. Therefore, researchers are strongly recommended
to interpret the findings with caution and reservation.
SEE ALSO: Interviews in Qualitative Research; Mixed Methods; Probability and
Hypothesis Testing; Qualitative Methods: Overview; Sampling: Quantitative Methods

References
Aliaga, M., & Gunderson, B. (2006). Interactive statistics (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
Clark-Carter, D. (2018). Quantitative psychological research: The complete student’s companion (4th ed.).
London: Routledge.
Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). London, England:
Routledge Falmer.
Creswell, J.W. (2007). Qualitative inquiry and research design (2nd ed.). London: Sage.
Hatch, E., & Farhady, H. (1982). Research design and statistics for applied linguistics. New York, NY:
Newbury House.
Kinn, S., & Curzio, J. (2005). Integrating qualitative and quantitative research methods. Journal of
Research in Nursing, 10(3), 317–36.
Larson-Hall, J. (2016). A guide to doing statistics in second language research using SPSS and R (2nd ed.).
New York, NY: Routledge.
Lazaraton, A. (2005). Quantitative research methods. In E. Hinkle (Ed.), Handbook of research in second
language teaching and learning (pp. 209–24). Mahwah, NJ: Erlbaum.
Lochmiller, C. R., & Lester, J. N. (2017). An introduction to educational research: Connecting methods to
practice. Thousand Oaks, CA: Sage Publications, Inc.
Mackey, A., & Gass, S. M. (2016). Second language research: Methodology and design (2nd ed.). New
York and London: Routledge.
Meyers, L. S., Gamst, G., & Guarino, A. J. (2013). Applied multivariate research: Design and
interpretation (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.
Murtonen, M. (2005). University students’ research orientations: Do negative attitudes exist toward
quantitative methods? Scandinavian Journal of Educational Research, 49(3), 263–80.
Onwuegbuzie, A. J., & Leech, N. L. (2003, November). On becoming a pragmatic researcher: The
importance of combining quantitative and qualitative methodologies. Paper presented at the annual
meeting of the Mid-South Educational Research Association, Biloxi, MS.

Suggested Readings
Booth, W. C., Colomb, G. G., & Williams, J. M. (2008). The craft of research (3rd ed.). Chicago, IL:
University of Chicago Press.
Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.).
London: Sage.
Dörnyei, Z. (2003). Questionnaires in second language research. Mahwah, NJ: Erlbaum.
Dörnyei, Z. (2007). Research methods in applied linguistics. Oxford: OUP.
Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston:
Pearson.
Hinkel, E. (Ed.). (2005). Handbook of research in second language teaching and learning. Mahwah, NJ:
Erlbaum.
Marczyk, G., DeMatteo, D., & Festinger, D. (2005). Essentials of research design and methodology.
Hoboken, NJ: Wiley.
Mills, G. E., & Gay, L. R. (2016). Educational research: Competencies for analysis and applications (11th
ed.). Boston: Pearson.
Morgan, S. E., Reichert, T., & Harrison, T. R. (2017). From numbers to words: Reporting statistical results
for the social sciences. New York and London: Routledge.
Perry, F. L., Jr. (2005). Research in applied linguistics: Becoming a discerning consumer. Mahwah, NJ:
Erlbaum.
Porte, G. K. (2002). Appraising research in second language learning: A practical approach to critical
analysis of quantitative research. Amsterdam, Netherlands: John Benjamins.
Snyder, P., Thompson, B., McLean, M., & Smith, B. (2002). Examination of quantitative methods used in
early childhood research: Linkage with recommended practice. Journal of Early Intervention, 25(2),
137–50.