Bambang Ruby S, NIM: 0203513037, Smester 1

PPs Unnes Pendidikan Bahasa Inggris (Kelas Khusus) 2013/2014

HANDOUT 4
Research in English Language Teaching
(The Psychometric Tradition)

Activity 1
What are variables, samples, and populations, and why are they important in
research?
Variable

: Nunan (1992) stated that a variable, as the term itself suggests, is

anything which does not remain constant. In our case, it includes
language proficiency, aptitude, motivation, and so on.

Population

: Saleh (2012) stated that population is the group of people or

objects that the researcher wants to know about by conducting
the study.

Sample

: Saleh (2012) stated that the sample is representative of the

population defined.

They are important components of research because of the significant impact

that they can have on the quality of the results/findings. The variables used as
main resources while population and sample used to know the influence of
variables.
What are the basic principles of sound experimental design?
The basic principles of experimental designs are randomization, replication and
local control. Each of them is described in the following subsections.
(1) Randomization. The first principle of an experimental design is randomization,
which is a random process of assigning treatments to the experimental units. The
random process implies that every possible allotment of treatments has the
same probability. An experimental unit is the smallest division of the
experimental material and a treatment means an experimental condition whose
effect is to be measured and compared. The purpose of randomization is to

remove bias and other sources of extraneous variation, which are not
controllable. Another advantage of randomization (accompanied by replication)
is that it forms the basis of any valid statistical test. Hence the treatments must
be assigned at random to the experimental units. Randomization is usually done
by drawing numbered cards from a well-shuffled pack of cards, or by drawing
numbered balls from a well-shaken container or by using tables of random
numbers.
(2) Replication. The second principle of an experimental design is replication;
which is a repetition of the basic experiment. In other words, it is a complete run
for all the treatments to be tested in the experiment. In all experiments, some
variation is introduced because of the fact that the experimental units such as
individuals or plots of land in agricultural experiments cannot be physically
identical. This type of variation can be removed by using a number of
experimental units. We therefore perform the experiment more than once, i.e.,
we repeat the basic experiment. An individual repetition is called a replicate. The
number, the shape and the size of replicates depend upon the nature of the
experimental material. A replication is used to:
(i) Secure more accurate estimate of the experimental error, a term which
represents the differences that would be observed if the same treatments were
applied several times to the same experimental units;
(ii) Decrease the experimental error and thereby to increase precision, which is a
measure of the variability of the experimental error; and
(iii) Obtain more precise estimate of the mean effect of a treatment, since ,
where denotes the number of replications.
(3) Local Control. It has been observed that all extraneous sources of variation
are not removed by randomization and replication. This necessitates a
refinement in the experimental technique. In other words, we need to choose a
design in such a manner that all extraneous sources of variation are brought
under control. For this purpose, we make use of local control, a term referring to
the amount of balancing, blocking and grouping of the experimental units.
Balancing means that the treatments should he assigned to the experimental
units in such a way that the result is a balanced arrangement of the treatments.
The main purpose of the principle of local control is to increase the efficiency of
an experimental design by decreasing the experimental error. The point to
remember here is that the term local control should not be confused with the
word control. The word control in experimental design is used for a treatment.
Which does not receive any treatment but we need to find out the effectiveness
of other treatments through comparison.

What do we mean by inferential statistics?

Inferential statistics are data which are used to make generalizations about a
population based on a sample. They rely on the use of a random sampling
technique designed to ensure that a sample is representative.
The first type of inferential statistic is a t-test. A t-test is used to compare the
average scores between two different groups in a study to see if the groups are
different from each other. The second basic type of inferential statistics is called
an analysis of variance or ANOVA. An analysis of variance is a test that compares
the average scores between three or more different groups in a study to see if
the groups are different from each other. In other words, an ANOVA is exactly
the same as a t-test, but it can analyze multiple groups at once.
For example, a teacher believes that different teaching styles result in different
scores when children take a test over the material. He might try lecturing for one
group of students, versus worksheets with a second group of students. He then
gives everyone the same test, and wants to compare the results. If he only had
these two groups, he would use a t-test to compare the scores. However, if he
wanted to add a third teaching style, which was having the students learn the
material on their own and then teach it to each other. If he now wants to
compare all three teaching styles to each other, he would use an analysis of
variance, or an ANOVA test.
When is it appropriate to use the following statistical procedures: t-test
analysis variance, correlation, Chi-square?
T-test: when researcher need to compare the average scores between two
different groups in a study to see if the groups are different from each other.
Analysis of variance (ANOVA): when researcher need to compares the average
scores between three or more different groups in a study to see if the groups are
different from each other.
Correlation: when researcher need to determine relationship between two
variables.
Chi-square: when researcher need to compares observed frequencies to
expected frequencies.

What is the difference between true experiments, quasi-experiments, and freeexperiments?

True experiments: Tuckman (1978) stated that it provides completely adequate
controls for all sources of internal invalidity. They represent no compromise
between experimental design requirements and the nature and reality of the
situation in which a study is being undertaken.
Quasi-experiments: Tuckman (1978) stated that quasi experimental design exist
for situations in which complete experimental control is difficult or impossible.
Free-experiments: Tuckman (1978) stated that free experimental designs do not
qualify as legitimate experimental designs because they do not control
adequately against the sources of internal invalidity.

Activity 2
The Two Ways of Classifying Variables
Nunan (1992) stated that a variable, as the term itself suggests, is anything which
does not remain constant. In our case, it includes language proficiency, aptitude,
motivation, and so on. In such a case, it is customary to distinguish between the
two variables by giving them different labels. The label given to the variable that
experimenter expects to influence the other is called the independent variable.
The variable upon which the independent variable is acting is called the
dependent variable.
Based on the explanation above, it is clearly known that variables in English
language teaching and learning divide into two categories. First, independent
variable that is teaching method. Secondly, dependent variable that is the result
of the students achievement.
The kind of situation requires experiment as an appropriate way of gathering
data
The teacher found the new innovative method in teaching and learning English.
He/she has used it and made development to his/her students. The teacher want
to persuade his/her new method is better than traditional method. So, he/she
should collect evidences from the teaching method and it correlation with the
students development.

The difficulties and solutions to conduct experiment in a school setting

Nunan (1992) argued that while your research design is becoming more rigorous,
it is still no rigorous enough to allow you to claim that there is a casual
relationship between the independent variable and the dependent variable.
There is always possibility that some factor other than the experimental
materials has brought about the observed differences in the scores.
From the explanation above, we know that there is some factor apart from
dependent and independent variables. We cant conclude the result only from
the correlation of both variables, but there is possibility we should observed
some factor which make our research acceptable.
Nunan (1992) stated that in order to appreciate the logic behind the procedures,
one must be familiar with the following statistical concepts: Mean, standard
deviation, normal distribution, and standard error.
The explanation above shows us to know the solution to conduct the experiment
in a school. We should follow the statistical concepts logically, those are: Mean,
standard deviation, normal distribution, and standard error. After we done the
concepts we could reasonably claim that our research is in true experiment.
The important of standard deviation for studying numerical data
Nunan (1992) stated that the standard deviation on the other hand, is the most
important measure of dispersion, giving us information on the extent to which a
set of scores varies in relation to the mean.
Based on the explanation, Standard deviation helps researcher in measuring the
variability of a mean. It used in evaluating values in records set to the mean and
measuring of dispersion.
For example, we have two classes whose mean reading scores are the same.
With only that information, we would be inclined to teach the two classes in the
same way. In our research, for example, we have many students throughout the
entire range of performance in the first class. In this situation we will need to
have teaching strategies more challenged. But in the second class they show low
Standard deviation, it means we don't have any challenged students. They're all
average, and our teaching strategy will be entirely different.

ANOVA as the best technique of data analysis

Nunan (1992) stated that when comparing more than two means, or more than
two groups, the appropriate test is the f-test, which is based on a procedure
called analysis of variance (ANOVA). The ability of ANOVA is to test the
significance of interactions between different variables.
Based on the explanation above, ANOVA can test more than one treatment is a
major advantage over other statistical analysis such as the t-test, it opens up
many testing capabilities. ANOVAs use an F-ratio as its significance statistic
which is variance because it is impossible to calculate the sample means
difference with more than two samples.
Activity 3
Intact- group comparison

Improved control for history but weak on

external validity

Pretest posttest control group design

does not control for history or selection

Posttest only control group design

True design with no testing effect

One- shot case study

does not control for selection

Equivalent time-samples design

True design with possible testing effect

Separate-sample pretest posttest

one group design repeated twice

Nonequivalent control group design

Imperfect control of selection but better than

nondesign

Posttest-only control group design

true design with no pretest bias

Time-series design

inadequate control of history

One-group pretest-posttest design

imperfect control of history

Factorial design

true design for dealing with multiple

independent variables

Patched-up design

combination of two non design

Three design in terms of controlling for history bias

Most adequate

pretest-posttest control group design

Next most adequate

time-series design

Least adequate

one-group pretest-posttest design

Four design in terms of controlling for selection bias

Most adequate

posttest-only control group design

Next most adequate

patched-up design

Next Least adequate

nonequivalent control group design

Least adequate

intact-group comparison

Prediction: student teachers who are randomly assigned to urban schools for
experience are more likely to choose urban schools for their first teaching
assignment than student teachers who are randomly assigned to nonurban
schools.
Construct an experimental design to test this prediction.
Based on the prediction above we should be better to use quasi experimental
design because the situations in which complete experimental control is difficult
or impossible. There are times when a comparison or control group cannot be
included in an experiment. One kind of quasi experimental design that suitable
for this case is time series design. It can be diagramed as follows:
O1O2O3O4
X
O5O6O7O8

Prediction: students given programmed math instruction will gain more in

math achievement than students not given this instruction, but this effect will
be more pronounced among high math aptitude students than among low.
Construct an experimental design to test this prediction.
Based on the prediction above it can be concluded that the best design is
Nonequivalent Control Group Design. There is possibility of bias because there is
not enough randomized of Ss. This design is identical to the previous one in all

respects except for the random assignment of subjects to conditions. It can be

diagramed below:
O1

O2

----------------------O3

O4

The procedures of this design are the same as for a true design except that intact
groups rather than randomly assigned ones are used, creating a control problem
in terms of selection bias. It has ability in controlling of selection of bias,
especially by using a pretest.
Which of the following circumstances necessitate the use of a quasiexperimental design?
Experimenter cannot assign Ss to condition
A patched up design has been created where this years first graders serve as
the control group for a treatment being tried on this years second graders.
Which validity threat is not controlled?
Maturation and History
Prediction: student teachers who choose urban schools for experience are
more likely to choose urban schools for their first teaching assignment than
student teachers who choose nonurban schools.
Based on the prediction above we should be better to use quasi experimental
design because the situations in which complete experimental control is difficult
or impossible. There are times when a comparison or control group cannot be
included in an experiment. One kind of quasi experimental design that suitable
for this case is time series design. It can be diagramed as follows:
O1O2O3O4
X
O5O6O7O8
A school decides to implement a dental hygiene program for all its students. It
predicts that cavities will be reduced as a result of this program.
a. Why must a quasi-experimental design be employed to test this prediction?
Because there is only a single group is available for study and the group
pattern of the experience with the treatment is highly predetermined- that is,

the researcher must expose the group to the treatment on some systematic
basis.
b. construct one
The design used for the prediction above is Equivalent Time samples Design.
It is diagramed below
X1

O1

X0

O2

X1

O3

X0

O4

The school takes the dental hygiene programme regularly for the week(X 1).
The following week the teacher checks students teeth (O 1). Then the teacher
ask a dentist to check again students teeth (X0) followed by the teachers
checking on the next week (O2). And it is done simultanously until (O4).
The analysis of the data in this study is set up as shown above. A comparison
of O1,O2,O3 and O4 allows the teacher to compare the experiences. The
interaction between the four measurements provides a check on differential
changes overtime,
Prediction : children from broken homes will create a greater discipline
problem in school ( as evidenced by demerits) than children from intact homes.
a. Why must a criterion-group design be employed to test this prediction?
We must use criterion- group design because the context of the prediction is
suitable for the requirements of this design. An attempt is made to determine
what characteristics are associated with the criterion group and have
presumably preceded and thereby caused the criterion behavior. Illustrated by
the directive versus nondirective teacher study, the criterion group approach
might better have called a naturalistic study.
b. construct one
The criterion group design can be diagramed as follows.
C

O1

O1

O2

O1

Prediction : an after-school dance program will improve the physical skills and
social skills of first graders.
a. why does the testing of this prediction call for Hawthorne control?
The prediction above calls for Hawthorne control because the experimental
intervention were after-school dance program for first graders and dependent
variables are physical skills and social skills. Then Hawthorne intervention might
take the form of technique in teaching dance to the children during the same
period of the time that the experimental group was experiencing it. The nontreatment control condition, on the other hand, would involve no contact
whatever between experimenter and subjects.
b. Construct a design for testing it.

X
H
Ep
En

Ep

O1

En

O2

Ep

O3

En

O4

Experimental (relevant) treatment.

Hawthorne control (irrelevant experience)
Positive teacher expectation
Neutral teacher expectation created.

A researcher has just designed a special program to increase verbal I.Q. It is a

series of classroom lesson. She wants to try it out in some schools.
a. Why would a Hawthorne control be a good idea?
A Hawthorne control is a good idea because it represents a systematic
intervention and interaction on the part of experimenter with the
subject; the purpose is to introduce a new procedure that is not
anticipated to have specific effects related to the effects of the treatment
or intervention being evaluated. In Hawthorne control experiment an
irrelevant, unrelated intervention is deliberately introduced in order to
create the Hawthorne effect which is often associated with intervention.

b. Why would teacher expectancy controls be a good idea?

Teacher expectancy controls be a good idea because the teacher would
believe that the experimental innovation would be successful. The
outcome then would be a combination of the treatment plus the
teachers expectation to success.
c. Construct a design to test this program
R

Ep

O1

En

O2

Ep

O3

En

O4

Experimental (relevant) treatment.

Hawthorne control (irrelevant experience)

Ep

Positive teacher expectation

En

Neutral teacher expectation created.