IOSR Journal of Agriculture and Veterinary Science (IOSR-JAVS)

e-ISSN: 2319-2380, p-ISSN: 2319-2372. Volume 5, Issue 2 (Sep. - Oct. 2013), PP 51-58
www.iosrjournals.org

Significance of Solomon four group pretest-posttest method in

True Experimental Research- A Study
P.LavanyaKumari
Assistant Professor, Dept. of Statistics & Mathematics, Agricultural College, Bapatla, AP, INDIA.

Abstract: Experimental research is important to society - it helps us to improve our everyday lives.Typically,
an experiment is constructed to be able to explain some kind of causation. This article is focused onvarious
issues involved in comparing related groups and measuring change with pretest and posttest data. Different
pretest-posttest designs are discussed in a manner that can help researchers to select suitable test in order to
maintain internal and external validity simultaneously. One real time example from agricultural sector is given
for better understanding of theme of the paper. Despite its complexity in analysis, Solomon four group methods
help the researchers in taking valid conclusions in any field of Research. Findings enumerated for each
methodcertainly provide guidance in drawing conclusions appropriately.

Keywords: Solomon four group test, pretest-posttest designs, internal &external validity
I.

Introduction

Experimental designs are often touted as the most "rigorous" of all research designs or, as the "gold
standard" against which all other designs are judged. True experimental design is regarded as the most accurate
form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical
analysis[3].In a true experimental study, a control and an experimental group are used. The dependent variable
in each group is observed before introduction of the independent variable. Then only the experimental group is
introduced to the independent variable. Observations of the dependent variable are made for both groups after
the exposure of the experimental to the independent variable.
Pretest-posttest designs are an expansion of the posttest only design with nonequivalent groups, one of
the simplest methods of testing the effectiveness of a treatment. In this design, which uses two groups, one
group is given the treatment and the results are gathered at the end. The control group receives no treatment,
over the same period of time, but undergoes exactly the same tests. Statistical analysis can then determine if the
intervention had a significant effect or not. One common example of this is in agriculture; one set of plots is
given a fertilizer, whereas the control group is given none, and this allows the researchers to determine if the
fertilizer really works. This type of design, whilst commonly using two groups, can be slightly more complex.
For example, if different levels of a fertilizer are tested, the design can be based around multiple groups.
Whilst this posttest only design does find many uses, it is limited in scope and contains many threats
to validity. It is very poor at guarding against assignment bias, because the researcher knows nothing about the
individual differences within the control group and how they may have affected the outcome. Even with
randomization of the initial groups, this failure to address assignment bias means that the statistical power is
weak.The results of such a study will always be limited in scope and, resources permitting; most researchers use
a more robust design, of which pretest-posttest designs are one. The posttest only design with non-equivalent
groups is usually reserved for experiments performed after the fact, such as anAgricultural researcher wishing to
observe the effect of a fertilizer that has already been administered.
For many true experimental designs, pretest-posttest designs are the preferred method to compare
participant groups and measure the degree of change occurring as a result of treatments or interventions.Pretestposttest designs grew from the simpler posttest designs, and address some of the issues arising with assignment
bias and the allocation of participants to groups. One example is Agriculture, where researchers want to monitor
the effect of insecticides upon yields of wheat. Other areas include evaluating the effects of counseling, testing
medical treatments, and measuring psychological constructs. The only stipulation is that the subjects must
be randomly assigned to groups, in a true experimental design, to properly isolate and nullify any nuisance
or confounding variables.

www.iosrjournals.org

51 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
II.Terminology
2.1 Confounding Variable
In any experiment there are many kinds of variables
that will affect the experiment. The independent variable is the
manipulation for the experiment and the dependent variable is
the measurement taken from that experiment. Confounding
variables are things in which have an effect on the dependent
variable, but were taken into account in the experimental
design.
For example, one wants to know if fertilizer X has an effect on getting more yields. The experimenter
must take care to design the experiment so that he can be very sure that the yields under observation must be
gained because of the influence of Fertilizer X, and that the improvement was not caused by other factors, called
confounding variables.
2.2 Internal and External validity
External and internal validity are not all-or-none, black-and-white, present-or-absent dimensions of an
experimental design. Validity varies along a continuum from low to high.
2.2.1. Internal validity is a property of scientific studies which reflects the extent to which a causal conclusion
based on a study is warranted. Such warrant is constituted by the extent to which a study minimizes bias. In
other words, to the degree that we are successful in eliminating confounding variables within the study itself is
referred to as internal validity.
In other words, Internal validity is the degree to which the experimental treatment makes a difference
in (or causes change in) the specific experimental settings [5]. The factors that threaten internal validity are:
history, maturation, pretest effects, instruments, and statistical regression toward the meanand differential
selection of participants, mortality, and interactions of factors (e.g., selection and maturation).
2.2.2 External validity is the validity of generalized (causal) inferences in scientific studies, usually based on
experiments as experimental validity. In other words, it is the extent to which the results of a study can be
generalized to other situations and to other people. Hence, a study that readily allows its findings to generalize
to the population at large has high external validity [14].Hence, External validity is the degree to which the
treatment effect can be generalized across populations, treatment variables, and measurement instruments.
Threats to external validity include: interaction effects of selection biases and treatment, reactive interaction
effect of pretesting, reactive effect of experimental procedures, and multiple-treatment interference [1].

III. Types of Experimental Designs

3.1 The Posttest Only Design with Non-Equivalent Control Groups
3.1.1 Requirements of the design
It must have two groups.
It uses a post-only measure.
It has two distributions (measures), each with an average and variation.
Assessment of treatment effect = statistical (i.e., non-chance) difference between the groups.
This is a two group design including an experimental group and a control group, where assignment to
groups occurs through randomization. The experimental group is exposed to the manipulation, the control group
is not. There is no pre measurement - a measure of how conditions would have been if there was no
manipulation. Both measures are taken after the manipulation occurred.
3.1.2Limitations of Posttest Only Design
There is no benchmark.
There is no random assortment to treatment groups.
This design recognizes the threats associated with pretesting effect and pretest-manipulation interaction
bias.
www.iosrjournals.org

52 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
3.2 The Two Group Control Group Design
This is, by far, the simplest and most common of the pretest-posttest designs, and is a useful way of
ensuring that an experiment has a strong level of internal validity. The principle behind this design is relatively
simple, and involves randomly assigning subjects between two groups, a test group and a control. Both groups
are pre-tested, and both are post-tested, the ultimate difference being that one group was administered the
treatment.
This test allows a number of distinct analyses, giving researchers the tools to filter out experimental
noise and confounding variables. The internal validity of this design is strong, because the pretest ensures that
the groups are equivalent. The various analyses that can be performed upon a two-group control group pretestposttest designs are (Fig 1).

3.2.1 Analysis and interpretation

By comparing the final posttest results between the two groups with the help of independent sample ttest, one can find the effectiveness of treatment. (C)
With the help of paired t-test one can observe how both groups changed from pretest to posttest,
whether one, both or neither improved over time. If the control group also showed a significant
improvement, then the researcher must attempt to uncover the reasons behind this. (A and A1)
Prior to the above tests, the researchers must compare the scores in the two pretest groups, to ensure
that the randomization process was effective using independent sample t-test. (B)
These tests evaluate the efficiency of the randomization process and also determine whether the group
given the treatment showed a significant difference.
3.2.2 Limitations of Pretest-Posttest Designs
The main problem with this design is that it improves internal validity but sacrifices external validity to
do so.
There is no way of judging whether the process of pre-testing actually influenced the results because
there is no baseline measurement against groups that remained completely untreated.
The other major problem, which afflicts many sociological and educational research programs, is that it
is impossible and unethical to isolate all of the participants completely.
3.2.3 Need for Solomon four group method
The two-group control group design is an exceptionally useful research method, as long as its
limitations are fully understood. For extensive and particularly important research, many researchers use
the Solomon four group methods, a design that is more costly, but avoids many weaknesses of the simple
pretest-posttest designs.
3.3 The Solomon Four Group Design
This design contains two extra control groups, which serve to reduce the influence of confounding
variables and allow the Solomon four group method researchers to test whether the pretest itself has an effect
on the subjects.Whilst much more complex to set up and analyze, this design type combats many of the internal
validity issues that can plague research. It allows the researcher to exert complete control over the variables and
allows the researcher to check that the pretest did not influence the results.
The Solomon four group test is a standard pretest-posttest two-group design and the posttest only
controls design. The various combinations of tested and untested groups with treatment and control
groups allow the researcher to ensure that confounding variables and extraneous factors have not influenced
the results.
www.iosrjournals.org

53 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
In the figure, A, A1, B and C are exactly the same as in the standard two group design.

3.3.1 Analysis and interpretation

The first two groups of the Solomon four group designs are designed and interpreted in exactly the
same way as in the pretest-post-test design discussed in previous section, and provide the same tests
upon randomization.
The comparison between the posttest results of groups C and D, marked by line 'D', allowsus to
determine if the actual act of pretesting influenced the results. If the difference between the posttest
results of Groups C and D is different from the Groups A and B difference, then the researcher can
assume that the pretest has had some effect upon the results.
The comparison between the Group B pretest and the Group D posttest allows the researcher to
establish if any external factors have caused a temporal distortion which was not included in present
study. For example, it shows if anything else could have caused the results shown and is a check
upon causality.
The Comparison between Group A posttest and the Group C posttest helps the researcher to determine
the effect that the pretest has had upon the treatment. If the posttest results for these two groups differ,
then the pretest has had some effect upon the treatment and the experiment is flawed.
The comparison between the Group B posttest and the Group D posttest shows whether the pretest
itself has affected behavior, independently of the treatment. If the results are significantly different,
then the act of pretesting has influenced the overall results and is in need of refinement.
3.3.2 Limitations of Solomon Four Group Design
The Solomon four group designs is one of the benchmarks for sociological and educational research,
and combats most of the internal and external validity issues apparent in lesser designs. Despite
the statistical power and results that are easy to generalize, this design does suffer from one major
drawback that prevents it from becoming a common method of research: the complexity.
A researcher using a Solomon four group design must have the resources and time to use four research
groups, not always possible in tightly funded research departments. Most schools and organizations are
not going to allow researchers to assign four groups randomly because it will disrupt their normal
practice. Thus, a non-random assignment of groups is essential and this undermines the strength of the
design.
Secondly, the statistics involved is extremely complex, even in the age of computers and statistical
programs. Unless the research is critical or funded by a large budget and extensive team of researchers,
most experiments are of the simpler pretest-posttest research designs. As long as the researcher is fully
aware of the issues withexternal validity andgeneralization, they are sufficiently robust and a Solomon
four group design is not needed.
3.3.3 Real time example
For the sake of clear understanding, a real time problem has been considered and is solved with three
different designs and results are compared.

IV. Methodology
4.1 Research approach
Survey method has been adopted to study the key objective of the Research i.e. whether training
programme for farmers conducted by Govt.Officials on modern cultivation methods is effective or not.
www.iosrjournals.org

54 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
4.2 Population
The population for the present study consists of large farmers who had more than 10 acres of cultivable
land in Chittoor district of Andhra Pradesh.
4.3 Sample and Sampling Technique
Three Stage Random sampling technique has been adopted to select the sample of farmers. For this
study four mandals have been selected and from each mandal one village has been selected randomly. From
each selected village, 24 farmers are again selected randomly. Hence, totally 96 farmers in four groups of size
24 haven been selected at random.
4.4 Study variable is Awareness score of farmers on modern cultivation.
A tool has been prepared to assess the knowledge levels of farmers with regard to modern cultivation
methods. Tool consists of 50 questions and measured with Yes or No type options. Reliability is found as
8.65 which is good and validity is checked.
4.5 Data
A study is designed to assess the awareness of farmers on modern cultivation techniques in agriculture.
As discussed in methodology, four groups of farmers had been considered and groups are labeled 1, 2, 3 and 4
respectively. Out of four groups, Group-2 and Group-4 have not been invited for awareness programme on
modern cultivation methods conducted by Govt. Officials. Group-1 and Group-3 had attended an awareness
programme. Group-1 and Group-2 had been interviewed before and after training whereas Group-3 and Group-4
had been questioned only after the training. Awareness scores are noted in table-1.
Notations:
Ex1Pre: Awareness Scores of group-1before attending training.
Ex1Post: Awareness Scores of group-1after attending training.
CG1Pre: Awareness Scores of group-2 before training who are not invited for training.
CG1Post: Awareness Scores of group-2 after training who are not invited/attended.
Ex2Post: Awareness Scores of group-3 after attending training.
CG2Post: Awareness Scores of group-4 after training who are not invited/attended.
Table-1: Awareness score of the farmers
Group-1

Group-2

Farmer
Number

Ex1Pre

Ex1Post

CG1Pre

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Mean

29
25
30
27
29
29
27
28
26
29
26
27
29
28
30
26
30
27
30
30
27
30
25
27
27.96

32
30
34
34
31
32
35
32
34
30
32
35
32
30
32
31
34
33
32
30
32
30
31
35
32.21

26
26
25
29
29
29
29
25
26
25
30
27
28
26
25
26
28
30
30
27
25
26
29
29
27.29

Group-3

Group-4

CG1Post

Ex2Post

CG2Post

27
27
29
27
27
29
28
30
27
27
30
29
27
30
26
26
28
26
27
27
28
30
28
26
27.75

35
35
32
31
32
31
34
35
33
35
34
35
35
32
35
35
34
35
34
33
33
34
33
34
33.71

30
25
28
27
25
30
26
29
27
28
25
30
29
26
25
25
25
30
27
29
29
27
27
25
27.25

www.iosrjournals.org

55 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
4.6 Statistical Analysis
Collected data is properly entered in computer for the sake of analysis. Appropriate Statistical
techniques such as Independent sample t-test and Paired t-tests have been applied with the help of SPSS version
20 and results are concluded as per the level of significance. Significant findings are presented graphically.

V.

Results and Discussions

Three different designs had been applied in order to test the main objective and findings have been
compared with respect to their effectiveness.
5.1

Solution with the Posttest Only Design with Non-Equivalent Control Groups
Line/Fig

Test No

D/Fig(2)

Group
CG2POST
EX2POST

Mean
27.25
33.71

Sd
1.89
1.33

Mean Diff

t-value

p-value

Remark

6.46

13.66**

0.00

Sig @ 1%

Findings of design 3.1

Using independent sample t-test, it can be concluded that there is significant difference between
CG2POSTand EX2POST (without pretest). Since the difference is statistically significant at 1% level, hence it can be
concluded that the training is highly effective in enhancing the awareness levels of farmers on modern
cultivation methods in agriculture.
As a remark, at this point one cannot give assurance whether the existing change only due to training or
any other intervening variable which is not included or studied at this time of survey. Hence further refinement
over conclusions is required which leads to the following design.
5.2

Solution with the Two Group Control Group Design

Line

Test No

Group Mean
EX1PRE 27.96
A /Fig(1) 1(Pt-test)
EX1POST 32.21
CG1PRE 27.29
A1/Fig(1) 2(Pt-test)
CG1POST 27.75
CG1PRE 27.29
B/Fig(1) 3(It-test)
EX1PRE 27.96
CG1POST 27.75
C/Fig(1) 4(It-test)
EX1POST 32.21
*significant @ 5% level
** significant @ 1% level
It-test : Independent sample t-test
Pt-test : Paired/related sample t-test

Sd
1.68
1.69
1.83
1.36
1.83
1.68
1.36
1.69

Mean Diff

t-value

p-value

Remark

4.25

8.25**

0.00

Sig @ 1%

0.46

0.93

0.36

NS

0.67

1.31

0.20

NS

4.46

10.06**

0.00

Sig @ 1%

Findings of design 3.2

Test#1 suggests that no difference found between before and after values in Control group but Test#2
reveals that there is significant difference between pre and post score of Experiment group which means
significant impact of treatment has been found on experiment group.
Further Test#3 suggests that the randomization process was effectively applied to get samples under
CG1Pre and EX1Pre because no significant difference has been found. Test C ensures that there is significant
difference between CG1Post and EX1Post.
Hence, it can be concluded that there treatment is very effective in enhancing the scores and perhaps there
may be some sort of influence of pre-test on score because the mean difference has been increased from
4.25 to 4.46 from test#1 to test#4.

www.iosrjournals.org

56 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
5.3 Solution with the Solomon four Group Control Group Design
Line

Test No

A/Fig(2)

1(Pt-test)

A1/Fig(2)

2(Pt-test)

B/Fig(2)

3(It-test)

C/Fig(2)

4(It-test)

D/Fig(2)

5(It-test)

E/Fig(2)

6(It-test)

F/Fig(2)

7(It-test)

G/Fig(2)

8(It-test)

Group
EX1PRE
EX1POST
CG1PRE
CG1POST
CG1PRE
EX1PRE
CG1POST
EX1POST
CG2POST
EX2POST
CG1PRE
CG2POST
EX1POST
EX2POST
CG1POST
CG2POST

Mean
27.96
32.21
27.29
27.75
27.29
27.96
27.75
32.21
27.25
33.71
27.29
27.25
32.21
33.71
27.75
27.25

Sd
1.68
1.69
1.83
1.36
1.83
1.68
1.36
1.69
1.89
1.33
1.83
1.89
1.69
1.33
1.36
1.89

Mean Diff

t-value

p-value

Remark

4.25

8.25**

0.00

Sig @ 1%

0.46

0.93

0.36

NS

0.67

1.31

0.20

NS

4.46

10.06**

0.00

Sig @ 1%

6.46

13.66**

0.00

Sig @ 1%

0.04

0.08

0.94

NS

1.50

3.41**

0.00

Sig @ 1%

0.50

1.05

0.30

NS

Findings of design 3.3

Test#2 suggests that no difference found between before and after values in Control group but Test#1
reveals that there is significant difference between pre and post score of Experiment group which
means significant impact of training has been found on experiment group.
Further Test#3 concludes that the randomization process was effectively applied to get samples under
CG1Pre and EX1Pre because no significant difference has been found.
Test#4 ensures that there is significant difference between CG1Post and EX1Post. Hence, it can be
concluded that the training is very effective in enhancing the awareness scores and perhaps there may
be some sort of influence of pre-test procedure on awareness scores because the mean difference has
been increased from 4.25 to 4.46 from Test#1 to Test#4.
Test#5 proves that there is significant difference between CG2POST and EX2POST (without pretest)at 1%
level, hence it can be inferred that the training is highly effective.
Both Test#4 and Test#5 suggest that there is significant impact of training but mean difference of
Test#5 is significantly higher than that of Test#4which is only due to pre-test procedure adopted for the
farmers in this study. That means farmers who had not been attended training might be enquired about
modern cultivation techniques and enriched themselves with some sort of informationdue to pretest.
Interestingly, Test#6 suggests that no external factors have been included unknowingly in this study.
Further Test#7 reveals that pretest has had effect upon the awareness scores hence the experiment is
flawed.
Finally, with the help of Test#8 one can conclude that the procedure of pre-test itself has affected
behavior or not. In this example it is proved because farmers who undergone pretest have inquired
about modern techniques and might have got some sort of information.
All observations from four groups regarding mean awareness scores of farmers have been showed in Fig-3.

Fig-3: Average awareness Score of farmers

40
35
30
25
20
15
10
5
0

34

32

28

EX1PRE

EX1POST

Group-1

27

28

27

CG1PRE

CG1POST

CG2POST

EX2POST

Group-3

Group-4

Group-2

www.iosrjournals.org

57 | Page

Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study
VI. Conclusion
Experiments are conducted to be able to predict the phenomenon. Present scenario of the research is
mainly based on experimental research which provides causality for the variation. There are several designs to
find causal relationship among variables. Amongst, Solomon four-group design(Design-3.3) is superior to
Posttest only design(Design-3.1) and Two group control group designs(Design-3.2) because, along with
controlling for effects of history, maturation, and pretesting, it allows for evaluation of the magnitudes of such
effects and higher degree of external validity despite difficulty in conducting the design.

Acknowledgements
Author is very grateful to the farmers, and farm workers who assisted wholeheartedly in collecting the
data from them at Nellore district.

References
[1]

J. Bellini and P. Rumrill, Research in rehabilitation counseling, Springfield, IL: Charles C.Thomas.

[2]

R.D. Bock, Basic issues in the measurement of change. in: Advances in Psychological and Educational Measurement, D.N.M.
DeGruijter and L.J.Th.Van der Kamp, eds, JohnWiley&Sons, NY, 1976, pp. 7596.

[3]

A.D. Bryk and H. I. Weisberg, Use of the nonequivalent control group design when subjects are growing, PsychologicalBulletin85
(1977), 950962.

[4]

I.S. Cahen and R.L. Linn, Regions of significant criterion difference in aptitude- treatment interaction research,
AmericanEducational Research Journal 8 (1971), 521530.

[5]

L.J. Cronbach and L. Furby, How should we measure change - or should we? PsychologicalBulletin74 (1970), 6880.

[6]

D.M. Dimitrov, S. McGee and B. Howard, Changes in students science ability produced By multimedia learning environments:
Application of the Linear Logistic Model for Change, School Science and Mathematics 102(1) (2002), 1522.

[7]

G.H. Fischer, Some probabilistic models for measuring change, in: Advances in Psychological and Educational
Measurement,D.N.M. DeGruijter and L.J.Th. Van der Kamp, eds,John Wiley & Sons, NY, 1976, pp. 97110.

[8]

G.H. Fischer and E. Ponocny-Seliger, Structural Rasch modeling, Handbook of the usage of LPCM-WIN 1.0,
Progamma,Netherlands, 1998.

[9]

R.K. Hambleton, H. Swaminathan and H. J. Rogers, Fundamentals of Item Response Theory, Sage, Newbury Park, CA,1991.

[10]

S.W. Huck and R.A. McLean, Using a repeated measures ANOVA to analyze data from A pretest-posttest design: A potentially
confusing task, Psychological Bulletin 82 (1975),511518.

[11]

S. Isaac and W.B. Michael, Handbook in research and evaluation 2nd. ed., EdITS, San Diego, CA, 1981.

[12]

E. Jennings, Models for pretest-posttest data: repeated measures ANOVA revisited, Journal of Educational Statistics 13 (1988),
273280.

[13]

K.G. Joreskog and D.Sorbom, Statistical models and methods for test-retest situations, in: Advances in Psychological
andEducational Measurement, D.N.M. DeGruijter and L.J.Th.Van der Kamp, eds, John Wiley & Sons, NY, 1976, pp. 135157.

[14]

L. Linn and J.A. Slindle, The determination of the significance of change between pre- and posttesting periods, Review of
Educational Research 47 (1977), 121150.

[15]

F.M. Lord, The measurement of growth, Educational and PsychologicalMeasurement16 (1956), 421437.

[16]

S. Maxwell, H.D. Delaney and J. Manheimer, ANOVA of residuals andANCOVA: Correcting an illusion by using model
comparisons and graphs, Journal of Educational Statistics 95 (1985), 136147.

[17]

G.J. Mellenbergh, A note on simple gain score precision, AppliedPsychologicalMeasurement 23 (1999), 8789.

[18]

J.E. Overall and J. A. Woodward, Unreliability of difference scores: A paradox for measurement of change,
PsychologicalBulletin82 (1975), 8586.

[19]

D. Rogosa, D. Brandt and M. Zimowski, A growth curve approach to the measurement of change, Psychological Bulletin 92 (1982),
726748.

[20]

I. Rop, The application of a linear logistic model describing the effects of preschool education on cognitive growth, in: Some D.M.
Dimitrov and P.D. Rumrill, Jr. / Pretest-posttest designs and measurement of change 165 mathematical models for social
psychology, W.H. KempfandB.H. Repp, eds, Huber, Bern, 1976.

[21]

D. Sorbom, A statistical model for the measurement of change in true scores, in: Advances in Psychological and Educational
Measurement, D.N.M. DeGruijter and L.J.Th. Van der Kamp, eds, John Wiley & Sons, NY, 1976, pp. 11591170.

www.iosrjournals.org

58 | Page