|


 

Brian Mennecke
College of Business
Iowa State University
The Source«

  e source of muc of t is information

comes from Campbell & Stanley«
± Campbell, D. . and J.C. Stanley (1963).
Experimental and Quasi-Experimental
Designs for Researc . C icago: Rand
McNally College Publis ing Company.
The Goal of Research

 r en one conducts science, t e goal is

to seek out t e trut .
± Question: How does one identify trut

 Experimentation is one mec anism for

identifying causation, w ic is a step
toward understanding ow one set of
factors influence anot er set of factors
Rausation and Positivism

 Positivism is a researc perspective t at as

as its premise t at inferences about cause
can be made.
 David Hume espoused t e conditions by
w ic inference could be made; t ese
include«
± Contiguity between t e cause and effect
± emporal precedence
± Constant conjunction (i.e., w en t e effect is seen,
t e cause is always present)
2ut how do we know something
is true?
 Some propositions are not true; ow do
we know w en somet ing is true or
not

 ne approac is to test for validity.
± Validity is a term used to describe w et er
t e conclusions one draws about a
proposition are true or false
Types of validity
 Internal Validity: How sure are we t at t e cause
leads to t e expected results
In ot er words, is it
appropriate for us to infer t at t e relations ip
between variables is causal
 External Validity: How sure are we t at we can
generalize t e finding of causation to ot er
populations, settings, or variables

 Construct Validity: How sure are we t at t e
variables we are using actually measure t e concept
(i.e., t e construct) t at we are seeking to measure

 Statistical Conclusion Validity: Do t e statistical tests
t at we perform accurately measure t e relations ips
between t e variables under study

Threats to Internal Validity
(Rampbell & Stanley)

 History: events t at occur between t e first and second measurement

t at are unrelated to t e experiment but t at could affect t e results.
 Maturation: C anges in t e participants t at occur as a function of t e
passage of time and not specific to t e experiment.
 esting:  e effects of taking a —
test on t e scores of a second test.
 Instrumentation: C anges in t e measurement instrument or c anges
or t e observers make c anges in t e obtained measurements.
 Statistical regression (toward to mean): Groups aving extreme scores
on t e pretest (or selected on t e basis of extreme scores) will tend to
ave scores closer to t e mean on t e posttest.
 Selection: Biases resulting in differentials selection of respondents for
t e comparison groups.
 Experimental mortality: Differential loss of respondents from t e
comparison groups.
 Selection-maturation interaction, ot er interaction effects:
Threats to External Validity
 Reactive or interaction effects of testing:  e pretest
itself mig t be a learning experience suc t at by
taking t e pretest students gain information t at will
affect posttest results
 Interaction of selection and t e experimental variable:
Different groups may respond differently to t e
experimental variable.
 Reactive effects of experimental arrangements:
Subjects respond differently because t ey know t ey
are in an experiment (i.e., t e Hawt orne effect)
 Multiple treatment interference: Multiple treatments
applied to t e same respondents; t e effects of prior
treatments cannot be erased.
rhat is the basis for asking
questions about causation?
  e source for all questions pertaining
to researc experimentation is t eory
± r y is t eory important

  eory s ould always drive researc

because it defines expectations about
t e relations ips t at exist between
variables.
2efore we get started«
 Some definitions:
± Construct: An idea or concept t at you are
attempting to measure
‡ Latent Construct: A construct t at cannot be measured
directly (e.g., group co esion)
± Independent Variable: Variables t at are presumed
to be t e cause of an effect being studied;
independent variables are „
  to examine
t eir impact on results
± Dependent Variables: Variables t at are observed
to understand t e result of causation.
± Hypot esis: A statement of a possible explanation
for causation. An ypot esis is tested by drawing
conclusions from an experimental examination of
t e variables t at are expected to be related
Types of Experimental Designs

 Pre-experimental designs: ne group designs and

designs t at compare pre-existing groups
 Quasi-experimental designs: Experiments t at ave
treatments, outcome measures, and experimental
conditions but t at do not use random selection and
assignment to treatment conditions.
 rue experimental designs: Experiments t at ave
treatments, outcome measures, and experimental
conditions and use random selection and assignment
to treatment conditions.  is is t e strongest set of
designs in terms of internal and external validity.
Pre-Experimental Designs

 Design 1: ne-S ot Case Study: A

single group is studied once after some
intervention/treatment t at is presumed
to cause c ange.
± For example, a training program is
implemented and participants are given a
posttest at t e conclusion of t e training.

X 
Pre-Experimental Designs
 Design 2: ne-Group Pretest-Posttest Design:
ne group, not randomly selected nor
randomly assigned, is given a pretest, followed
by a treatment/intervention, and finally a
posttest.  ere is no comparison group.
Generally done wit intact groups.
± For example, a classroom teac er gives er
students a pretest t en implements an instructional
strategy followed by a posttest.

1 X 2
Pre-Experimental Designs
 Design 3:  e Static-Group Comparison: ne group
w ic as experienced a treatment/intervention (X) is
compared to anot er group t at as not ad t e
intervention.  e groups are not randomly selected nor
randomly assigned and are generally pre-existing
groups.  ere is no pre-observation/pretest.
± For example, comparison of GRE scores for students w o
attended a rural ig sc ool versus t ose w o attended an
urban ig sc ool.

X1 
X2 
True Experimental Designs
 Design 4: Pretest-Posttest Control Group Design: ne
group is administered a treatment w ile t e ot er is
not; all groups are observed before and after t e
treatment is administered.
± For example, 50 fres man students are randomly selected to
participate in a tutoring study. Half are randomly assigned to a
tutor for t eir first semester and alf are not. All students are
given a pretest at t e beginning of t e term and a posttest at
t e end of t e term.

R 1 X 2
R 1 2
True Experimental Designs
 Design 5: Solomon Four-Group Design:  is design involves four
experimental groups. wo of t e groups parallel t e structure of
Design 4 w ile t e remaining two groups include no pre-test (so
t at t e effects of t e pretest can be evaluated).
± For example, 100 fres man students are randomly selected to
participate in a tutoring study. 25 are randomly assigned to a tutor for
t eir first semester and given a pretest. 25 are randomly assigned to
a group w ere no tutor is assigned and t ey are given a pretest.
Anot er 25 are randomly assigned to a tutor but not given a pretest.
 e remaining 25 are randomly assigned to a group w ere no tutor is
assigned and t ey are not given a pretest.
r ew!

R 1 X 2
R 1 2
R X 2
R 2
True Experimental Designs
 Design 6: Posttest nly Control Group Design: ne
group is administered a treatment w ile t e ot er is
not; all groups are observed after t e treatment is
administered BU not before t e treatment.
± For example, students are randomly assigned to two groups of
50 eac .  e experimental (treatment) group receives a new
teac ing met od during a special class session.  e second
group (t e control) receives a traditional teac ing met od
during a special class session. No pretest is used for eac
group. Issues suc as existing grades, SA scores, and ot er
factors are examined as covariates.

R X 2
R 2
Ñuasi-Experimental Designs
 Design 7:  e ime-Series Experiment:  is
design involves periodic measurements of
some group or individuals and t e introduction
of a c ange into t e conditions during t e
series.
± For example, studying a group of workers over time
and taking several measures of productivity during
t is period. At some point a new work process is
introduced and measures of productivity are taken
over several weeks following t e intervention.

1 2 3 X 4 5 6
Ñuasi-Experimental Designs
 Design 8: Equivalent ime-Samples Designs:
 is design involves periodic introduction of
treatments followed by measurements wit t e
treatments varied consistently over time.
± For example, to study t e effect on student
discussions of aving an observer appear in a
classroom. At time period one, an observer is
present and a measure of discussion level is made.
At time two, no observer is present and a measure
of discussion level is made. At time t ree an
observer is present, a measure is taken. At time
four an observer is not present, a measure is taken.
Etc.

X1  X2  X1  X2 
Ñuasi-Experimental Designs
 Design 9:  e Equivalent Materials Design:  is
design involves giving equivalent samples of materials
to subjects, imparting interventions, and t en making
observations.
± For example, subjects are asked to complete a survey
instrument about t eir opinions related to current events.  e
students are t en split into two groups and given two different
sets of (falsified) survey results indicating ow ot er students
answered t e survey. Bot groups are t en asked to complete
t e survey again to observe ow t ey respond.

Experimental Materials A() X0 

Experimental Materials B() X0 
Ñuasi-Experimental Designs
 Design 10: Nonequivalent Control Group:  is design
involves an experimental and control group wit bot
given pretests and posttest; owever, t ese groups are
not randomly selected because t ey constitute
naturally assembled groups (e.g. classrooms).  e
assignment of X (t e treatment) to one group or t e
ot er is randomly selected by t e researc er.
± For example, four sections of a course are c osen to
participate in a study of teac ing met ods. Half are randomly
assigned a new teac ing met od and alf are not. All are
given pretests at t e beginning of t e term and all are given
posttests at t e end of t e semester.

 X 
 
Ñuasi-Experimental Designs
 Design 11: Counterbalanced Designs: In t is design all subjects
receive all treatments but in a different order. Eac treatment
occurs once at eac time period and once for eac treatment
group. A Latin-square design is a type of counterbalanced design
in w ic four treatments are applied to four naturally assembled
pools of subjects.

± For example, consider a study of t e effect of different training

met ods on learning. Subjects are placed into four groups (A,B,C, D)
for different training met ods, X1-X4.

Group A X1 X2 X3 X4

Group B X2 X4 X1 X3
Group C X3 X1 X4 X2
Group D X4 X3 X2 X1
Ñuasi-Experimental Designs
 Design 12:  e Separate Sample Pretest-Posttest Design: ften
used wit large populations (i.e., in public opinion studies) w ere
t e researc er cannot randomize or segregate subgroups for
different treatments. wo equivalent groups are identified, one
sample is measured prior to t e treatment and a different (but
equivalent) sample is measured after t e treatment.  is design is
also called t e "simulated before and after" design.

± For example, 100 community members are randomly surveyed

concerning t eir opinions about local government policies. A PR
campaign is t en conducted for six weeks. A follow-up survey is t en
conducted wit 100 different residents w o are randomly selected.

R  X
R X 
Ñuasi-Experimental Designs
 Design 13:  e Separate Sample Pretest-Posttest
Control Group Design:  is design is similar to Design
12; owever, a control group is added to t e design.
± For example, consider t e PR campaign described in Design
12. In t is case, t e same design is used, but, in addition, t e
measurements are made in a similar nearby city w ere no PR
campaign is run.

R  X
R X 

R 
R 
Ñuasi-Experimental Designs
 Design 15: Recurrent Institutional Cycle Design (A "Patc ed-Up" Design):
 is is an approac used in field researc . A researc er begins wit an
inadequate design and t en adds features to control for one or more
sources of invalidity.  e result is an "inelegant accumulation of
precautionary c ecks."  e researc er is aware of rival interpretations
(sources of internal invalidity) and incrementally identifies ot er data t at
would rule out rivals.  e design exploits contextual features to refine t e
researc as it progresses.
± For example, t is design would combine a longitudinal and cross sectional
structure. ne group will be exposed to X and measured at t e same time as
a second group t at is just about to be exposed to X. A comparison of t e two
groups would be able to be made because it is equivalent to a static group
comparison.  e second group would be remeasured (posttest), w ic would
make t e design comparable to t e one group pretest-posttest design.

Group A X 1

Group B 1 X 2
`y Research Agenda

 So, w at type of researc approac do

you t ink I use

General Research Themes

 Geograp ic Information Systems (GIS)

and Location Intelligence
± Studies of t e use of GIS as a decision
support tool
±  e use of GIS in Businesses and
rganizations
± Location intelligence and t e use of
location in decision making
± Perceptions of space and geograp y
General Research Themes

 Studies of eams, Collaboration, and

tec nology
± Virtual eams
± eam History
± Individual C aracteristics
General Research Themes

 Virtual rorlds
±  e application of Vr to education and
learning
± Perceptions of avatars, space and location
in Vrs
± Legal, tax, and social issues in Vrs
± Communication and collaboration in Vrs
General Research Themes

 Mobile Commerce, Computing, and

Virtual eams
± Mobile Device Interfaces
± Impressions of Mobile Device Users
± Applications of Mobile Devices
General Research Themes

 Applications of Conjoint to IS Researc

± Human Resources
± Information Systems Analysis
± I Planning
General Research Themes

 I Adoption and Implementation

± User Acceptance of Mobile Devices
±  e Use of Mobile Devices in Commerce
General Research Themes

  e Application of I for raining and

Learning
±  e Application of ec nology in
Education
±  e Role of Communication ec nology in
Learning
A Recent Study

 Question: r at is t e impact of video

conference tec nology and training
met odology on student learning
 IV:
± raining Mode:
‡ Enactive Mastery
‡ Vicarious Experience
± Communication Media
‡ Face to Face
‡ Video Conferencing
Results






Œ
 
Results
ests f Bet ee - u je ts ffe ts

 t ril:  - tl
  III 
rc f rs f  r  i .
rrct l 
 Œ.Œ  . .Œ .
Itrc t .   .  .
.
I

I
 .  . Œ.  .
 
Œ.   Œ.  .  .

I

I 
.  
. .Π.

I I  .  . .
.

I * I I  .
  .
 .Π.

rrr Œ.  . 
tl . 
rrct tl 
Œ. Œ
.  r = .
 (st  r = .Œ)