Causal Design and Marketing

Experiments
Chapter 4
Causal Design and Marketing
Experiments
1. The Quest of Causality
2. Experimentation
3. Quasi-Experimentation
4. Managerial Aspects of Experimentation and
Quasi Experimentation
5. Four Experimental Design Procedures
!"#$"#%$!" '('
%
1 The Quest of Causality
I know quite certainly that I have no special gift.
Curiosity, obsession, and dogged endurance have
brought me my ideas. (Albert Einstein)
The new advertising campaign caused 10% increased
in the sales.
The new sales training program has resulted in lower
sales force turnover.
Are there other possible factors or events that could
have led to the observed changes?
Rigorously and honestly answering the question lies at
the heart of investigations of causality and forms a
major part of conclusive marketing research.
!"#$"#%$!" '('
)
1 The Quest of Causality
Managers and researchers must be able to
discern the conditions under which proper
causal statements can be formulated and
claimed to hold.
In marketing, not all conditions allow for
tight, accurate causal statements.
!"#$"#%$!" '('
"
1 The Quest of Causality
Deterministic Causation: The ordinary concept
of causality that presumes that an effect always
follows a cause; differs from the scientific concept
of probabilistic causation conceptualizing effects
in terms of their statistical probability.
Probabilistic Causation: The notion, common in
the philosophy and science, that research can
never truly prove causality, only infer with some
degree of confidence.
!"#$"#%$!" '('
*
1 The Quest of Causality
Necessary Conditions for Causality
The scientific concept is different and complex
as held by a common person. There is
difference between the so-called common
sense concepts of causality and scientific
concept.
The commonsense view hold that a single
event (the cause) always results in another
event (the effect).
!"#$"#%$!" '('
+
1 The Quest of Causality
In the commonsense causality the effect always
follows whenever the cause take place. We refer
to this as deterministic causation (X causes Y).
In contrast scientific notion specifies the effect
only as being probable. This is called probabilistic
causation (we can infer but we cannot prove this).
Statisticians also state that there is always a
chance of error because data is never perfect (e.g.,
biases, coding errors, missing values, or entire
variable). Such phenomena is even true in hard
sciences.
!"#$"#%$!" '('
,
1 The Quest of Causality
Marketing effects are caused by multiple forces
acting at once and, at best, we can only infer a
causal relationship; we can never demonstrate it
definitively.
Under the following three conditions researchers
can claim to have made causal inferences, and all
should be met.
1. Concomitant Variation
2. Time Occurrence of Variables
3. Elimination of other Possible Causal Factors
!"#$"#%$!" '('
-
1 The Quest of Causality
Concomitant Variation is the extent to which a
cause, X, and an effect, Y, occur together or vary
together in the way predicted by a hypothesis
under consideration.
For example if a company does advertising to
improve the attitude of people that would result
in rising sales. We can say that there is a
concomitant variation between the attitude and
sales (improved attitude causes the sales to
increase)
!"#$"#%$!" '('
.
1 The Quest of Causality
There can some issues here to understand the
probabilistic causation.
1. Reverse Causation (people becoming more
experienced with the cars)
2. Omitted Variable (the company may have
improved the quality of the cars)
3. Insufficient Variation (advertising may have
increased the in-store visits )

!"#$"#%$!" '('
!$
1 The Quest of Causality
Time Occurrence of Variables is that the one
event cannot cause another event if it occurs after
the other event. The causing event must occur
either before or simultaneously with the effect.
We can also do some research about the changing
attitude of people towards the company before
and after and if there is an improvement then we
can say that hypothesis would be tenable.
And if sales only improved after the car purchase
then hypothesis is untenable.
!"#$"#%$!" '('
!!
1 The Quest of Causality
Elimination of other Possible Causal Factors should
be done as there can be a case where other possible
causes were not systematically eliminated, and the
research design did not allow for the identification of
the true causal relationship.
Consider a case where a scientist wants to check the
effect of artificial sweetener on sugar level of a person
with a usual soft drink. She infers that there is
causality present and artificial sweetener is not
effective. Is the scientist right at this conclusion.
!"#$"#%$!" '('
!%
2 Experimentation
A. Basic Definitions in Experimentation
1) Experiments
2) Treatments
3) Test Units
4) Dependent Variables
5) Extraneous Variables (Types : History, Maturation, Testing,
Instrumentation, Statistical Regression, Selection Bias, Test Unit
Mortality)
6) Experimental Design
7) Validity in Experimentation
B. Rigorously Defining Experiments X-O-R Syntax
C. Three Pre-Experimental Design (One-Shot Case Study, One Group
Pre-Test-Post-Test Design, Static Group Comparison)
D. Three True-Experimental Design (Pre-Test-Post-Test Control
Group Design)
!"#$"#%$!" '('
!)
2 Experimentation
Experimentation is the fundamental tool to
help identify causal relationships.
The objective of an experiment is to measure
the effect of explanatory (independent)
variables on another (dependent) variable of
interest while controlling for other variables
that might confuse the ability to make causal
inferences.
!"#$"#%$!" '('
!"
2 Experimentation
Experiment: A rigorous investigation in which the
researcher controls several (independent) variables and
measures or observe their effects on one or more dependent
variables.
Treatment: In experimental design, the manipulated
independent variable.
Test Unit: In experimental design, the entity to whom (or
to which) treatments are presented and whose responses to
the treatments are measured.
Extraneous variables: Any variable, other than specific to
the treatment administered to test units, that may affect the
response of the test units to the treatments, including
history effect, maturation effect, testing effect,
instrumentation effect, statistical regression effect, selection
bias, and test unit mortality; also referred to as confound.
!"#$"#%$!" '('
!*
2 Experimentation
Confounding Variable (Confound): Any variable other than those
specific to the treatment, that may effect how test units respond to
the treatments; makes it difficult to attribute effects to variables
controlled or manipulated by the experimenter.
Experimental Design: A form of investigation in which the
researcher can directly control one of more (independent) variables
to study their effects on other (dependent) variable.
Internal validity: A way to evaluate experiments by verifying that
changes in the dependent or criterion variable truly arose from
changes in the independent or treatment variables alone.
External Validity: A method of evaluating the results of
experiments; formally, the degree to which conclusions would hold
under other, presumably identical circumstances.
History Effect: The occurrence of events outside of, but taking
place at the same time as, the experiment that can affect the
dependent variable.

!"#$"#%$!" '('
!+
2 Experimentation
Maturation Effect: An effect similar to the history effect except that it
pertains to the changes in the experimental units themselves over time.
Testing Effect: An effect that can come about when the pretest itself exerts
an influence on how participants perform on the post-test.
Direct (main) Testing Effect: Occurs when the first of two observations
affects the second.
Interactive (reactive) Testing Effect: In an experiment, the effect of the
test units pretreatment measurement on the reaction to the treatment.
Instrumentation Effect: In statistical models, an effect that arises when
instruments, observers, or scorers change over the course of an
experiment.
Statistical Regression Effect: When, in an experiment, test units are
selected (for exposure to the treatment) based on an extreme pretreatment
score; usually considered poor research practice.
Test Unit Mortality: A serious problem that occurs when test units
withdraw from an experiment before their role is complete.
!"#$"#%$!" '('
!,
2 Experimentation
One-shot Case Study Design: A simple form of pre-experimental design
where a single group of test units is first exposed to a treatment and a
measurement is then taken on the dependent variables without random
assignment of test units to the treatment group.
One Group Pre-test-post-test Design: A simple form of pre-experimental
design, equivalent to a one-shot case study design but where an additional
pre-test measurement is taken before the treatment.
Static Group Comparison Design: An experimental design in which two
treatment groups, one that has been exposed to the treatment and one that
has not, are observed after the treatment has been presented.
Control Group: A group that is comparable to the treatment group in
terms of measurable characteristics but did not receive the treatment.
Baseline: A figure that can be used to provide context for a comparison
with another figure.
!"#$"#%$!" '('
!-
2 Experimentation
Generally experiments help to answer following question
types:
a) Can we increase profits by servicing small accounts
through mail rather than walk-ins.
b) Can we increase sales by acquiring more shelf space.
c) Will the addition of some chemical (stannous fluoride) to
our toothpaste reduce user cavities.
d) Does the frequency of a sales call in a specific time period
help increase sales of a specific account.
e) What is better color or black and white in a print ad.
f) What is the optimal promotional techniques.
g) Is it necessary to change the attitude of subject to increase
sales.
!"#$"#%$!" '('
!.
2 Experimentation
An Experiment is carried out when one of more
independent variables is deliberately
manipulated or controlled by an experimenter in
a planned fashion and the effects on the
dependent variable (or variables) is measured.
In surveys or observational studies there is no
manipulation of independent variables by the
researcher. This is the fundamental difference
between experimental and non-experimental
research.
!"#$"#%$!" '('
%$
2 Experimentation
In searching for causal relationships in non-
experimental situation the researcher must
proceed in reverse: observe the effect and then
search for a cause.
In this situation one can never be sure of the
proper time order of occurrence of variables
and effects of other possible independent
variables that have been excluded from
consideration.
!"#$"#%$!" '('
%!
2 Experimentation
Treatments are the manipulated alternatives or
independent variables whose effects are then
measured.
Examples in marketing are plentiful including
product design, packaging or name, advertising
themes, price levels, distribution strategies, and
promotional incentives.
Treatments can be a nominal scale based which
means that ordinal, interval, and ratio are also
acceptable.
!"#$"#%$!" '('
%%
2 Experimentation
For example if we are interested in sales
response the treatment can range from
different ad designs (nominal) to various
promotional price-reduction percentages.
Whatever the data type, researcher should
clearly define treatments to be used in the
research.
!"#$"#%$!" '('
%)
2 Experimentation
The Test Units are the entities to whom (or to
which) the treatments are presented and whose
response to the treatments is measured.
It is common in marketing for both people and
physical locations such as stores, geographical
areas to be used as test units.
For example people (test units) may be asked to
try a product (the treatment) and then have their
attitudes and opinions towards it measured
(dependent variables).
!"#$"#%$!" '('
%"
2 Experimentation
Alternatively, different aisle display
(treatments) may be set up in supermarkets
(test units) and sales levels (dependent
variables) cab be measured.
Conceptualizing a marketing experiment in
these terms (test units, treatments, and
dependent measures) during the planning
phase helps avoid conceptual and analytical
complications later.
!"#$"#%$!" '('
%*
2 Experimentation
The Dependent Variables are the measures taken
on the test units.
Typical marketing examples are sales, preference,
awareness, willingness to purchase, attitudes,
and a host of related concepts.
It is desirable for dependent variable to be
measured at interval level to allow for ease of
analysis, so that regression analysis can be used.
!"#$"#%$!" '('
%+
2 Experimentation
The Extraneous Variables are all variables
other than the treatments that potentially
affect the response of the test units to the
treatments. These variables can distort the
dependent variable measures in such a way as
to weaken or or entirely invalidate a
researchers ability to make causal inferences,
they are also called nuisance factors
!"#$"#%$!" '('
%,
2 Experimentation
For example, a book publisher attempting to
measure the responses of buyers to two different
cover designs would want to keep other aspects
of the book the same for each buyer group. If the
publisher allowed extraneous variables like price,
book dimensions, title, or paper quality to vary
between buyer groups, she could not be sure that
she was measuring the effects of cover.
Those other variables are said to confound the
experiment and are often referred to as confounds.
!"#$"#%$!" '('
%-
2 Experimentation
Researcher has three possible courses of action
with respect to extraneous variables.
1. When practicable, an extraneous variable can be
controlled, price or dimensions of book can be held
constant.
2. If physical control is not possible, the assignment of
treatments to test units may be randomized. The
book publisher could randomly different prices to all
buyers. In experiments with human test units, this
usually takes the form of randomly assigning the test
units to different treatments.
!"#$"#%$!" '('
%.
2 Experimentation
In this way it is hoped that extraneous factors that
could plausibly affect the outcome (such as
education, age, or product experience) are equally
represented in each treatment group. In marketing
research we can hardly control human behavior
and we have to rely on randomization. Control
and randomization are not either-or. If we like to
test a book cover with group of kids and another
group of adults, we have to randomize to get the
valid inference.
!"#$"#%$!" '('
)$
2 Experimentation
3. The third way to control the effects of extraneous
variables is through the use of specific
experimental design (on following slides) that
accomplish this purpose.
If physical control, randomization, and design
features do not eliminate the differential
effects of extraneous variables among
treatment groups, the experiment has been
confounded, and no causal statements are
possible.
!"#$"#%$!" '('
)!
2 Experimentation
As mentioned earlier we call such an
extraneous variable a confounding variable or
simply a confound.
For example, suppose we are using two cities
as our test units and it rains in one city and
not in the other. If rain affects the dependent
variable (say, the no. of car washes), the
experiment has been confounded and rain
was the confounding variable.
!"#$"#%$!" '('
)%
2 Experimentation
An Experimental Design involves the
specification of treatments to be manipulated,
test units to be used, dependent variables to
be measured, and procedures for dealing with
extraneous variables.
In Validity in Experimentation two concepts
of validity are relevant in experimentation:
internal validity and external validity.
!"#$"#%$!" '('
))
2 Experimentation
Internal Validity is the basic minimum validity
that must be present in an experiment before any
conclusion about treatment effects can be made.
It relates to whether the observed effects on the
test units could have been caused by variables
other than the treatment, that is, by extraneous
variables.
Without internal validity the experiment is
confounded and largely useless for purpose of
causal inference.
!"#$"#%$!" '('
)"
2 Experimentation
External Validity is concerned with the extent to
which the experimental results can be generalized
beyond the experiment.
To what populations, geographic areas, treatment
variables, and measurement variables can the
measured effect be protected.
For example student researcher often rely on other
students or stores near campus. Even if their
experiments are otherwise flawless, it would be
premature to claim that results extend outside the
realm of their fellow students or vicinity.
!"#$"#%$!" '('
)*
2 Experimentation
Ideally an experimental design should be
strong on both types of validity; unfortunately
one type must often be traded against the
other.
For example, an advertiser may ask
respondents to view advertisements in a lab or
rented space in a mall. Can any effect in such
environments be generalized to home viewing
environments?
!"#$"#%$!" '('
)+
2 Experimentation
B. Rigorously Defining Experiments
For researchers to collaborate and understand
the results of one anothers experiments, a
common language is required.
To facilitate our discussion of specific
experimental designs, we use a set of symbols
common across social science in general and
marketing research applications in particular,
referred to as X-O-R Syntax.
!"#$"#%$!" '('
),
2 Experimentation
X represents the exposure of a test group to an
experimental treatment, the effects of which are
to be determined. This is often referred to as a
treatment.
O refers to process of observation or
measurement of the dependent variable on test
units (i.e., within the test group).
R indicates that individuals have been assigned
at random to separate the treatment groups or
that groups themselves have been allocated at
random to separate treatments.
!"#$"#%$!" '('
)-
2 Experimentation
Types of Extraneous Variables
Extraneous variables needed to be controlled
to ensure that the experiment has not been
confounded or experiment is internally valid.
History Effect refers to the occurrence of
specific events that are external to the
experience but that take place during the
course of the experiment.
!"#$"#%$!" '('
).
2 Experimentation
For example, consider a design
O
1
X
1
O
2
Where O
1
and O
2
measures the dollar sales of
personnel and X
1
represents a new sales training
program. The difference O
1
- O
2
is the
measurement of the treatment effect.
However, there can be some other reasons for it;
improvement in general business conditions.
The greater the length of time, the greater the
chance of history effect confounding an
experiment of this type.
!"#$"#%$!" '('
"$
2 Experimentation
Maturation Effect is similar to history except
that it is concerned with changes in the
experimental units themselves that occur with
the passage of time.
For example, getting older, getting hungrier,
developing fatigue, and under going many
types of learning or experience.
!"#$"#%$!" '('
"!
2 Experimentation
Testing Effect is concerned with the possible
effects on the experiment of taking a measure on
the dependent variable before presentation on the
treatment. There are two kinds of testing effect.
Direct or Main Testing Effect, it occurs when the first
observation affects the second observation. For
example if we ask respondents to fill the same
questionnaire before and after the treatment, they may
respond differently as now they are expert with it, or
just to express variety of opinion. The internal validity
of the experiment is compromised.

!"#$"#%$!" '('
"%
2 Experimentation
Reactive or Interactive Testing Effect is a situation
test units permanent measurement affects the reaction
to the treatment. It influences external validity.
For example a pretreatment questionnaire that asks
questions about shampoo may sensitize the
respondent to the shampoo market and distort the
awareness levels of a new introduction (the
treatment). Hence, the measured effect cannot be
generalized to other potential customers who were not
similarly sensitized. There should not be even subtle cues
from the researcher or from the environment to the
respondents.
!"#$"#%$!" '('
")
2 Experimentation
Instruments Effect refers to changes in the
calibration of the measuring instrument used
or changes in the observers or scorers.
If sales were measured in a different store or
geographic location after the study, or if the
accounting calculated net sales differently, the
difference O
1
-O
2
could be explained by this
change in instrumentation.
!"#$"#%$!" '('
""
2 Experimentation
Statistical Regression Effects occur where test units have
been selected for exposure to the treatment on the basis of
an extreme pre-treatment score.
For example, suppose that in the sales training example
only poorly performing salespeople had been given the
new training program. Subsequent sales increases might be
attributed to the regression effect, because random
occurrences such as weather, family problems, or luck
helped define good or poor performance of the salespeople
in the pretreatment measures. Subsequent random
occurrences will make some of the poor performers do
better the following year, thus confounding the experiment.
!"#$"#%$!" '('
"*
2 Experimentation
Selection Bias refers to the assigning of test units
to treatment groups in such a way that the
groups differ on the dependent variable before
the presentation of the treatments.
Test units should be randomly assigned to
treatment groups.
If test units self-select their own groups or are
assigned to groups on the basis of researcher
judgment, the possibility of selection bias exists.
!"#$"#%$!" '('
"+
2 Experimentation
Test Unit Mortality refers to test units
withdrawing from experiment while it is in
progress.
!"#$"#%$!" '('
",
2 Experimentation
Three Pre-Experimental Designs questions
the internal experimental validity.
These can be used in the exploratory research
but should be avoided in the actual
experiment.
!"#$"#%$!" '('
"-
2 Experimentation
One Shot Case Study design is presented
symbolically as follows:
X
1
O
1
A single group of test units is first exposed to
treatment X
1
, and then a measurement is taken on
the dependent variable.
The test units were self selected or selected by the
experimenter using some criterion, hopefully
reflecting representativeness, and not
convenience sample.
!"#$"#%$!" '('
".
2 Experimentation
For example, a manager requests volunteer to take part
in a new sales training program, and a measure of their
new sales performance is taken sometime after the
program is completed.
The impossibility of drawing meaningful conclusions
from such design should be apparent.
The level of O
1
is the result of many uncontrolled
factors. Thus, history, maturation, selection, and
mortality problems all render this design internally
invalid.
!"#$"#%$!" '('
*$
2 Experimentation
One Group Pre-Test-Post-Test-Design is
represented symbolically as follows.
O
1
X
1
O
2
There is a pre-test measurement of sales
performance to the one-shot study design and
it provides a baseline O
1
.
!"#$"#%$!" '('
*!
2 Experimentation
There can be problems like
a) maturation (team getting experienced through time),
b) Testing (pre-test could have affected performance),
c) Instrumentation (prices of goods could have changed),
d) Selection Bias (test units could have self-selected),
e) Mortality (some test could have dropped out), and
f) Regression (test units may have selected at a
particularly bad year).
Even if the design is
R O
1
X
1
O
2
we would be able to get rid of selection bias
!"#$"#%$!" '('
*%
2 Experimentation
Static Group Comparison Design uses two
treatment groups. One that has been exposed
to the treatment and the other one has not.
Both groups are observed only after the
treatment has been presented, and test units
are not randomly assigned to the groups.
Symbolically, the design is as follows
!"#$"#%$!" '('
*)
2 Experimentation
Group 1: X
1
O
1


Group 2: O
2
Group 2 is called a control group because it has
not received the treatment and so may serve as a
baseline for comparison.
In marketing we often define the control group
treatment as the current level of marketing
activity. This can be shown symbolically
Experimental Group 1: X
1
O
1
Control Group 2: X
2
O
2
!"#$"#%$!" '('
*"
2 Experimentation
Here X
2
is the baseline marketing program
with which we wish to compare X
1
.
For example, in trying the new sales training
program on some people while keeping the
old one intact for some of them.
There are few problems here also e.g. test
units have been selected randomly, mortality
issues.
!"#$"#%$!" '('
**
2 Experimentation
Three True Experimental Designs
A true experimental design is one where
researcher is able to eliminate all extraneous
variables as competitive hypotheses to the
treatment, at least in theory.
!"#$"#%$!" '('
*+
2 Experimentation
Pre-test-Post-test Control Group Design is
presented symbolically as follows.
Experimental Group: R O
1
X
1
O
2


Control Group: R O
3
O
4
Here, X
1
is the treatment of interest (if there is a
baseline then there could be a possible X
2
.
!"#$"#%$!" '('
*,
2 Experimentation
Random assignment of the groups eliminates selection
bias as a potential confounding variable.
The premise here is that all extraneous variables
operate equally on the both the experimental group and
the control group. The only difference between the
groups is the presentation of the treatment to the
experimental group.
Therefore, O
2
O
1
is the sum of the treatment effect
and the effects of the extraneous variables, whereas O
4

O
3
accounts for the extraneous variables only.
!"#$"#%$!" '('
*-
2 Experimentation
In symbols, O
2
- O
1
= EXT+TE
O
3
- O
4
= EXT
Where TE is the treatment effect and EXT is
the sum of all extraneous effect (history,
maturation, testing, instrumentation,
regression, test unit mortality).
(O
2
- O
1
) (O
4
O
3
) = TE
We have found out the effect of independent
variable.
!"#$"#%$!" '('
*.
2 Experimentation
All threats to the internal validity are
addressed in this design. But there is a lacking
of not mentioning the extraneous variables
and their effects are assumed to be common.
If the extraneous variable is acting different on
one of the group then it would be
confounding.
!"#$"#%$!" '('
+$
3 Quasi Experimentation
In designing a research, the researcher often
creates artificial environments to have control
over independent and extraneous variables.
In this case, there would be higher internal
validity and less external validity.
One response to this problem is the quasi-
experimental design.
!"#$"#%$!" '('
+!
3 Quasi Experimentation
To confer internal validity the researcher tries
to control for extraneous and independent
variables, but this would raise the questions
regarding external validity.
Control over when and to whom of
measurement but lacks control of when and to
whom of exposure.
!"#$"#%$!" '('
+%
3 Quasi Experimentation
Time Series Experiment is the periodic
measurement on the dependent variables for
some test units.
Consumer purchase panels providing periodic
activity (Os). A marketer may undertake new
advertising campaign (X).
Here the marketer has control over
advertising timing but not advertising
exposure.
!"#$"#%$!" '('
+)
3 Quasi Experimentation
A: the campaign has
both short and long run
positive effects.
B: there is a short-run
positive effect.
C: delayed but seems a
long run effect.
D, E, F: cannot be
inferred as changes
already occuring.
!"#$"#%$!" '('
+"
4 Managerial Aspects
Comparison with other procedures: Most the
techniques used in marketing are descriptive
ones (secondary data, observation, surveys,
panels and web log analysis and simulation)
Constraints regarding time, money and other
resources may limit the feasibility of causal
research.
Researcher can be right inferring the causal
relationship but not 100% sure.
!"#$"#%$!" '('
+*
4 Managerial Aspects
Laboratory vs. Field Experiments
Laboratory would be showing off TV
commercials to test units while field would
running comm. during actual TV programs.
There are issues of validity, accuracy, cost, control
variables, and others.
Laboratory Experiments would have lower
external validity, low cost, and require less time.
!"#$"#%$!" '('
++
4 Managerial Aspects
Limitations of Experimentation: It is not possible
to control for extraneous variables.
In field experiments lack of cooperation can be an
issue.
Lack of knowledge of a experiment procedures.
Experiments are costly, time consuming, and
requires large amount of data for expert analysis.
People would be test units most of the time and
care should be taken to avoid biasedness.
!"#$"#%$!" '('
+,
4 Managerial Aspects
Stages in Conducting an Experiment
1. State the problem
2. Formulate hypothesis (es).
3. Construct experimental design.
4. Be sure that design answers the question.
5. Analysis of data (before actually conducting the
experiments)
6. Perform the experiment.
7. Apply statistical procedures (whether effects are real or
there is noise)
8. Draw conclusion while paying attention to the internal
and external validity.
!"#$"#%$!" '('
+-