Effect Sizes

The Effect Size
The effect size (ES) makes meta-analysis possible

The ES encodes the selected research findings on a
numeric scale
There are many different types of ES measures, each
suited to different research situations
Each ES type may also have multiple methods of
computation
Practical Meta-Analys
Examples of Different Types of

Effect Sizes
Standardized mean difference
Odds-ratio
Group contrast research

Treatment groups
Naturally occurring groups
Inherently continuous construct
Group contrast research
Treatment groups
Naturally occurring groups
Inherently dichotomous construct
Correlation coefficient
Association between variables research

Effect Sizes
Risk ratio
Group differences research (naturally occurring groups)

Commonly used by epidemiologist and medical meta-analyses
Inherently dichotomous construct
Easier to interpret than the odds-ratio

Effect Sizes
Proportion
Standardized gain score
Central tendency research

HIV/AIDS prevalence rates
Proportion of homeless persons found to be alcohol
abusers
Gain or change between two measurement points on the
same variable
Reading speed before and after a reading improvement
class
Others?
What Makes Something an Effect

Size
for
Meta-analytic
Purposes
The
type of ES must be comparable across the
collection of studies of interest

This is generally accomplished through standardization
Must be able to calculate a standard error for that type of
ES
The standard error is needed to calculate the ES weights, called

inverse variance weights (more on this latter)
All meta-analytic analyses are weighted
The Standardized Mean Difference
X X G2
ES G1
s pooled
s pooled
s12 n1 1 s22 n2 1
n1 n2 2
Represents a standardized group contrast on an

inherently continuous measure
Uses the pooled standard deviation (some situations use
control group standard deviation)
Commonly called d or occasionally g
The Correlation Coefficient
ES r
Represents the strength of association between two

inherently continuous measures
Generally reported directly as r (the Pearson product
moment coefficient)
The Odds-Ratio
The odds-ratio is based on a 2 by 2 contingency

table, such as the one below
Frequencies
Success
Failure
Treatment Group
Control Group
ad
ES
bc
The Odds-Ratio is the odds of success in the

treatment group relative to the odds of success in the
control group.
The Risk Ratio
The risk ratio is also based on data from a 2 by 2

contingency table, and is the ratio of the probability of
success (or failure) for each group
a / (a b )
ES
c / (c d )
Unstandardized Effect Size Metric
If you are synthesizing are research domain that using a

common measure across studies, you may wish to use
an effect size that is unstandardized, such as a simple
mean difference (e.g., dollars expended)
Multi-site evaluations or evaluation contracted by a
single granting agency
10
Effect Size Decision Tree for Group

Differences Research (Page 58 of
Book)
All dependent variables are
inherently dichotomous
Difference between proportions

as ES [see Note 1]
Group contrast on dependent variable
Odds ratio; Log of the odds ratio

as ES
All dependent variables are
inherently continuous
All dependent variables measured

on a continuous scale
All studies involve same

measure/scale
Unstandardized mean difference

ES
Studies use different

measures/scales
Standardized mean difference

ES
Some dependent variables measured

on a continuous scale, some
artificially dichotomized
Some dependent variables are

inherently dichotomous, some are
inherently continuous
Standardized mean difference ES;

those involving dichotomies
computed using probit [see Note 2]
or arcsine [see Note 3]
Do separate meta-analyses for
dichotomous and continuous
variables
11
Methods of Calculating the

Standardized Mean Difference
The standardized mean difference probably has more

methods of calculation than any other effect size type
See Appendix B of book for numerous formulas and
methods.
12
Degrees of Approximation to the ES

Value
Depending
of
Method
of
Direct calculation based on means and standard deviations
Computation
Algebraically equivalent formulas (t-test)
Great
Poor
Good
Exact probability value for a t-test

Approximations based on continuous data (correlation coefficient)
Estimates of the mean difference (adjusted means, regression B
weight, gain score means)
Estimates of the pooled standard deviation (gain score standard
deviation, one-way ANOVA with 3 or more groups, ANCOVA)
Approximations based on dichotomous data
13

Direction Calculation Method
ES
X1 X 2
X1 X 2
2
2
s pooled
s1 (n1 1) s2 (n2 1)
n1 n2 2
14

Algebraically Equivalent Formulas:
n1 n2
ES t
n1n2
ES
F (n1 n2 )
n1n2
independent t-test
two-group one-way ANOVA
exact p-values from a t-test or F-ratio can be converted

into t-value and the above formula applied
15

A study may report a grouped frequency distribution
from which you can calculate means and standard
deviations and apply to direct calculation method.
16

Close Approximation Based on Continuous Data -Point-Biserial Correlation. For example, the correlation
between treatment/no treatment and outcome measured
on a continuous scale.
ES
2r
1 r2
17

Estimates of the Numerator of ES -- The Mean

Difference
difference between gain scores

difference between covariance adjusted means
unstandardized regression coefficient for group membership
18

Estimates of the Denominator of ES -Pooled Standard Deviation
s pooled se n 1
standard error of the mean
19

s pooled
MSbetween
F
MSbetween
one-way ANOVA >2 groups

2
(
X
n
)
j j
2
X
n
j j
nj
k 1
20

s pooled
s gain
2(1 r )
standard deviation of gain

scores, where r is the correlation
between pretest and posttest
scores
21

s pooled
MS error df error 1
2
1 r
df error 2
ANCOVA, where r is the

correlation between the
covariate and the DV
22

s pooled
SS B SS AB SSW
df B df AB dfW
A two-way factorial ANOVA

where B is the irrelevant factor
and AB is the interaction
between the irrelevant factor
and group membership (factor
A).
23

Approximations Based on Dichotomous Data
ES probit ( p group1 ) probit ( p group 2 )

the difference between the probits transformation
of the proportion successful in each group
converts proportion into a z-value
24

3
log Odds ratio
ES
this represents the rescaling of the logged

odds-ratio (see Sanchez-Meca et al 2004
Psychological Methods article)
25

2
ES 2
N 2
ES
2r
1 r
chi-square must be based on

a 2 by 2 contingency table
(i.e., have only 1 df)
phi coefficient
26
27
28
Formulas for the Correlation

Coefficient
Results typically reported directly as a correlation

Any data for which you can calculate a standardized
mean difference effect size, you can also calculate a
correlation type effect size
See appendix B for formulas
29
Formulas for the Odds Ratio
Results typically reported in one of three forms:
Frequency of successes in each group

Proportion of successes in each group
2 by 2 contingency table
Appendix B provides formulas for each situation
30
Data to Code Along With the ES
The effect size
May want to code the data from which the ES is calculated

Confidence in ES calculation
Method of calculation
Any additional data needed for calculation of the inverse variance weight
Sample size
ES specific attrition
Construct measured
Point in time when variable measured
Reliability of measure
Type of statistical test used
31
Issues in Coding Effect Sizes
Which formula to use when data are available for

multiple formulas
Multiple documents/publications reporting the same data
(not always in agreement)
How much guessing should be allowed
sample size is important but may not be presented for both

groups
some numbers matter more than others
32

Effect Sizes

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Effect Sizes

Transféré par

Droits d'auteur :

Formats disponibles

The Effect Size

The effect size (ES) makes meta-analysis possible

Examples of Different Types of

Standardized mean difference

Group contrast research

Association between variables research

Examples of Different Types of

Group differences research (naturally occurring groups)

Examples of Different Types of

Standardized gain score

Central tendency research

What Makes Something an Effect

collection of studies of interest

The standard error is needed to calculate the ES weights, called

The Standardized Mean Difference

Represents a standardized group contrast on an

The Correlation Coefficient

Represents the strength of association between two

The odds-ratio is based on a 2 by 2 contingency

The Odds-Ratio is the odds of success in the

The Risk Ratio

The risk ratio is also based on data from a 2 by 2

Unstandardized Effect Size Metric

If you are synthesizing are research domain that using a

Effect Size Decision Tree for Group

Difference between proportions

Group contrast on dependent variable

Odds ratio; Log of the odds ratio

All dependent variables measured

All studies involve same

Unstandardized mean difference

Studies use different

Standardized mean difference

Some dependent variables measured

Some dependent variables are

Standardized mean difference ES;

Methods of Calculating the

The standardized mean difference probably has more

Degrees of Approximation to the ES

Exact probability value for a t-test

Methods of Calculating the

Methods of Calculating the

two-group one-way ANOVA

exact p-values from a t-test or F-ratio can be converted

Methods of Calculating the

Methods of Calculating the

Methods of Calculating the

Estimates of the Numerator of ES -- The Mean

difference between gain scores

Methods of Calculating the

standard error of the mean

Methods of Calculating the

one-way ANOVA >2 groups

Methods of Calculating the

standard deviation of gain

Methods of Calculating the

ANCOVA, where r is the

Methods of Calculating the

A two-way factorial ANOVA

Methods of Calculating the

ES probit ( p group1 ) probit ( p group 2 )

Methods of Calculating the

this represents the rescaling of the logged

Methods of Calculating the