META

ANALYSIS
OVERVIEW

Objectives
Following the course, you will be able to:
!Define

Meta-analysis

!Select

Studies for a Meta-analysis

!Identify

different types of Models

!Calculate
!Interpret

Summary Effects

Results of a Meta-analysis

What does it mean?

QUANTITATIVE: NUMBERS
SYSTEMATIC: METHODICAL
COMBINING: PUTTING TOGETHER
PREVIOUS RESEARCH: WHATS ALREADY DONE
CONCLUSIONS: NEW KNOWLEDGE

The popularity of meta analyses

Number of Meta Analysis publica7ons are steadily

increasing since 1993. We graphed the counts of journal
ar7cles included meta analysis as publica7on type
from Pubmed, from years 1993 through 2004

Four Steps of Meta Analysis

!Iden7fy your studies
!Determine eligibility of studies
!Inclusion: which ones to keep
!Exclusion: which ones to throw out

!Abstract Data from the studies

!Analyze data in the studies sta7s7cally

Identify Studies: Overview

!Steps:
!Search

Your Personal Files First

!Search Electronic Databases
!Review Reference Lists of Articles
!Contact Experts and Researchers
!Retrieve New Articles
!Evaluate
!Set

Quality of the Studies

Up Eligibility Criteria

Searching Electronic Databases

!First,

Define a Search Strategy

!Limitations

of Databases

!incomplete and imperfect

!language problems
!problems

queries

with fugitive literature

!Publication

Bias is important

!What is publication bias

!How to deal with publication

bias

Evaluating Study Quality

!Define

Study Quality Criteria Early

!Set

Up A Good Scoring System

!Develop A Form for Assessment
!Calculate

Quality for each Study

!Use this for Sensitivity Analysis
!stratify

studies according to quality

Defining Eligibility of Studies

Select Eligible Studies Based On:
!Study Designs
!Years of Publication
!Language
!Choice among multiple articles
!Sample-size or follow-up issues
!Similarity of Exposure and/or Rx
!Completeness of information

Abstract Data - Review!

Steps:
!Identify Relevant Articles
!Sort out Eligible Articles
!Set up a Form for Abstraction
!Enter the Eligible Studies
!Use this as your database

Keep some, throw out others

! Cannot include all studies
! Keep the ones with

! high levels of evidence

! good quality

! check with QUOROM guidelines

! Usually, MA done with RCTs

! Case series, and case reports denitely out
! Selec7on problems are major problems
! read the ar7cle I sent

MA = Meta Analysis; RCT = Randomized

Controlled Trial

Figure 1. Flow of informa7on through the dierent phases of a systema7c review.

Moher D, LiberaH A, Tetzla J, Altman DG, et al. (2009) Preferred ReporHng Items for SystemaHc Reviews and Meta-Analyses: The PRISMA
Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097
hbp://www.plosmedicine.org/arHcle/info:doi/10.1371/journal.pmed.1000097

Figure 1. Flow of informa7on through the dierent phases of a systema7c review.

Moher D, LiberaH A, Tetzla J, Altman DG, et al. (2009) Preferred ReporHng Items for SystemaHc Reviews and Meta-Analyses: The PRISMA
Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097
hbp://www.plosmedicine.org/arHcle/info:doi/10.1371/journal.pmed.1000097

How to Abstract Data: Guidelines

!Create a spreadsheet (Excel)
!For each study, create the following columns:
! name of the study
! name of the author, year published
! number of par7cipants who received interven7on
! number of par7cipants who were in control arm
! number who developed outcomes in interven7on
! number who developed outcomes in control arm

Spreadsheet Data for Strepto Study

We got like 22 studies to do our meta analysis, after
all

We created seven columns

trial: trial identity code
trialname: name of trial
year: year of the study
pop1: study population
deaths1: deaths in study
pop0: control population
deaths0: deaths in control

Abstract Data - Review!

Steps:
!Identify Relevant Articles
!Sort out Eligible Articles
!Set up a Form for Abstraction
!Enter the Eligible Studies
!Use this as your database
Statistical Analysis is next...

Statistical Analysis - Overview

!Select

An Estimate of Effect
!Choose An Effects Measure
!Select An Effects Model
!For Each Model:
!Calculate
!Calculate
!Calculate

!Perform

Summary Effect Size

Confidence Intervals
Q-statistic for Homogeneity

Sensitivity Analysis

Selecting Estimate of Effect

!Choose

Only One Estimate

!For RCTS, choose the one with
!Once

!For

randomized always randomized

nonrandomized trials, choose:

!estimate adjusted only for age

!that, and for a known confounder
!the most adjusted estimate
!estimate presented in the abstract

Choosing An Effect Measure

!RCTs or Cohort Studies
!Rate

Difference between Treatment and

Control Groups
!Ratio of Disease Rates
!Case

Control Studies

!Odds

Ratio
!Rate Ratio

Treatment Eect and Eect sizes

! In MEDICINE eect size as TREATMENT EFFECT

Odds raHos, risk raHos or risk dierences

! In SOCIAL SCIENCES eect size simply as an EFFECT SIZE

Standardized mean dierences, correlaHons

BOTH terms, TREATMENT EFFECT and EFFECT SIZE can refer to
any of the above indices.

WHEN APPROPRIATE TO USE:

! EFFECT SIZE quanHfy the relaHonship between two

variables or dierence between two groups

! TREATMENT EFFECT quanHfy the impact of a

deliberate intervenHon,

THREE MAJOR CONSIDERATIONS IN THE

CHOICE OF AN EFFECT SIZE INDEX:
! Eect sizes from the dierent studies should be

comparable to one another in the sense that they

measure the same thing

! Eect size can be computed

! Eect size should have good technical properHes

GOAL OF META-ANALYSIS?
TO ESTIMATE THE OVERALL, OR COMBINED
EFFECT

NOMENCLATURE

STATISTICAL MODELS:
!Fixed Eect
!Random Eects

FIXED EFFECT MODEL

!TRUE EFFECT SIZE: (Unknown)

all studies in the meta analysis share a

common (true) eect size

all factors that could inuence the eect

size are the same in all studies; true eect size
is the same in all studies

FIXED EFFECT MODEL: TRUE EFFECT SIZE

Fixed Eect Model True eects and

sampling error

Performing a xed eect meta-analysis

!Start with the observed eects and esHmate

the populaHon eect

1.

Compute the weighted mean

a. Assign weight for each study: Wi = 1/Vy

Vy within study variance for study ( i )

Performing a xed eect meta analysis:

!

Weighted Mean (M)

M = Wi Yi / Wi
Sum of the products WiYi (eect size mulHplied by weight) divided by the
sum of weights

Performing a Fixed eect meta analysis

!The Variance of the summary eect is

esHmated as the reciprocal of the sum of

weights, or

VM = 1 / Wi

Performing a Fixed Eect Meta Analysis

!EsHmated standard error of the summary eect

is the square root of the variance:

SE M = VM

Performing xed eect meta analysis

95% Lower and upper limits for the summary
eect are esHmated as
LL M = M 1.96 x SE M
UL M = M + 1.96 x SE M

Performing a xed eect Meta analysis

Finally, Z Value to test the null hypothesis that
the common true eect is zero
Z = M / SEM

Performing a xed eect meta analysis

One tailed test the p value is given by:
p = 1 ( |Z|)
Where we choose + if the dierence is in the
expected direcHon, otherwise

Performing a Fixed Eect Meta analysis

For a 2 tailed test:
p = 2 [ 1 ( (|Z|))]
where (Z) is the standard normal cummulaHve
distribuHon. In Excel as the funcHon
=NORMSDIST(Z).

PERFORMING A RANDOM EFFECT META

ANALYSIS
ASSUMPTION: THAT THE TRUE EFFECTS ARE
NORMALLY DISYTRIBUTED

RANDOM EFFECTS: IMPACT OF SAMPLING

ERROR

RANDOM EFFECT MODEL

PERFORMING A RANDOM-EFFECTS META

ANALYSIS
!

START WITH THE OBSERVED EFFECTS AND TRY TO ESTIMATE THE

POPULATION EFFECT

ESTIMATE THE OVERALL MEAN (WEIGHTED MEAN), WHERE THE WEIGHT

ASSIGNED TO EACH STUDY IS THE INVERSE OF THAT STUDYS VARIANCE

STUDYS TOTAL VARIANCE IS THE SUM OF WITHIN-STUDY VARIANCE AND T2

(BETWEEN STUDIES VARIANCE)

ESTIMATING TAU-SQUARED:
!

Method: Method of moments (or the DerSimonian and Laird)

Where:

EsHmaHng TAU-SQUARED:

ESTIMATING THE MEAN EFFECT SIZE:

(RANDOM EFFECT MODEL)

ESTIMATING THE MEAN EFFECT SIZE:

(RANDOM EFFECT MODEL)

ESTIMATING THE MEAN EFFECT SIZE:

(RANDOM EFFECT MODEL)

FIXED-EFFFECT VERSUS RANDOM-EFFECTS

MODELS

FIXED-EFFFECT VERSUS RANDOM-EFFECTS

MODELS

FIXED-EFFFECT VERSUS RANDOM-EFFECTS

MODELS
!

WHEN TO USE FIXED EFFECT:

1.

ALL THE STUDIES INCLUDED IN THE ANALYSIS ARE FUNCTIONALLY IDENTICAL

2.

THE GOAL IS TO COMPUTE THE COMMON EFFECT SIZE FOR THE IDENTIFIED
POPULATION, AND NOT TO GENERALIZE TO OTHER POPULATIONS

FIXED-EFFFECT VERSUS RANDOM-EFFECTS

MODELS
!

WHEN TO USE RANDOM EFFECT:

1. STUDIES PERFORMED BY RESEARCHERS OPERATING INDEPENDENTLY

2. WHEN THE GOAL IS TO GENERALIZE TO A RANGE OF SCENARIOS

Fixed Effects Model

!Methods:
!Mantel

Haenszel Method
!Petos Method
!General Variance Based Methods
!For Rate Difference
!For Rate Ratios
!When only RR and 95 CI given

!Tests

of Homogeneity

!Calculation

of Q Statistic

Mantel Haenszel Method

!Download

Spreasheet Calculator
!Strength of Mantel Haenszel
!Very

powerful
!Widely Used
!Limitation
!Cannot

Control For Confounding!

Petos Method
!Similar

to Mantel Haenszel

!Download

Calculator
!Simpler Computation
!No

Control for Confounding

!Good for RCTs
!Requires

2 X 2 Table

Variance-Based Methods
!Download Calculators For:
!Rate Difference
!For only Relative Risk and

CI
!Strengths and Limitations
!Good For Rate Differences
!Computationally Intensive

95%

Tests of Homogeneity
!Establish

Null Hypothesis that Effect

Sizes Are Equal in All of the Studies
[FAIL TO REJECT NULL]

!Tested

By Using Q-statistic

!Q-statistic

is distributed as chi-square
distribution with degree of freedom = n-1
where n = number of studies

Random Effects Model

!Download

The Calculator!

!Strengths

and Limitations:

!Can

Generalize the Conclusions

!Computationally Intensive

Continuous Outcomes
!Measurement
!Outcome

Measured in Same Scale

!Download

!Essentially
!Useful

Scale: Continuous

Spreadsheet Calculator!

Extension of ANOVA

For Integrating Social Science

Research Data

Sensitivity and Publication Bias

!Conduct

Sensitivity Analysis

!Stratify

studies by their quality rating

!Compare Fixed and Random Effects
!Deal

With Publication Bias

!Construct

A Funnel Plot

Funnel Plot: what and how to read

Plots the eect size against the
sample size of the study
To study a funnel plot, look at
its LOWER LEFT corner, thats
where nega7ve or null studies
are located
If EMPTY, this indicates
PUBLICATION BIAS
Note that here, the plot ts in
a funnel, and that the lee
corner is not all that empty,
but we cannot rule out
publica7on bias

Forest Plot

The dofed line passes

across null, or 1.0
The Risk Es7mate of each
study is lined up on each
side of the dofed line, with
95% CI spread as the line
The diamond in the below
is the summary es7mate
The two ends of the
diamond indicate 95% CI

The size of the black square box indicates weight of

the study
They call it a forest plot so that you dont miss the wood for the trees!

Conclusion

!Presentation

of Meta-analysis or
Literature Synthesis

!Download
!More

The Graph Maker

Resources...