CLINICAL APPRAISAL OF

JOURNALS ON THERAPY
Joy Grace G. Jerusalem MD, FPCS, FPSGS
 CLINICAL QUESTION
P - patient population with a certain disease
or condition

E - exposure/treatment/intervention

O - outcomes treatment intends to prevent

or promote
 EXAMPLE
Among children 1-5 years of age, how
effective is zinc supplementation compared to
placebo , in preventing acute diarrhea?
 RESEARCH QUESTION

• Usually two exposures

– Experimental treatment
– Control therapy
• Neutral control- placebo
• Active control- most effective treatment at the time
of investigation
 RESEARCH QUESTION

• Outcome
– Dichotomous- summarized in proportions
– Continuous- summarized as means
 APPRAISING DIRECTNESS

• Comparing research question with the

clinical question

• Very, narrow focused questions answered

in research
 APPRAISING DIRECTNESS

• Direct answers to clinical questions rare

– Use available information
– Caution in using indirect evidence

• Look for closer match if the article DOES

NOT provide a reasonably close answer to
the question you are asking
 APPRAISING VALIDITY

• Depends almost entirely on how fair the

comparisons are
• Absence of bias
• 8 questions to answer
 APPRAISING VALIDITY
1. WERE PATIENTS RANDOMLY ASSIGNED TO
TREATMENT GROUPS?

• Best technique to ensure true

comparability
• Equalize known and unknown variables
• Randomized clinical trial
 APPRAISING VALIDITY

• WORDS TO LOOK FOR

– Randomization
– Randomized
– Randomly allocated/assigned
 APPRAISING VALIDITY

• HOW TO ACHIEVE RANDOMIZATION

– Coin toss
– Table of random numbers
– Computer-generated random sequences
 APPRAISING VALIDITY
2. WAS ALLOCATION CONCEALED?

• Random order to which patients are

assigned to treatment groups is the
allocation sequence
• Allocation concealment strategies
eliminates selection bias
 APPRAISING VALIDITY
ALLOCATION CONCEALMENT STRATEGIES
• Sealed opaque envelopes
• Assign a third party not involved with the
study
• Computer allocation
 APPRAISING VALIDITY
WHY CONCEAL ALLOCATION?

• Caregivers can knowingly or unknowingly

disturb allocation sequence to get a
favorable result
 APPRAISING VALIDITY
• WHAT TO LOOK FOR
– Patient recruiter contacts a remote center to
obtain treatment assignments
– Use of sealed, numbered envelopes to assign
allocation
– Indistinguishable vehicles or packages
containing medications
 APPRAISING VALIDITY
3. WERE BASELINE CHARACTERISTICS
SIMILAR AT THE START OF THE TRIAL?
• Should be very similar
• Factor of successful randomization and
allocation concealment
• Comparisons are usually presented as a
table with columns for treatment and
control groups and sometimes, p-values
 APPRAISING VALIDITY
4. WERE PATIENTS BLINDED TO TREATMENT
ASSIGNMENT?
• Patients unaware of treatment modality
• Single blind
• Outcomes influenced by knowledge of
treatment modality
 APPRAISING VALIDITY
LOOK FOR
• Data with regards to preparation of
treatments- identical preparations
• Blinding may be difficult or even impossible
when the interventions involve diet,
educational maneuvers or surgical
procedures
 APPRAISING VALIDITY
5. WERE CAREGIVERS BLINDED TO
TREATMENT ASSIGNMENT?
• Affects way patients are followed up
• Caregivers unaware of treatment
modality
 APPRAISING VALIDITY
LOOK FOR

• Double-blind
• Use of identical preparations
 APPRAISING VALIDITY
6. WERE OUTCOME ASSESSORS BLINDED TO
TREATMENT ASSIGNMENT?
• Identify
– Patients
– Caregivers
– Study personnel responsible for deciding
response to therapy
 APPRAISING VALIDITY
LOOK FOR
• Identify assessors
• Statement saying outcome assessor is
blinded
• Clinical data was withheld
• Assessments were independent
 APPRAISING VALIDITY
7. WERE ALL PATIENTS ANALYZED IN THE
GROUPS TO WHICH THEY WERE
ORIGINALLY RANDOMIZED?
• Concerns subjects who do not comply or
continue treatment protocols
– analyzed in their original treatment group
(intention to treat analysis) or
– should they be excluded (censored analysis)
 APPRAISING VALIDITY
• REASONS WHY NON-COMPLIANT PATIENTS
SHOULD BE ANALYZED IN THEIR ORIGINAL
GROUPS
– Ability to comply considered part of the
performance of the therapy
– Removing them could disturb balance achieved
by random allocation
 APPRAISING VALIDITY
WHAT TO LOOK FOR
• Intention-to-treat analysis
• Per protocol analysis
• Check number of patients randomized
against number of patients analyzed
 APPRAISING VALIDITY
8. WAS FOLLOW-UP RATE ADEQUATE?
• Drop out- number of patients lost to
follow-up
• The greater the number the more validity
is threatened
 APPRAISING VALIDITY
WHEN DO YOU WORRY ABOUT DROP-OUTS?
Step 1: count number of patients with
bad outcomes in each treatment
group
Step 2: count number of drop-outs in
each treatment group
 APPRAISING VALIDITY
• WHEN DO YOU WORRY ABOUT DROP-OUTS?
Step 3: create worst case scenario for
treatment group
Step 4: create best scenario for
treatment group
Step 5: were conclusions of best and worst
scenarios significantly different
 APPRAISING VALIDITY
Certain study with N = 30 for each group, 6 drop-outs in the treatment
group and 7 in the control group

TREATMENT CONTROL
STEP 1 DEATHS = 5/24 DEATHS = 9/27
STEP 2 DROP-OUTS = 6 DROP-OUTS = 7
STEP 3 DEATHS = 5+6/24+6 = 11/30 DEATHS = 9+0/23+7 = 9/30
STEP 4 DEATHS = 5+0/24+6 = 5/30 DEATHS = 9+7/23+7 = 16/30
STEP 5 COMPARE RESULTS OF STEPS 3 AND 4
 APPRAISING VALIDITY

• Sensitivity analysis
• Drop-out insignificant: robust
• Drop-out significant: soft
 APPRAISING RESULTS
1. HOW LARGE WAS THE EFFECT OF
TREATMENT?
• Magnitude expressed by comparing
outcomes in both groups
– Continuous variables
– Dichotomous variables
 APPRAISING RESULTS
• Continuous results
– Mean difference

Mean difference = mean control - mean tx

 APPRAISING RESULTS
• Dichotomous results
– Relative risk (RR)
– Relative risk reduction (RRR)
– Absolute risk reduction (ARR)
– Number needed to treat (NNT)
 APPRAISING RESULTS
RISK
- probability of a bad outcome in people
with disease
- relate numbers with a baseline or control
 APPRAISING RESULTS
• Relative reductions are constant and
appear to be bigger
• ARR, RRR, RR are interpretations based on
‘harmful’ events
 APPRAISING RESULTS
RELATIVE RISK

RR = new risk RR (in decimals)

original risk <1.0 tx beneficial
= R(t) 1.0 no effect
R(c) >1.0 tx harmful
 APPRAISING RESULTS
ABSOLUTE RISK REDUCTION

ARR = risk change ARR (in percent)

>0% tx beneficial
= R(c)-R(t) 0% no effect
<0% tx harmful
 APPRAISING RESULTS
RELATIVE RISK REDUCTION

RRR = risk change RRR (in percent)

original risk >0% tx beneficial
0% no effect
= R(c)-R(t) <0% tx harmful
R(c)
 APPRAISING RESULTS
NUMBER NEEDED TO TREAT

NNT = 100
ARR
- number of patients needed to treat to
prevent one adverse outcome
- Measures effectiveness of treatment
 APPRAISING RESULTS
• Dichotomous results
– Use rates
• How fast events accumulate in both groups through
time (ex. Survival curves)
– Hazards ratio
• rate of outcomes of treatment divided by the rate
of outcomes in control
• This expression is similar to RR
 APPRAISING RESULTS
outcome Summary of Comparison of
result w/in @ results b/w 2 groups
group
dichotomous proportion RRR, ARR, RR

rate Hazard
ratio=rate(t)/rate(c)

continuous mean mean difference

 APPRAISING RESULTS
2. HOW PRECISE WAS THE ESTIMATE OF
THE TREATMENT EFFECT?
- ARR, RRR, RR not exact values
- Does not reflect uncertainty due to
sample size limitations
- Confidence interval
 APPRAISING RESULTS
• Confidence intervals
– Interval estimate which provides a range of
possible values of the treatment effect
– 95% level of confidence
– The narrower the interval, the more precise
 APPRAISING RESULTS
FOUR BASIC TENETS
1. When both ends of the CI are on the side
of benefit, the treatment is definitely
beneficial.
2. When both ends of the CI are on the side
of harm, the treatment is definitely
harmful.
 APPRAISING RESULTS
3. When one end reflects important benefit and
the other end reflects important harm, then the
study is inconclusive.
4. When one end reflects a small unimportant
benefit and the other end reflects a small
unimportant harm, then the two treatments are
equal.
 INTERPRETING 95% CI
Measure of Interpretation of point Tx better than control Tx worse than control Inconclusive The two are equivalent
effectiveness estimates

RRR 0%- Tx beneficial Both ends of 95% CI Both ends of 95% CI 95% CI straddles 0%; 95% CI straddles 0%;
~ 0%- Tx no effect >0% <0% either end is far from either end very close to
< 0%- Tx harmful 0% 0%

ARR 0%- Tx beneficial Both ends of 95% CI Both ends of 95% CI 95% CI wide; straddles 95% CI narrow;
~ 0%- Tx no effect >0% <0% 0% straddles 0%
< 0%- Tx harmful

RR <1.0 Tx beneficial Both ends of 95% CI Both ends of 95% CI 95% CI is wide; 95% CI narrow;
~ 1.0 Tx no effect <1.0 >1.0 straddles 1.0 straddles 1.0
> 1.0 Tx harmful
 APPRAISING RESULTS
• If confidence intervals not stated
– Use freely downloadable programs
– Get the help of a statistician
– Compute it yourself
• Control events
• Treatment events
• Total population
 ASSESSING APPLICABILITY
• Decide if the results can be applied to your
patients
• Issues affecting applicability
– Biologic
– Socioeconomic
 ASSESSING APPLICABILITY
BIOLOGIC ISSUES AFFECTING APPLICABILITY
• Sex
• Co-morbidities
• Race
• Age
• Pathology
 ASSESSING APPLICABILITY
SOCIOECONOMIC ISSUES AFFECTING
APPLICABILITY
• Patient compliance problems
• Provider compliance problems
 INDIVIDUALIZING THE RESULTS
• Individualize benefits, risks and costs
• Patient factors
• Disease factors
• Treatment factors
 STEPS TO ESTIMATE THE
INDIVIDUALIZED CHANGES IN RISK
1. Estimate your individual patient’s risk for
an event without treatment (Rc)
2. Estimate the RR using the study results
3. Estimate your individual patient’s risk for
an event with treatment (Rt)
4. Estimate the individualized ARR
5. Estimate the individualized NNT or NNH
 Thank you for your kind
attention!
• References
– Dans, A.L.; Dans, L.F., Silvestre, M.A.A.;
Painless Evidence-Based Medicine; John Wiley
and Sons Ltd., 2008
– Guyatt, G; Drummond, R.; Users’ Guides to the
Medical Literature: A Manual of Evidence-
Based Clinical Practice; American Medical
Association, 2002
 WORKSHOP
• Clinical scenario:
– A 3 yr old female consults at the ER for moderate
grade fever of 6 hours duration. PE showed Temp of
38.5, (+) Koplik’s spots. Rest of the PE is unremarkable.
She had exposure to measles a week ago. CXR is
normal. The most likely diagnosis is measles. The
mother is curious whether her daughter should be
given antibiotics to prevent the development of
pneumonia.
 ARTICLE
Prophylactic antibiotics to prevent
pneumonia and other complications after
measles: community based randomised
double-blind placebo controlled trial in
Guinea-Bissau; May-Lill Garly, Carlitos Balé,
Cesário Lourenco Martins, Hilton C Whittle,
Jens Nielsen, Ida M Lisse, Peter Aaby BMJ,
doi:10.1136/bmj.38989.684178.AE
(published 23 October 2006)
 DIRECTNESS
• How does the clinical CEO compare with
the research CEO?
 VALIDITY
1. Were patients randomly assigned to
treatment groups?
2. Was allocation concealed?
 VALIDITY
3. Were baseline characteristics similar at
the start of the trial?
 VALIDITY
4. Were patients blinded to treatment
assignment?
5. Were caregivers blinded to treatment
assignment?
6. Were outcome assessors blinded to
treatment assignment?
 VALIDITY
7. Were all patients analyzed in the groups
to which they were originally randomized?
 VALIDITY
8. Was follow-up rate adequate?
 DROP OUT RATES
(+) PNEUMONIA TOTAL
COTRI 1 46
PLACEBO 6 38
 BEST CASE SCENARIO
(+) PNEUMONIA TOTAL
COTRI 1+0 46 + 1
PLACEBO 6+2 38 + 2
 WORST CASE SCENARIO
(+) PNEUMONIA TOTAL
COTRI 1+1 46 + 1
PLACEBO 6+0 38 + 2
 APPRAISING RESULTS
How large was the treatment effect?
• Tendency to just look at the results and conclusion
• Clinical interpretation of results :
Is evidence abt Tx important?
Is the result impressive?
Will the giving of Tx be practical?
• Need to come up with useful clinical expressions of Tx effect
• Need to determine the size and magnitude of the potential
benefits of Tx
• Concepts of ARR, RR, RRR, NNT: clinically useful ms of
effects of Tx
 APPRAISING RESULTS
• The results for pneumonia frequency are stated in 1st paragraph of
section on “Treatment failure due to pneumonia or hospital
admission’ on page 4. The final adjusted Odds ratio w/ 95% CI, using
true ITT analysis, & adjusted for age group is stated in the last
sentence of said section: 0.20 (0.03 to 1.00)
• Based on table 3, for pneumonia, the Rt=2%, the Rc=16%
Relative risk= Rt/ Rc
= 0.02 / 0.16
= 0.125
• The risk of developing pneumonia is increased 0.125x if Cotri is given
• The risk of developing pneumonia is decreased 1/0.125x if Cotri is
given
• The risk of developing pneumonia is decreased 8x if Cotri is given
 APPRAISING RESULTS
• Relative risk reduction = (1-RR) x 100
= 1 – 0.125
= 0.875 x 100
= 87.5%
• The risk of pneumonia is decreased by
87.5% if Cotri is given relative to placebo
 APPRAISING RESULTS
• Absolute risk reduction = Rc – Rt
= 0.16 – 0.02
= 0.14
• The absolute risk of pneumonia is
decreased by 0.14 if Cotri is given
• ARR vs RRR: Clinicians more likely to
accept Tx when RRR is reported and reject
Tx when ARR is reported
 APPRAISING RESULTS
• The Number Needed to Treat (NNT) as stated in the study
as stated on page 4 is 7 (4-48).
• Interpret:
– 1. You need to treat 7 pxs w/ Cotri to prevent 1 from
developing pneumonia.
– 2. Result is imprecise as seen from wide CI of the NNT.
• There is a potential benefit based on trends but since this
is not statistically significant due to low sample size.
 APPRAISING RESULTS
How precise was the estimate of the
treatment effect?
• Based on OR= 0.20 (0.03 to 1.00):
• Interpretation:
– 1. The odds of developing pneumonia w/ measles is decreased 0.2 x if
Cotri is given.
• The odds of developing pneumonia w/ measles is decreased 1/ 0.2 x if Cotri is given.
• The odds of developing pneumonia w/ measles is decreased 5 x if Cotri is given.
– 2. The result is not significant since the confidence interval crosses 1.
– 3. The result is not precise as seen from wide CI, indicative probably of
inadequate sample size.
 APPLICATION
• How can I apply the results to patient care?
– Were the study patients similar to the patient in my
practice?
• Yes. Patient in clinical scenario fulfills the
inclusion/exclusion criteria of the study..

– Were all clinically important outcomes considered?

• Yes. Efficacy & safety outcomes were considered.