3.

02 Critical Appraisal: Article on Therapy

Dr. Sison || September 2014 RESEARCH
Transcribers: Francisco, A., Francisco, E., Fudotan, Gabon,
Editors: Gomez, Arellano

OBJECTIVES USERS’ GUIDES FOR ARTICLE ON THERAPY (RCT)

 Understand the principles of critical
appraisal in therapeutics  Validity
 Be able to appraise published research and judge its reliability o Assignments of treatments to patient must be
 Be able to assess the relevance of published research to your randomized
PRACTICE
o Follow up must be complete
LEGEND
o Patients analyzed in the groups should be
Remember Previous randomized
Lecturer Book Trans Comm o Patients, health workers, and study personnel
(Exams) Trans
should be “blind” to treatment
    
o Groups must be similar at the start of the trial
o Groups should be treated equally, aside from the
experimental intervention
EVIDENCE BASED MEDICINE (EBM)  Results
o Observe how large the treatment effect was
 Process of systematically finding, appraising and using o Determine how precise the estimate of the
research findings as the basis for clinical decision treatment effect was
 Applicability
o Results should be applied to the patient care
o Clinically important outcomes should considered
o The likely treatment benefits should be worth the
potential harms & costs

TYPES OF BIAS

Selection bias

 Errors in choice of subjects &/or subject assignments to

study groups
 Due to differences in characteristics of the subjects being
compared
o Differences in age, severity of disease, etc. between
The 5 A’s of EBM the groups being compared
 Patients’ assignment to treatments were not randomized
1. Ask a clinical question
2. Acquire the best evidence “Missing data” bias
3. Appraise the evidence
4. Apply the evidence
 Attrition bias: Losses to follow-up = Dropouts
5. Assess your performance
 Missing records
 Absence of data leads to imbalance in the outcomes being
CRITICAL APPRAISAL
compared
 “Critical appraisal is the assessment of evidence by  Failure of subjects to follow up may be related to the
systematically reviewing its relevance, validity and results to outcome being studied
specific situations.” –Chambers, R (1998)
Information Bias
Why Do Critical Appraisal?

 Published research is not always reliable  Improper data collection

 Only an estimated 2% is judged clinically relevant  Errors in gathering of data, exposures & outcomes
 There are a lot of reading materials in the web which are not  Use of Subjective exposures/outcomes
peer reviewed like WIKIPEDIA  Differences in the way subjects of 1 group are treated
 Want to give best possible treatment to our patients compared to the other group
o Patients nowadays are well informed  Patients, health workers, and study personnel were not
o Explain the limitations of certain researches for the “blind” to treatment
patient to believe what we say
 There are many conflicting research results, many different AVOIDING BIAS
kind of studies, and a lot of commissioned studies by drug
companies. We do critical appraisal to determine which Avoiding Selection Bias
results to accept as the truth.
 To determine of Selection Bias is present:
o Randomized?
o Look at “proverbial table 1”
o If there are differences in baseline characteristics,
statistical adjustments should have been done
Page 1 of 5
[RES] 3.02 Critical Appraisal: Article on Therapy – Dr.Sison

 Randomization
o “Random allocation”
o Each subject is given an equal chance of being
assigned to either experimental or control group
o It removes investigator bias in the allocation of
subjects
o It produces study groups comparable w/ respect to
known as well as unknown variables.

 Alternate methods to ensure similar baseline characteristics:

o Restriction
 Narrow down inclusion and exclusion criteria
o Matching
 May produce additional selection bias
o Stratified/ Subgroup Analysis (Data analysis)
o Combination of the above

 Experimental grp & Control grp should have SIMILAR

characteristics at baseline EXCEPT for the
Intervention/Exposure being studied.
Example:
Table 2. The data shows that the Mean tumor size is not
comparable having a p value less than 0.05, what you can do are
the following: a.) Stratified analysis or b.) Multivariate analysis
 Stratified analysis
 Calculate the outcomes for the factor



Multivariate analysis
 Statistically significant factors are combined and
analyzed for assocation w/ outcome

 Use the Proverbial Table 1 in order to check for p-values. P-

values will dictate whether your groups have similar baseline
characteristics or not

Avoiding Missing Data Bias

 All subjects entered into the study should be accounted for.

(No dropouts, losses to follow-up, missing records)
 Follow up should be adequate and complete
 Inadequate follow-up less subjects develop outcome
overestimation of “favorable” outcomes
 To ensure adequate follow-up time:
o All patients should be followed up to the end of the risk
period of developing the outcome
 Example:
- Early Breast cancer survival: at least 10 yrs
- Post op wound infection: at least 30 days
o If not, estimation of outcome should be done using
statistical tests
Table 1. Proverbial table 1  Ex. Kaplan Meier life table/survival curves

 If p value is > 0.05 = there is no significant  Reasons for Losses to Follow-up / Drop-outs
difference between values, therefore the o Too ill
characteristics being observed in the control and o Already cured
experimental groups are comparable (similar) o Migrated
 If p value is < 0.05 = there is a significant difference  Solutions for Losses to Follow-up / Drop-outs
between values, therefore the characteristics being o Assume WORST CASE SCENARIO
observed are not comparable

Page 2 of 5
[RES] 3.02 Critical Appraisal: Article on Therapy – Dr.Sison

 When is missing data significant? Effects of Non – blinding

o “ 5 and 20” rule of thumb  Administering tx
 Fewer than 5% loss probably leads to little o Treatment bias
bias while greater than 20% loss threatens  Investigators favors 1 group ->
validity overestimate of effect
 >20% = significant drop out rate  Subject (Patient)
o Missing/Drop-out rate > Diff in outcome rates be the o Observation bias
groups  Tend to act in a way that will please
o IF Dropout rate is SIGNIFICANT: Do WORST CASE investigator
SCENARIO (WCS) o Placebo effect
 Tend to feel good with any form of tx
 WORST CASE SCENARIO  Measuring response
o Assume that the worst outcome happened to o Same as treatment bias
dropouts (D.O.) in the experimental group & the
best outcome happened to the drop outs (D.O.) in Blinding
the control  Placebo
o Example: o Indistinguishable from active tx in appearance,
taste, texture but lacking the active ingredient
o Automatically results in double blinding
o Considered BEST way of blinding

When blinding is not possible:

 Surgical procedures
o MRM vs. BCT (Subject and MD cannot be
blinded)
 Solution
o Use of objective exposure and outcome
measures
 Survival, mortality: (NOT prone to
misinterpretation)
o Use of validated scoring systems
 Cosmetic appearance: Use cosmesis
score
 ain: use of Pain score
Figure 2. In the given example, the worst (added to deaths)  SSI: use of objective criteria for dx
were assumed to the D.O. in the experimental group (Chemotx
+) while the best (added to survived) were assumed to the Exposure and Outcome criteria
control (chemotx -) group  Objective
o Less prone to errors in interpretation
 In the example, the survival rates differ from one o Easily measured
another: o Examples:
o Uncorrected table: surv rate A > surv rate B  Exposures(biopsy, culture studies)
o WCS table: surv rate A < surv rate B  Outcomes (survival, death)
o This difference means that the number of drop  Subjective
outs affected the outcome of the study o Require some judgment
o Therefore, we can say that the conclusion of o Often difficult to measure
the study is NOT valid since the number of o Examples:
dropouts have significantly altered the results  Exposures (rashes and pain)
 However, if for example, the survival rates are the same  Outcomes (quality of life and disability)
such that:
o Uncorrected table: surv rate A > surv rate B LOOKING AT THE RESULTS
o WCS table: surv rate A > surv rate B
Rule: DATA ANALYSIS basically compares the outcome in
o We can say that the conclusion of the study is
the experimental group vs. the outcome in the control group
VALID because the number of dropouts have
not significantly altered the results

Avoiding Information Bias

 Experimental group and control group should be treated

and followed up with equal intensity.
Blinding – a process wherein the researcher makes sure that
neither the subject, investigator or data collector knows of the
subject’s assigned group.
Page 3 of 5
WHAT ARE THE RESULTS?
 What is the result in experimental group?
 What is the result in the control group?
 How large is the difference in result
 Is it statistically significant?
 How precise is it?
[RES] 3.02 Critical Appraisal: Article on Therapy – Dr.Sison

 ARR = outcome in control group – outcome in experimental

group
o If ARR > 0, tx is harmful
o If ARR < 0, tx is beneficial
 Outcome rate (I+) experimental group: a/a+b o If ARR = 0, tx has no effect
 Outcome rate (I-) control group: c/c+d
If type of outcome is harmful
RISK RATIO (mortality, recurrence, complication rate)
 RR = outcome in experimental group / outcome in control
Relative Risk (RR): group
o (I+) / (I-) o If numerator > denominator, RR > 1, tx is
harmful.
o If numerator < denominator, RR < 1, tx is
beneficial.
o If numerator = denominator, RR = 1, tx is
 useless.
 ARR = outcome in control group – outcome in experimental
Relative Risk Reduction (RRR) or Relative Risk Increase group
(RRI)
o If ARR > 0, tx is beneficial
o If ARR < 0, tx is harmful
o If ARR = 0, tx has no effect

LOOKING AT THE APPLICABILITY OF THE RESULTS

RISK DIFFERENCE
Relevance
 Define the population studied
o Inclusion and exclusion criteria
 Endpoints
o Clinical relevance
o Primary vs. secondary
o Sub groups analysis
 Balance of beneficial outcomes against side-effects

Are the results reliable?

 Confidence Interval (CI): > or = 95%
 P-value: < or = 0.05

Is the difference “clinically significant”?

Example: Does prophylactic antibiotic prevent the development of  Clinical relevance is different from statistical significance.
pneumonia in children with measles? o The result may be statistically significant but not
clinically significant
Results may be statistically significant but NOT clinically
significant to change present practice

Are the likely treatments benefits worth the potential harm

 Risk of Pneu w/ Ab: and costs?
o A/ A+B  10/100 = 0.10
 Risk of Pneu w/o Ab:
o C / C+D  20/100 = 0.20
 RR:
o (A/ A+B) / (C/ C+D)  0.10/0.20 = 0.50
 ARR = 0.20 – 0.10 = 0.10
 NNT = 1/0.10 = 10

INTERPRETING THE COMPARISONS

If type of outcome is beneficial

(Survival, tx success, cure rate)  NNT = 1 / ARR
 RR = outcome in experimental group / outcome in control o You need to treat # patients produce 1 benefit.
group o Low NNT: more beneficial
o If numerator > denominator, RR >1, tx is  NNH = NNT x % side effect (in decimal)
beneficial. o You need to harm # patients to produce 1
o If numerator < denominator, RR < 1, tx is benefit.
harmful. o Low NNH: more harmful
o If numerator = denominator, RR = 1, tx is  Cost/life saved = (unit cost x #doses x duration x NNT)
useless. o You need to spend # cost to produce 1 benefit
Page 4 of 5
[RES] 3.02 Critical Appraisal: Article on Therapy – Dr.Sison

 Cost – risk benefit

o Need to treat a number of persons to save 1 life
at a certain cost and produce an adverse effect
o Save a number of lives at a certain cost and at
the same time produce an adverse effect
 Example
o NNT = 14
o NNH = 7
o Cost / benefit = P100
o Therefore, you need 14 patients to save 1 life at
a cost of 100 pesos but you produce to adverse
effects

Where to search
 Dynamed
 Pubmed Clinical Queries
 Cochrane
 Clinical Evidence
 Sumsearch
 EBSCO CINAHL
 TRIP
 EBM Online
 CLINICAL RESEARCH CRITIQUES
 ACP Journal Club
 Bandolier
 Essential Evidence Plus
 Bestbets
 CAT Bank (Critically Appraised Topics)

 REMEMBER:

(I+) or Outcome rate (I+) experimental group: A/ A+B

(I-) or Outcome rate control group: C/ C+D

(RR) or Relative Risk: (A/ A+B) / (C/ C+D)

(RRR) or Relative Risk Reduction: (1-RR) x 100

(ARR) or Absolute Risk Reduction: (C/ C+D) – (A/ A+B)

(NNT) or Number needed to treat: 1/ARR

(NNH) or Number needed to harm: NNT x % side effect in decimal

Cost per life saved: (Unit cost) x (# doses) x (duration) x (NNT)

MINI Quiz
1. What is Process of systematically finding, appraising and
using research findings as the basis for clinical decision?
2. What are errors in choice of subjects &/or subject
assignments to study groups?
3. These are errors in gathering of data, exposures &
outcomes
4. T or F. In the worst case scenario, the worst is assumed
to the drop outs in the control group and the best is
assumed in the experimental group.
5. Results that are statistically significant are always
clinically significant

Answers :
1.EBM, 2.selection bias, 3.information bias, 4.F, 5.F
Page 5 of 5