Binary Outcome Tests Guide

Tests for Binary/Categorical
outcomes
Binary or categorical outcomes
(proportions)

Outcome
Variable
Are the observations correlated? Alternative to the chi-
square test if sparse
cells:
independent correlated
Binary or
categorical
(e.g.
fracture,
yes/no)
Chi-square test:
compares proportions between
more than two groups

Relative risks: odds ratios
or risk ratios

Logistic regression:
multivariate technique used
when outcome is binary; gives
multivariate-adjusted odds
ratios
McNemars chi-square test:
compares binary outcome between
correlated groups (e.g., before and
after)

Conditional logistic
regression: multivariate
regression technique for a binary
outcome when groups are
correlated (e.g., matched data)

GEE modeling: multivariate
correlated (e.g., repeated measures)

Fishers exact test: compares
proportions between independent
groups when there are sparse data
(some cells <5).

McNemars exact test:
correlated groups when there are
sparse data (some cells <5).

(proportions)

Outcome
Variable
cells:
Binary or
categorical
(e.g.
fracture,
yes/no)
Chi-square test:

or risk ratios (for 2x2 tables)

ratios
after)



(some cells <5).

Chi-square test
Probiotics group Placebo group p-value Adjusted OR(95% CI) p-value
Cumulative incidence at
12 months
12/33 (36.4%) 22/35 (62.9%) 0.029* 0.243(0.0750.792) 0.019
*Significant difference between the groups as determined by Pearson's chi-square test.
p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding,
and delivery by cesarean section.

Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind,
randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.
Table 3. Cumulative incidence of eczema at 12 months of age
From an RCT of probiotic supplementation during pregnancy to prevent
eczema in the infant:
Chi-square test
Statistical question: Does the proportion of infants with
eczema differ in the treatment and control groups?
What is the outcome variable? Eczema in the first
year of life (yes/no)
What type of variable is it? Binary
Are the observations correlated? No
Are groups being compared and, if so, how many?
Yes, two groups
Are any of the counts smaller than 5? No, smallest is
12 (probiotics group with eczema)
chi-square test or relative risks, or both
Chi-square test of
Independence
Chi-square test allows you to compare proportions between 2
or more groups (ANOVA for means; chi-square for
proportions).

Example 2
Asch, S.E. (1955). Opinions and social
pressure. Scientific American, 193, 31-
35.
The Experiment
A Subject volunteers to participate in a
visual perception study.
Everyone else in the room is actually a
conspirator in the study (unbeknownst
to the Subject).
The experimenter reveals a pair of
cards
The Task Cards
Standard line Comparison lines
A, B, and C
The Experiment
Everyone goes around the room and says
which comparison line (A, B, or C) is correct;
the true Subject always answers last after
hearing all the others answers.
The first few times, the 7 conspirators give
the correct answer.
Then, they start purposely giving the
(obviously) wrong answer.
75% of Subjects tested went along with the
groups consensus at least once.
Further Results
In a further experiment, group size
(number of conspirators) was altered
from 2-10.

Does the group size alter the proportion
of subjects who conform?
The Chi-Square test

Conformed?

Number of group members?

2

4

6

8

10

Yes

20

50

75

60

30

No

80

50

25

40

70

Apparently, conformity less likely when less or more group
members
20 + 50 + 75 + 60 + 30 = 235
conformed
out of 500 experiments.

Overall likelihood of conforming =
235/500 = .47
Expected frequencies if no
association between group
size and conformity

Conformed?

Number of group members?

2

4

6

8

10

Yes

47

47

47

47

47

No

53

53 53

53

53

Do observed and expected differ more
than expected due to chance?
Chi-Square test
=
expected
expected) - (observed
2
2
_
85
53
) 53 70 (
53
) 53 40 (
53
) 53 25 (
53
) 53 50 (
53
) 53 80 (

47
) 47 30 (
47
) 47 60 (
47
) 47 75 (
47
) 47 50 (
47
) 47 20 (

2 2 2 2 2
2 2 2 2 2
2
4
~
= _
Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4
Chi-Square test
=
expected
expected) - (observed
2
2
_
Rule of thumb: if the chi-square statistic is much greater than its degrees of freedom,
indicates statistical significance. Here 85>>4.
85
53
) 53 70 (
53
) 53 40 (
53
) 53 25 (
53
) 53 50 (
53
) 53 80 (

47
) 47 30 (
47
) 47 60 (
47
) 47 75 (
47
) 47 50 (
47
) 47 20 (

2 2 2 2 2
2 2 2 2 2
2
4
~
= _
Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4
Interpretation
Group size and conformity are not
independent, for at least some categories of
group size
The proportion who conform differs between
at least two categories of group size
Global test (like ANOVA) doesnt tell you
which categories of group size differ
Caveat
**When the sample size is very small in
any cell (<5), Fishers exact test is
used as an alternative to the chi-square
test.

Review Question 1
I divide my study population into smokers, ex-smokers,
and never-smokers; I want to compare years of
schooling (a normally distributed variable) between the
three groups. What test should I use?

a. Repeated-measures ANOVA.
b. One-way ANOVA.
c. Difference in proportions test.
d. Paired ttest.
e. Chi-square test.

Review Question 1
and never-smokers; I want to compare years of
schooling (a normally distributed variable) between the
three groups. What test should I use?

b. One-way ANOVA.
d. Paired ttest.
e. Chi-square test.

Review Question 2
and never-smokers; I want to compare the proportions
of each group that went to graduate school. What test
should I use?

b. One-way ANOVA.
d. Paired ttest.
e. Chi-square test.

Review Question 2
should I use?

b. One-way ANOVA.
d. Paired ttest.
e. Chi-square test.

Review Question 2
should I use?

b. One-way ANOVA.
d. Paired ttest.
e. Chi-square test.

(proportions)

Outcome
Variable
cells:
Binary or
categorical
(e.g.
fracture,
yes/no)

Chi-square test:


ratios
after)



(some cells <5).

Risk ratios and odds ratios
Probiotics group Placebo group p-value Adjusted OR(95% CI) p-value
Cumulative incidence at
12 months
12/33 (36.4%) 22/35 (62.9%) 0.029* 0.243(0.0750.792) 0.019
*Significant difference between the groups as determined by Pearson's chi-square test.
p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding,
and delivery by cesarean section.

Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind,
randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.
Table 3. Cumulative incidence of eczema at 12 months of age
From an RCT of probiotic supplementation during pregnancy to prevent
eczema in the infant:
Corresponding 2x2 table

Treatment Placebo

+

12

22

-

21

13

Treatment Group
Eczema
Statistical question: Does the proportion of infants with
eczema differ in the treatment and control groups?
What is the outcome variable? Eczema in the first
year of life (yes/no)
Yes, binary
Are any of the counts smaller than 5? No, smallest is
12 (probiotics group with eczema)
chi-square test or relative risks, or both
Odds vs. Risk (=probability)

If the risk is Then the odds
are
(50%)
(75%)
1/10 (10%)
1/100 (1%)
Note: An odds is always higher than its corresponding probability,
unless the probability is 100%.
1:1
3:1
1:9
1:99
Absolute risk difference in eczema
between treatment and placebo:
36.4%-62.9%=-26.5% (p=.029, chi-
square test).
Risk ratio:

Corresponding odds ratio:
58 . 0
% 9 . 62
% 4 . 36
=
34 . 0
%) 9 . 62 1 /( % 9 . 62
%) 4 . 36 1 /( % 4 . 36
=
There is a 26.5%
decrease in absolute risk,
a 42% decrease in relative
risk, and a 66% decrease
in relative odds.
Why do we ever use an odds
ratio??
We cannot calculate a risk ratio from a case-
control study (since we cannot calculate the
risk of developing the disease in either
exposure group).
The multivariate regression model for binary
outcomes (logistic regression) gives odds
ratios, not risk ratios.
The odds ratio is a good approximation of the
risk ratio when the disease/outcome is rare
(~<10% of the control group)
Interpretation of the odds
ratio:
The odds ratio will always be bigger
than the corresponding risk ratio if RR
>1 and smaller if RR <1 (the harmful or
protective effect always appears larger)
The magnitude of the inflation depends
on the prevalence of the disease.

The rare disease assumption
RR OR
E D P
E D P
E D P
E D P
E D P
E D P
= ~ =
) ~ / (
) / (
) ~ / (~
) ~ / (
) / (~
) / (
1
1
When a disease is rare:
P(~D) = 1 - P(D) ~ 1
The odds ratio vs. the risk ratio
1.0 (null)
Odds ratio
Risk ratio Risk ratio
Odds ratio
Odds ratio
Risk ratio
Risk ratio
Odds ratio
Rare Outcome
Common Outcome
1.0 (null)
When is the OR is a good
approximation of the RR?
General Rule of
Thumb:
OR is a good
approximation as long
as the probability of the
outcome in the
unexposed is less than
10%
(proportions)

Outcome
Variable
cells:
Binary or
categorical
(e.g.
patency,
revision)
Chi-square test:


ratios
after)



(some cells <5).


Recall

Split-face trial:
Researchers assigned 56 subjects to apply
SPF 85 sunscreen to one side of their faces
and SPF 50 to the other prior to engaging
in 5 hours of outdoor sports during mid-
day.
Sides of the face were randomly assigned;
subjects were blinded to SPF strength.
Outcome: sunburn
Russak JE et al. JAAD 2010; 62: 348-349.
Results:
Table I -- Dermatologist grading of sunburn after an average of 5 hours of
skiing/snowboarding (P = .03; Fishers exact test)

Sun protection factor Sunburned Not sunburned
85 1 55
50 8 48

The authors use Fishers exact test to compare 1/56 versus 8/56. But this
counts individuals twice and ignores the correlations in the data!
McNemars test
Statistical question: Is SPF 85 more effective than SPF
50 at preventing sunburn?
What is the outcome variable? Sunburn on half a
face (yes/no)
Are the observations correlated? Yes, split-face trial
Yes, two groups (SPF 85 and SPF 50)
Are any of the counts smaller than 5? Yes, smallest
is 0
McNemars test exact test (if bigger numbers, would
use McNemars chi-square test)
Correct analysis of data
Table 1. Correct presentation of the data from: Russak JE et
al. JAAD 2010; 62: 348-349. (P = .016; McNemars test).

SPF-50 side
SPF-85 side Sunburned Not sunburned
Sunburned 1 0
Not sunburned 7 48
Only the 7 discordant pairs provide useful information for the analysis!
McNemars exact test
There are 7 discordant pairs; under the null
hypothesis of no difference between
sunscreens, the chance that the sunburn
appears on the SPF 85 side is 50%.
In other words, we have a binomial
distribution with N=7 and p=.5.
Whats the probability of getting X=0 from a
binomial of N=7, p=.5?

Probability =

Two-sided probability =
0078 . 5 . 5 .
0 7
7
0
=
|
.
|
\
|
0156 . 0078 . 5 . 5 . 0078 . 5 . 5 .
7 0
7
7
0 7
7
0
= =
|
.
|
\
|
+ =
|
.
|
\
|
McNemars chi-square test
Basically the same as McNemars exact
test but approximates the binomial
distribution with a normal distribution
(works well as long as sample sizes in
each cell >=5)
(proportions)

Outcome
Variable
cells:
Binary or
categorical
(e.g.
patency,
revision)
Chi-square test:


ratios
McNemars test: compares
binary outcome between correlated
groups (e.g., before and after)



(some cells <5).


Recall: Political party and
drinking
Drinking by political affiliation
Recall: Political party and
alcohol
This association could be analyzed by a ttest
or a linear regression or also by logistic
regression:

Republican (yes/no) becomes the binary
outcome.
Alcohol (continuous) becomes the predictor.

Logistic regression
Statistical question: Does alcohol drinking
predict political party?
What is the outcome variable? Political party
Are groups being compared? No, our
independent variable is continuous
logistic regression
The logistic model
ln(p/1- p) = o + |
1
*X
Logit function
=log odds of the
outcome
The Logit Model (multivariate)
)... ( ) ( )
) ( 1
) (
ln(
2 2 1 1
X X
D P
D P
+ + =
o
Logit function (log odds)
Baseline odds
Linear function of
risk factors for
individual i:
|
1
x
1
+ |
2
x
2
+ |
3
x
3
+
|
4
x
4

Review question 7
If X=.50, what is the logit (=log odds) of X?

a. .50
b. 0
c. 1.0
d. 2.0
e. -.50

Review question 7
If X=.50, what is the logit (=log odds) of X?

a. .50
b. 0
c. 1.0
d. 2.0
e. -.50

Example: political party and
drinking
Model:
Log odds of being a Republican (outcome)=
Intercept+ Weekly drinks (predictor)

Fit the data in logistic regression using a
computer

Fitted logistic model:
Log Odds of being a Republican = -.09 -1.4* (d/wk)

Slope for
drinking can be
directly
translated into
an odds ratio:
25 . 0
4 . 1
=
e
Interpretation: every 1 drink more per week decreases your odds of
being a Republican by 75% (95% CI is 0.047 to 1.325; p=.10)
To get back to ORs
)... ( ) (
2 2 1 1
) ( 1
) (
disease of odds
X X
e
D P
D P
+ +
=
=
o
)... ( ) ( )
) ( 1
) (
ln(
2 2 1 1
X X
D P
D P
+ + =
o
Adjusted Odds Ratio
Interpretation
unexposed for the disease of odds
exposed for the disease of odds
= OR
) 1 ( ) 0 (
) 1 ( ) 1 (
smoking alcohol
smoking alcohol
e
e
| | o
| | o
+ +
+ +
=

= =
) 1 (
) 0 (
) 1 (
) 1 (
smoking
alcohol
smoking
alcohol
e e e
e e e
|
| o
|
| o
) 1 (
) 1 (
1
alcohol
alcohol
e
e
|
|
=
Adjusted odds ratio,
continuous predictor
unexposed for the disease of odds
exposed for the disease of odds
= OR
) 19 ( ) 1 ( ) 1 (
) 29 ( ) 1 ( ) 1 (
age smoking alcohol
age smoking alcohol
e
e
| | | o
| | | o
+ + +
+ + +
=

= =
) 19 ( ) 1 (
) 1 (
) 29 ( ) 1 (
) 1 (
age smoking
alcohol
age smoking
alcohol
e e e e
e e e e
| |
| o
| |
| o
) 10 (
) 19 (
) 29 (
age
age
age
e
e
e
|
|
|
=
Practical Interpretation
interest of factor risk
) (
rf
OR e
x
=
|
The odds of disease increase multiplicatively by e

for for every one-unit increase in the exposure,
controlling for other variables in the model.
Multivariate logistic regression
Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp 2225-2230.
Logistic regression
Statistical question: What factors are associated with
anosmia (and hyposmia)?
What are the outcome variables? anosmia vs.
normal olfaction (and hyosmia vs. normal)
Are groups being compared? We want to examine
multiple predictors at once, so we need multivariate
regression.
multivariate logistic regression
Multivariate logistic regression
Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp 2225-2230.
Interpretation:
being a smoker
increases your
odds of anosmia
by 658% after
adjusting for older
age, nasal
polyposis, asthma,
inferior turbinate
hypertrophy, and
septal deviation.
Logistic regression in cross-
sectional and cohort studies
Many cohort and cross-sectional studies report ORs
rather than RRs even though the data necessary to
calculate RRs are available. Why?
If you have a binary outcome and want to adjust for
confounders, you have to use logistic regression.
Logistic regression gives adjusted odds ratios, not risk ratios.
These odds ratios must be interpreted cautiously (as
increased odds, not risk) when the outcome is common.
When the outcome is common, authors should also report
unadjusted risk ratios and/or use a simple formula to
convert adjusted odds ratios back to adjusted risk ratios.
Example, wrinkle study
A cross-sectional study on risk factors for
wrinkles found that heavy smoking
significantly increases the risk of prominent
wrinkles.
Adjusted OR=3.92 (heavy smokers vs.
nonsmokers) calculated from logistic regression.
Interpretation: heavy smoking increases risk of
prominent wrinkles nearly 4-fold??
The prevalence of prominent wrinkles in non-
smokers is roughly 45%. So, its not possible to
have a 4-fold increase in risk (=180%)!

Raduan et al. J Eur Acad Dermatol Venereol. 2008 Jul 3.
Interpreting ORs when the
outcome is common
If the outcome has a 10% prevalence in the
unexposed/reference group*, the maximum possible
RR=10.0.
For 20% prevalence, the maximum possible RR=5.0
For 30% prevalence, the maximum possible RR=3.3.
For 40% prevalence, maximum possible RR=2.5.
For 50% prevalence, maximum possible RR=2.0.

*Authors should report the prevalence/risk of the outcome in the
unexposed/reference group, but they often dont. If this number is not given,
you can usually estimate it from other data in the paper (or, if its important
enough, email the authors).

Interpreting ORs when the
outcome is common
Formula from: Zhang J. What's the Relative Risk? A Method of Correcting the Odds
Ratio in Cohort Studies of Common Outcomes JAMA. 1998;280:1690-1691.
) ( ) 1 ( OR P P
OR
RR
o o
+
=
Where:
OR = odds ratio from logistic regression (e.g., 3.92)
P
0
= P(D/~E) = probability/prevalence of the outcome in the
unexposed/reference group (e.g. ~45%)
If data are from a cross-sectional or cohort study, then you can
convert ORs (from logistic regression) back to RRs with a simple
formula:
For wrinkle study
Zhang J. What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort
Studies of Common Outcomes JAMA. 1998;280:1690-1691.
69 . 1
) 92 . 3 45 (. ) 45 . 1 (
92 . 3
smokers non vs. smokers
=
+
=
RR
So, the risk (prevalence) of wrinkles is increased by
69%, not 292%.

Sleep and hypertension
study
OR
hypertension
= 5.12 for chronic insomniacs who sleep
5 hours per night vs. the reference (good sleep)
group.
OR
hypertension
= 3.53 for chronic insomiacs who sleep
5-6 hours per night vs. the reference group.
Interpretation: risk of hypertension is increased
500% and 350% in these groups?
No, ~25% of reference group has hypertension. Use
formula to find corresponding RRs = 2.5, 2.2
Correct interpretation: Hypertension is increased
150% and 120% in these groups.

-Sainani KL, Schmajuk G, Liu V. A Caution on Interpreting Odds Ratios. SLEEP, Vol. 32, No. 8, 2009 .
-Vgontzas AN, Liao D, Bixler EO, Chrousos GP, Vela-Bueno A. Insomnia with objective short sleep duration is
associated with a high risk for hypertension. Sleep 2009;32:491-7.
Review problem 8
In a cross-sectional study of heart disease in middle-aged
men and women, 10% of men in the sample had
prevalent heart disease compared with only 5% of
women. After adjusting for age in multivariate logistic
regression, the odds ratio for heart disease comparing
males to females was 1.1 (95% confidence interval:
0.801.42). What conclusion can you draw?

a. Being male increases your risk of heart disease.
b. Age is a confounder of the relationship between gender and heart
disease.
c. There is a statistically significant association between gender and
heart disease.
d. The study had insufficient power to detect an effect.

Review problem 8
In a cross-sectional study of heart disease in middle-aged
men and women, 10% of men in the sample had
prevalent heart disease compared with only 5% of
women. After adjusting for age in multivariate logistic
regression, the odds ratio for heart disease comparing
males to females was 1.1 (95% confidence interval:
0.801.42). What conclusion can you draw?

a. Being male increases your risk of heart disease.
b. Age is a confounder of the relationship between gender and
heart disease.
c. There is a statistically significant association between gender and
heart disease.
d. The study had insufficient power to detect an effect.

Review topic: Diagnostic
Testing and Screening Tests
Characteristics of a diagnostic test
Sensitivity= Probability that, if you truly
have the disease, the diagnostic test
will catch it.

Specificity=Probability that, if you truly do
not have the disease, the test will
register negative.
Calculating sensitivity and
specificity from a 2x2 table

+ -

+

a

b

-

c

d

Screening Test
Truly have disease
b a
a
+
=
Sensitivity
d c
d
+
=
Specificity
Among those with true
disease, how many test
positive?
Among those without the
disease, how many test
negative?
a+b
c+d
Hypothetical Example

+ -

+

9

1

-

109

881

Mammography
Breast cancer ( on biopsy)
Sensitivity=9/10=.90
10
990
Specificity= 881/990 =.89
1 false negatives out of 10
cases
109 false positives out of 990
Positive predictive value
The probability that if you test positive
for the disease, you actually have the
disease.
Depends on the characteristics of the
test (sensitivity, specificity) and the
prevalence of disease.
Calculating PPV and NPV from
a 2x2 table

+ -

+

a

b

-

c

d

Screening Test
Truly have disease
c a
a
+
=
PPV
d b
d
+
=
NPV
Among those who test
positive, how many truly have
the disease?
Among those who test
negative, how many truly do
not have the disease?
a+c b+d
Hypothetical Example

+ -

+

9

1

-

109

881

Mammography
PPV=9/118=7.6%
118 882
Prevalence of disease = 10/1000 =1%
NPV=881/882=99.9%
What if disease was twice as
prevalent in the population?

+ -

+

18

2

-

108

872

Mammography
sensitivity=18/20=.90
20
980
specificity=872/980=.89
Sensitivity and specificity are characteristics of the test, so they dont
change!
What if disease was more
prevalent?
PPV=18/126=14.3%
126 874
Prevalence of disease = 20/1000 =2%
NPV=872/874=99.8%

+ -

+

18

2

-

108

872

Mammography
Conclusions
Positive predictive value increases with
increasing prevalence of disease
Or if you change the diagnostic tests to
improve their accuracy.
Review question 9
In a group of patients presenting to the hospital casualty department
with abdominal pain, 30% of patients have acute appendicitis. 70% of
patients with appendicitis have a temperature greater than 37.5C; 40%
of patients without appendicitis have a temperature greater than 37.5C.

a. The sensitivity of temperature greater than 37.5C as a marker for
appendicitis is 21/49.
b. The specificity of temperature greater than 37.5C as a marker for
c. The positive predictive value of temperature greater than 37.5C as a
marker for appendicitis is 21/30.
d. The predictive value of the test will be the same in a different population.
e. The specificity of the test will depend upon the prevalence of appendicitis
in the population to which it is applied.

Review question 9
In a group of patients presenting to the hospital casualty department
with abdominal pain, 30% of patients have acute appendicitis. 70% of
patients with appendicitis have a temperature greater than 37.5C; 40%
of patients without appendicitis have a temperature greater than 37.5C.

a. The sensitivity of temperature greater than 37.5C as a marker for
b. The specificity of temperature greater than 37.5C as a marker
for appendicitis is 42/70.
c. The positive predictive value of temperature greater than 37.5C as a
marker for appendicitis is 21/30.
d. The predictive value of the test will be the same in a different population.
e. The specificity of the test will depend upon the prevalence of appendicitis
in the population to which it is applied.

Binary Outcome Tests Guide

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Binary Outcome Tests Guide

Transféré par

Droits d'auteur :

Formats disponibles

Tests for Binary/Categorical

Vous aimerez peut-être aussi