NCSS Statistical Software NCSS.

com

Chapter 536

Binary Diagnostic
Tests – Paired Samples
Introduction
An important task in diagnostic medicine is to measure the accuracy of two diagnostic tests. This can be done by
comparing summary measures of diagnostic accuracy such as sensitivity or specificity using a statistical test.
Often, you want to show that a new test is similar to another test, in which case you use an equivalence test. Or,
you may wish to show that a new diagnostic test is not inferior to the existing test, so you use a noninferiority test.
All of these hypothesis tests are available in this procedure for the important case when the diagnostic tests
provide a binary (yes or no) result.

Experimental Design
Suppose you are interested in comparing the sensitivities of two diagnostic tests for a particular disease (or
condition). Each test provides a binary (yes or no) response. Further suppose you draw a random sample of
subjects from the population with the disease and administered both diagnostic tests to each subject in random
order. Assume that Test 1 is a new (experimental or treatment) test that will replace Test 2, the existing (standard
or reference) test, if it is found to be better.
The results of such a study can be displayed in a 2-by-2 table in which the Test 1 result is shown as the rows and
the Test 2 result is shown as the columns.

Test 2 Result
Test 1 Result Positive Negative Total
Positive X11 X10 m1
Negative X01 X00 m0
Total n1 n0 N

Data such as this can be analyzed using standard techniques for comparing two correlated proportions which are
presented in the chapter on Two Correlated Proportions. Such a table was originally analyzed using McNemar’s
Test. However, procedures with better statistical properties have recently been proposed. See for example Nam
and Blackwelder (2002).

Sensitivity
Sensitivity is the proportion of those that have the condition for which the diagnostic test is positive. Since this
design assumes that the subjects come from the population of individuals with the disease, the sensitivity can be
calculated.

536-1
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

Specificity
Specificity is the proportion of those that do not have the condition for which the diagnostic test is negative. To
study specificity, a separate study would have to be conducted in which subjects were drawn from the population
of individuals without the disease. The data from a such a study could be analyzed with this procedure by
changing the meaning of positive and negative. Instead of positive meaning that the person had the disease,
positive would mean that the diagnostic test result matched the true condition of the subject. Likewise, negative
would mean that the diagnostic test result did not match the true condition. In the procedure printouts, you would
substitute specificity for sensitivity.

Comparing Sensitivity and Specificity

Suppose you arrange the results of two diagnostic tests into two 2-by-2 tables as follows:

Test 2 Result
Test 1 Result Positive Negative Total
Positive X11 X10 m1
Negative X01 X00 m0
Total n1 n0 N

Hence, the study design include N = N1 + N0 patients.

The hypotheses of interest when comparing the sensitivities (Se) of two diagnostic tests are either the difference
hypotheses
H O : Se1 − Se2 = 0 versus H A: Se1 − Se2 ≠ 0
or the ratio hypothesis
H O : Se1 / Se2 = 1 versus H A: Se1 / Se2 ≠ 1
Similar sets of hypotheses may be defined for the difference or ratio of the specificities (Sp) as
H O : Sp1 − Sp2 = 0 versus H A: Sp1 − Sp2 ≠ 0
and
H O : Sp1 / Sp2 = 1 versus H A: Sp1 / Sp2 ≠ 1
Note that the difference hypotheses usually require a smaller sample size for comparable statistical power, but the
ratio hypotheses may be more convenient.
The sensitivities are estimated as
m1 n1
Sˆe1 = and Sˆe2 =
N N
The sensitivities of the two diagnostic tests may be compared using either their differences or their ratios. Hence,
the comparison of the sensitivity reduces to the problem of comparing two correlated binomial proportions. The
formulas used for hypothesis testing and confidence intervals are the same as presented in the chapter on testing
two correlated proportions. We refer you to that chapter for further details.

536-2
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

Data Structure
This procedure does not use data from a dataset. Instead, you enter the values directly into the 2-by-2 table on the
panel.

Procedure Options
This section describes the options available in this procedure.

Data Tab
Enter the data values directly on this panel.

Data Values
X11
This is the number of patients that responded positively to both diagnostic tests. The value entered must be a non-
negative number.
X10
This is the number of patients that tested positive using Test 1, but negative using Test 2. The value entered must
be a non-negative number.
X01
This is the number of patients that tested negative using Test 1, but positive using Test 2. The value entered must
be a non-negative number.
X00
This is the number of patients that responded negatively to both diagnostic tests. The value entered must be a non-
negative number.

Confidence Interval Method

Difference C.I. Method
This option specifies the method used to calculate the confidence intervals of the sensitivity differences. These
methods are documented in detail in the Two Correlated Proportions chapter. We recommend the score method
proposed by Nam (1990).
Ratio C.I. Method
This option specifies the method used to calculate the confidence intervals of the sensitivity ratios. These methods
are documented in detail in the Two Correlated Proportions chapter. The recommended method is score method
proposed by Nam and Blackwelder (2002).

Report Options
Alpha – Confidence Intervals
The confidence coefficient to use for calculating the confidence limits in proportions. 100 x (1 - alpha)%
confidence limits will be calculated. This must be a value between 0 and 0.5. The most common choice is 0.05.

536-3
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

Alpha – Hypothesis Tests

This is the significance level of the hypothesis tests, including the equivalence and noninferiority tests. Typical
values are between 0.01 and 0.10. The most common choice is 0.05.
Proportion Decimals
The number of digits to the right of the decimal place to display when showing proportions on the reports.
Probability Decimals
The number of digits to the right of the decimal place to display when showing probabilities on the reports.

Equivalence or Non-Inferiority Settings

Max Equivalence Difference
This is the largest value of the difference between the two sensitivities that will still result in the conclusion of
equivalence. When running equivalence tests, this value is crucial since it defines the interval of equivalence.
Usually, this value is between 0.01 and 0.20.
Note that this value must be a positive number.
Max Equivalence Ratio
This is the largest value of the ratio of the two sensitivities that will still result in the conclusion of diagnostic
equivalence. When running equivalence tests, this value is crucial since it defines the interval of equivalence.
Usually, this value is between 1.05 and 2.0.
Note that this value must be greater than one.

Example 1 – Binary Diagnostic Test of Paired Samples

This section presents an example of how to enter data and run an analysis. In this example, a sample of 50
individuals known to have a certain disease was selected. For this study, Test 1 refers to a new, cheaper, less-
invasive diagnostic test and Test 2 refers to the standard diagnostic test that is currently being used. The results
are summarized into the following table:

Test 2 Result
Test 1 Result Positive Negative Total
Positive 31 5 36
Negative 4 10 14
Total 35 15 50

You may follow along here by making the appropriate entries or load the completed template Example 1 by
clicking on Open Example Template from the File menu of the Binary Diagnostic Tests – Paired Samples
window.

1 Open the Binary Diagnostic Tests – Paired Samples window.

• On the menus, select Analysis, then Proportions, then Binary Diagnostic Tests – Paired Samples. The
Binary Diagnostic Tests – Paired Samples procedure will be displayed.
• On the menus, select File, then New Template. This will fill the procedure with the default template.

536-4
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

2 Enter the data.

• Select the Data tab.
• In the X11 box, enter 31.
• In the X10 box, enter 5.
• In the X01 box, enter 4.
• In the X00 box, enter 10.

3 Set the other options.

• Set the Difference C.I. Method to Score (Nam RMLE).
• Set the Ratio C.I. Method to Score (Nam Blackwelder)
• Set the Max Equivalence Difference to 0.2.
• Set the Max Equivalence Ratio to 1.25.

4 Run the procedure.

• From the Run menu, select Run Procedure. Alternatively, just click the green Run button.

Data and Proportions

Counts Table Proportions

Test 1 (New) Test 2 (Standard) Result Test 2 (Standard) Result

Result Positive Negative Total Positive Negative Total
Positive 31 5 36 0.6200 0.1000 0.7200
Negative 4 10 14 0.0800 0.2000 0.2800
Total 35 15 50 0.7000 0.3000 1.0000

Row Proportions Column Proportions

Test 1 (New) Test 2 (Standard) Result Test 2 (Standard) Result

Result Positive Negative Total Positive Negative Total
Positive 0.8611 0.1389 1.0000 0.8857 0.3333 0.7200
Negative 0.2857 0.7143 1.0000 0.1143 0.6667 0.2800
Total 0.7000 0.3000 1.0000 1.0000 1.0000 1.0000

These reports display the counts that were entered along with various proportions that make interpreting the table
easier. Note that Test 1’s sensitivity of 0.7200 and Test 2’s sensitivity of 0.7000 are displayed in the margins of
the Table Proportions table.

Sensitivity Confidence Intervals

Lower 95.0% Upper 95.0%
Statistic Test Value Conf. Limit Conf. Limit
Sensitivity (Se1) 1 0.7200 0.5833 0.8253
Sensitivity (Se2) 2 0.7000 0.5625 0.8090
Difference (Se1-Se2) 0.0200 -0.1094 0.1511
Ratio (Se1/Se2) 1.0286 0.8524 1.2491

Notes:
Sensitivity: proportion of those that actually have the condition for which the diagnostic test is positive.
Difference confidence limits based on Nam's RMLE method.
Ratio confidence limits based on Blackwelder and Nam's method.

This report displays the sensitivity for each test as well as corresponding confidence interval. It also shows the
value and confidence interval for the difference and ratio of the sensitivity. Note that for a perfect diagnostic test,
the sensitivity would be one. Hence, the larger the values the better.
Note that the type of confidence interval for the difference and ratio is specified on the Data panel.

536-5
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

Confidence Intervals for the Odds Ratio

Lower 95.0% Upper 95.0%
Statistic Value Conf. Limit Conf. Limit
Exact Conditional Binomial 1.2500 0.2690 6.2995
Maximum Likelihood 1.2500 0.3357 4.6549

Notes:
Odds Ratio = Odds(True Condition = +) / Odds(True Condition = -)
where
Odds(Condition) = P(Positive Test | Condition) / P(Negative Test | Condition)

This report displays estimates of the odds ratio as well as its confidence interval.

Hypothesis Tests about Sensitivity Difference

Null Test Conclusion
Test Test Hypothesis Statistic Prob at the 5.0%
Name Sides (H0) Value Level Level
Nam 2 Se1-Se2=0 0.1111 0.7389 Cannot Reject H0
Nam Lower 1 Se1-Se2<=0 0.3333 0.3694 Cannot Reject H0
Nam Upper 1 Se1-Se2>=0 0.3333 0.6306 Cannot Reject H0

This report displays the results of hypothesis tests comparing the sensitivities of the two diagnostic tests using
Nam’s test. Note that for this test, identical test results are obtained from either the test of differences or test of
ratios.

Tests of Equivalence
Reject H0
Lower Upper and Conclude
90.0% 90.0% Lower Upper Equivalence
Prob Conf. Conf. Equiv. Equiv. at the 5.0%
Statistic Level Limit Limit Bound Bound Significance Level
Difference (Se1-Se2) 0.0051 -0.0859 0.1274 -0.2000 0.2000 Yes
Ratio (Se1/Se2) 0.0247 0.8833 1.2039 0.8000 1.2500 Yes

Notes:
Equivalence is concluded when the confidence limits fall completely inside the equivalence bounds.
Difference confidence limits based on Nam's RMLE method.
Ratio confidence limits based on Blackwelder and Nam's method.

This report displays the results of the equivalence tests of sensitivity, one based on the difference and the other
based on the ratio. Equivalence is concluded if the confidence limits are inside the equivalence bounds.
Prob Level
The probability level is the smallest value of alpha that would result in rejection of the null hypothesis. It is
interpreted as any other significance level. That is, reject the null hypothesis when this value is less than the
desired significance level.
Note that for many types of confidence limits, a closed form solution for this value does not exist and it must be
searched for.
Confidence Limits
These are the lower and upper confidence limits calculated using the method you specified. Note that for
equivalence tests, these intervals use twice the alpha. Hence, for a 5% equivalence test, the confidence coefficient
is 0.90, not 0.95.

536-6
© NCSS, LLC. All Rights Reserved.
NCSS Statistical Software NCSS.com
Binary Diagnostic Tests – Paired Samples

Lower and Upper Bounds

These are the equivalence bounds. Values of the difference (ratio) inside these bounds are defined as being
equivalent. Note that this value does not come from the data. Rather, you have to set it. These bounds are crucial
to the equivalence test and they should be chosen carefully.
Reject H0 and Conclude Equivalence at the 5% Significance Level
This column gives the result of the equivalence test at the stated level of significance. Note that when you reject
H0, you can conclude equivalence. However, when you do not reject H0, you cannot conclude nonequivalence.
Instead, you conclude that there was not enough evidence in the study to reject the null hypothesis.

Tests Showing the Sensitivity Non-inferiority of Test2 Compared to Test1

Reject H0
Lower Upper and Conclude
90.0% 90.0% Lower Upper Equivalence
Prob Conf. Conf. Equiv. Equiv. at the 5.0%
Statistic Level Limit Limit Bound Bound Significance Level
Diff. (Se1-Se2) 0.0051 -0.0859 0.1274 -0.2000 0.2000 Yes
Ratio (Se1/Se2) 0.0247 0.8833 1.2039 0.8000 1.2500 Yes

Notes:
H0: The Sensitivity of Test2 is inferior to Test1.
Ha: The Sensitivity of Test2 is noninferior to Test1.
The noninferiority of Test2 compared to Test1 is concluded when the upper c.l. < upper bound.
Difference confidence limits based on Nam's RMLE method.
Ratio confidence limits based on Blackwelder and Nam's method.

This report displays the results of two noninferiority tests of sensitivity, one based on the difference and the other
based on the ratio. Report definitions are identical with those above for equivalence.

536-7
© NCSS, LLC. All Rights Reserved.