Vaccine (2008) 26, 27232732

available at www.sciencedirect.com

journal homepage: www.elsevier.com/locate/vaccine

Use of continuous results to compare ELISAs for the detection of antibodies to non-structural proteins of foot-and-mouth disease virus
Aldo Dekker a,, Donal Sammin b, Matthias Greiner c, Ingrid Bergmann d, David Paton e, Santina Grazioli f, Kris de Clercq g, Emiliana Brocchi f
a

Central Veterinary Institute of Wageningen University and Research Centre, P.O. Box 65, 8200 AB Lelystad, The Netherlands Central Veterinary Research Laboratory, Backweston, Celbridge, Co. Kildare, Ireland c Federal Institute for Risk Assessment, P.O. Box 330013, D 14191 Berlin, Germany d Centro Panamericano de Fiebre Aftosa OPS/OMS (PANAFTOSA), Avenue President Kennedy 7778, ZC 20001-970 Rio de Janeiro, Brazil e Institute for Animal Health (IAH), Ash Road, Pirbright, Surrey GU24 0NF, UK f Istituto Zooprolattico Sperimentale della Lombardia e dellEmilia Romagna (IZSLER), Via Bianchi 7/9, 25124 Brescia, Italy g CODA-CERVA-VAR, Department of Virology, Epizootic Diseases Section, Groeselenberg 99, B-1180 Ukkel, Belgium
b

Received 21 December 2007; received in revised form 5 March 2008; accepted 12 March 2008 Available online 11 April 2008

KEYWORDS
Foot-and-mouth disease; DIVA test; ROC analysis; Likelihood ratio; Diagnostics; ELISA; NSP

Summary Six tests for detection of antibodies against the non-structural proteins of footand-mouth disease virus (FMDV) were compared at an international workshop in Brescia, Italy in 2004 on the basis of dichotomous test results. However, as results from all of these assays were also available on a continuous scale, validation was extended by calculating and subsequently analysing the receiver-operator characteristic (ROC) curves and likelihood ratios (LR) for each test method. For the purposes of these analyses, test results for a total of 1337 sera were selected from the Brescia workshop dataset, 237 sera that had been obtained from cattle exposed to FMDV and 1100 sera obtained from cattle that were not exposed to the virus; sera from exposed cattle were considered to be true positives and sera from nonexposed cattle were considered to be true negatives. Analysis of ROC curves showed that at specicities of both 99 and 99.5%, the IZS-Brescia and the Ceditest ELISA had signicantly better detection rates in exposed cattle than the other ELISAs. The ROC analysis conrms the previous nding that the IZS-Brescia and the Ceditest ELISAs have both better detection rates in exposed cattle combined with a high specicity. The analysis of likelihood ratios provides information

Corresponding author. Tel.: +31 320238603/8800; fax: +31 320238668. E-mail address: Aldo.Dekker@wur.nl (A. Dekker). URL: http://www.cvi.wur.nl (A. Dekker).

0264-410X/$ see front matter 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.vaccine.2008.03.052

2724

A. Dekker et al.
that may be very useful in the interpretation of test results, and a working example is presented to show how these likelihood ratios might be used in an objective approach to deciding the true infection status of surveyed populations. 2008 Elsevier Ltd. All rights reserved.

Introduction
Previous studies have shown that serum antibody specic for the non-structural protein 3ABC is a reliable marker of foot-and-mouth disease (FMD) virus replication in vaccinated cattle [1,2]. The NCPanaftosa ELISA and western blot technique developed by Bergmann et al. have been accepted as the OIE standard test system for differentiating the infected from vaccinated animals [3]. However, neither the NCPanaftosa nor the western blot were available in Europe and had not been validated for use in vaccinated European livestock. Therefore, it was decided to compare the NCPanaftosa ELISA (the OIE standard screening test) with ve ELISAs available in Europe using sera from European livestock in an international workshop setting. Initial comparative analyses based on dichotomous test results (i.e. where each test result was designated as either positive or negative) have already been reported [4]. However, as all six ELISAs generated results on a continuous scale, it is possible to compile further information about test performance by analysing these continuous test results. Reciever Operating Characteristic (ROC) analysis is one method often used for this purpose. ROC analysis originated from signal detection theory, as a means of modelling how well a receiver is able to detect a signal in the presence of noise. Its key feature is the distinction between two separate performance measures, the hit rate or true positive rate (the equivalent of the diagnostic sensitivity for a laboratory test method) and the false alarm rate or false positive rate (equivalent to [1 diagnostic specicity] for a laboratory test method). Using ROC analysis it is possible to compare the performance of ELISAs at each possible cut-off value. A validation study not only provides useful information on comparative performance of different tests but also provides essential information on the validity of a particular test result in any given population. The ELISA results, which are a series of OD measurements of varying magnitude, do not have to be interpreted simply as either positive or negative. Interpretation on a continuous scale allows a weighting to be placed on the magnitude of the test result on the assumption that the more positive (or the more negative) the test result, the more likely this result is to be a true reection of the status of the animal under test. Likelihood ratios (LR) may be used to facilitate interpretation of continuous results [5]. The LR of a positive test result is the ratio of the probability of a true positive test result and the probability of a false positive test result. An advantage in using LRs, is that several LRs may be combined in series (including LRs based on criteria other than laboratory test results) to establish the overall odds of an individual or a population having a target disorder (this is further exemplied in the working example). The present paper describes how six different ELISAs, previously evaluated on the basis of dichotomous results, were compared using ROC curves and LRs, both of which

were derived from results, on a continuous scale, available for each ELISA. It also considers how the LRs derived for one of these test methods might be used to assist in the interpretation of a positive test result.

Materials and methods

Sera
For this analysis we selected cattle sera and results from a large comparative study described previously [4], but we used only sera from non-exposed cattle obtained from countries known to be free of FMD as true negatives and only sera from exposed cattle obtained from experiments where animals were intentionally exposed to FMDV in an experimental setting as true positives. The true negatives consisted of a single serum sample from each of 1100 cattle, 425 of which had been vaccinated against FMD and 675 of which were not vaccinated. The true positives consisted of a single serum sample from each of 237 cattle sampled 21 days or more after exposure to FMDV (randomly selected from all of the data available, in the case of sequentially sampled animals), 173 of these cattle were vaccinated prior to exposure to the virus and 56 had never been vaccinated.

ELISAs
As described previously, each of the six ELISAs (NCPanaftosa screening ELISA produced by PANAFTOSA, Rio de Janeiro, Brazil; IZS-Brescia 3ABC trapping-ELISA produced by IZS, Brescia, Italy; Ceditest FMDV-NS produced by Prionics Lelystad B.V., The Netherlands; SVANOVIR FMDV-Ab ELISA produced by Svanova, Upsala, Sweden; CHEKIT FMD Antibody ELISA produced by IDEXX Laboratories, Westbrook, USA; UBI FMDV NS ELISA produced by United Biomedical Inc., New York, USA) were performed according to the manufacturers instructions [4]. The optical density (OD) value obtained for each test serum (in each ELISA) was compared with the OD value of a known positive control serum to give a percent positivity (PP) value (NCPanaftosa, IZS-Brescia, Svanovir and CHECKIT ELISA), a test/control ratio (UBI ELISA) or in the case of the Ceditest ELISA, a blocking ELISA, a percent inhibition (PI) value.

ROC analysis
For the ROC analysis we used a library developed for SPlus (Insightful Corporation, Seattle, USA) by Atkinson and Mahoney [6]. This library not only produced a graphical output of the result, but also facilitated a statistical comparison of the areas under the curves for the various ELISAs. Two separate ROC curves were produced for each ELISA, one based on the results obtained from vaccinated cattle (173 cattle exposed to FMDV and 425 cattle that were not exposed

To compare ELISAs detection of antibodies to non-structural proteins virus to FMDV) and the other for data from non-vaccinated cattle (56 cattle exposed to FMDV and 675 cattle not exposed to FMDV). Using the quantile function in S-Plus, cut-off values were determined at which diagnostic specicities of 97.5, 99 and 99.5% were obtained for each ELISA. At each of the aforementioned predened levels of diagnostic specicity the detection rate in exposed cattle of each ELISA was determined for both vaccinated and non-vaccinated cattle. Using the McNemars Chi Square test (S-Plus), statistical pair-wise comparisons of these observed detection rates were performed.

2725

whether exposed to FMDV or not), was taken as the response variable and the ELISA result as the explanatory variable [7]. The LR for a positive result was calculated for the various ELISAs according to the following formula: Likelihood ratio = e(X X ) In the formula above, is the estimate of the coefcient from the logistic regression model, X the observed result of the ELISA and X the result of the ELISA where the LR would have been 1. All statistical analyses were performed in S-Plus.

Likelihood ratio analysis

Likelihood ratio analysis was performed using logistic regression, where the exposure status of each animal (i.e.

Figure 1 Histogram of ELISA results obtained for cattle of different vaccination and infection status with the Ceditest ELISA. (A) Vaccinated cattle (173 cattle exposed and 425 cattle not exposed tot FMD virus) and (B) non-vaccinated cattle (56 cattle exposed and 675 cattle not exposed to FMD virus).

Figure 2 ROC curves based on test results obtained for each of the six ELISAs. (A) Vaccinated cattle, 173 exposed and 425 not exposed to FMD virus and (B) non-vaccinated cattle, 56 exposed and 675 not exposed to FMD virus.

2726

A. Dekker et al. in exposed vaccinated cattle, followed by the IZS-Brescia (68.8%) and Ceditest (68.6%) ELISAs whereas at both 99 and 99.5% specicity, differences in performance of the ELISAs were even more pronounced and the IZS-Brescia and Ceditest ELISAs had signicantly better detection rates in exposed vaccinated cattle than all other ELISAs. It should be noted that both the UBI and CHECKIT ELISAs, which had signicantly lower detection rates when used to test sera from vaccinated cattle, had comparable detection rates to all of the other ELISAs when applied to sera from non-vaccinated cattle (Table 2B). A logistic regression model [7] was used to calculate the LR for every possible ELISA result both for vaccinated cattle and for non-vaccinated cattle (Table 3). The LR curves for each ELISA together with their 95% condence intervals were calculated with data derived from vaccinated cattle (Fig. 3). In each case, a very sharp increase in LR can be seen above a certain test value, e.g. with the Ceditest ELISA the LR increases dramatically above 60% inhibition. Based on the parameters from the model (Table 3) the LRs were calculated at the cut-off values recommended by the manufacturers of each ELISA and at the cut-offs that were used for the pair-wise comparison of the ROC curves to provide for each of three given levels of specicity (Table 4). At the higher of the two cut-off values recommended for both the NCPanaftosa and CHECKIT ELISAs and at the single cut-off value recommended for each of the other ELISAs, LRs ranged between 0.5 and 3.8 for non-vaccinated cattle and between 1.6 and 5.9 for vaccinated cattle.

Results
The histogram illustrating the range of ELISA results obtained with the Ceditest, using the selected vaccinated and non-vaccinated cattle, shows a large overlap between the true positive and true negative population in vaccinated cattle, whereas in the non-vaccinated cattle this overlap is much smaller (Fig. 1). A similar pattern was apparent in histograms representing data from each of the other ELISAs (data not shown). From the ROC curves generated using results from vaccinated cattle (Fig. 2A) it is clear that there are differences between the ELISAs with respect to the detection rate in exposed cattle, especially within the 4080% range of diagnostic specicities (i.e. within the interval of 0.20.6 on the x-axis of the graph). In pair-wise comparisons of the area under the ROC curve for each ELISA (when used to test sera from vaccinated cattle), a statistically signicant difference was evident between the CHECKIT and UBI ELISAs and all of the other ELISAs (p < 0.01, Table 1A). The ROC curves generated using results for sera from non-vaccinated cattle (Fig. 2B) curved close to the upper left hand corner indicating that ELISAs have a high detection rate and/or specicity and that there was no appreciable difference in the performance of the different ELISAs, when used in non-vaccinated cattle as against vaccinated cattle. No signicant difference was observed between any of the ELISAs in such pair-wise comparisons when using data derived from non-vaccinated cattle (Table 1B). As the ELISAs vary enormously in terms of detection rate in exposed cattle at diagnostic specicities within the range from 40 to 80%, cut-off values were selected that would provide for higher specicities (of 97.5, 99 or 99.5%). Significant differences were seen between ELISAs at these cut-offs (Table 2). At 97.5% specicity, the result is similar to the pair-wise comparison, with the NCPanaftosa, IZS-Brescia, Ceditest and Svanovir ELISAs performing signicantly better than the UBI and CHECKIT ELISAs in vaccinated cattle (Table 2A). At this cut-off point (97.5% specicity), the NCPanaftosa ELISA had the highest detection rate (69.4%)

Discussion
In this paper the comparative evaluation of the six ELISAs described previously [4] was extended by using the continuous results available from the same database for each of the ELISAs. As for the previous paper, the twin objectives of this analysis were: (i) to establish if there was a signicant difference in performance between the six ELISAs and (ii) to assist in the interpretation of results obtained

Table 1

Pair-wise comparison of the six ELISAs based on areas under the ROC curves NCPanaftosa IZS-Brescia Ceditest Svanovir CHEKIT

A: Vaccinated cattle IZS-Brescia Ceditest Svanovir CHEKIT UBI

** **

** **

** ** ** **

B: Non-vaccinated cattle IZS-Brescia Ceditest Svanovir CHEKIT UBI

A: vaccinated cattle (173 exposed and 425 not exposed to FMD virus) and B: non-vaccinated cattle (56 exposed and 675 not exposed to FMD virus). () Not signicant. ** Signicant p < 0.01.

To compare ELISAs detection of antibodies to non-structural proteins virus

Table 2 Pair-wise comparison of ELISAs based on detection rate for exposed cattle at different levels of specicity NCPanaftosa A: Vaccinated cattle 97.5% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI 99% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI 99.5% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI B: Non-vaccinated cattle 97.5% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI 99% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI 99.5% specicity NCPanaftosa IZS-Brescia Ceditest Svanovir Chekit UBI IZS-Brescia Ceditest Svanovir Chekit

2727

** **

** **

** ** * **

69.4%a 68.8% 68.6% 64.2% 56.6% 50.9% 60.1% 68.2% 67.4% 53.2% 50.3% 36.4% 46.2% 63.6% 66.3% 49.1% 43.4% 32.9%

** *

** ** ** ** ** **

** **

** **

** **

** ** ** ** ** **

**

** *

100.0%a 100.0% 100.0% 98.2% 98.2% 100.0% 100.0% 98.2% 98.2% 98.2% 96.4% 98.2% 75.0% 98.2% 80.4% 91.1% 92.9% 92.9%

**

* **

**

A: vaccinated cattle (173 exposed and 425 not exposed tot FMD virus) B: non-vaccinated cattle (56 exposed and 675 not exposed to FMD virus). () Not signicant. a Detection rate at the given level of specicity. * Signicant p < 0.05. ** Signicant p < 0.01.

from these ELISAs. From the histogram (Fig. 1A), it can be seen that sera from some of the vaccinated cattle that were exposed to FMD virus did not show a response in the Ceditest ELISA. Most of the sera from exposed vaccinated cattle that tested negative by the Ceditest ELISA also tested negative

by each of the other ELISAs. This nding might indicate that the cattle selected as true positives, although exposed to FMD virus, were not actually infected. However, in 36 of the 48 (75%) vaccinated exposed cattle that had a percentage inhibition lower than 50% in the Ceditest NS ELISA,

2728
Table 3 Parameters for calculation of the Likelihood ratios for each ELISA NCPanaftosa Non-vaccinated cattle (prev = 0.08) Alpha Beta SE beta X = (ln(prev/(1 prev)) alpha)/beta Vaccinated cattle (prev = 0.29) Alpha Beta SE beta X = (ln(prev/(1 prev)) alpha)/beta 5.36 0.13 0.02 21.96 2.19 0.13 0.01 10.04 IZS-Brescia 9.07 0.36 0.09 18.50 2.67 0.22 0.03 7.92 Ceditest 12.12 0.18 0.03 52.85 3.60 0.09 0.01 30.56 Svanovir 9.09 0.15 0.02 44.25 3.64 0.09 0.01 29.09

A. Dekker et al.

CHECKIT 6.44 0.17 0.03 23.37 2.09 0.08 0.01 14.83

UBI 7.21 19.46 3.05 0.24 2.22 11.81 1.35 0.11

and for which information on the infection status was available, it was shown that these cattle had been infected (i.e. viral replication was indicated by detection of virus or by increased serum antibody levels post-exposure). In

fact, virus was detected in probang samples collected 28 days or longer after exposure from 12 out of 44 (27%) of the cattle that demonstrated little or no antibody response to non-structural proteins. This indicates that vaccination

Figure 3 Likelihood ratios together with 95% condence intervals for each ELISA determined in 598 vaccinated cattle of which 173 had been exposed to FMD virus and 425 had not been exposed; the arrow(s) in each graph indicate the manufacturers recommended cut-off value(s).

To compare ELISAs detection of antibodies to non-structural proteins virus might reduce the replication of virus or otherwise inhibit the serological response to non-structural proteins of FMDV. Even though additional information on the true infection status was lacking for the remainder of these exposed cattle, all sera collected at least 21 days after exposure to FMDV were considered true positive for the purposes of the analyses described in this paper. This criterion leads to a slightly conservative estimate of the detection rate in infected cattle. In the ROC analysis, the NCPanaftosa, IZS-Brescia and Ceditest ELISAs gave the largest areas under the curve, followed by the ROC curves for the Svanovir, CHECKIT and UBI ELISAs, conrming previous ndings that the former three tests were superior in performance to the latter three [4]. ROC curves drawn for the different ELISAs and based on data from vaccinated cattle, intersected when superimposed on the same graph (Fig. 2A), making it difcult to select the best performing ELISA by visual comparison alone. For this reason the detection rate for each ELISA when used on exposed cattle was compared at three different levels of specicity. As evident from visual inspection of the ROC curve, the Svanovir ELISA does not perform well when a cut-off with a higher specicity is selected, but more remarkably, the NCPanaftosa ELISA, which performs best of all the ELISAs at a specicity of 97.5%, has signicantly lower detection rate in exposed cattle than the IZS-Brescia and Ceditest ELISAs when used at 99 or 99.5% specicity. So in situations where both a high specicity and a high sensitivity are needed, the IZS-Brescia and Ceditest ELISAs would be the screening tests of choice. It should be noted, however, that in South America, the NCPanaftosa ELISA is used very effectively as a screening test in combination with a conrmatory

2729

western blot test, the latter increasing the specicity of the test system. Although the ELISAs described in this paper were primarily developed to discriminate between infected and vaccinated animals, these tests may also be used to detect infection in non-vaccinated cattle. It was clear from the study by Brocchi et al., that all 6 ELISAs were more sensitive in non-vaccinated populations than in vaccinates [4]. Using pair-wise comparison (ROC analysis) of data from non-vaccinated cattle, no signicant difference was evident between ELISAs at specicities of 97.5 or 99%. However, when used in non-vaccinated cattle and interpreted at a cut-off which gave a specicity of 99.5%, the detection rate in exposed cattle of both NCPanaftosa and Ceditest ELISAs was signicantly lower that that of the IZS-Brescia ELISA. This may be explained by what appeared to be non-specic reactions obtained on initial testing of sera from a relatively small number of cattle (4 out of 675 cattle tested), which when retested with the same ELISA mostly yielded a lower test result. The lower retest results for these problematic sera has already been described and demonstrates that test specicity may be increased by retesting positive sera using the same ELISA [4]. A more complete analysis of results after retesting was not possible because only the sera that gave an unexpected result on the rst test were retested. Our ROC curves are based on the detection rate in exposed cattle and specicity of the ELISA. Although knowledge of the detection rate in exposed cattle and specicity of an ELISA can assist in the interpretation of ELISA results, it would be more useful to know the probability that an individual animal is infected given a positive (or negative) ELISA result. The positive predictive value of a test is one

Table 4

Likelihood ratios for each ELISA at different cut-off values NCPanaftosa IZS-Brescia 10 1.6 NA 9.9 1.5 11.5 2.2 14.6 4.5 10 0.0 NA 10.1 0.0 15.7 0.4 18.4 1.0 Ceditest 50 5.6 NA 41.2 2.5 43.0 3.0 45.5 3.7 50 0.6 NA 50.8 0.7 59.8 3.6 74.0 47.4 Svanovir 48 5.9 NA 42.0 3.4 51.4 8.2 57.4 14.4 48 1.7 NA 42.4 0.8 53.8 4.1 61.1 12.3 CHECKIT 30 3.4 20 1.5 22.7 1.9 33.4 4.4 45.9 12.1 30 3.1 20 2 16.5 0.3 24.4 1.2 28.6 2.4 UBI 0.23 4.0 NA 0.19 2.5 0.36 18.8 0.42 39.1 0.23 0.8 NA 0.13 0.1 0.24 0.9 0.36 9.7

A: Vaccinated cattle Manufacturers selected cut-off Likelihood ratio at recommended cut-off Lower cut-off valuea Likelihood ratio at lower cut-off value Cut-off giving a specicity of 97.5% Likelihood ratio at 97.55% specicity Cut-off giving a specicity of 99% Likelihood ratio at 99% specicity Cut-off giving a specicity of 99.5% Likelihood ratio at 99.5%specicity Manufacturers selected cut-off Likelihood ratio at recommended cut-off B: Non-vaccinated cattle Lower cut-off value Likelihood ratio at lower cut-off value Cut-off giving a specicity of 97.5% Likelihood ratio at 97.5% specicity Cut-off giving a specicity of 99% Likelihood ratio at 99% specicity Cut-off giving a specicity of 99.5% Likelihood ratio at 99.5% specicity

20 3.6 10 1.0 11.4 1.2 26.8 8.6 40.8 51.6 20 0.8 10 0.2 10.4 0.2 19.3 0.7 51.5 47.6

NA = not applicable. a Manufacturers of two ELISAs recommended two separate cut-off values.

2730 way of expressing the probability that an individual which tests positive actually has the target disorder but the use of this parameter is dependent on knowing the prevalence of that disorder in the population of interest [8]. The likelihood ratio for a positive test result provides an alternative measure of this probability, the use of which does not require us to know the prevalence of the target disorder. As the LR is the ratio of true positive test results to false positive test results, it is not surprising that the LR for each ELISA increases dramatically above a certain test value (as graphically illustrated in Fig. 3). For example, the probability of a non-exposed animal giving a result above 60% inhibition with the Ceditest ELISA is very close to zero and 67% inhibition was the highest value found in sera from non-exposed vaccinated cattle tested by this ELISA in the present study (Fig. 1A). Consequently, with a result above 60%, the LR for the Ceditest approaches innity. Likelihood ratios have great value in the interpretation of diagnostic test results. However, various pieces of information (derived from clinical, epidemiological and laboratory investigations) have to be taken into consideration when a decision is taken whether a herd is infected with FMDV. A herd is more likely to be infected where there is a history of dangerous contact with known infected animals or where infection has been identied in a nearby herd or premises. The likelihood of infection also increases with the number of animals in a herd that test positive during surveillance and with increasing magnitude of the positive test result for any one or more of those animals. Intuitively, we might combine this information and decide that the probability of infection in a herd is high or low. In Appendix A a working example is provided to exemplify the manner in which likelihood ratios derived from different variables may be combined to give an overall probability or odds of infection in a particular group of animals. Although using likelihood ratios in this way may provide a potentially valuable objective basis for decision-making, by quantifying and combining different probabilities, the application of such a system has to be validated. The working example illustrates some of the possibilities and limitations in the approach. What it perhaps best illustrates, is the necessity of having reliable data for decision-making purposes regardless of how decisions are made, e.g. on within herd prevalence of infection in infected herds. The results in this paper can help to design an evidence based decision scheme for recovering the FMD free status after emergency vaccination has been applied.

A. Dekker et al.

Figure 4 The relationship between pre-test odds and distance to the nearest infected premises (Estimation based on Fig. 1 in Keeling et al. [9]).

Acknowledgement
This work was supported by the EU (Contract SSPE No. 503603 FMD ImproCon).

Appendix A. A working example to illustrate the principle of combining likelihood ratios to provide an objective basis for decision-making
Scenario: Emergency FMD vaccination has been deployed in the face of an outbreak in a country previously recognised by the OIE as FMD free without vaccination.

Post-vaccination serological surveillance was performed as specied in EC directive 2003/85; it did not commence until at least 30 days after vaccination was completed and at least 30 days after the last outbreak was detected and it involved whole herd testing. The objective of the surveillance was to detect vaccinated herds that had been exposed to the virus and were likely to harbour carriers; these herds were to be depopulated. The Ceditest FMDV-NS ELISA was used for all testing and had a diagnostic sensitivity of 86.4% for detection of vaccinated carrier cattle between 28 and 100 days after exposure to the virus and a diagnostic specicity of 98.1% [4]. The pre-test odds for each sampled herd were based on the proximity of that herd to the nearest known infected premises (IP) using data extrapolated from a paper by Keeling et al., which describes the spread of FMD in unvaccinated animals in the UK during 2001 [9]. The relationship between pre-test odds and distance from the nearest IP is graphically illustrated in Fig. 4. For herds at distances of 1, 3 and 10 km from an IP, the pre-test odds calculated on this basis were 0.05, 0.007 and 0.003, respectively. The rst likelihood ratio (LR1) was based on the number of animals in a herd of 100 cattle which gave a positive test result in the Ceditest using the manufacturers recommended cut-off value of 50 PI. This was calculated by comparing binomial probabilities of detecting specic numbers of positive animals in two herds of this size, one an infected herd and the other an uninfected herd (Table 5). An apparent prevalence (seroprevalence) of 3.35% was assumed for vaccinated, infected herds as this was the mean seroprevalence obtained on testing, with the Ceditest, two vaccinated and infected Israeli dairy herds that had not shown clinical signs (2/119 and 6/120 animals tested positive, respectively) [10]. The second likelihood ratio (LR2) was based on the magnitude of the largest PI value obtained in testing the herd with the Ceditest. LR2 was calculated using logarithmic regression as described elsewhere in this paper such that PI values of 55, 70 and 90 gave ratios of 8.6, 32.2 and 189.1, respectively.

To compare ELISAs detection of antibodies to non-structural proteins virus

Table 5

2731

Likelihood ratio based on binomial probabilities of detecting different numbers of positives in a herd of 100 animals Uninfected false positive results = 1.9% (%) 14.69 28.44 27.27 17.25 8.10 3.01 0.92 0.24 0.05 0.01 0.00 Infected apparent prevalence = 3.35% (%) 3.32 11.50 19.72 22.31 18.74 12.46 6.83 3.18 1.28 0.45 0.14 LR 0.226208 0.404483 0.723259 1.293265 2.312495 4.134988 7.393799 13.2209 23.64038 42.27152 75.58599

Number positive 0 1 2 3 4 5 6 7 8 9 10

The formula pre-test odds LR1 LR2 = post-test odds was applied to various combinations of distance from an IP (1, 3 or 10 km), number of positives in a herd of 100 cattle (1, 5 or 10) and largest test value (55, 70 and 90 PI). The resulting post-test odds and equivalent post-test probabilities were ranked in order of magnitude (Table 6). Two dotted lines representing threshold values have been super-

imposed on this table for the purposes of illustration. The lower threshold value is set at post-test odds of 0.05 and the higher threshold at odds of 20; both threshold values have been chosen on an entirely arbitrary basis. Using these threshold values, any herd with post-test odds exceeding 20 would be culled (as the benets of further testing and follow-up investigation are outweighed by the costs, and

Table 6 ratios Distance

Post-test odds (and probabilities) for different levels of pre-test odds and different levels of each of two likelihood Pre-test odds Number of positives 10 10 10 10 10 10 5 10 10 1 5 10 5 5 1 5 1 5 1 1 5 1 5 1 5 1 1 LR1 75.6 75.6 75.6 75.6 75.6 75.6 1.3 75.6 75.6 0.4 1.3 75.6 1.3 1.3 0.4 1.3 0.4 1.3 0.4 0.4 1.3 0.4 1.3 0.4 1.3 0.4 0.4 Highest PI value 90 70 90 90 55 70 90 70 55 90 70 55 90 90 70 55 90 70 90 55 70 70 55 70 55 55 55 LR2 189.1 32.3 189.1 189.1 8.6 32.3 189.1 32.3 8.6 189.1 32.3 8.6 189.1 189.1 32.3 8.6 189.1 32.3 189.1 8.6 32.3 32.3 8.6 32.3 8.6 8.6 8.6 Post-test odds 739.5 126.5 100.4 42.5 33.6 17.2 12.7 7.3 4.6 4.0 2.2 1.9 1.7 0.7 0.7 0.6 0.5 0.3 0.2 0.2 0.12 0.09 0.08 0.04 0.03 0.02 0.01 Post-test probability (%) 99.9 99.2 99.0 97.7 97.1 94.5 92.7 87.9 82.0 79.8 68.4 65.9 63.2 42.1 40.4 36.5 35.0 22.7 18.5 15.3 11.1 8.4 7.3 3.7 3.2 2.4 1.0

Depopulate 1 0.052 1 0.052 3 0.007 10 0.003 1 0.052 Further testing 3 0.007 1 0.052 10 0.003 3 0.007 1 0.052 1 0.052 10 0.003 3 0.007 10 0.003 1 0.052 1 0.052 3 0.007 3 0.007 10 0.003 1 0.052 10 0.003 3 0.007 3 0.007 No further action 10 0.003 10 0.003 3 0.007 10 0.0

2732 perhaps the risks, incurred by any further delay in designating the herd as infected and proceeding with depopulation) and any herd with odds below 0.05 would be considered uninfected whereas herds with post-test odds between these two thresholds would be subject to further investigation and testing.

A. Dekker et al.
[4] Brocchi E, Bergmann IE, Dekker A, Paton DJ, Sammin DJ, Greiner M, et al. Comparative evaluation of six ELISAs for the detection of antibodies to the non-structural proteins of foot-and-mouth disease virus. Vaccine 2006;24(4748): 696679. [5] Radack KL, Rouan G, Hedges J. The likelihood ratio an improved measure for reporting and evaluating diagnostic test results. Arch Pathol Lab Med 1986;110(8):68993. [6] Atkinson B, Mahoney D. ROC for Splus. In http://mayoresearch. mayo.edu/mayo/research/biostat/splusfunctions.cfm; 2004. [7] Simel DL, Samsa GP, Matchar DB. Likelihood ratios for continuous test resultsmaking the clinicians job easier or harder? J Clin Epidemiol 1993;46(1):8593. [8] Choi YK, Johnson WO, Thurmond MC. Diagnosis using predictive probabilities without cut-offs. Stat Med 2006;25(4, February):699717. [9] Keeling MJ, Woolhouse MEJ, Shaw DJ, Matthews L, Chase Topping M, Haydon DT, et al. Dynamics of the 2001 UK foot and mouth epidemic: stochastic dispersal in a heterogeneous landscape. Science 2001;294(5543):8137. [10] Yadin H, Chai D, Brener J, Oved Z, Hadany Y, Kusak A. Screening for FMD virus in vaccinated herds affected by eld infection. In: Session of the research group of the European commission for the control of foot-and-mouth disease. 2004.

References
[1] Mackay DKJ, Forsyth MA, Davies PR, Berlinzani A, Belsham GJ, Flint M, et al. Differentiating infection from vaccination in foot and mouth disease using a panel of recombinant, non structural proteins in ELISA. Vaccine 1998;16(5):44659. [2] Sorensen KJ, Madsen KG, Madsen ES, Salt JS, Nqindi J, Mackay DKJ. Differentiation of infection from vaccination in foot and mouth disease by the detection of antibodies to the non structural proteins 3D 3AB and 3ABC in ELISA using antigens expressed in baculovirus. Arch Virol 1998;143(8): 146176. [3] Bergmann IE, Malirat V, Neitzert E, Beck E, Panizzutti N, Sanchez C, et al. Improvement of a serodiagnostic strategy for foot-and-mouth disease virus surveillance in cattle under systematic vaccination: a combined system of an indirect ELISA-3ABC with an enzyme-linked immunoelectrotransfer blot assay. Arch Virol 2000;145(3):47389.