ORIGINAL RESEARCH

Small Pulmonary Artery Defects Are Not Reliable Indicators of

Pulmonary Embolism
Wallace T. Miller, Jr.1, Lawrence A. Marinari2, Eduardo Barbosa, Jr.1, Harold I. Litt1, James E. Schmitt1, Anton Mahne3,
Victor Lee4, and Scott R. Akers5
1
Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia Pennsylvania; 2Department of
Medicine, and 4Department of Radiology, The Bryn Mawr Hospital, Bryn Mawr, Pennsylvania; 3Department of Radiology and
Outpatient Diagnostic Testing, Cooper University Health Care, Camden, New Jersey; and 5Veterans Affairs Medical Center,
Philadelphia, Pennsylvania

Abstract (short-axis diameter) of the lesion, and (3) diminishing quality of

Objectives: To evaluate the rate of agreement of pulmonary
embolism diagnosis in computed tomography (CT) pulmonary
angiogram studies and to evaluate the rate of inaccurate
interpretations in the community hospital setting.
Methods: Using the keywords “pulmonary embolism/embolus/
emboli,” the radiology information system was searched for CT
pulmonary angiograms performed over a 3-year period at three
U.S. community hospitals. Studies containing probable or definite
pulmonary emboli were independently reviewed by four subspecialty
thoracic radiologists.
Measurements and Main Results: Agreement about the
presence of pulmonary embolism progressively decreased with
decreasing diameter of pulmonary vascular lesions (P , 0.0001).
There was a sharp fall in observer agreement for pulmonary
embolism of subsegmental lesions (P , 0.0001). The frequency
of agreement decreased with decreasing quality of the imaging
examination (P , 0.0001). Community radiologists were prone to
false-positive pulmonary embolism diagnosis of subsegmental and/
or small pulmonary arterial defects. The probability of a false-positive
diagnosis and indeterminate examinations progressively increased
with: (1) more peripheral location of the lesion, (2) decreased size
the CT examination. Forty-eight of 177 (27%) of subsegmental
vascular defects identified by community radiologists were deemed
indeterminate, and 27 of 177 (15%) of subsegmental vascular defects
were judged to be false positive for pulmonary embolism by the
consensus diagnosis. Fifty-four of 274 (20%) vascular defects with
short axis less than 6 mm were indeterminate for pulmonary
embolism, and 37 of 274 (14%) of vascular defects with short axis
less than 6 mm were false positive for pulmonary embolism. Eleven
of 13 (85%) of vascular lesions identified as pulmonary emboli
on the lowest-quality CT examinations were false positive or
indeterminate for pulmonary embolism. False-positive
examinations were most often due to respiratory motion artifact
(19/38, 50%).
Conclusions: There is relatively poor interobserver agreement for
subsegmental and/or small pulmonary artery defects, especially in
CT pulmonary angiograms degraded by technical artifacts. These
factors can lead to an increased frequency of inaccurate interpretation
or indeterminate diagnosis of subsegmental and/or small defects.
Caution is indicated in interpreting the significance of small vascular
defects in CT pulmonary angiograms.
Keywords: pulmonary embolism; X-ray computed tomography;
diagnostic errors; false-positive reactions; community hospitals

(Received in original form February 25, 2015; accepted in final form April 8, 2015 )
Supported by the Sharpe-Strumia Research Foundation.
Author Contributions: W.T.M. takes responsibility for the content of the study as a whole, including study design, data collection, data analysis and
interpretation, and the writing of the manuscript. L.A.M. contributed substantially to the study design, data analysis and interpretation, and the writing of the
manuscript. E.B., H.I.L., A.M., V.L., and S.R.A. contributed to data collection and interpretation and the writing of the manuscript. J.E.S. contributed to
statistical analysis, data interpretation, and the writing of the manuscript.
Correspondence and requests for reprints should be addressed to Wallace T. Miller, Jr., M.D., Department of Radiology, Perelman School of Medicine University
of Pennsylvania, Silverstein 1, 3400 Spruce Street, Philadelphia PA 19104. E-mail: millerw@uphs.upenn.edu
Ann Am Thorac Soc Vol 12, No 7, pp 1022–1029, Jul 2015
Copyright © 2015 by the American Thoracic Society
DOI: 10.1513/AnnalsATS.201502-105OC
Internet address: www.atsjournals.org

1022 AnnalsATS Volume 12 Number 7 | July 2015

 ORIGINAL RESEARCH

Pulmonary embolism is the third most
commonly diagnosed acute cardiovascular
disease after myocardial infarction and
cerebrovascular accident (1). The signs and
symptoms of pulmonary embolism are not
specific, and clinicians rely on imaging
examinations for the diagnosis. Computed
tomography (CT) pulmonary angiography
is fast, noninvasive, and widely available,
and has emerged as the modality of choice
for the diagnosis of pulmonary embolism
(2, 3).
Prior studies have suggested that the
interobserver agreement of the presence
of a pulmonary embolism is dependent
on the location of the embolus (central
vs. peripheral) and that there is a high
false-positive rate in the diagnosis of
subsegmental pulmonary arteries (2, 3). The
purpose of this study was to investigate
interobserver agreement as a function of
lesion size and location and how it affects
the reliability of a diagnosis of pulmonary
embolism as identified by CT pulmonary
angiograms in routine clinical practice.
Methods
Study Design
The study was approved by the center’s
Institutional Review Board (Main Line
Health IRB: F/N-R09-2783BLP) and
is Health Insurance Portability and
Accountability Act compliant. Informed
consent was waived due to the retrospective
nature of the project. Using the keywords
“pulmonary embolism/embolus/emboli,”
the radiology information system was
searched for CT pulmonary angiograms
performed from January 1, 2006 to
December 31, 2008 at three suburban
community hospitals in the mid-Atlantic
region of the United States. We identified
8,414 reports that fulfilled criteria. We
excluded 514 examinations from further
review because they were: (1) non–
pulmonary embolism protocol studies
(n = 369), (2) repeat examinations (n = 140),
or (3) thrombus within the stump of the
pulmonary artery after lobectomy (n = 5).
One of the authors (A.M.) reviewed
the reports of the remaining 7,900 CT
pulmonary angiograms for the following:
(1) presence or absence of pulmonary
embolism, (2) location of pulmonary embolism,
and (3) confidence of pulmonary
embolism. The confidence of pulmonary
embolism in the report was scored on
a 5-point scale: 1, definitely no
pulmonary embolism; 2, probably
no pulmonary embolism; 3,
indeterminate pulmonary embolism; 4,
probable pulmonary embolism; 5, definite
pulmonary embolism. Reports with
descriptions like “multiple pulmonary
emboli present” or “small filling defect
consistent with a pulmonary embolism”
were scored as a “5” (definite pulmonary
embolism). Reports using qualifiers such as
“probable,” “suspicious for,” or “likely to
represent” when describing a defect were
scored as a “4” (probable pulmonary
embolism). Reports using the terms “could
represent” or “possibly representing”
pulmonary embolism were scored as “3”
(indeterminate pulmonary embolism).
Reports with qualifiers such as “probably
not” or “not likely to represent” pulmonary
embolism were scored as “2” (probably no
pulmonary embolism). If the report noted
no pulmonary embolism but expressed
a limitation in interpretation based on
suboptimal quality of the study, the report
was scored as a “2.” Reports with strong
descriptors of the absence of pulmonary
embolism, such as “no evidence of
pulmonary embolism” or “no pulmonary
embolism,” were scored as “1” (definite
absence of pulmonary embolism).
Of the 7,900 examinations, a total
of 728 examinations were interpreted as
a score of 4 to 5 for pulmonary embolism
and represented our study group.
Examinations that were interpreted as
inadequate to evaluate for pulmonary
embolism (106), interpreted has having no
pulmonary embolism (score 1–2, 7,035
examinations), or interpreted as being
indeterminate for pulmonary embolism
(score 3, 31 examinations) were excluded.
A thoracic radiologist (Reader 1) with
19 years of subspecialty experience reviewed
all 728 CT pulmonary angiograms for the
presence of pulmonary embolism and
evaluated the specific lesions that were
identified by the primary reader. In all cases,
the location of the presumed embolus was
identified in the report, by slice location
and lobe, by electronic arrows or circles, by
the pulmonary segment, or in some cases by
more general terms like “multiple small
emboli in both lung bases.”
Reader 1 scored each study for the
likelihood of pulmonary embolism using
the same 5-point scale and identified the
largest pulmonary embolism based on the
generation of pulmonary artery (pulmonary
trunk, right/left pulmonary artery, lobar
pulmonary artery, segmental pulmonary
artery, subsegmental pulmonary artery) in
which the embolus was identified. The
short-axis diameter of the largest pulmonary
embolism was measured by electronic
calipers. Each study was scored for technical
quality using a 5-point scale: 5, excellent
quality (no technical limitation); 4,
good quality (minor technical limitations
not likely to affect the detection of
pulmonary embolism); 3, moderate quality
(technical defects that could affect the
detection of small pulmonary embolism);
2, poor quality (examination in which only
large central pulmonary embolism can
reliably be detected); and 1, nondiagnostic
study. This initial review suggested a high
false-positive rate in the diagnosis of
pulmonary embolism for peripheral and/or
small pulmonary artery defects. Additional
radiologist review was sought to buttress
the conclusions concerning false-positive
diagnosis.
Three additional thoracic radiologists
with 10 (Reader 2), 8 (Reader 3), and 2
(Reader 4) years of subspecialty experience
reviewed a subset of the CT pulmonary
angiogram examinations. We attempted to
include all segmental and subsegmental
lesions in the second review both because
our initial review indicated a problem with
small, peripheral lesions and because prior
studies have indicated lower interobserver
agreement for more peripheral pulmonary
emboli (2, 3). We were able to include
177 of 179 subsegmental and 192 of 195
segmental lesions. Five lesions could not be
successfully transferred to an independent
workstation where the second review was
performed and were excluded. We included
all examinations with main and lobar
pulmonary embolism that were scored as
a 1 to 3 for likelihood of PE by Reader 1
and a random sampling of the remainder,
including 19 of 126 (15%) lesions of the
main pulmonary arteries and 27 of 228
(12%) lesions of the lobar pulmonary
arteries. Readers 2 to 4 randomly reviewed
the subset and scored both the likelihood of
pulmonary embolism and the quality of
the examination, using our 5-point scales.
A composite study quality was derived by
averaging the study quality across the four
reviewers.
Each imaging examination was scored
as positive, negative, or indeterminate for
the presence of pulmonary embolism by the
five readers (the original report and the four

Miller, Marinari, Barbosa, et al.: Small Pulmonary Artery Defects and Predicting PE 1023
 ORIGINAL RESEARCH

reviewers). A score of a “4” or “5” was
considered positive for pulmonary
embolism; a score of “1” or “2” was
considered negative for pulmonary
embolism, and a score of “3” was
considered indeterminate for pulmonary
embolism. If four of the five readers agreed
on the presence of pulmonary embolism,
then the examination was considered
positive for pulmonary embolism.
For purposes of evaluating the
community radiologist, if four of four
thoracic radiologists agreed on the absence
of pulmonary embolism, then the
examination was considered negative for
pulmonary embolism and a false-positive
interpretation by the community
radiologist. For purposes of evaluating
the thoracic radiologists, if three of four
thoracic radiologists agreed on the absence
of pulmonary embolism, the examination
was considered negative for pulmonary
embolism and a false-positive
interpretation for the dissenting thoracic
radiologist.
CT Examinations
CT examinations were performed on four
multidetector spiral CT scanners (GE
Medical Systems, Milwaukee, WI): Light
speed QXi (n = 22), Lightspeed Ultra
(n = 190), Light speed 16 (n = 116), and
Lightspeed VCT (n = 87). Ninety-four
percent (n = 708) were performed with
a slice thickness of 1.25 mm and 6%
(n = 46) with a slice thickness of 2.5 mm.
Intravenous contrast used included:
ioversol 320 mg I/mL (Optiray 320;
Covidien, Dublin, Ireland) (n = 27), ioversol
350 mg I/mL (Optiray 350; Covidien)
(n = 315), and iodixanol 320 mg I/mL
(Visipaque 320; GE Healthcare, Milwaukee,
WI) (n = 51). In 22 cases, the type of
contrast was not recorded.
Table 1. Locations of the largest pulmonary artery defect in the original and second
review computed tomography studies
Location Original Dataset
Main 127 (17)
Lobar 228 (30)
Segmental 207 (27)
Subsegmental 197 (26)
Total 759
Data are presented as n (%).
Statistical Analysis
The data were subsequently imported
into the R computing environment for
statistical analysis (4). Several metrics
of interrater reliability were calculated
on the sample as a whole and after
subsampling by pulmonary embolism
size, location, and CT pulmonary
angiogram scan quality. First, the simple
fraction of cases with complete or
near-complete agreement (e.g., five- or
four-rater agreement) was calculated.
Interrater reliability was estimated using
Randolph’s Free Marginal Multirater
kappa, which accounts for differences
in group prevalence (5, 6). Ninety-five
percent confidence intervals were
calculated via bootstrap using 1,000
replicates. Fisher exact test was used to
compare differences in agreement based
on pulmonary embolism size, location,
and scan quality. Post hoc pairwise
differences also were assessed with Fisher
Table 2. Agreement for pulmonary embolism as a function of pulmonary embolism
location, pulmonary embolism diameter, and study quality
5 of 5
Readers Agree
Second Review
Subset
19 (4.6)
27 (6.5)
192 (46)
177 (43)
415
exact test, with control for multiple
testing using Bonferroni correction. All
P values reported are two sided, with
a P value less than 0.05 considered
significant.
Results
Community Radiologists
The radiology group that initially
interpreted the studies was a practice
covering three suburban, northeastern
United States hospitals. Forty-two
participating community staff radiologists
each interpreted between 3 and 35 CT
pulmonary angiogram examinations. Of the
42 radiologists, all were board certified and
31 had subspecialty training (7 body
imaging, 6 interventional radiology, 7
neuroradiology, 5 breast imaging, 4
pediatric radiology, 2 nuclear medicine). Six
board-certified moonlighting radiologists
4 of 5 Indeterminate* Kappa
Readers Agree

Image Review Location

Evaluation by the community radiologists Main (n = 19) 17 (89) 2 (11) 0 0.94
Lobar (n = 27) 24 (89) 2 (7.4) 1 (3.7) 0.92
and Reader 1 was performed using the Segment (n = 192) 169 (88) 12 (6.3) 11 (5.7) 0.90
picture archiving system (McKesson Subsegment (n = 177) 70 (40) 59 (33) 48 (27) 0.53
Radiology, McKesson, San Francisco, CA). Diameter
Images were available in the axial, sagittal, .9.0 (N = 42) 40 (95) 0 2 (4.8) 0.95
and coronal planes. Readers 2 through 4 6.1–9.0 (N = 99) 87 (88) 8 (8.1) 4 (4.0) 0.91
3.1–6.0 (N = 207) 133 (64) 41 (20) 33 (16) 0.72
reviewed examinations on a thin-client 0–3.0 (N = 67) 20 (30) 26 (39) 21 (31) 0.46
viewing platform (AQNet; Terarecon, Study quality
Foster City, CA) where Digital Imaging and 4.0–5.0 (N = 251) 198 (79) 31 (12) 22 (8.7) 0.84
Communications in Medicine (DICOM)- 3.0–3.9 (N = 151) 81 (54) 38 (25) 32 (21) 0.63
2.0–2.9 (N = 13) 1 (7.7) 6 (46) 6 (46) 0.22
formatted images were loaded, patient-
specific information was removed, and the Numbers in parentheses are percentages.
patient order was randomized. *Fewer than four of five readers agreeing on the presence or absence of pulmonary emboli.

1024 AnnalsATS Volume 12 Number 7 | July 2015

 ORIGINAL RESEARCH

interpreted 12 examinations (1.6%). The A

group practice had a diagnostic radiology 1.0

Free Marginal Multirater Kappa

residency. Of the 759 examinations, 261
studies (35%) were reviewed by 26 residents 0.8
in conjunction with staff radiologists.
0.6
Location of Pulmonary Artery Lesions
The locations of pulmonary artery lesions in 0.4
the original and second review subsets are
listed in Table 1. In the original dataset, 0.2
locations of pulmonary artery lesions were
nearly uniformly distributed between
0.0
central and peripheral locations. By study
design, the second review subset was highly Main Lobar Segmental Subsegment
skewed toward smaller lesions in the PE Location
segmental and subsegmental arteries.
B
1.0
Free Marginal Multirater Kappa
Agreement among Readers
Table 2 reports agreement among readers
0.8
based on pulmonary embolism location,
pulmonary embolism diameter, and study
0.6
quality. The frequency of agreement
decreased and the proportion of
indeterminate lesions increased with more 0.4
peripheral location of the pulmonary artery
defect. 0.2
Kappa scores typically are rated on the
following scale: poor (0.00), slight (0.01– 0.0
0.20), fair (0.21–0.40), moderate (0.41– >9 6–9 3–6 <3
0.60), substantial (0.61–0.80), or almost PE size (mm)
perfect (0.81–1.0) agreement. By this scale,
agreement among readers was almost C
perfect for main, lobar, and segmental 1.0
Free Marginal Multirater Kappa

emboli. However, there was notable fall

in agreement with subsegmental lesions,
which was only moderate, with a k = 0.53
(Figure 1A). Five-rater agreement was
significantly different based on lesion
location (P , 0.0001). Post hoc analysis
demonstrated significant differences
between subsegmental lesions and all other
locations (P , 00001), with other pairwise
group comparisons not reaching statistical
significance (P . 0.9999). There was lower
five-reader agreement for subsegmental
lesions than for larger branches (40%
[70/177] vs. 88% [210/238], P , 0.0001).
Similarly, subsegmental lesions were judged
as indeterminate for pulmonary embolism
more commonly than for larger vessels
(27% [48/177] vs. 5% [12/238], P , 0.0001).
The frequency of agreement decreased
and the indeterminate fraction increased
as the short-axis diameter of the defect
decreased (Table 2). Agreement regarding
lesions with a short axis greater than 6 mm
was nearly perfect, but fell for lesions with
short axis of 6 mm or smaller (Figure 1B).
There were statistically significant group
0.8
0.6
0.4
0.2
0.0
High Intermediate Low
Scan Quality
Figure 1. (A) Agreement among five readers as a function of lesion location. (B) Agreement as
a function of pulmonary embolism size. (C) Agreement as a function of study quality. Bars are the
95% confidence intervals. PE = pulmonary embolism.
differences in agreement by size for the difference of agreement for lesions 6 to
entire population (P , 0.0001). There was 9 mm and smaller size groups (P , 0.0001).
a significant difference between agreement There was a significant difference of
for lesions less than 3 mm and all other agreement for lesions greater than 9 mm
size groups (P , 0.0001). There was and groups of less than 6 mm (P , 0.0001).
a significant difference of agreement for However, there was not a significant
lesions 3 to 6 mm and all other size groups difference in agreement between emboli of
(P , 0.0001). There was a significant size 6 to 9 mm and emboli of size greater

Miller, Marinari, Barbosa, et al.: Small Pulmonary Artery Defects and Predicting PE 1025
 ORIGINAL RESEARCH

than 9 mm. Five-reader agreement for Table 3. Frequency of false-positive interpretations by individual readers as a function
lesions with a short-axis diameter of less of location of pulmonary embolism, diameter of pulmonary embolism, and study quality
than or equal to 6 mm was significantly
lower than larger lesions (56% [153/274] vs. R1 R2 R3 R4 Community Radiologist (%)
90% [127/141], P , 0.0001), and frequency
of indeterminate lesions was increased Location
(20% [54/274] vs. 4% [6/141], P , 0.0001). Main (N = 19) 0 0 0 0 2
The frequency of agreement decreased Lobar (N = 27) 0 0 0 0 2
with decreasing quality of examination Segment (N = 192) 0 1 (0.5) 2 (1.0) 0 7 (3.6)
Subsegment (N = 177) 7 (4.0) 1 (0.6) 2 (1.1) 0 27 (15)
(P , 0.0001), with post hoc analysis Size
demonstrating significant differences in .9.0 (N = 42) 0 0 0 0 0
five-rater agreement between all pairwise 6.1–9.0 (N = 99) 0 0 1 (1.0) 0 1 (1.0)
levels of scan quality (P = 0.0014 to 3.1–6.0 (N = 207) 3 (1.4) 0 2 (1.0) 0 26 (13)
0–3.0 (N = 67) 4 (6.0) 2 (3.0) 1 (1.5) 0 11 (16)
,0.0001) (Table 2). Agreement regarding Study quality
lesions in studies with a quality of 4 4.0–5.0 (N = 251) 4 (0.4) 0 4 (0.4) 0 8 (0.3)
(“minor technical limitations not likely 3.0–3.9 (N = 151) 3 (2.0) 2 (1.3) 0 0 25 (17)
to affect the detection of pulmonary 2.0–2.9 (N = 13) 0 0 0 0 5 (38)
embolism”) or better was substantial
False positive for community radiologist is defined as four of four thoracic radiologists interpreted
(k = 0.84) but fell progressively with qualities the examination as negative for pulmonary embolism. False positive for the thoracic radiologists was
less than 4 (Figure 1C). Agreement of all defined as three of three other thoracic radiologists interpreted the examination as negative for
five readers for lesions on examinations pulmonary embolism. R1, R2, R3, R4 indicate thoracic radiology reviewers 1 through 4.
of quality greater than or equal to 4 was
significantly better than examinations of
lower quality (79% [198/251] vs. 50% [82/ 0.7% [1/141]). Furthermore, of the highest- lesions, and motion artifact and mucous
164], P , 0.0001), and the indeterminate quality examinations (quality score > 4), plug are typically confused with peripheral
fraction was less than those of lower quality 0.3% (8/251) were false positive compared lesions.
(9% [22/251] vs. 23% [38/164], P , 0.0001). with 18% (30/164; P , 0.0001) in those
with lower quality (quality score , 4). Accuracy of Thoracic Radiologists
Accuracy of Interpretations by Motion artifact (Figure 2) accounted There was a range of false-positive rates
Community Radiologists for 50% (19/38) of false-positive lesions among chest radiologists, with a low of 0%
Community radiologists were prone to (Table 4). Other less-common causes false-positive interpretations to a maximum
false-positive diagnosis (Table 3). The included streak artifact, pulsation artifact, of 1.7% (7/415) false-positive interpretations
consensus diagnosis identified 38 of 759 mucous plug in a bronchus adjacent to the of pulmonary embolism. The likelihood of
examinations (5.0%) as false positive for pulmonary artery, and hilar lymph nodes a false-positive interpretation increased with
pulmonary embolism. The probability of (Figure 3). The number of causes other more peripheral location and decreasing
a false-positive diagnosis increased with (1) than motion artifact are small but suggest short-axis diameter of the lesion. A
more peripheral location of the lesion, (2) that streak artifact and lymphadenopathy maximum of 4.0% (7/177) of subsegmental
decreased short-axis diameter of the lesion, are typically confused with centrally located lesions compared with 0.8% (2/238,
and (3) decreasing quality of the CT
examination (Table 3).
False-positive reports were more likely
to describe subsegmental lesions than more
central lesions (15% [27/177] vs. 4.6%
[11/238]; P = 0.0002). Because not all main
and lobar emboli examinations were
reviewed by the entire panel, we cannot
absolutely determine the positive predictive
values for all locations. However, assuming
that Reader 1 identified all false-positive
pulmonary embolisms in the main and
lobar arteries, the positive predictive values
for locations of emboli were 98.4% for
main pulmonary emboli, 99.1% for lobar
pulmonary emboli, 96.6% for segmental
pulmonary emboli, and only 86.3% for
subsegmental emboli. Figure 2. This 46-year-old woman complained of chest pain. (A) Computed tomography pulmonary
The false-positive rate of lesions less angiogram through the left lower lobe shows apparent filling defects (arrows) in several subsegmental
than or equal to 6 mm was greater than arteries. (B) Lung windows of the same slice show moderate blurring of the pulmonary vessels
lesions greater than 6 mm (14% [37/274] vs. indicating motion artifact as the most likely cause for the lesions seen in the pulmonary arteries.

1026 AnnalsATS Volume 12 Number 7 | July 2015

 ORIGINAL RESEARCH

P , 0001) of central lesions were false Table 4. Causes of 38 false-positive examinations as judged by thoracic radiologists
positive for pulmonary embolism. Six
percent (4/67) of lesions with short axis less Location Motion Streak Pulsation Mucous Plug Lymph Node No Agreement
than or equal to 3 mm were false positive
compared with 0.9% (3/348, P = 0.015) of Main 0 1 0 0 1 0
lesions with short axis greater than 3 mm. Lobar 0 1 0 0 0 1
Scan quality differences were not associated Segment 3 1 0 0 0 3
with false-positive interpretations by Subsegment 16 0 1 2 0 8
Total 19 (50) 3 (7.8) 1 (2.6) 2 (5.2) 1 (2.6) 12 (32)
thoracic radiologists.
There was a range of false-negative Numbers in parenthesis are percentages. Cause established if more than three readers identified the
rates among chest radiologists, with a low same cause. Fewer than three readers classified as no agreement on cause of false positive.
of 0% false-negative interpretations to
a maximum of 4.8% (20/415) incidence
false-negative interpretations of pulmonary 26 proximal vessels, but is impossible to study quality, analogous to decreases in
embolism (Table 5). The likelihood of directly inspect each subsegmental artery. agreement among radiologists.
a false-negative diagnosis increased with Radiologists rely on “peripheral vision” to Of subsegmental pulmonary emboli
more peripheral location and decreasing draw their eye to subsegmental lesions, with identified by community radiologists, 27%
short-axis diameter of the pulmonary artery relatively low accuracy. (48/177) were judged to be indeterminate
lesion. As high as 11% (19/177) of Our thoracic radiologist reviewers and 15% (27/177) false positive by
subsegmental lesions compared with 0.8% rated the quality of the examination based consensus diagnosis. In most cases, false-
(2/238, P , 0.0001) of central lesions were on a subjective assessment of how the positive lesions represented artifacts,
false negative for a thoracic radiologist, and quality of the examination might influence primarily motion artifact, and streak artifact
up to 16% (11/67) of lesions with a short the detection of pulmonary embolism. We (Figure 2). Hilar lymph nodes and small
axis less than 3 mm compared with 2.6% found that they could detect falls in quality peripheral mucous plugs were also
(9/348, P , 0.0001) of larger lesions were that would result in errors of pulmonary confused with pulmonary embolism (7, 8)
missed by a thoracic radiologist. embolism detection. Agreement about the (Figure 3). Because the majority of false-
presence of pulmonary embolism fell as positive lesions represented artifacts, it is
radiologists identified decreasing quality of not surprising that decreasing quality of the
Discussion examinations. examination led to increasing incidence of
Compared with thoracic radiologists, false-positive interpretations.
Our study indicates that the interpretation community radiologists were more likely to Several other studies have described
of subsegmental and/or small pulmonary identify false-positive defects as pulmonary the inaccuracies associated with small CT
artery defects has a high rate of inaccuracy embolism. The probability of a false-positive pulmonary angiography lesions. Stein and
among both community radiologists examination was directly correlated with colleagues reported a false-positive rate of
and academic thoracic radiologists. subsegmental location and small size of the 32% (15/47) of segmental defects and 75%
Interobserver agreement fell relatively defect and was inversely correlated with (6/8) of subsegmental defects (3). Among
continuously as the lesion diameter
decreased (Figure 1B). Using post hoc
review by a panel of academic chest
radiologists to define a “gold standard,” Table 5. Frequency of false-negative interpretations by individual readers as a function
this result suggests that there is a direct of location of pulmonary embolism, diameter of pulmonary embolism, and study quality
correlation between the size of a pulmonary
embolism and the reliability of detection. R1 R2 R3 R4
By necessity, subsegmental pulmonary
emboli must be small in size because of Location
the small caliber of the vessel. However, Main (N = 19) 0 0 0 0
interobserver agreement dropped sharply Lobar (N = 27) 0 0 0 0
Segment (N = 192) 0 0 3 (1.5) 1 (0.5)
from segmental to subsegmental pulmonary Subsegment (N = 177) 0 8 (4.5) 3 (1.7) 19 (11)
artery lesions (Figure 1A), suggesting that Size
additional factors other than lesion size .9.0 (N = 42) 0 0 0 0
contribute to inaccuracy in detection of 6.1–9.0 (N = 99) 0 1 (1.0) 3 (3.0) 2 (2.0)
subsegmental pulmonary embolism. We 3.1–6.0 (N = 207) 0 4 (1.9) 2 (0.9) 7 (3.3)
0–3.0 (N = 67) 0 3 (4.5) 1 (1.5) 11 (16)
suspect that this factor is the sheer number Study quality
of subsegmental vessels. There is one 4.0–5.0 (N = 251) 0 4 (1.6) 4 (1.6) 14 (5.6)
pulmonary trunk, two main pulmonary 3.0–3.9 (N = 151) 0 4 (2.6) 2 (1.3) 6 (4.0)
arteries, five lobar arteries, and 18 2.0–2.9 (N = 13) 0 0 0 0
Total (N = 415) 0 8 (1.9) 6 (1.4) 20 (4.8)
segmental arteries, and hundreds of visible
subsegmental pulmonary arteries. It is Numbers in parenthesis are percentages. Note: by study design, Reader 1 could have no false
a reasonable task to look at each of the negatives.

Miller, Marinari, Barbosa, et al.: Small Pulmonary Artery Defects and Predicting PE 1027

Figure 3. This 70-year-old woman with a history of breast cancer had progressive dyspnea,
diaphoresis, and lightheadedness. (A) The small low attenuation lesion (arrow) in the right lower lobe
was interpreted initially as representing a small subsegmental pulmonary embolus. (B) Sagittal oblique
reconstructions of the lesion show it to represent a small mucus plug.
patients with a low clinical probability of
pulmonary embolism, CT pulmonary
angiography had a 42% false-positive rate
(3). Perrier and colleagues demonstrated
a false-positive rate of 0% in the main
pulmonary artery, 15% in the lobar
pulmonary arteries, and 38% in the
segmental pulmonary arteries (2).
Since the emergence of CT pulmonary
angiography as the primary diagnostic
study for pulmonary embolism, there has
been a near doubling in the detection
of pulmonary embolism without
a corresponding decrease in mortality (8).
Some have suggested that this phenomenon
is due to the detection of small clinically
insignificant pulmonary embolism (2, 9–
16). Our study suggests the increase in
detection of pulmonary embolism may in
part be due to false-positive diagnosis.
Although our thoracic radiologists
performed better than the community
radiologists at detecting pulmonary
embolism, their performance was also
adversely affected by subsegmental location
and/or small size. Up to 4.0% (7/177) of
subsegmental lesions identified by one
thoracic radiologist were judged false
positive by the remaining three thoracic
radiologists, and up to 10% (19/195) of
subsegmental emboli determined by
consensus as positive for pulmonary
embolism were read as negative
examinations by one thoracic radiologist.
The false-negative rate is likely an
underrepresentation of the true rate of
false-negative examinations because our
1028
reviewers knew that another radiologist had
already diagnosed a pulmonary embolism
and they were looking more diligently for
small lesions. This study was not designed
to measure the false-negative rate of
identifying pulmonary emboli but indicates
that even experienced readers can miss
a significant percentage of small peripheral
emboli. Our study suggests that both
small short-axis diameter and peripheral
location play a role in false-negative
interpretations.
Given our results, we suggest three
policies regarding subsegmental pulmonary
artery defects and small lesions in any
location.
1. Subsegmental pulmonary artery defects
have a high probability of inaccurate
interpretation. A diagnosis of pulmonary
embolism should be made only when
several readers agree that the lesion(s)
represent pulmonary embolism.
2. Small defects (,6 mm short axis)
identified on lower-quality examinations
have a high probability of inaccurate
interpretation. A diagnosis of pulmonary
embolism should be made only when
several readers agree that the lesion(s)
represent pulmonary embolism.
3. Radiologists should report the quality of
imaging examinations. We believe that
this is best reported by indicating to
what generation of pulmonary arteries
(main, lobar, segmental, subsegmental)
the interpreting radiologist can reliably
confirm or exclude a diagnosis of
pulmonary embolism. For example:
AnnalsATS Volume 12 Number 7 | July 2015
ORIGINAL RESEARCH
“The imaging exam is degraded by
motion artifact, and emboli can only be
reliably interpreted to the level of the
segmental arteries.”
The primary limitation of this study is the
absence of definitive proof of pulmonary
embolism. We have relied on a five-radiologist
panel as the arbiter of “truth.” A second
limitation is the evaluation of a single
community radiology group. It is unknown if
these results apply to practicing radiologists in
general; however, there is no obvious reason,
other than random variation, why this group
would be different from other radiologists.
Our community radiologists were
evaluating clinical cases, whereas our
thoracic radiologists were involved in
a research study. Thus, the consequences of
a false-negative study were higher for the
community radiologists than the thoracic
radiologists, which may have led to a greater
probability of community radiologist calling
a pulmonary artery defect a positive lesion.
Our study did not review the 6,095 CT
pulmonary angiography examinations
interpreted as negative for pulmonary
embolism; therefore, we have no measure of
the incidence of false-negative studies. We
selected this design because it was our intent
to study “overdiagnosis” rather than
“underdiagnosis” of pulmonary embolism.
Many recent studies have demonstrated
that overdiagnosis is likely to be a greater
problem than underdiagnosis in the
interpretation of CT pulmonary
angiograms (2, 6, 7, 9–13).
In conclusion, our study indicates that
there is often lack of agreement among
radiologists concerning subsegmental and/
or small vascular defects in CT pulmonary
angiograms. Furthermore, a review by
a panel of thoracic radiologist specialists
suggests that community radiologists have
a tendency to falsely interpret imaging
artifacts as subsegmental and/or small
pulmonary emboli. A confident diagnosis of
pulmonary embolism for subsegmental or
small pulmonary artery defects should only
be made if multiple readers agree on the
presence of an embolism. Diminished
examination quality, especially as a result
of motion artifact or poor vascular
opacification, increases the likelihood of
inaccurate interpretations for peripheral
and/or small pulmonary embolism. n
Author disclosures are available with the text
of this article at www.atsjournals.org.
 ORIGINAL RESEARCH
References
1 British Thoracic Society Standards of Care Committee Pulmonary
Embolism Guideline Development Group. British Thoracic Society
guidelines for the management of suspected acute pulmonary
embolism. Thorax 2003;58:470–483.
2 Perrier A, Roy PM, Sanchez O, Le Gal G, Meyer G, Gourdier AL,
Furber A, Revel MP, Howarth N, Davido A, et al. Multidetector-row
computed tomography in suspected pulmonary embolism. N Engl J
Med 2005;352:1760–1768.
3 Stein PD, Fowler SE, Goodman LR, Gottschalk A, Hales CA, Hull RD,
Leeper KV Jr, Popovich J Jr, Quinn DA, Sos TA, et al.; PIOPED II
Investigators. Multidetector computed tomography for acute
pulmonary embolism. N Engl J Med 2006;354:2317–2327.
4 R Development Core Team. R: a language and environment for
statistical computing. Vienna, Austria; 2014.
5 Randolph JJ. Free-marginal multirater kappa: an alternative to Fleiss’
fixed marginal multirater kappa. Joensuu University Learning and
Instruction Symposium, 2005.
6 Dahiya A, Bolen M, Grimm RA, Rodriguez LL, Thomas JD, Marwick TH;
AR Concordance Investigators. Development of a consensus
document to improve multireader concordance and accuracy of
aortic regurgitation severity grading by echocardiography versus
cardiac magnetic resonance imaging. Am J Cardiol 2012;110:
709–714.
7 Beigelman C, Chartrand-Lefebvre C, Howarth N, Grenier P. Pitfalls in
diagnosis of pulmonary embolism with helical CT angiography. AJR
Am J Roentgenol 1998;171:579–585.
8 Remy-Jardin M, Remy J, Artaud D, Deschildre F, Fribourg M, Beregi JP.
Spiral CT of pulmonary embolism: technical considerations and
interpretive pitfalls. J Thorac Imaging 1997;12:103–117.
Miller, Marinari, Barbosa, et al.: Small Pulmonary Artery Defects and Predicting PE
9 Wiener RS, Schwartz LM, Woloshin S. Time trends in pulmonary
embolism in the United States: evidence of overdiagnosis. Arch
Intern Med 2011;171:831–837.
10 Ryu JH, Olson EJ, Pellikka PA. Clinical recognition of pulmonary
embolism: problem of unrecognized and asymptomatic cases. Mayo
Clin Proc 1998;73:873–879.
11 Musset D, Parent F, Meyer G, Maı̂tre S, Girard P, Leroyer C, Revel MP,
Carette MF, Laurent M, Charbonnier B, et al.; Evaluation du Scanner
Spiralé dans l’Embolie Pulmonaire study group. Diagnostic strategy
for patients with suspected pulmonary embolism: a prospective
multicentre outcome study. Lancet 2002;360:1914–1920.
12 Ghaye B, Szapiro D, Mastora I, Delannoy V, Duhamel A, Remy J, Remy-
Jardin M. Peripheral pulmonary arteries: how far in the lung does
multi-detector row spiral CT allow analysis? Radiology 2001;219:
629–636.
13 Schoepf UJ, Goldhaber SZ, Costello P. Spiral computed tomography
for acute pulmonary embolism. Circulation 2004;109:2160–2167.
14 Carrier M, Righini M, Wells PS, Perrier A, Anderson DR, Rodger MA,
Pleasance S, Le Gal G. Subsegmental pulmonary embolism diagnosed
by computed tomography: incidence and clinical implications. A
systematic review and meta-analysis of the management outcome
studies. J Thromb Haemost 2010;8:1716–1722.
15 Coche E, Verschuren F, Keyeux A, Goffette P, Goncette L, Hainaut P,
Hammer F, Lavenne E, Zech F, Meert P, et al. Diagnosis of acute
pulmonary embolism in outpatients: comparison of thin-collimation
multi-detector row spiral CT and planar ventilation-perfusion
scintigraphy. Radiology 2003;229:757–765.
16 Eyer B, Goodman L, Washington L. Clinicians’ response to radiologists’
reports of isolated subsegmental pulmonary embolism or
inconclusive interpretation of pulmonary embolism using MDCT.
AJR 2005;184:623–628.
1029