Diagnostic Systematic Reviews: A Road Map (Version 3)

Define a focused 4-part diagnostic review question (Patient/Disease, Index test, Reference standard & Outcomes)
PubMed, Embase, BIOSIS,
Web of Science, CENTRAL,
MEDION, & subject-specific
databases;
Contact authors, experts,
6
companies; citation tracking

Review guidelines on diagnostic reviews1 - 4, and guidelines on primary

studies5 and prepare a protocol

Search directly or via

EndNote; avoid language
restrictions at this stage;
involve a librarian

Identify appropriate databases/sources for diagnostic studies6

Software
suggestions:
9
EndNote , Reference
Manger, ProCite

Run searches on all relevant databases and sources

Use filters for
diagnostic studies
(e.g. PubMed Clinical
Queries filters for
7, 8
diagnosis)

Save all citations (titles/abstracts) in a reference manager9

Document search strategy
These citations are ready for first screen (N0)

Reviewer 1 screens all titles/abstracts and

makes selections for second screen
Software
suggestions:
EndNote, Reference
Manger, ProCite

Need clear inclusion

& exclusion criteria

Reviewer 2 screens all titles/abstracts and

makes selections for second screen

Reviewers meet and resolve disagreements on citations they do not agree on

The final number (N) selected after this process is ready for second screen (full text review)

Excluded after second screen

Keep a log of
excluded studies with
reasons for exclusion

Get full texts of all articles identified for

second screen (N)

Articles considered eligible after full-text review (by two

reviewers) is the final set of studies for inclusion (n0)

Included in the final analysis (ni)

Each article gets a unique ID number

Paper data
extraction forms
10
(after pilot test)

Reviewer 1 extracts data (including quality

assessment) from the final selected articles
Collect outcomes
as TP, FP, FN &
TN; or ROC data

Exploration of
heterogeneity: graphical
methods (e.g. Galbraith
plots), subgroup analyses,
17
and meta-regression
19

Use QUORUM or
20
MOOSE as general
guides for report writing
(acknowledging that they
are not meant for
diagnostic reviews)

Use many overlapping

approaches to get full text
articles (particularly foreign);
takes time and effort!

Excluded from the final

analysis (ne)
Contact authors for
missing data; use email!
Reminders help!!

Reviewer 2 extracts data (including quality

assessment) from the final selected articles

Reviewers meet and resolve disagreements

Compute inter-rater reliability (e.g. Kappa statistic)
The final data after this process is ready for data entry
Enter data into database manager software

Software suggestions:
Access, Excel

Avoid printing
citations at this stage

Import data and analyze using software13, 14, 15

Tabulate study characteristics
Forest plots of sensitivity and specificity
Look for correlation between TPR and FPR
Search for threshold effect; perform SROC analyses16
Search for heterogeneity, and reasons for heterogeneity17
Pool Se, Sp, LR and DOR if appropriate
Consider subgroup and sensitivity analyses
Explore possibility of publication bias18

Interpret & discuss results;

Discuss generalizability of results, and limitations of the review
Make recommendations for practice or policy, and research

Consider blinded data

extraction (hiding author
names, etc.)
Quality criteria: study design,
blinding, verification,
sampling, test description,
appropriate reference standard,
prospective data collection and
1, 2, 11, 12
others.
Software suggestions:
13
Meta-Test for forest plots,
pooling Se/Sp and SROC;
14, 15
for pooling DOR and
Stata
likelihood ratios, and metaregression

You made it!

Celebrate!!!

Madhukar Pai, Berkeley Systematic Reviews Group: www.medepi.org/meta October 2003 [madhupai@uclink.berkeley.edu]

Diagnostic Systematic Reviews: A Road Map (Version 3): Links and References
1.
2.
3.
4.
5.

6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.

Cochrane Methods Group on Systematic Review of Screening and Diagnostic Tests: Recommended
Methods. Cochrane Collaboration, 1996. http://www.cochrane.org/cochrane/sadtdoc1.htm.
Deville WL, Buntinx F, Bouter LM, Montori VM, De Vet HC, Van Der Windt DA, Bezemer P.
Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Med Res Methodol
2002; 2(1): 9. http://www.biomedcentral.com/1471-2288/2/9
Deeks JJ. Systematic reviews of evaluations of diagnostic and screening tests. In: Egger M, Smith GD,
Altman DG, eds. Systematic reviews in health care. Meta-analysis in context. London: BMJ Publishing
Group, 2001: 248282. www.systematicreviews.com
Irwig L, Macaskill P, Glasziou P, Fahey M. Meta-analytic methods for diagnostic test accuracy. J Clin
Epidemiol 1995; 48: 11930.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D,
Rennie D, de Vet HC; Standards for Reporting of Diagnostic Accuracy steering group. Towards
complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ 2003;
326: 4144. http://bmj.bmjjournals.com/cgi/content/full/326/7379/41
Finding studies for systematic reviews: a basic checklist for researchers. NHS Center for Reviews and
Dissemination, University of York: http://www.york.ac.uk/inst/crd/revs.htm
Pai M. Cheat sheet for using search filters in PubMed. Berkeley Systematic Reviews Group:
http://www.medepi.org/meta/litsearch.html
McKibbon A, et al. PDQ Evidence-based Principles and Practice. Hamilton: BC Decker, 1999.
EndNote Tip Sheet: Importing References into EndNote:
http://www.endnote.com/support/helpdocs/EN7-ImportRefs.pdf
Sample data extraction form for diagnostic reviews: Berkeley Systematic Reviews Group:
http://www.medepi.org/meta/forms.html
Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM.
Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999; 282: 10616.
Jaeschke R, Guyatt G, Lijmer J. Diagnostic tests. In: Guyatt G, Rennie D, eds. Users guides to the
medical literature. A manual for evidence-based clinical practice. Chicago: AMA Press, 2002: 121
140. http://www.usersguides.org/
Lau J. Meta-Test version 0.6. New England Medical Center, Boston, 1997. Available at
http://www.cochrane.org/cochrane/sadt.htm.
Sterne JAC, Bradburn MJ, Egger M. Meta-analysis in Stata. In: Egger M, Smith GD, Altman DG, eds.
Systematic reviews in health care. Meta-analysis in context. London: BMJ Publishing Group, 2001:
347369. http://www.bmjpg.com/books/sysrev/chapter18.pdf
Pai M, McCulloch M, Colford JM. Meta-analysis commands in Stata. Berkeley Systematic Reviews
Group: http://www.medepi.org/meta/softwar.html
Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new
meta-analytic method. Med Decis Making 1993; 13: 313321.
Lijmer JG, Bossuyt PM, Heisterkamp SH. Exploring sources of heterogeneity in systematic reviews of
diagnostic tests. Stat Med 2002; 21: 152537.
Song F, Khan KS, Dinnes J, Sutton AJ. Asymmetric funnel plots and publication bias in meta-analyses
of diagnostic accuracy. Int J Epidemiol 2002; 31: 8895.
Moher D, Cook DJ, Eastwood S et al. Improving the quality of reports of meta-analyses of randomized
controlled trials: the QUORUM statement. Lancet 1999;354:1896-1900. http://www.consortstatement.org/
Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a
proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
JAMA 2000 Apr 19;283(15):2008-12. http://www.consort-statement.org/

Websites with useful resources for conducting diagnostic reviews:

Cochrane Group on Screening and Diagnostic Tests: http://www.cochrane.org/cochrane/sadtdoc1.htm

Berkeley Systematic Reviews Group: www.medepi.org/meta
MEDION Database of diagnostic reviews: http://www.mediondatabase.nl/