3/7/2016

www.medscape.com/viewarticle/706399_print

www.medscape.com

HowtoCriticallyAppraiseanArticle
JaneMYoungMichaelJSolomon
NatClinPractGastroenterolHepatol.20096(2):8291.

SummaryandIntroduction
Summary

Criticalappraisalisasystematicprocessusedtoidentifythestrengthsandweaknessesofaresearcharticleinorderto
assesstheusefulnessandvalidityofresearchfindings.Themostimportantcomponentsofacriticalappraisalarean
evaluationoftheappropriatenessofthestudydesignfortheresearchquestionandacarefulassessmentofthekey
methodologicalfeaturesofthisdesign.Otherfactorsthatalsoshouldbeconsideredincludethesuitabilityofthe
statisticalmethodsusedandtheirsubsequentinterpretation,potentialconflictsofinterestandtherelevanceofthe
researchtoone'sownpractice.ThisReviewpresentsa10stepguidetocriticalappraisalthataimstoassistclinicians
toidentifythemostrelevanthighqualitystudiesavailabletoguidetheirclinicalpractice.
Introduction

Topracticeevidencebasedmedicine,cliniciansneedtoapplythefindingsofscientificresearchtothecircumstancesof
individualpatientsaspartoftheirclinicaldecisionmakingprocess.Clinicians,therefore,mustbeabletoselectand
appraisescientificliteraturethatisrelevanttotheirfield,understandtheimplicationsofresearchfindingsforindividual
patients,elicitpatients'ownpreferencesanddevelopanappropriatemanagementplanbasedonthecombinationof
thisinformation.Eachofthesetaskspresentsitsownchallenges,butthesheervolumeofmedicalliteraturemeans
thatthefirststep(thatofselectingandappraisingscientificevidence)canbedaunting.Thenumberofnewmedical
researcharticlespublishedeachyearcontinuallyincreases,andmorethan12,000newarticles,includingpapersonin
excessof300randomizedcontrolledtrials(RCTs),areaddedtotheMEDLINEdatabaseeachweek. [1,2]Onepractical
waythatclinicianscanmanagethis'informationoverload'[2]istodevelopefficientskillsincriticalappraisal,which
enablethemfocusononlythehighestqualitystudiesthatwillguidetheirclinicalpracticeandtoextrapolate
informationwhennecessaryfromstudiesoflessrigorousdesignifhighqualitytrialsareunavailable.
Criticalappraisalhasbeendefinedasthe"...applicationofrulesofevidencetoastudytoassessthevalidityofthe
data,completenessofreporting,methodsandprocedures,conclusions,compliancewithethicalstandards,etc.The
rulesofevidencevarywithcircumstances." [3]Althoughthemethodologicalcriteriabywhichthevalidityofastudyis
assessedwillvaryaccordingtoitsdesign,somegeneralprinciplesunderpintheevaluationofanyresearchstudy.
Variousguidelinesandassessmenttoolshavebeendevelopedtoprovideastructuredapproachtotheprocessof
criticalappraisalforclinicians. [414]
Despitetheplethoraofdocumentsavailabletoguidetheprocess,no'goldstandard'instrumentforcriticalappraisal
exists.Thecriteriausedtoassessthevalidityandrelevanceofscientificliteraturearenotstatictheymustevolvewith
improvementsinunderstandingoftheimportantsourcesofbiasinherentindifferentstudydesigns,andincreased
awarenessofthepotentialinfluenceofothernonmethodologicalfactors,suchasconflictsofinterest. [15]Astructured
approachtocriticalappraisalcouldpotentiallyimprovethequalityofthisprocess,andsimplechecklistscanbeuseful
toscreenoutresearchthatisoflowqualityoroflittlerelevance. [16]ThisReviewpresentsaguidetothecritical
appraisalprocess.

SelectionandCriticalAppraisalofResearchLiterature
Tenkeyquestions()canbeusedtoassessthevalidityandrelevanceofaresearcharticle.Thesequestionscanassist
clinicianstoidentifythemostrelevant,highqualitystudiesthatareavailabletoguidetheirclinicalpractice.

http://www.medscape.com/viewarticle/706399_print

1/14

3/7/2016

www.medscape.com/viewarticle/706399_print

Box1.TenQuestionstoAskwhenCriticallyAppraisingaResearchArticle

Isthestudyquestionrelevant?
Doesthestudyaddanythingnew?
Whattypeofresearchquestionisbeingasked?
Wasthestudydesignappropriatefortheresearchquestion?
Didthestudymethodsaddressthemostimportantpotentialsourcesofbias?
Wasthestudyperformedaccordingtotheoriginalprotocol?
Doesthestudytestastatedhypothesis?
Werethestatisticalanalysesperformedcorrectly?
Dothedatajustifytheconclusions?
Arethereanyconflictsofinterest?
IstheStudy'sResearchQuestionRelevant?

Evenifastudyisofthehighestmethodologicalrigor,itisoflittlevalueunlessitaddressesanimportanttopicand
addstowhatisalreadyknownaboutthatsubject. [17]Theassessmentofwhethertheresearchquestionisrelevantis
inevitablybasedonsubjectiveopinion,aswhatmightbecrucialtosomewillbeirrelevanttoothers.Nonetheless,the
firstquestiontoaskofanyresearcharticleiswhetheritstopicisrelevanttoone'sownfieldofwork.
DoestheStudyAddAnythingNew?

Scientificresearchendeavorisoftenlikenedto'standingontheshouldersofgiants',becausenewideasandknowledge
aredevelopedonthebasisofpreviouswork. [18]Seminalresearchpapersthatmakeasubstantivenewcontributionto
knowledgearearelativerarity,butresearchthatmakesanincrementaladvancecanalsobeofvalue.Forexample,a
studymightincreaseconfidenceinthevalidityofpreviousresearchbyreplicatingitsfindings,ormightenhancethe
abilitytogeneralizeastudybyextendingtheoriginalresearchfindingstoanewpopulationofpatientsorclinical
context. [17]
WhatTypeofResearchQuestionDoestheStudyPose?

Themostfundamentaltaskofcriticalappraisalistoidentifythespecificresearchquestionthatanarticleaddresses,as
thisprocesswilldeterminetheoptimalstudydesignandhaveamajorbearingontheimportanceandrelevanceofthe
findings.Awelldevelopedresearchquestionusuallyidentifiesthreecomponents:thegrouporpopulationofpatients,
thestudiedparameter(e.g.atherapyorclinicalintervention)andtheoutcomesofinterest. [10]Ingeneral,clinical
researchquestionsfallintotwodistinctcategories,below.
QuestionsAbouttheEffectivenessofTreatment.Thesetypesofquestionsrelatetowhetheronetreatmentis
betterthananotherintermsofclinicaleffectiveness(benefitandharm)orcosteffectiveness.
QuestionsAbouttheFrequencyofEvents.Suchquestionsrefertotheincidenceorprevalenceofdiseaseorother
clinicalphenomena,riskfactors,diagnosis,prognosisorpredictionofspecificclinicaloutcomesandinvestigationson
thequalityofhealthcare.
http://www.medscape.com/viewarticle/706399_print

2/14

3/7/2016

www.medscape.com/viewarticle/706399_print

WastheStudyDesignAppropriatefortheResearchQuestion?

Studiesthatanswerquestionsabouteffectivenesshaveawellestablishedhierarchyofstudydesignsbasedonthe
degreetowhichthedesignprotectsagainstbias.MetaanalysesofwellconductedRCTsandindividualRCTsprovide
themostrobustevidencefollowedbynonrandomizedcontrolledtrials,cohortstudies,casecontrolstudies,andother
observationalstudydesigns. [19,20]However,insomecircumstances,RCTsareeithernotfeasibleorconsidered
ethicallyinappropriate.Theseissuesaremorecommoninnonpharmaceuticaltrials,suchasthoseofsurgical
procedures.Onereviewofgastrointestinalsurgicalresearchfoundthatonly40%ofresearchquestionscouldhave
beenansweredbyanRCT,evenwhenfundingwasnotanimpediment.Patients'preferences,therarityofsome
conditions,andtheabsenceofequipoiseamongsurgeonsprovedtobethemajorobstaclestoperformingRCTsof
gastrointestinalsurgeryinthissetting. [21]WhenanRCTisnotfeasible,thespecificreasonsthatprecludeitsusewill
determinethetypeofalternatestudydesignthatcanbeused. [21]Observationalstudies,ratherthanRCTs,arethe
mostappropriatestudydesignforresearchquestionsonthefrequencyofevents.
DidtheStudyMethodsAddresstheKeyPotentialSourcesofBias?

Inepidemiologicalterms,thepresenceofbiasdoesnotimplyapreconceptiononthepartoftheresearcher,butrather
meansthattheresultsofastudyhavedeviatedfromthetruth. [3]Biascanbeattributedtochance(e.g.arandom
error)ortothestudymethods(systematicbias).Randomerrordoesnotinfluencetheresultsinanyparticulardirection,
butitwillaffecttheprecisionofthestudy [22]bycontrast,systematicbiashasadirectionandresultsinthe
overestimationorunderestimationofthe'truth'.Systematicbiasesarisefromthewayinwhichthestudyisconducted,
beithowstudyparticipantswereselected,howdatawascollected,orthroughtheresearchers'analysisor
interpretation. [23]
Differentstudydesignsarepronetovaryingsourcesofsystematicbias.Oncethestudydesignofagivenarticlehas
beenidentified,werecommendthatcliniciansuseoneoftheavailabledesignspecificcriticalappraisalcheckliststo
decidewhetherthestudyinquestionisofhighquality.TheCriticalAppraisalSkillsProgramme(CASP)includessuch
toolsandtheprogramcoordinatorshavedevelopedseparatechecklistsfortheappraisalofsystematicreviews,RCTs,
cohortstudies,casecontrolstudies,diagnosticteststudies,economicevaluationsandqualitativeresearchthateach
comprise10questions. [9]TheyhavebeendevelopedfromtheUsers'guidestothemedicalliteratureseriesofarticles
thatwereoriginallypublishedintheJournaloftheAmericanMedicalAssociation.Thesearticlesarenowavailablein
bookform[5]andarereadilyaccessibleontheinternet. [9]
SystematicReviewsandMetaanalyses

Ameticulous,standardizedprotocolisusedinasystematicreviewtoidentify,criticallyappraiseandsynthesizeallthe
relevantstudiesonaparticulartopic.Somesystematicreviewsmaythenproceedtoametaanalysis,inwhichthe
resultsfromindividualstudiesarecombinedstatisticallytoproduceasinglepooledresult. [3]Althoughplanningto
undertakeasystematicrevieworametaanalysisprospectivelyispossible, [24]themajorityofthesetypesofarticleare
retrospectiveandariskofbiasexists,whicharisesfromtheselectionofstudiesandthequalityoftheseprimary
sources. [25]Publicationbias,whichresultsfromtheselectivepublicationofstudieswithpositivefindings,isof
particularconcern,asitdistortsoverallperceptionsofthefindingsonaparticulartopic. [26,27]
TheQUORUM(QualityofReportingofMetaAnalyses)statementprovidesacomprehensiveframeworkfor
assessmentsofthequalityofreportinginmetaanalysesandsystematicreviews. [25,28]Inaddition,theAMSTAR[29]
assessmenttool,whichcomprises11questions,hasbeendevelopedfortheappraisalofsystematicreviews,andthis
toolortheCASPchecklist [9]couldbemoreusefulthantheQUORUMstatementforclinicianswhowishtoundertakea
rapidappraisalofthesetypesofarticles.Keymethodologicalpointstoconsiderintheappraisalofsystematicreviews
andmetaanalysesarelistedin.
Box2.KeyMethodologicalPointstoConsiderintheAppraisalofSystematicReviewsandMetaanalyses

Wereallrelevantstudiesincluded(i.e.wasthesearchcomprehensive,diditexcludearticlesonthebasisof
http://www.medscape.com/viewarticle/706399_print

3/14

3/7/2016

www.medscape.com/viewarticle/706399_print

publicationstatusorlanguageandwasthepotentialforpublicationbiasassessed)?
Wereselectedarticlesappraisedanddataextractedbytwoindependentreviewers?
Wassufficientdetailprovidedabouttheprimarystudies,includingdescriptionsofthepatients,interventionsand
outcomes?
Wasthequalityoftheprimarystudiesassessed?
Didtheresearchersassesstheappropriatenessofcombiningresultstocalculateasummarymeasure?
SystematicreviewsandmetaanalysesarenotrestrictedtoRCTsalone.TheMOOSE(MetaAnalysisOfObservational
StudiesinEpidemiology)guidelineshavebeendevelopedasacorollaryoftheQUORUMstatementformetaanalyses
ofnonRCTs. [30]
RandomizedControlledTrials

InanRCT,therandomallocationofparticipantsshouldensurethattreatmentgroupsareequivalentintermsofboth
knownandunknownconfoundingfactorsanydifferencesinoutcomesbetweengroupscan,therefore,beascribedto
theeffectoftreatment. [31]Studydesignalone,however,willnotguardagainstbiasifcrucialaspectsofthestudy
protocolaresuboptimal.Thepotentialforselectiveenrollmentofpatientsintothestudycanbeoneanimportant
sourceofbiasifthegrouptowhichindividualswillbeallocatedisknownorcanbeguessed. [32]Centralizedmethodsof
randomization,forexampleacomputergeneratedallocation,arepreferabletolessconcealedmethods,suchasuseof
colorcodedformsorpseudorandomsequencesbasedonmedicalrecordnumbersordaysoftheweek. [31]Failureto
concealtheallocationsequencehasbeenshowntoresultinagreaterdistortionoftheresultsthanlackofdouble
blindinganothermajorsourceofbiasinRCTs. [33]
TheCONSORT(ConsolidatedStandardsofReportingTrials)statementflowchart(Figure1)isfunctionallyequivalent
totheQUORUMstatementforsystematicreviews,andprovidesacomprehensivetoolwithwhichtoassessthe
standardofreportinginrandomizedtrials. [34]KeypointstoconsiderintheappraisalofanRCTarelistedin.
Box3.KeyMethodologicalPointstoConsiderintheAppraisalofRandomizedControlledTrials

Wastheprocessoftreatmentallocationtrulyrandom?
Wouldparticipantshavebeenabletoknoworguesstheirtreatmentallocation?
Wereparticipantsandresearchers'blinded'toparticipants'treatmentgroup?
Wereoutcomesassessedobjectively?
Wereallparticipantswhowererandomlyallocatedatreatmentaccountedforinthefinalanalysis?
Wereallparticipants'dataanalyzedinthegrouptowhichtheywererandomlyallocated?a
aSeesectiononintentiontotreatanalysisunder'Werethestatisticalanalysesperformedcorrectly?'

http://www.medscape.com/viewarticle/706399_print

4/14

3/7/2016

www.medscape.com/viewarticle/706399_print

Figure1.

Consolidatedstandardsofreportingtrials(CONSORT)statementflowchartforthestandardreportingandappraisalof
randomizedcontrolledtrials.WithpermissionfromCONSORT
CohortStudies

Cohort,orlongitudinal,studiesinvolvefollowinguptwoormoregroupsofpatientstoobservewhodevelopsthe
outcomeofinterest.Prospectivecohortstudieshavebeenlikenedtonaturalexperiments,asoutcomesaremeasured
inlargegroupsofindividualsoverextendedperiodsoftimeintherealworld. [35]Cohortstudiescanalsobeperformed
retrospectivelysuchstudiesusuallyinvolveidentifyingagroupofpatientsandfollowinguptheirprogressbyexamining
recordsthathavebeencollectedroutinelyorforanotherpurpose,suchasmedicaldata,deathregistryrecordsand
hospitaladmissiondatabases.
Themajormethodologicalconcernwithcohortstudiesistheirhighpotentialforselectionbiasandconfoundingfactors.
Theseproblemsareparticularlyrelevantwhencohortstudies(ornonRCTs)areusedtoevaluatetherapeutic
interventions.Inthissituation,thetreatmentthatsomeonereceivesisdeterminedbythepatient'sorclinician's
preferences,referralpatterns,currenttreatmentparadigmsorlocalpolicy. [36]Importantdifferencesarelikelytoexist
betweenpatientswhoreceivedisparatetreatmentsandthesedifferences,ratherthanthetreatmentitself,mightbe
responsiblefortheobservedoutcomes.Althoughsomepotentialconfoundingfactorscanbemeasuredandaccounted
[37]

forintheanalysis, suchadjustmentsaremoredifficultinretrospectivethanprospectivestudies,asdataon
http://www.medscape.com/viewarticle/706399_print

5/14

3/7/2016

www.medscape.com/viewarticle/706399_print

forintheanalysis, [37]suchadjustmentsaremoredifficultinretrospectivethanprospectivestudies,asdataon
importantpotentialconfoundersmightnothavebeencollected,ormightbeofpoorquality.
TheSTROBE(StrengtheningtheReportingofObservationalStudiesinEpidemiology)statementisthecorollaryofthe
QUORUMandCONSORTstatementsforobservationalstudies,includingcohort,casecontrolandcrosssectional
studies. [38]Keymethodologicalfeaturestoconsiderintheappraisalofcohortstudiesarelistedin.
Box4.KeyMethodologicalPointstoConsiderintheAppraisalofaCohortStudy

Isthestudyprospectiveorretrospective?
Isthecohortrepresentativeofadefinedgrouporpopulation?
Wereallimportantconfoundingfactorsidentified?
Wereallimportantexposuresand/ortreatments,potentialconfoundingfactorsandoutcomesmeasuredaccurately
andobjectivelyinallmembersofthecohort?
Werethereimportantlossestofollowup?
Wereparticipantsfollowedupforasufficientlengthoftime?
CasecontrolStudies

Casecontrolstudiesarealwaysretrospectivebytheirverynaturethecasepatientsareselectedbecausetheyhave
alreadydevelopedtheoutcomeofinterest(e.g.adisease).Dataarethencollectedaboutfactorsthatmighthave
influencedthisoutcome,andtheseexposuresarecomparedwiththoseofagroupofpeoplewhodifferfromthecase
patientsonlyinthattheyhavenotdevelopedtheoutcomeofinterest.Casecontrolstudiesareidealforthe
investigationofriskfactorswhentheoutcomeofinterestisrare,asitwouldtaketoolongtorecruitaprospective
cohort.
Majormethodologicaldifficultieswithcasecontrolstudiesaretheselectionofappropriatecontrolindividualsandthe
possibilityof'recallbias'(apatient'ssubjectiveinterpretationofwhatcausedtheirconditioncanaltertheirrecallof
certaineventsorexperiences).Controlsshouldbedrawnfromexactlythesamepopulationasthecases,andtheonly
differencebetweencontrolsandcasesshouldbethatthecontrolshavenotdevelopedtheconditionofinterest.
Althoughobjectivemeasuresofpossiblecausativefactorsarepreferable,casecontrolstudiesoftenrelyonparticipants'
recall,andpatientsmightbemorelikelytoremembercertaineventsorexperiencesthancontrols. [39]Keyaspectsto
considerwhenassessingacasecontrolstudyarelistedin.
Box5.KeyMethodologicalPointstoConsiderintheAppraisalofaCaseControlStudy

Werethecasesclearlydefined?
Werethecasesrepresentativeofadefinedpopulation?
Howwerethecontrolsselectedandweretheydrawnfromthesamepopulationasthecases?
Werestudymeasuresidenticalforcasesandcontrols?
Werestudymeasuresobjectiveorsubjectiveandisrecallbiaslikelyiftheyweresubjective?

http://www.medscape.com/viewarticle/706399_print

6/14

3/7/2016

www.medscape.com/viewarticle/706399_print

CrosssectionalAnalyses

Crosssectionalstudiesprovidea'snapshot'inwhichallparameters(exposuresandoutcomes)areassessedatthe
sametimeexamplesofcrosssectionaldesignsincludeoneoffsurveysandauditsofpractice.Keymethodological
pointstoconsiderintheappraisalofacrosssectionalstudyarelistedin.
Box6.KeyMethodologicalPointstoConsiderintheAppraisalofaCrosssectionalStudy

Wasthestudysampleclearlydefined?
Wasarepresentativesampleachieved(e.g.wastheresponseratesufficientlyhigh)?
Wereallrelevantexposures,potentialconfoundingfactorsandoutcomesmeasuredaccurately?
Werepatientswithawiderangeofseverityofdiseaseassessed?
CaseSeries

Caseseriesprovidelowlevelevidenceabouttherapeuticeffectivenesshowever,thesearticlesareverycommonin
medicalliterature.Keymethodologicalissuestoconsiderwhenassessingsucharticlesarelistedin.
Box7.KeyMethodologicalPointstoConsiderintheAppraisalofaCaseStudy

Werecasesidentifiedprospectivelyorretrospectively?
Arethecasesarepresentativesample(e.g.aconsecutiveseriesofindividualsrecruitedfrommultiplecenters)
andsimilartopatientsinyourpractice?
Wereallrelevantexposures,potentialconfoundingfactorsandoutcomesmeasuredaccurately?
StudiesthatAssesstheAccuracyofDiagnosticTests

Thesestudiesareusuallycrosssectionalindesign,butpossessanumberofspecificmethodologicalissuesthatshould
beconsideredinadditiontothosenotedabove. [40]Toinvestigatetheaccuracyofadiagnostictest,itisperformedona
sampleofpatientsandtheresultsarecomparedwiththoseofareferenceorgoldstandarddiagnostictest. [41]The
levelofagreementbetweentheinvestigatedtestandthegoldstandarddiagnostictestcanthenbereportedeitherin
termsofthesensitivityandspecificity,orlikelihoodratio. [4,41]
TheSTARD(StandardsfortheReportingofDiagnosticAccuracyStudies)websiteprovidesadetailedflowchart(Figure
2)and25itemchecklistforstandardizedreportingandappraisalofstudiesthatassesstheaccuracyofdiagnostictests.
[42,43]TheCASPalsoprovidesasimilar,butmoresimple,toolforthistypeofstudy. [9]Importantfeaturestoconsider
whenappraisingastudyofdiagnosticaccuracyarelistedin.
Box8.KeyMethodologicalPointstoConsiderintheAppraisalofaStudyofDiagnosticAccuracy

Doesthesampleofpatientsrepresentthefullspectrumofpatientswithandwithoutthediagnosisofinterest?
Wasthereacomparisonwithanappropriate'goldstandard'test?
Didallpatientsreceiveboththetestunderevaluationandthesame'goldstandard'test?
Werethetestsperformedindependentlywithblindingofassessorstotheresultsofthe'goldstandard'test?
http://www.medscape.com/viewarticle/706399_print

7/14

3/7/2016

www.medscape.com/viewarticle/706399_print

Werethecutoffsthatwereusedtoclassifypatientsashavingapositivetestresultclearlydescribed?

Figure2.

Standardsforthereportingofdiagnosticaccuracystudies(STARD)statementflowchartforthestandardreportingand
appraisalofstudiesexaminingtheaccuracyofdiagnostictests.WithpermissionfromSTARD
EconomicEvaluations

Economicevaluationstudiesfocusoncostefficiency,orwhichtreatmentcanprovidethegreatestbenefitfortheleast
cost. [44]Severaltypesofeconomicevaluationstudiesexist,includingcostbenefit,costeffectivenessandcostutility
analyses,allofwhichdifferinhowtheymeasurehealthbenefits. [45]Animportantfeatureofcriticalappraisalofany
costanalysisisanassessmentofhowwellthevariouscostsandconsequencesofindividualtreatmentshavebeen
identifiedandmeasured.TheCASPhasdevelopedachecklisttoaidwiththeappraisalofeconomicevaluation
studies. [9]
WastheStudyPerformedinLinewiththeOriginalProtocol?

Deviationsfromtheplannedprotocolcanaffectthevalidityorrelevanceofastudy.Oneofthemostcommonproblems
encounteredinclinicalresearchisthefailuretorecruittheplannednumberofparticipants.Anestimatesuggeststhat
morethanathirdofRCTsrecruitlessthan75%oftheirplannedsample. [46]Thisdeviationfromthestudyplannotonly
http://www.medscape.com/viewarticle/706399_print

8/14

3/7/2016

www.medscape.com/viewarticle/706399_print

potentiallyreducestheextenttowhichtheresultsofthestudycanbegeneralizedtorealworldsituations,because
thosewhoactuallywererecruitedmightbedifferentfromthosewhoweren'tforsomereason,butalsoreducesthe
powerofthestudytodemonstratesignificantfindings.Otherdifferencestotheoriginalprotocolmightincludechanges
totheinclusionandexclusioncriteria,variationintheprovidedtreatmentsorinterventions,changestotheemployed
techniquesortechnologies,andchangestothedurationoffollowup.
DoestheStudyTestaStatedHypothesis?

Ahypothesisisaclearstatementofwhattheinvestigatorsexpectthestudytofindandiscentraltoanyresearchasit
statestheresearchquestioninaformthatcanbetestedandrefuted. [3]Anullhypothesisstatesthatthefindingsofa
studyarenodifferenttothosethatwouldhavebeenexpectedtooccurbychance.Statisticalhypothesistesting
involvescalculatingtheprobabilityofachievingtheobservedresultsifthenullhypothesisweretrue.Ifthisprobabilityis
low(conventionallylessthan1:20orP<0.05),thenullhypothesisisrejectedandthefindingsaresaidtobe
'statisticallysignificant'atthatacceptedlevel.
Studyhypothesesmustcruciallybeidentifiedapriori(thatis,beforethestudyisconducted,andaredevelopedfrom
theoryorpreviousexperience).Ifthestudyinvestigatesthestatisticalsignificanceofassociationsthatwerenot
prespecifiedintheoriginalhypothesis(posthocanalysis),suchanalysesarepronetofalsepositivefindingsbecause,
atasignificancelevelof5%(P=0.05),1in20associationstestedwillbesignificant(positive)bychancealone.When
alargenumberofsuchtestsareconductedsomefalsepositiveresultsarehighlylikelytooccur.Anotherimportant
considerationittocheckthatalldatarelevanttothestatedstudyobjectiveshavebeenreported,andthatselected
outcomeshavenotbeenomitted.
Wheretreatmentsforamedicalconditionalreadyexist,trialscanbedesignedtotestwhetheranewtherapyhas
similarefficacytoanexistingone.Thistypeoftrialiscalledanequivalenceornoninferioritytrial,asitspurposeisto
establishthatthenewtreatmentisnoworsethantheexistingone. [47]Equivalencestudiesrequirethatthedegreeof
outcomedifferenceatwhichthetwotreatmentswillnotbeconsideredequivalentbedeterminedinadvance. [48]For
example,researchersmightdecidethatiftheprimaryoutcomeforanewtreatmentisnogreaterthan5%worsethan
thatoftheexistingtreatment,thetwotreatmentswillbeconsideredtobeequivalent.Equivalencestudiesdetermine
whetheranewtreatmentisatleastasgoodasanexistingtreatmentsothatdecisionsaboutwhichtreatmentto
administertoagivenpatientcanbemadeonthebasisofcriteria,suchascostoreaseofadministration. [47,48]
TheCONSORTstatementforrandomizedtrialshasbeenextendedtoincorporateguidelinesforreportingequivalence
studies. [49]Akeyquestionwhenappraisingthistypeofstudyiswhetherthetrialresultswereanalyzedappropriately
foranequivalencestudy.Ifastudyisdesignedtoshowthatanewtreatmentisatleastasgoodasanexisting
treatment,statisticalmethods,forconventionaltestingofahypothesisthatonetreatmentissuperiortoanothershould
notbeused.Appropriateanalysisoftheresultsinanequivalencestudyofteninvolvescalculatingconfidenceintervals
forthetreatmenteffect,anddeterminingwhethertheselimitsarewithinthepredeterminedmarginofnoninferiority. [48]
Anotherkeyquestioniswhetherthesamplesizewascalculatedcorrectlyforanequivalencestudy,asthesetypesof
studyusuallyrequirealargersamplesizethanacorrespondingsuperioritytrial. [49]
WeretheStatisticalAnalysesPerformedCorrectly?

Assessingtheappropriatenessofstatisticalanalysescanbedifficultfornonstatisticians.However,allquantitative
researcharticlesshouldincludeasegmentwithintheir'Method'sectionthatexplainsthetoolsusedinthestatistical
analysisandtherationaleforthisapproach,whichshouldbewrittenintermsthatareappropriateforthejournal's
readership.Inparticular,theapproachtodealingwithmissingdataandthestatisticaltechniquesthathavebeen
appliedshouldbespecifiedpatientswhoarelostinfollowupandmissingdatashouldbeclearlyidentifiedinthe
'Results'section.Originaldatashouldbepresentedinsuchawaythatreaderscancheckthestatisticalaccuracyofthe
paper.
AnimportantconsiderationinthestatisticalanalysisofRCTsiswhetherintentiontotreat(ITT)orperprotocolanalyses
http://www.medscape.com/viewarticle/706399_print

9/14

3/7/2016

www.medscape.com/viewarticle/706399_print

wereconducted.AccordingtotheITTprinciple,participants'dataareanalyzedwithreferencetothegrouptowhich
theywererandomlyallocated,regardlessofwhethertheyactuallyreceivedtheallocatedtreatment.ITTanalysesare
preferred,becausetheymaintaintherandomizationandensurethatthetwotreatmentgroupsarecomparableat
baseline. [50]However,ifalotofparticipantsarenonadherantoralargeproportioncrossovertoothertreatments,an
ITTanalysiswillbesomewhatconservativeandtheresultsmightbedifficulttointerpret.Inthissituation,aper
protocolanalysisthatincludesonlythosepatientswhocompliedwiththetrialprotocolcanbeusedtosupplementthe
ITTanalysis.Asperprotocolanalysesareatincreasedriskofselectionbias,theyshouldnotusuallybeusedasthe
primarymethodofanalysisunlessacompellingreasonexiststojustifythisapproach. [50]TheCONSORTflowchart
(Figure1)enablestheflowofparticipantsandthegroupsusedintheanalysisofthetrialtobeclearlyidentified. [34]
DotheDataJustifytheConclusions?

Thenextconsiderationiswhethertheconclusionsthattheauthorspresentarereasonableonthebasisofthe
accumulateddata.Sometimesanoveremphasisisplacedonstatisticallysignificantfindingsthatinvokedifferences
thataretoosmalltobeofclinicalvaluealternatively,someresearchersmightdismisslargeandpotentiallyimportant
differencesbetweengroupsthatarenotstatisticallysignificant,oftenbecausesamplesizesweresmall.Otherissuesto
bewaryofarewhethertheauthorsgeneralizedtheirfindingstobroadergroupsofpatientsorcontextsthanwas
reasonablegiventheirstudysample,andwhetherstatisticallysignificantassociationshavebeenmisinterpretedto
implyacauseandeffect.
AreThereanyConflictsofInterest?

Conflictsofinterestoccurwhenpersonalfactorshavethepotentialtoinfluenceprofessionalrolesorresponsibilities. [51]
Membersofaresearchteammustmakejudgmentsthathavethepotentialtoaffectthesafetyoftheparticipantsand
thevalidityoftheresearchfindings.Researchersareinapositiontodecidewhichstudieswillbeconductedintheir
unit,whichpatientswillbeinvitedtoparticipateinastudyandwhethercertainclinicaloccurrencesshouldbereported
asadverseevents. [52]Thesedecisionsrequireresearcherstoactwithintegrityandnotforpersonalorinstitutionalgain.
Potentialfinancialconflictsofinterestincludethereceiptofsalaryandconsultationfeesfromthecompanythathas
sponsoredtheresearchandownershipofstocksandsharesorotherpecuniaryinterests,suchaspatentsrelatedtothe
research. [52]Unitsthatrecruitresearchparticipantsmightbepaidapercapitafeeforeverypatientenrolled,whichcan
begreaterthantheexpensesinvolved. [53]Manypotentialfinancialsourcesofconflictsofinterest,suchasindustry
fundingforeducationalevents,travelorgifts,areincreasinglyrecognizedbothwithinthecontextofdailyclinical
practiceandresearch. [54]However,otherpotentialconflictsareinherenttotheresearchsetting.Anexampleisthat
medicalresearchers'statusandfutureresearchincomeisdependentonthesuccessoftheirresearch. [55]
Identificationofapotentialconflictofinterestisnotsynonymouswithhavinganactualconflictofinterestorpoor
researchpractice.Potentialconflictsofinterestareextremelycommon,andthemostimportantquestionsarewhether
theyhavebeenrecognizedandhowtheyhavebeendealtwith. [56]Amainmechanismfordealingwithpotential
conflictsofinterestisopendisclosure. [56]Intheprocessofcriticallyappraisingaresearcharticle,oneimportantstepis
tocheckforadeclarationaboutthesourceoffundingforthestudyand,ifapotentialconflictofinteresthadbeen
identifiedforastatementabouthowthisconflictwasmanaged.Forexample,theresearchersmightstatespecifically
thatthesponsoringagencyhadnoinputintotheresearchprotocol,dataanalysisorinterpretationofthefindings.Many
journalsnowroutinelyrequireauthorstodeclareanypotentialfinancialorotherconflictsofinterestwhenanarticleis
submitted.Thereadermustthendecidewhetherthedeclaredfactorsareimportantandmighthaveinfluencedthe
validityofthestudy'sfindings.

Conclusions
Criticalappraisalisasystematicprocessthroughwhichthestrengthsandweaknessesofaresearchstudycanbe
identified.Thisprocessenablesthereadertoassessthestudy'susefulnessandwhetheritsfindingsaretrustworthy.
Themostimportantcomponentofcriticalappraisaliscarefulassessmentofthestudydesignhowever,othersteps,
http://www.medscape.com/viewarticle/706399_print

10/14

3/7/2016

www.medscape.com/viewarticle/706399_print

suchasevaluationofthestatisticalmethodsused,interpretationofthefindingsandpotentialconflictsofinterestare
alsoessential.Finally,considerationoftheimportanceoftheresearchtoone'sownpatientswillhelpcliniciansidentify
themostrelevant,highqualitystudiesavailabletoguidetheirclinicalpractice.

CLICKHEREforsubscriptioninformationaboutthisjournal.

KeyPoints
Criticalappraisalisasystematicprocessusedtoidentifythestrengthsandweaknessesofaresearcharticle

Criticalappraisalprovidesabasisfordecisionsonwhethertousetheresultsofastudyinclinicalpractice

Differentstudydesignsarepronetovarioussourcesofsystematicbias

Designspecific,criticalappraisalchecklistsareusefultoolstohelpassessstudyquality

Assessmentsofotherfactors,includingtheimportanceoftheresearchquestion,theappropriatenessof
statisticalanalysis,thelegitimacyofconclusionsandpotentialconflictsofinterestareanimportantpartofthe
criticalappraisalprocess
References

1.DrussBGandMarcusSC(2005)Growthanddecentralisationofthemedicalliterature:implicationsfor
evidencebasedmedicine.JMedLibrAssoc93:499501
2.GlasziouPP(2008)Informationoverload:what'sbehindit,what'sbeyondit?MedJAust189:8485
3.LastJE(Ed.2001)ADictionaryofEpidemiology(4thEdn).NewYork:OxfordUniversityPress
4.SackettDLetal.(2000).EvidencebasedMedicine.HowtoPracticeandTeachEBM.London:Churchill
Livingstone
5.GuyattGandRennieD(Eds2002).Users'GuidestotheMedicalLiterature:aManualforEvidencebased
ClinicalPractice.Chicago:AmericanMedicalAssociation
6.GreenhalghT(2000)HowtoReadaPaper:theBasicsofEvidencebasedMedicine.London:Blackwell
MedicineBooks
7.MacAuleyD(1994)READER:anacronymtoaidcriticalreadingbygeneralpractitioners.BrJGenPract44:83
85
8.HillAandSpittlehouseC(2001)Whatiscriticalappraisal.EvidencebasedMedicine3:18
9.PublicHealthResourceUnit(2008)CriticalAppraisalSkillsProgramme(CASP).
10.NationalHealthandMedicalResearchCouncil(2000)HowtoReviewtheEvidence:SystematicIdentification
andReviewoftheScientificLiterature.Canberra:NHMRC
http://www.medscape.com/viewarticle/706399_print

11/14

3/7/2016

www.medscape.com/viewarticle/706399_print

11.ElwoodJM(1998)CriticalAppraisalofEpidemiologicalStudiesandClinicalTrials(2ndEdn).Oxford:Oxford
UniversityPress
12.AgencyforHealthcareResearchandQuality(2002)Systemstoratethestrengthofscientificevidence?
EvidenceReport/TechnologyAssessmentNo47,PublicationNo02E019Rockville:AgencyforHealthcare
ResearchandQuality
13.CrombieIK(1996)ThePocketGuidetoCriticalAppraisal:aHandbookforHealthCareProfessionals.London:
BlackwellMedicinePublishingGroup
14.HellerRFetal.(2008)Criticalappraisalforpublichealth:anewchecklist.PublicHealth122:9298
15.MacAuleyDetal.(1998)RandomisedcontrolledtrialoftheREADERmethodofcriticalappraisalingeneral
practice.BMJ316:113437
16.ParkesJetal.Teachingcriticalappraisalskillsinhealthcaresettings(Review).CochraneDatabaseof
SystematicReviews2005,Issue3.Art.No.:cd001270.doi:10.1002/14651858.cd001270
17.MaysNandPopeC(2000)Assessingqualityinqualitativeresearch.BMJ320:5052
18.HawkingSW(2003)OntheShouldersofGiants:theGreatWorksofPhysicsandAstronomy.Philadelphia,PN:
Penguin
19.NationalHealthandMedicalResearchCouncil(1999)AGuidetotheDevelopment,Implementationand
EvaluationofClinicalPracticeGuidelines.Canberra:NationalHealthandMedicalResearchCouncil
20.USPreventiveServicesTaskforce(1996)Guidetoclinicalpreventiveservices(2ndEdn).Baltimore,MD:
Williams&Wilkins
21.SolomonMJandMcLeodRS(1995)Shouldwebeperformingmorerandomizedcontrolledtrialsevaluating
surgicaloperations?Surgery118:456467
22.RothmanKJ(2002)Epidemiology:anIntroduction.Oxford:OxfordUniversityPress
23.YoungJMandSolomonMJ(2003)Improvingtheevidencebaseinsurgery:sourcesofbiasinsurgicalstudies.
ANZJSurg73:504506
24.MargiticSEetal.(1995)Lessonslearnedfromaprospectivemetaanalysis.JAmGeriatrSoc43:435439
25.SheaBetal.(2001)Assessingthequalityofreportsofsystematicreviews:theQUORUMstatementcompared
toothertools.InSystematicReviewsinHealthCare:MetaanalysisinContext2ndEdition,122139(EdsEgger
M.etal.)London:BMJBooks
26.EasterbrookPHetal.(1991)Publicationbiasinclinicalresearch.Lancet337:867872
27.BeggCBandBerlinJA(1989)Publicationbiasanddisseminationofclinicalresearch.JNatlCancerInst81:
107115
28.MoherDetal.(2000)Improvingthequalityofreportsofmetaanalysesofrandomisedcontrolledtrials:the
QUORUMstatement.BrJSurg87:14481454
29.SheaBJetal.(2007)DevelopmentofAMSTAR:ameasurementtooltoassessthemethodologicalqualityof
systematicreviews.BMCMedicalResearchMethodology7:10
http://www.medscape.com/viewarticle/706399_print

12/14

3/7/2016

www.medscape.com/viewarticle/706399_print

30.StroupDFetal.(2000)Metaanalysisofobservationalstudiesinepidemiology:aproposalforreporting.Meta
analysisOfObservationalStudiesinEpidemiology(MOOSE)group.JAMA283:20082012
31.YoungJMandSolomonMJ(2003)Improvingtheevidencebaseinsurgery:evaluatingsurgicaleffectiveness.
ANZJSurg73:507510
32.SchulzKF(1995)Subvertingrandomizationincontrolledtrials.JAMA274:14561458
33.SchulzKFetal.(1995)Empiricalevidenceofbias.Dimensionsofmethodologicalqualityassociatedwith
estimatesoftreatmenteffectsincontrolledtrials.JAMA273:408412
34.MoherDetal.(2001)TheCONSORTstatement:revisedrecommendationsforimprovingthequalityofreports
ofparallelgrouprandomizedtrials.BMCMedicalResearchMethodology1:2
35.RochonPAetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:1.Roleanddesign.BMJ330:
895897
36.MamdaniMetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:2.Assessingpotentialfor
confounding.BMJ330:960962
37.NormandSetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:3.Analyticalstrategiestoreduce
confounding.BMJ330:10211023
38.vonElmEetal.(2007)Strengtheningthereportingofobservationalstudiesinepidemiology(STROBE)
statement:guidelinesforreportingobservationalstudies.BMJ335:806808
39.SuttonTyrrellK(1991)Assessingbiasincasecontrolstudies:properselectionofcasesandcontrols.Stroke22:
938942
40.KnottnerusJ(2003)Assessmentoftheaccuracyofdiagnostictests:thecrosssectionalstudy.JClinEpidemiol
56:11181128
41.FurukawaTAandGuyattGH(2006)Sourcesofbiasindiagnosticaccuracystudiesandthediagnosticprocess.
CMAJ174:481482
42.BossyutPMetal.(2003)TheSTARDstatementforreportingstudiesofdiagnosticaccuracy:explanationand
elaboration.AnnInternMed138:W1W12
43.STARDstatement(StandardsfortheReportingofDiagnosticAccuracyStudies).
44.RafteryJ(1998)Economicevaluation:anintroduction.BMJ316:10131014
45.PalmerSetal.(1999)Economicsnotes:typesofeconomicevaluation.BMJ318:1349
46.RussSetal.(1999)Barrierstoparticipationinrandomizedcontrolledtrials:asystematicreview.JClin
Epidemiol52:11431156
47.TinmouthJMetal.(2004)Areclaimsofequivalencyindigestivediseasestrialssupportedbytheevidence?
Gastroentrology126:17001710
48.KaulSandDiamondGA(2006)Goodenough:aprimerontheanalysisandinterpretationofnoninferiority
trials.AnnInternMed145:6269
49.PiaggioGetal.(2006)Reportingofnoninferiorityandequivalencerandomizedtrials:anextensionofthe
http://www.medscape.com/viewarticle/706399_print

13/14

3/7/2016

www.medscape.com/viewarticle/706399_print

CONSORTstatement.JAMA295:11521160
50.HeritierSRetal.(2007)Inclusionofpatientsinclinicaltrialanalysis:theintentiontotreatprinciple.In
InterpretingandReportingClinicalTrials:aGuidetotheCONSORTStatementandthePrinciplesof
RandomizedControlledTrials,9298(EdsKeechA.etal.)StrawberryHills,NSW:AustralianMedical
PublishingCompany
51.NationalHealthandMedicalResearchCouncil(2007)NationalStatementonEthicalConductinHuman
Research8990Canberra:NHMRC
52.LoBetal.(2000)Conflictofinterestpoliciesforinvestigatorsinclinicaltrials.NEnglJMed343:16161620
53.KimSYHetal.(2004)Potentialresearchparticipants'viewsregardingresearcherandinstitutionalfinancial
conflictsofinterests.JMedEthics30:7379
54.KomesaroffPAandKerridgeIH(2002)Ethicalissuesconcerningtherelationshipsbetweenmedical
practitionersandthepharmaceuticalindustry.MedJAust176:118121
55.LittleM(1999)Research,ethicsandconflictsofinterest.JMedEthics25:259262
56.LemmensTandSingerPA(1998)Bioethicsforclinicians:17.Conflictofinterestinresearch,educationand
patientcare.CMAJ159:960965
ReprintAddress
JaneMYoung,SurgicalOutcomesResearchCentre(SOuRCe),RoyalPrinceAlfredHospital,POBoxM157,
MissendenRoad,NSW2050,Australia.Email:jyoung@email.cs.nsw.gov.au
NatClinPractGastroenterolHepatol.20096(2):8291.2009NaturePublishingGroup
ThiswebsiteusescookiestodeliveritsservicesasdescribedinourCookiePolicy.Byusingthiswebsite,youagreetothe
useofcookies.
close

http://www.medscape.com/viewarticle/706399_print

14/14