Académique Documents
Professionnel Documents
Culture Documents
betweentwoBigFivefactors
Somethingisnotquiterightinthewoodshed
PaulBarrett&JeanPierreRolland*
*Universit deParisQuest
NanterreLaDfense,France
April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 2 of16
AuntAdaDoomistheinfamousmadwomanintheatticofStellaGibbonscomedynovelColdComfort
Farm(1932);hermindbecameunhingedwhenasachildshesawsomethingnastyinthewoodshed.
Theliteraryphrasemaynottotallycapturetheeffectoftheobservationswemakebelow,but
somethingisnotquiterightaboutthefollowingmetaanalyticresultsreportedinaseriesofstudies
since1993.
Wedonotwishtodwellontheprosandconsofmetaanalysis,butratherwefindourselvesquestioning
theimplicitunderstandingthatmetaanalysisisalwayscapableofrevealingtheexpectedpopulation
correlationbetweenattributes.ThepaperbyLeLorier,J.,Gregoire,G.,Benhaddad,A.,Lapierre,J.,&
Derderian,F.(1997)Discrepanciesbetweenmetaanalysesandsubsequentlargerandomized,controlled
trials.TheNewEnglandJournalofMedicine,337,8,536542isperhapsthemostfamousstudyshowing
thatmetaanalysisdoesnotalwaysproduceaccurateestimatesofpopulationparameters,andthe
recentstudybySchonemann,P.H.,&Scargle,J.D.(2008)AGeneralizedPublicationBiasModel.Chinese
JournalofPsychology,50,1,2129,helpstoexplainwhy.
Ofspecificinterestherethougharethevariousmetaanalyticestimatesofpopulationcorrelations
betweentwospecificBigFivepersonalitytestscales,EmotionalStabilityandConscientiousness.These
arethetwomostimportantbroadpersonalityfactorsassociatedmetaanalyticallywithjob
performance.
Anaside[Feb,2010]:arecentpaperfromDeRaad,B.,Barelds,D.P.H.,Levert,E.,Ostendorf,F.,Mlacic,
B.,DiBlas,L.,Hrebckov,M.,Szirmk,Z.,Szarota,P.,Perugini,M.,Church,A.T.,&Katigbak,M.S.(2010)
Onlythreefactorsofpersonalitydescriptionarefullyreplicableacrosslanguages:Acomparisonof14
traittaxonomies.JournalofPersonalityandSocialPsychology,98,1,160173.Thethreefactorsthey
findreplicableare:
Extraversion
(+)active,chatty,cheerful,dynamic,energetic,enthusiastic,extraverted,exuberant,lively,open,
outgoing,sociable,talkative,vigorous,vivacious
()bashful,closed,introverted,lonely,passive,quiet,reserved,shy,silent,solitary,taciturn,timid,
unsociable,untalkative,withdrawn
Agreeableness
(+)accommodating,agreeable,benevolent,conciliatory,friendly,gentle,goodnatured,kindhearted,
lenient,meek,mild,patient,peaceful,sympathetic,tolerant
()aggressive,bossy,brusque,choleric,coldhearted,despotic,domineering,fierce,hottempered,
intolerant,irritable,overbearing,quarrelsome,shorttempered,stubborn
Conscientiousness
(+)careful,conscientious,diligent,disciplined,dutiful,hardworking,industrious,methodical,
meticulous,orderly,organized,precise,scrupulous,thorough,tidy
()absentminded,careless,chaotic,disorderly,disorganized,frivolous,imprudent,inaccurate,
irresponsible,lax,lazy,negligent,rash,undisciplined,untidy
InterestinglytheyfindEmotionalStability/Anxietyisnotreplicableacrosscultures.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 3 of16
TheEvidence
Therelevantcorrelationappears(albeitnotreferencedinthepaper)inTable6,p.669,of:
Ones,D.S.,Viswesvaran,C.,&Reiss,A.D.(1996)Roleofsocialdesirabilityinpersonalitytestingfor
personnelselection:Theredherring.JournalofAppliedPsychology,81,6,660679.
Ones(1993)isnotreferencedinthepaper,butthecitationmayrefertoOnes,D.S.(1993).Theconstruct
validityofintegritytests.Unpublisheddoctoraldissertation,UniversityofIowa,IowaCity,IA.Whats
importanthereisthattheseareestimatedpopulationcorrelations.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 4 of16
Mount,M.K.,Barrick,M.R.,Scullen,S.M.,&Rounds,J.(2005)HigherorderdimensionsoftheBig
FivepersonalitytraitsandtheBigSixvocationalinteresttypes.PersonnelPsychology,58,2,447478.
Table3,p.463ofthispapernowshowsatruecorrelationof0.52insteadof0.26.
LookingatTable2onpage462,wesee..
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 5 of16
The0.26correlationreportedinOnesetal(1993)liesjustoutsidethe90%credibilityvaluereportedin
Mountetal(2005).Itwouldliewithina95%interval,soarareresultperhapsbutnotimpossible.But,
therangeofthatcredibilityintervalinMount,etal.isverywide.
Itshouldbenotedthatthisistheestimatedtruescorecorrelationbetweenthesetwoattributesusedin
Table4,p.845intherecentpaperbySchmidt,F.L.,Shaffer,J.A.,&Oh,IS.(2008)Increasedaccuracyfor
rangerestrictioncorrections:implicationsfortheroleofpersonalityandgeneralmentalabilityinjob
andtrainingperformance.PersonnelPsychology,61,4,827868,andnottheonereportedbyOnes
(1993).
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 6 of16
Steel,P.,Schmidt,J.,&Shultz,J.(2008)Refiningtherelationshipbetweenpersonalityand
subjectivewellbeing.PsychologicalBulletin,134,1,138161.
FromTable6,p.148
Thiscorrelationiscomputedover37studies,utilizing17,464cases(anaverageof472perstudyas
notedonpage14,para2,column1).
GiventheBigFiveattributeEmotionalStabilityisnowcalledNeuroticism,wethinkitislikelythe
scoringwasreversed(highscoresequatetohigherEmotionalInstability),whichwouldexplainthe
reversalinsignofthecorrelationbetweenEmotionalStability(Neuroticism)andConscientiousness
attributes.
Sofarwehavecorrelationsof0.26,0.52,andnow0.33allpopulation/truevalueestimates.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 7 of16
Meriac,J.P.,Hoffman,B.J.,Woehr,D.J.,&Fleisher,M.S.(2008)Furtherevidenceforthevalidityof
assessmentcenterdimensions:Ametaanalysisoftheincrementalcriterionrelatedvalidityof
dimensionratings.JournalofAppliedPsychology,93,5,10421052.
FromTable3,p.1047,wesee
ThecorrelationbetweenNeuroticismandConscientiousnessiscomputedusing1009casesincludedin6
independentstudies(fromTable1,p.1046).
Sonowwehavecorrelationsof0.24,0.26,0.33,and0.52;allpopulation/truevalueestimates.
NoticehowfarthecorrelationbetweenExtraversionandNeuroticismhasdroppedinthisstudyfrom
thevalueshownonthepreviouspage,0.33downto0.02.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 8 of16
Hogan,J.,Barrett,P.,&Hogan,R.(2007)Personalitymeasurement,faking,andemployment
selection.JournalofAppliedPsychology,92,5,12701285.
Dataarefrom5,266individualswhowereadministeredtheHoganPersonalityInventoryontwo
occasions(T1andT2),whoseHPIscalescoreswereconvertedintotheirBigFiveequivalentsusinga
formuladevelopedbySmith,B.&Ellingson,J.E.(2002)Substancevsstyle:anewlookatSocial
Desirabilityinmotivatingcontexts.JournalofAppliedPsychology,87,2,211219.FromTable2,p.1276
Note,thisisnotmetaanalyticdata,justaverylargesampleof5,266cases.
Ifwecorrecteachcorrelationfortheunreliabilityofeachscale,theT1correlationbecomes0.67,that
forT2becomes0.70.
Table1:Summaryofresultsfromthevariousevidencebases to
Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266(samecasesasT1) 0.70
Itspossiblethattheestimatedpopulationcorrelationsallliewithina95%credibilityintervalextending
from0.20to0.80.Butisthisofanypracticalvalueexcepttoindicatethatsometruecorrelationis
positive,andliessomewherebetweennearzeroandverylarge?
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 9 of16
Ultimately,wethinkwehavethedefinitiveanswerusingthe2007HoganNormativeSample
datasetof156,614cases.ScoringthedataintotheBigFivescalesusingtheSmithandEllingson
formula,wecancomputewhatarelikelytobethenearestrealisticpopulationestimatesoftheattribute
correlationsforaUSpopulation.Further,ifwewishtoassumeeachindividualpossessesatruescore
onanattribute,wecancorrectthesecorrelationsfortheunreliabilityofeachscaletocreatethetrue
scorecorrelationswhichcanbecompareddirectlytothosepresentedinTable1.
Inordertocorrecteachrawscorecorrelationforscaleunreliability,weneedthecompositealphasfor
eachoftheBigFivescales(compositebecauseeachBigFivescaleisconstructedfromfourHPIHICs).
Table2presentsthesecorrelationalresults:
Table2:RawScoreandTrueScoredisattenuatedcorrelationsbetweenEmotionalStabilityand
ConscientiousnessusingtheBigFiverescoredHPIUSNormativedatafortheHoganPersonality
Inventory
Correlations(Big5FactorscoresfromNormativeHPIN=156614.sta)
Markedcorrelationsaresignificantatp<.05000
N=156614(Casewisedeletionofmissingdata)
Variable Emotional Stability Extraversion Openness Agreeableness Conscientiousness
Emotional Stability 0.75 0.16 0.28 0.25 0.37
Extraversion 0.20 0.83 0.27 0.34 0.06
Openness 0.36 0.33 0.80 0.22 0.28
Agreeableness 0.38 0.49 0.32 0.58 0.32
Conscientiousness 0.53 0.08 0.39 0.52 0.66
Therawscorecorrelationsareabovethediagonal,thetruescoredisattenuatedvaluesaregiveninthe
lowerhalfofthematrix(theshadedcells),thealphareliabilityforeachscaleisgiveninthemain
diagonal.
AddingtherelevantvaluetoTable1:
Table3:Summaryofresultsfromthevariousevidencebases to
Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266 0.70
HoganHPINormativeUSdataset 156,614 0.53
Ofthefourmetaanalyses(rows14),onlyone(Mount2005)cameclose,usingaquarterofthesample
sizeoftheonethatyieldedapopulationestimateof0.33.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 10 of16
TheEvidence:TwoMoreStudies
Judge,T.A.,vanVianen,A.E.M.,&DePater,I.E.(2004)Emotionalstability,coreselfevaluations,
andjoboutcomes:Areviewoftheevidenceandanagendaforfutureresearch.HumanPerformance,
17,3,325346.
FromTable2,p.330)wesee
Thesedatawerecomputedfromthesummary(uncorrected)dataprovidedinJudge,T.A.,Erez,A.,
Bono,J.E.,&Thoresen,C.J.(2002)Aremeasuresofselfesteem,neuroticism,locusofcontrol,and
generalizedselfefficacyindicatorsofacommoncoreconstruct?JournalofPersonalityandSocial
Psychology,83,3,693710,Table8,p.704
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 11 of16
Revelle,W.,Wilt,J.,&Rosenthal,A.(inpress)Individualdifferencesincognition:Newmethods
forexaminingthepersonalitycognitionlink.ToappearinAleksandraGruszka,GeraldMatthews,and
BlazejSzymura(editors):HandbookofIndividualDifferencesinCognition:Attention,Memoryand
ExecutiveControl.Draft,April2008,downloadedfrom:
http://personalityresearch.net/revelle/publications/rwr.08.pdf
Frompage22,Table12wesee:
Thesamplesizeforthiscorrelationwas>50,000usingtheinnovativeSyntheticAperturePersonality
Assessmentsamplingprocedure(see:
http://www.personalityproject.org/revelle/publications/sapa.mpa.key.pdfformoredetailsaboutthis
studyandthesamplingtechnique;thetabledataabovearesimilartothedatareportedinslide#74).
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 12 of16
TheFinalPicture
Incorporatingthesestudieswiththosefrom to ,wecansummarizetheeightstudiesresultsin
Table4.
Table4:Summaryofresultsfromthevariousevidencebases to
Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266 0.70
HoganHPINormativeUSdataset 156,614 0.53
Judgeetal(2004) 1,885 0.49
Revelleetal >50,000 0.17
So,ourestimatesfortherelationshipbetweenEmotionalStabilityandConscientiousnessrangefrom
0.17throughto0.70.Clearly,thisrangeofpopulationestimatesindicatesthatsomethingisnotright
here.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 13 of16
InPsychology,attributeswiththesamenamearenotnecessarilythesameatall
Thecluetowhatmightbecausalforthiswidelyvaryingrangeofestimatesisgivenintwopapers:
Rich,G.A.,Bommer,W.H.,MacKenzie,S.B.,Podsakoff,P.M.,&Johnson,J.L.(1999)Applesand
applesorapplesandoranges?AMetaAnalysisofobjectiveandsubjectivemeasuresofsalesperson
performance.JournalofPersonalSelling&SalesManagement,XIX,4,4152.
Abstract
Thegoalofthisstudywastoexaminetherelationshipbetweenobjectiveandsubjectivemeasuresof
salespersonperformance.Theresultsofametaanalysisof21studieswithatotalsamplesizeof4,092
participantsindicatedanoverallmeancorrectedcorrelationof.447,indicatingthatthetwomeasures
sharedonlyabout20%ofvariance.Althoughamoderatorsubgroupanalysisfoundthatthecorrected
meancorrelationwassomewhathigherincertainsituations,thefindingsgenerallysuggestthat
objectiveandsubjectivemeasuresofsalespersonperformancearenotinterchangeable,andthatthe
choiceofthemostappropriatemeasuremayrequireatradeoffbetweenaccuratelytappingthedomain
oftheperformanceconstructandminimizingmeasurementerror.
Thisisaclearindicationthatmetaanalyseswhichutilizestudiescomposedofasamenamecriterion
(hereSalesPerformance)arelikelytoyieldquitedifferentpopulationestimates)ifthecriterionis
notexactlythesame.
Frompp.4142ofthispaper:
Inthesalesmanagementliterature,performancehasbeenmeasuredinavarietyofdifferentways.For
example,roughlyhalfofthestudies(53.3%ofthereportedcorrelations)includedinChurchill,Ford,
Hartley,andWalker's(1985)metaanalysisofthedeterminantsofsalespersonperformancemeasured
performanceusingsubjectiveevaluationsobtainedfrommanagers,peers,orselfreports.Theotherhalf
(46.7%ofthereportedcorrelations)measuredvolume,salescommissions,orpercentofquota.
However,inmostinstances,noattemptwasmadetoexplainwhyonetypeofmeasurewasusedas
opposedtoanother.Theimplicitassumptionappearstobethatobjectiveandsubjectivemeasuresof
performanceareinterchangeable,andthatthedomainofsalesperformanceisadequatelycapturedby
eithersubjectiveratingsorobjectiveresults.Thus,practitionersandresearchersalikeareassumingthat
thesetwoclassesofmeasuresare"applesandapples"andthattheymaybetreatedinterchangeably.
Thesameargumentmighteasilybemadeforthatglobalcriterion,jobperformance.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 14 of16
Woods,S.A.(2009).Thecomparativevaliditiesofsixpersonalityinventories.Proceedingsofthe
DivisionofOccupationalPsychology,BritishPsychologicalSociety,AnnualConference2009reportedin
detailinWoods,S.A.(2009).TheStructuresandValiditiesofFiveWorkrelatedPersonalityInventories.
UnpublishedWorkingPaper(Jan30th,2009).TheworkingpapermayberequestedfromS.A.Woods,
AstonBusinessSchoolhttp://www.abs.aston.ac.uk/newweb/staff/detail.asp/sfldStaffID=A0000731).
Abstract
Thisstudyexaminedthestructures,convergentvalidities,andcriterionvaliditiesoffiveworkrelated
personalityinventories(theHoganPersonalityInventory.theOccupationalPersonalityQuestionnaire.
theSixteenPersonalityFactorQuestionnaire.thePersonalityandPreferencesInventory,ProfileMatch).
Asampleof371individualsfromtheUKworkingpopulationcompletedvariouscombinationsofthefive
inventories,plusameasureofthelexicalBigFiveandseveralcriterionscales.Overall,theresults
indicatedsensibleandinterpretablefactorstructuresfortheinventories(withtheexceptionofthe
PersonalityandPreferencesInventory),theoreticallymeaningfulconvergentvalidities,andsimilar
criterionvaliditymagnitudesandpatterns.Itwasconcludedthatthesedatagiveconfidenceintheuse
oftheseinventoriesforresearchandpractice,andrevealimportantsimilaritiesandconsistenciesin
theirvalidities.
AsubselectionofTable23onp.30presentsasummaryofthemeancorrelationsforeachparticularBig
Fivescale.Thetableshowsthemeancorrelationbetweenthe5differenttestsmeasuresofeachofthe
BigFiveattributes.So,given5personalitytests,eachmeasuringExtraversion,Woodscorrelateseach
testsscoresonExtraversionwiththosefromtheothertests(pairwise);producing((5x5)5)/2possible
correlations.Theaverageisthentakenofthesecorrelations.
Whatthesedataindicateisthatthereisonlyabroad
levelofagreementbetweenfivemajorworkrelated
personalitytestswhentheirscalesarereexpressed
(whereappropriate)asBigFivepersonality
constructs.
Whereastheevidencebasessupportingeach
questionnairemightwellbesubstantiveand
excellent,anymetaanalysiswhichincorporated
thesetestsaspartofastudylookingatthe
correlationbetweenpersonalityconstructs,oreven
thepredictiveaccuracyofaspecifiedoutcome,is
goingtobeinfluencedbythelackofagreement
betweentestscales.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 15 of16
Anaside
Maraun,M.D.(1998)MeasurementasaNormativePractice:ImplicationsofWittgenstein'sPhilosophy
forMeasurementinPsychology.Theory&Psychology,8,4,435461...
B.Thedifficultyfacedbypsychologistsinmeasuringisnotmathematicalorempiricalinnature,butis
insteadthattheconceptstheywishtohaveenterintotheirmeasurementoperationsaretypicallyofthe
commonorgardenvariety.Theseconceptshavenotoriouslycomplicatedgrammars.Inlightof(5),this
generatesseriousdifficulties.
C.Therelativelackofsuccessofmeasurementinthesocialsciencesascomparedtothephysical
sciencesisattributabletotheirsharplydifferentconceptualfoundations.Inparticular,thephysical
sciencesrestonabedrockoftechnicalconcepts,whilepsychologyrestsonawebofcommonorgarden
psychologicalconcepts.Thefactof(B)completestheexplanation."
andonpp452453andonwards..
"ConstraintsonMeasurementinPsychology
IfWittgenstein'sviewsarecorrect,thendifficultiesinpsychologicalmeasurementmaybeexplainedby
thefactthatpsychologyhasmischaracterizedmeasurement.However,whatifonedidconsiderthe
grammarsofpsychologicalconcepts.Whatwouldgrammarsayabouthowtomeasurepsychological
concepts?Ibelievethatagrammaticalinvestigationofpsychologicalconcepts(e.g.asinTerHark,1990)
reveals(a)theobviousfactthat,asitstands,commonorgardenpsychologicalconceptsarenot
measurable,and(b)theexistenceofgrammaticalconstraintsonthepossibilityofmeasurement
involvingcommonorgardenpsychologicalconcepts.Theseconstraintsmay,inpart,explainthe
enduringdifficultyfacedbypsychologyinattemptingtomeasure,and,inparticular,ascomparedtothe
physicalsciences(forrelateddiscussionsofthelatterpoint,seeCampbell,1921;Schonemann,1994).
While(a)and(b)mayhavethetoneofoverstatement,Ibelievethatitisneverthelessinterestingto
brieflyconsidereach.Todosorequiresthattwotypesofconceptbedistinguished:(1)technical
conceptsand(2)commonorgardenconcepts.Atechnicalconceptisaconceptdefinedbyaspecialized
orexpertcommunity,andemployedwithinanarrow,technicalfieldofapplication.Acommonor
gardenconcept,ontheotherhand,isaconceptwithacommonemploymentineverydaylife(Baker&
Hacker,1982).Commonorgardenconceptsaretaught,learnedandunderstoodbythepersononthe
street,andhavemeaningsthataremanifestinbroad,normativelinguisticpractices.They,inaddition,
aremoreapttobethesourceofconfusioninthecontextofscientificinvestigation...."
AnObviousConclusion?
MetaAnalyticpopulationestimateswillbeunreliable(unstable)unlessidenticalattributesand
predictors/outcomesareusedineverycontributingstudy.Thismakessensewhenyouthinkaboutwhat
metaanalysiswasreallydesignedforcorrectingsamplingerrorsduetosmall,individualstudy
attemptsatexaminingconstructrelationsandeffectsizes.
Asyoudepartfromthisrule,independentstudymetaanalyticestimatesofsupposedlythesame
phenomenawillshowdegreesofvariationinroughproportiontothevariationinmeaningamongst
samenameattributeswhichcompriseconstituentstudies.
Somethingisdefinitelynotquiterightinthewoodshed.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009
http://www.pbarrett.net/stratpapers/metacorr.pdf 16 of16
Appendix1:
TheabstracttothepaperbyLeLorier,J.,Gregoire,G.,Benhaddad,A.,Lapierre,J.,&Derderian,F.(1997)
Discrepanciesbetweenmetaanalysesandsubsequentlargerandomized,controlledtrials.TheNew
EnglandJournalofMedicine,337,8,536542.
Background
Metaanalysesarenowwidelyusedtoprovideevidencetosupportclinicalstrategies.However,large
randomized,controlledtrialsareconsideredthegoldstandardinevaluatingtheefficacyofclinical
interventions.
Methods
Wecomparedtheresultsoflargerandomized,controlledtrials(involving1000patientsormore)that
werepublishedinfourjournals(theNewEnglandJournalofMedicine,theLancet,theAnnalsofInternal
Medicine,andtheJournaloftheAmericanMedicalAssociation)withtheresultsofmetaanalyses
publishedearlieronthesametopics.Regardingtheprincipalandsecondaryoutcomes,wejudged
whetherthefindingsoftherandomizedtrialsagreedwiththoseofthecorrespondingmetaanalyses,
andwedeterminedwhetherthestudyresultswerepositive(indicatingthattreatmentimprovedthe
outcome)ornegative(indicatingthattheoutcomewithtreatmentwasthesameorworsethanwithout
it)attheconventionallevelofstatisticalsignificance(P<0.05).
Results
Weidentified12largerandomized,controlledtrialsand19metaanalysesaddressingthesame
questions.Foratotalof40primaryandsecondaryoutcomes,agreementbetweenthemetaanalyses
andthelargeclinicaltrialswasonlyfair(kappa=0.35;95percentconfidenceinterval,0.06to0.64).The
positivepredictivevalueofthemetaanalyseswas68percent,andthenegativepredictivevalue67
percent.However,thedifferenceinpointestimatesbetweentherandomizedtrialsandthemeta
analyseswasstatisticallysignificantforonly5ofthe40comparisons(12percent).Furthermore,ineach
caseofdisagreementastatisticallysignificanteffectoftreatmentwasfoundbyonemethod,whereas
nostatisticallysignificanteffectwasfoundbytheother.
Conclusions
Theoutcomesofthe12largerandomized,controlledtrialsthatwestudiedwerenotpredicted
accurately35percentofthetimebythemetaanalysespublishedpreviouslyonthesametopics.
StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009