Home>MethodValidationandRobustness

MethodValidationandRobustness
May01,2006
ByIraKrull[1] ,MichaelE.Swartz[2]
LCGCNorthAmerica
Volume24,Issue5

Reproducibility?Ruggedness?Robustness?Tomanypeople,alloftheseterms
meanthesamething.Butinreality,inthewordsofapopularchildren'sprogram,
"oneofthesethingsisnotliketheother."Anearlier"ValidationViewpoint"column(1)
touchedbrieflyonthistopic,butthiscolumnwillexplorethetopicinalittlemore
detail.So,forthepurposesofthisdiscussion,the"Rword"wearemostinterestedin
isrobustness.However,firstweneedafewclarifications.
TermsandDefinitions
Therearetwoguidelinedocumentsimportanttoanymethodvalidationprocess(2,3):
USPChapter1225:ValidationofCompendialMethodsandtheInternational
ConferenceonHarmonization(ICH)Guideline:ValidationofAnalyticalProcedures:
TextandMethodologyQ2(R1).WhiletheUSPisthesolelegaldocumentintheeyes
oftheFDA,thisarticledrawsfrombothguidelinesasappropriatefordefinitionsand
methodology.
RuggednessisdefinedinthecurrentUSPguidelineasthedegreeofreproducibility
oftestresultsobtainedbytheanalysisofthesamesamplesunderavarietyof
conditions,suchasdifferent
laboratories
analysts
instruments
reagentlots
elapsedassaytimes
assaytemperaturesand
days.
Thatis,itisameasureofthereproducibilityoftestresultsunderthevariationin
conditionsnormallyexpectedfromlaboratorytolaboratoryandfromanalystto
analyst.Theuseofthetermruggedness,however,isnotusedbytheICH,butis
certainlyaddressedinguidelineQ2(R1)underintermediateprecision(within
laboratoryvariationsdifferentdays,analysts,equipment,andsoforth)and
reproducibility(betweenlaboratoryvariationsfromcollaborativestudiesappliedto
thestandardizationofthemethod).ItisalsofallingoutoffavorwiththeUSP,as
evidentinrecentlyproposedrevisionstochapter1225,wherereferencesto
ruggednesshavebeendeletedtoharmonizemorecloselywithICH,usingtheterm
"intermediateprecision"instead(4).

BoththeICHandtheUSPguidelinesdefinetherobustnessofananalytical
procedureasameasureofitscapacitytoremainunaffectedbysmallbutdeliberate
variationsinproceduralparameterslistedinthedocumentation,providingan
indicationofthemethod'sorprocedure'ssuitabilityandreliabilityduringnormaluse.
Butwhilerobustnessshowsupinbothguidelines,interestinglyenough,itisnotin
thelistofsuggestedortypicalanalyticalcharacteristicsusedtovalidateamethod
(again,thisapparentdiscrepancyischanginginrecentlyproposedrevisionstoUSP
chapter1225[3]).

Robustnesstraditionallyhasnotbeenconsideredasavalidationparameterinthe
strictestsensebecauseusuallyitisinvestigatedduringmethoddevelopment,once
themethodisatleastpartiallyoptimized.Whenthoughtofinthiscontext,evaluation
ofrobustnessduringdevelopmentmakessenseasparametersthataffectthemethod
canbeidentifiedeasilywhenmanipulatedforselectivityoroptimizationpurposes.
Evaluatingrobustnesseitherbeforeoratthebeginningoftheformalmethod
validationprocessalsofitsintothecategoryof"youcanpaymenow,oryoucanpay
melater."Inotherwords,investingalittletimeupfrontcansavealotoftime,energy,
Figure1:Fullfactorialdesignexperimentsforfourfactors:pH,flowrate,wavelength,and
percentorganicinthemobilephase.Runsareindictedbythedots.
andexpenselater.
Duringarobustnessstudy,methodparametersarevariedintentionallytoseeifthe
methodresultsareaffected.Thekeywordinthedefinitionisdeliberate.Inliquid
chromatography(LC),examplesoftypicalvariationsare
mobilephasecomposition
number,type,andproportionoforganicsolvents
buffercompositionandconcentration
pHofthemobilephase
differentcolumnslots
temperature
flowrate
wavelength
gradientvariations
holdtimes
slopeand
length.
Robustnessstudiesalsoareusedtoestablishsystemsuitabilityparameterstomake
surethevalidityoftheentiresystem(includingboththeinstrumentandthemethod)
ismaintainedthroughoutimplementationanduse.

Figure2:Fractionalfactorialdesignrobustnessstudyforafivefactorexperiment:pH,flow
rate,wavelength,percentorganic,andtemperature.Runsareindictedbythedots.
Whatisthebottomline?Thetermsrobustnessandruggednessrefertoseparateand
distinctmeasurablecharacteristicsorassomerefertoit,parametersexternal
(ruggedness)orinternal(robustness)tothemethod.Aruleofthumb:ifitiswritten
intothemethod(forexample,30C,1.0mL/min,254nm),itisarobustnessissue.Ifit
isnotspecifiedinthemethod(forexample,youwouldneverspecify:Steverunsthe
methodonTuesdaysoninstrumentsix),itisaruggednessintermediateprecision
issue.Inaddition,itisagoodideatoalwaysevaluatetheexternalruggedness
intermediateprecisionparametersseparatefromtheinternalrobustnessparameters.
Foryears,analystshaveconductedboth
RobustnessStudyExperimentalDesignoptimizationandrobustnessstudies

TableI:PlackettBurmandesignin12runsforupto11factors.High(1)andlow(1)valuescorrespondtochromatographicvariablesorfactors.

accordingtoaunivariateapproachchangingasinglevariableorfactoratatime.
Performingexperimentsinthismannermostlikelyresultedfrombeingtrainedas
scientists(onevariableatatime!)asopposedtoastatistician.Thisapproach,while
certainlyinformative,canbetimeconsuming,andoften,importantinteractions
betweenvariablessuchaspHchangeswithtemperatureremainundetected.
Multivariateapproachesallowtheeffectsofmultiplevariablesonaprocesstobe
studiedsimultaneously.
Inamultivariateexperiment,varyingparameterssimultaneously,ratherthanoneata
time,canbemoreefficientandcanallowtheeffectsbetweenparameterstobe
observed.
Therearefourcommontypesofmultivariateexperimentaldesignapproaches:
Comparative,usedtochoosebetweendifferentalternatives.
Responsesurfacemodeling,usedtohitatarget,minimizeormaximizea
response.
Regressionmodeling,usedtoquantifydependenceofresponsevariableson
processinputs.
Screening,usedtoidentifywhichfactorsareimportantorsignificant.
Thechoiceofwhichdesigntousedependsupontheobjectiveandthenumberof
parameters,referredtoasfactors,thatneedtobeinvestigated.Forchromatographic
studies,thetwomostcommondesignsareresponsesurfacemodelingand
screening.Responsesurfacemodelingcommonlyisusedformethoddevelopment,
butthefocusoftheremainderofthiscolumnwillbeonscreening,becauseitisthe
mostappropriatedesignforrobustnessstudies.Thereferencesincludemoredetail
beyondwhatispresentedhere,andadditionalinformationonvariousexperimental
designs(47).
Fullfactorialdesignrunscanreallystarttoaddupwhen
ScreeningDesignsinvestigatinglargenumberoffactorsforninefactors,512runs
Screeningdesignsareanefficientwaytoidentifythecriticalfactorsthataffect
robustnessandareusefulforthelargernumbersoffactorsoftenencounteredina
chromatographicmethod.Therearethreecommontypesofscreeningexperimental
designsthatcanbeused:fullfactorial,fractionalfactorial,andPlackettBurman
designs.
wouldbeneeded!(Withouteventakingintoaccountreplicate
FullFactorialDesignsinjections.)Inaddition,thedesignpresentedinFigure1
Inafullfactorialexperiment,allpossiblecombinationsoffactorsaremeasured.Each
experimentalconditioniscalleda"run,"andtheresultsarecalled"observations."
Theexperimentaldesignconsistsoftheentiresetofruns.Acommonfullfactorial
designisonewithallfactorssetattwolevelseach,ahighandlowvalue.Ifthereare
kfactors,eachattwolevels,afullfactorialdesignthenhas2kruns.Inotherwords,
usingfourfactors,therewouldbe24 or16designpointsorruns.Tofurtherillustrate
thepoint,Figure1illustratesafullfactorialdesignrobustnessstudyforfourfactors
pH,flow,wavelength,andpercentorganicinthemobilephase.
assumeslinearresponsesbetweenfactors,butinmany
FractionalFactorialDeignscases,curvatureispossible,necessitatingcenterpoint

TableII:Examplerobustnessfactorselectionandlimitsforanisocraticmethod.Theselimitsare
examplesonlyandshouldbechosenaccordingtoexpectedlaboratoryandinstrumentvariations.
runs,furtherincreasingthenumberofruns.Forthisreason,fullfactorialdesigns
usuallyarenotrecommendedformorethanfivefactorstominimizetimeand
expense.Fractionalfactorialdesignworksmostlyduetothe"scarcityofeffects
So,whatdoyoudoifyouwanttoinvestigatemorefactors,withorwithoutcenter
points?Acarefullychosenfractionorsubsetofthefactorcombinationsmightbeall
thatisnecessary,whichisreferredtoasfractionalfactorialdesigns.Ingeneral,a
"degreeoffractionation(2p ),"suchas1/2,1/4,andsoforthoftherunscalledforin
thefullfactorialdesignarechosen,asshowninFigure2.Inthepreviousexample,
withninefactorsresultingin512runsforafullfactorialdesign,fractionalfactorial
designscanaccomplishthesameevaluationinaslittleas32runs(usinga1/16
fraction:512/16,or2kp .Thelatterisarrivedatbytakingthefullfactorial2k*2p or
2kp ).

Figure3(a):ExampleexperimentalconditionsthatmightbeusedforaPlackettBurmandesignforaneightfactorultra
performanceLCexperiment.Shownaretheexperimentaldesignsetupwiththefactornamesandnominal,upper,and
lowervalues.
principle"thatstatesthatwhiletherecanbemanyfactors,fewcanactuallybe
important,andexperimentsareusuallydominatedbymaineffectsnoteveryvariable
interactswitheveryothervariable.Therefore,themostcriticalissueinfractional
factorialdesignisselectionoftheproperfractionofthefullfactorialtostudy.Thereis,
ofcourse,apricetobepaidforreducingthenumberofrunsandthatisthatnotall
factorscanbedetermined"freeandclear,"butarealiasedorconfoundedwithother
factors,andthedesignresolutionreferstothedegreeofconfounding.Fullfactorial
designshavenoconfounding,andhavearesolutionofinfinity.Fractionalfactorial
designscanberesolution3(somemaineffectsconfoundedwithsometwolevel

interactions),resolution4(somemaineffectsconfoundedwiththreelevel
interactions),andresolution5(somemaineffectsareconfoundedwithfourlevel
interactions).Ingeneral,theresolutionofadesignisonemorethanthesmallest
orderinteractionthatamaineffectisconfounded(aliased)with.Wherepossible,
importantfactorsshouldnotbealiasedwitheachother.Chromatographicknowledge
andthelessonslearnedduringmethoddevelopment(forexample,whataffectsthe
separationthemost?)areveryimportantforselectingtheproperfactorsandfraction.
Butdonotworry,runscanalwaysbeaddedtofractionalfactorialstudiesif
ambiguitiesresult.Forrobustnesstesting,itusuallyis
PlackettBurmanDesignssufficienttodeterminewhetheramethodisrobustto

Figure3(b):ExampleexperimentalconditionsthatmightbeusedforaPlackettBurmandesignforaneightfactorultra
performanceLCexperiment.Shownarethesuggestedexperimentaldesignandconditionsforthe12
chromatographicruns.
manychanges,ratherthantodeterminethevalueofeachindividualeffect,and
PlackettBurmandesignsareveryefficientscreeningdesignswhereonlymain
effectsareofinterest.PlackettandBurmanpublishedtheirnowfamouspaperback
in1946describingtheireconomicalexperimentaldesignsinmultiplesoffourrather
thanapoweroftwo(9),andPlackettBurmandesignsfrequentlyhavebeenreported
intheliteratureforchromatographicrobustnessstudies(see,forexample,references
1012).TableIshowsagenericPlackettBurmandesignforthe12runsneededto
evaluate11factors,accordingtothegeneralformulaN1factors.Figures3aand
3billustrateanexampleofsomeactualexperimentalconditionsthatmightbeused
foraPlackettBurmandesignforaneightfactorultraperformanceLCexperiment.A
PlackettBurmandesignisatypeofresolution3twolevelfractionalfactorialdesign
wheremaineffectsarealiasedwithtwowayinteractions.PlackettBurmandesigns
alsoexistfor20(19factors),24(23factors),and28(27factors)rundesigns,but
theseareseldomusedinchromatographyasthereisrarelytheneedtoevaluateso
manyfactors.IninstanceswhereN1factorsdonotresultinamultipleoffour,
"dummy"factorsareused.
DeterminingtheFactors,MeasuringtheResultsTypicallyfactorsarechosen
AtypicalLCmethodconsistsofmanydifferentparametersthatcanaffecttheresults.
Variousinstrument,mobilephase,andevensampleparametersmustbetakeninto
account.Eventhetypeofmethod(isocraticversusgradient)candictatenumbersand
importanceofvariousfactors.

TableIII:Examplerobustnessfactorselectionandlimitsforagradientmethod.Factorsandlimits
listedhereareinadditiontomanyofthefactorsconsideredinanisocraticmethod.
symmetricallyaroundanominalvalue,orthevaluespecifiedinthemethod,forming
anintervalthatslightlyexceedsthevariationsthatcanbeexpectedwhenthemethod
isimplementedortransferred(1315).Forexample,ifthemethodcallsforpremixing
amobilephaseof60%methanol,thenthehigh(1)andlow(1)factorsmightbe
58%and62%methanol,orsomesimilarrangeexpectedtobracketthevariabilitya
properlytrainedanalystusingproperlaboratoryapparatuscanbeexpectedto
measure.Inthecaseofinstrumentsettings,manufacturers'specificationscanbe
usedtodeterminevariability.Iftheinstrumentisbeingusedtogeneratethemobile
phase,gradients,orsetthetemperature,forexample,thentherangeshouldbracket
thosespecifications.Ultimately,therangeevaluatedshouldnotbeselectedtobeso
widethattherobustnesstestwillpurposelyfail,buttorepresentthetypeofvariability
routineencounteredinthelaboratory.Foranychromatographicrun,therearea
TableIIlistssomefactorlimitsforanisocraticmethodinwhichthemobilephaseis
premixed.Mobilephasecomposition,flowrate,temperature,andwavelengthallare
considered.GradientmethodfactorlimitsarelistedinTableIII.Gradientmethods
representaslightlydifferentfactorselectionchallengeinadditiontosomeofthe
factorsthatneedtobeconsideredforanisocraticmethod,gradienttimingshouldbe
takenintoconsideration.

Figure4:Exampleprobabilityplot.Inthisexamplethefactoreffectsareplottedagainstalinear
distributiondeparturesfromthelinewouldindicaterobustnessissues.
myriadofresultsgenerated.Typicalresultsinvestigatedforrobustnessstudies
includecriticalpairresolution,efficiency(N),retentiontimeofthemaincomponents,
tailingfactor,area,height,andquantitativeresultslikeamounts.Notethatthereare
differentwaysofmeasuringsomeofthesefactorsstandardoperatingprocedures
(SOPs)orotherdocumentationshouldspecifyhowtheresultswerecalculated.Ifa
quantitativeresultisdesired,itisnecessarytomeasurebothsamplesandstandards.
Replicationofthedesignpointsalsocanimprovetheestimateoftheeffects.In
addition,itisagoodideatomeasureresultsformultiplepeaks,ascompoundswill
responddifferentlyaccordingtotheirownphysicochemicalcharacteristicsfor
example,ionizableandneutralcompoundsmightbepresentinthesamemixture.
Chromatographydatasystems(CDS)alsoareavailablethat
AnalyzingtheResultsperformmanyoftherequisitecalculationsandreportingfor
Oncethedesignhasbeenchosen,thefactorsandlimitsdetermined,andthe
chromatographicresultsgenerated,therealworkstarts.Allofthatdatamustbe
analyzed,andatthispoint,manyanalyticalchemistsbegintosearchouttheir
residentstatistician.Ultimately,thelimitsuncoveredbytherobustnessstudy,
determinedinthedatatreatmentorobservedinthegraphs,areusedtosetsystem
suitabilityspecifications.
Whileconsultingandcollaboratingwithagoodstatisticianforrobustnessstudiesis
alwaysagoodidea,therearemanytoolsavailabletoassisttheanalystinanalyzing
theresults.
Statisticalsoftwareisavailablefromavarietyofsources.Thereareaddinprograms
forExcel,andpopularcommerciallyavailablesoftwaresuchasSPSS(Chicago,
Illinois,orspss.com[3] )JMP(Cary,NorthCarolina,orJMP.com[4] )orMinitab(State
College,Pennsylvania,orminitab.com[5] ).Thirdpartysoftwareaddstothevalidation
process,asittooneedstobevalidated.

Figure5:Exampleeffectsplot.Factoreffectscanbeeitherpositiveornegative.Themagnitudeanddirectionofthebarindicatesthemagnitudeoftheeffect.
robustnessstudies(16).Butunlikethirdpartysoftware,CDSshavetheadvantage
thatthedataaretraceable,validationneedonlybeperformedonce,andtheentire
audittrail,relationaldatabase,andreportingfeaturescanbeusednotjustto
generatethedata,butalsotoanalyzeandreportthedataduringmethodvalidation.
Whileacomprehensivestatisticaldiscussionisoutsidethescopeofthiscolumn,
severalgoodreferencesareavailableformoredetail(57).Awealthofinformation
alsocanbefoundontheinternetsimplyby"Googling"thetermofinterestorfor
statisticinformationingeneral.
Essentially,theanalystmustcomparethedesignresults,ortheresultsobtainedfrom
theexperimentsrun,forexample,accordingtoTableI,orFigure3,tothenominal
results.Regressionanalysisandcalculationofstandardorrelativeerrorarecommon
waystolookatthedata.Analysisofvariance(ANOVA),whichisatestthatmeasures
thedifferencebetweenthemeansofgroupsofdata,isanothercommonwayof
makingthecomparison.SometimescalledanFtest,ANOVAiscloselyrelatedtothe
ttest.Themajordifferenceisthat,wherethettestmeasuresthedifferencebetween
themeansoftwogroups,anANOVAteststhedifferencebetweenthemeansoftwo
ormoregroups.
Theprimarygoalofanyrobustnessstudyistoidentifythekeyvariablesorfactors
thatinfluencetheresultorresponse,andgraphsareaneasywaytoobservethe
effectsataglance.Effectsplotsandprobabilityplots,aretwocommonwaysto
representrobustnessdata,andmostgeneralpurposestatisticalsoftwareprograms
canbeusedtogeneratetheseplots.AsillustratedinFigure4,inaprobabilityplotthe
dataareplottedinsuchawaythatthepointsshouldapproximatelyformastraight
line.Departuresfromthestraightlineindicatedeviationsinthedatathataffect
robustness.Differenttypesofprobabilityplots,callednormalorhalfnormal
probabilityplots,areusedtofurtherqualifythedata.Normalprobabilityplotsare
usedtoassesswhetherornotthedatasetisapproximatelynormallydistributed.
Halfnormalprobabilityplotscanidentifytheimportantfactorsandinteractions.
Aneffectsplotisanothertypeofgraphicalrepresentation,asdepictedinFigure5.
Similartoabarchart,orhistogram,theeffectsplotalsocanidentifytheimportant
factorsandinteractions.
DocumentationandReportingConclusionAcknowledgmentsReferences
Asthesayinggoes,ifitisnotwrittendown,ordocumentedinareport,itislikeit
neverhappened.Properdocumentationoftherobustnessstudyisofcourse
essentialtothemethodvalidationprocess.Aproperlyconstructedreportmust
includetheexperimentaldesignusedforthestudy,allofthegraphsusedtoevaluate
thedata,andtablesofinformationincludingthefactorsevaluatedandthelevels,and
thestatisticalanalysisoftheresponses.Thefactorlimits,andanysystemsuitability
specificationsarrivedatalsoshouldbetabulated.Aprecautionarystatementalso
shouldbeincludedforanyanalyticalconditionsthatmustbesuitablycontrolledfor
measurementsthataresusceptibletovariationsintheprocedure.
Aproperlydesigned,executed,andevaluatedrobustnessstudyisacritical
componentofanymethodvalidationprotocol.Inadevelopmentlaboratory,a
robustnessstudycanprovidevaluableinformationaboutthequalityandreliabilityof
themethod,andisanindicationofhowgoodajobwasdoneindevelopingand
optimizingthemethod,indicatingwhetherornotfurtherdevelopmentoroptimization
isnecessary.Whenperformedearlyinthevalidationprocess,arobustnessstudy
canprovidefeedbackonwhatparameterscanaffectthemethodifnotproperly
controlled,andhelpinsettingsystemsuitabilityparametersforwhenthemethodis
implemented.Indeed,asrecommendedearlier,performingtherobustnessstudy
beforeevaluatingadditionalmethodvalidationparameterscansavealotoftimeand
effort,perhapspreventingthenecessitytorevalidate.
TheauthorswouldliketoacknowledgethecontributionsofJimMorgado(Pfizer,
Groton,Connecticut)andLaurenWood(WatersCorporation,Milford,Massachusetts)
forcontributionstothismanuscript.
MichaelE.Swartz"ValidationViewpoint"CoEditorMichaelE.SwartzisaPrincipal
ScientistatWatersCorp.,Milford,Massachusetts,andamemberofLCGC'seditorial

advisoryboard.
IraS.Krull"ValidationViewpoint"CoEditorIraS.KrullisanAssociateProfessorof
chemistryatNortheasternUniversity,Boston,Massachusetts,andamemberof
LCGC'seditorialadvisoryboard..
(1)USP29,Chapter1225,30503053(2006).
(2)Int.Conf.Harmonization,HarmonizedTripartiteGuideline,ValidationofAnalytical
Procedures,TextandMethodology,Q2(R1),November2005.
(3)PharmacopeialForum31(2)Mar.Apr.,549(2005).
(4)G.E.P.Box,W.G.Hunter,andJ.S.Hunter,StatisticsforExperimenters(JohnWiley
andSons,NewYork,1978).
(5)J.C.MillerandJ.N.Miller,StatisticsforAnalyticalChemistry(EllisHorwood
Publishers,Chichester,UK,1986).
(6)NIST/SEMATECHeHandbookofStatisticalMethods,
http://www.itl.nist.gov/div898/handbook[6]
(7)P.C.MeierandR.E.Zund,StatisticalMethodsinAnalyticalChemistry(JohnWiley
andSons,NewYork,1993).
(8)R.L.PlackettandJ.P.Burman,Biometrika33,305(1946).
(9)R.Ragonese,M.Mulholland,andJ.Kalman,J.Chromatogr.A870,45(2000).
(10)A.N.M.Nguyet,L.Tallieu,J.PlaizierVercammen,D.L.Massart,andY.V.
Heyden,J.Pharm.Biomed.Analysis21,119(2003).
(11)SW.SunandHT.Su,J.Pharm.Biomed.Analysis29,881894(2002).
(12)Y.V.Heyden,A.Nijhuis,B.G.M.SmeyersVerbek,andD.L.Massart,J.Pharm.
Biomed.Analysis24,723753(2001).
(13)Y.V.Heyden,F.Questier,andD.L.Massart,J.Pharm.Biomed.Analysis18,43
56(1998).
(14)F.Questier,Y.V.Heyden,andD.L.Massart,J.Pharm.Biomed.Analysis18,
287303(1998).
(15)P.Lukalay,andJ.Morgado,LCGC24(2),150156(2006).
2016AdvanstarCommunications,Inc.Allrightsreserved.Reproductioninwholeorinpartisprohibited.Pleasesendanytechnicalcommentsorquestionstoourwebmasters.

SourceURL:http://www.chromatographyonline.com/methodvalidationandrobustness
Links:
[1]http://www.chromatographyonline.com/irakrull
[2]http://www.chromatographyonline.com/michaeleswartz
[3]http://spss.com/
[4]http://JMP.com/
[5]http://minitab.com/
[6]http://www.itl.nist.gov/div898/handbook/