Interpreting Wait Events

InterpretingWaitEvents
WhatareWaitEvents?
CommonWaitEvents
CollectingWaitEvents
TracingWaitEvents
10gWaitInterfaceEnhanments
EventWaitAnalysiswithStatspack
HowWaitEventscanbeusedtoimproveperformance(withgoodexamples)
Moreexampleson10g
WhatareWaitEvents?
"WaitEvent"Defined
Atanygivenmoment,everyOracleprocessiseitherbusyservicingarequestorwaitingforsomethingspecifictohappen.By"busy"wemeanthattheprocesswishesto
usetheCPU.Forexample,adedicatedserverprocessmightbesearchingthebuffercachetoseeifacertaindatablockisinmemory.Thisprocesswouldbesaidtobe
busyandnotwaiting.TheARC0process,meanwhile,mightbewaitingforLGWRtosignalthatanonlineredologneedsarchiving.Inthiscase,theARC0processis
waiting.
ThekerneldevelopersatOraclehavedefinedalistofeverypossibleeventthatanOracleprocesscouldbewaitingfor.Atanymoment,everyOracleprocessthatisnot
busyiswaitingforoneoftheseeventstooccur.SupposeanapplicationhassubmittedaCOMMITstatementandtheserverprocessiswaitingfortheLGWRprocessto
signalthattheredologbufferhasbeenflushedtodisk.Thiswaiteventiscalled"logfilesync."Anotherdedicatedserverprocessmightbewaitingforarowlevellockon
theINVOICEStabletobefreedsothataSELECTFORUPDATEstatementcancontinue.Thatwaiteventiscalled"enqueue."
Byqueryingv$views,wecanseewhateventsprocessesarewaitingontoanamazinglevelofdetail.Forexample,wemightlearnthatadedicatedserverprocesshas
beenwaiting30millisecondsfortheoperatingsystemtoreadeightblocksfromdatafile42,startingatblock18042.Wecanalsoseesummaryinformationofhowmuch
timeeachOracleprocesshasspentwaitingoneachtypeofwaiteventforthedurationoftheprocess.Inaddition,wecandirectanOracleprocesstowritedetailedwait
eventdatatoatracefileforlateranalysisusingTKPROF.
WhyWaitEventInformationisUseful
Usingthewaiteventinterface,youcangetinsightsintowheretimeisbeingspent.Ifareporttakesfourhourstocomplete,forexample,thewaiteventinterfacewilltellyou
howmuchofthatfourhourswasspentwaitingfordiskreadscausedbyfulltablescans,diskreadscausedbyindexlookups,latchcontention,andsoon.
Thewaiteventinterfaceprovidesmuchmoreinformationtoworkwiththancachehitratiosdo.Yougetdatathatcantouchuponsomanydifferentareasofyourdatabase
suchasdiskI/O,latches,parallelprocessing,networktraffic,checkpoints,androwlevellocking.Atthesametime,yougetincredibledetailsuchasthefilenumberand
blocknumberofablockbeingreadfromdisk,orthenameofalatchbeingwaitedonalongwiththenumberofretries.
Thewaiteventinterfacewillnotalwaysgiveyoualloftheinformationyouneedinordertodiagnoseandsolveaproblem,butitwillcertainlypointyouintheright
direction.Youmightthinkthebuffercacheistoosmallbecausethecachehitratioisonly70%,butinfact,theapplication'sslowresponsetimecouldbecausedbylatch
contentioninthesharedpool,abottleneckinthelogwriter,oranyofanumberofotherthings.
PriortoOracle10g,Oracle'sdefaultoptimizermodewascalledchoose.Inthechooseoptimizermode,Oraclewillexecutetherulebasedoptimizerifthereareno
statisticspresentforthetableitwillexecutethecostbasedoptimizerifstatisticsarepresent.Thedangerwithusingthechooseoptimizermodeisthatproblemscanoccur
incaseswhereoneOracletableinacomplexqueryhasstatisticsandtheothertablesdonot.
StartinginOracle10g,thedefaultoptimizermodeisall_rows,favoringfulltablescansoverindexaccess.Theall_rowsoptimizermodeisdesignedtominimizecomputing
resourcesanditfavorsfulltablescans.Indexaccess(first_rows)addsadditionalI/Ooverhead,buttheyreturnrowsfaster.
WhenonlysometablescontainCBOstatistics,Oraclewillusethecostbasedoptimizationandestimatestatisticsfortheothertablesinthequeryatruntime.Thiscan
causesignificantslowdownintheperformanceoftheindividualquery.
CommonWaitEventNamesandWhatTheyMean
Althoughtherearemanydifferenttypesofwaitevents,themajorityofthemcomeupveryinfrequentlyortendnottobesignificant.Inpractice,onlyafewdozenwait
eventstendtobeofinteresttomostDBAs.You'llseedifferentwaiteventssurfacingindifferentenvironmentsbasedonwhichOraclefeatureshavebeenimplementedand
whichcapabilitiesofthedatabasearebeingtaxedthemost.Forexample,thePXwaiteventswon'tappearifyouaren'tusingparallelquery,andthevirtualcircuitstatus
waiteventwon'tappearifyouarenotusingthemultithreadedserverarchitecture(orsharedserverarchitectureasitismorerecentlycalled).Alongthoselines,thelogfile
syncandenqueuewaiteventsprobablywon'tbeprevalentinaprimarilyreadonlysystem.Herearesomeofthemostcommonwaiteventsandwhattheymean:
WaitEvent
Description
bufferbusywaits
Thesessionwantstoaccessadatablockthatiseither1)currentlynotinmemory,butanotherprocesshasalreadyissuedanI/Orequestto
readtheblockintomemory,or2)inmemorybutinanincompatiblemode(currentversusconsistent,forexample).Forexampleanother
sessionisusingthatblockviaaninsert,updateordelete.
controlfileparallelwrite
ThesessionhasissuedmultipleI/Orequestsinparalleltowriteblockstoallcontrolfiles,andiswaitingforallofthewritestocomplete.
controlfilesequentialread Thesessioniswaitingforblockstobereadfromacontrolfile.
dbfileparallelread
ThesessionhasissuedmultipleI/Orequestsinparalleltoreadblocksfromdatafilesintomemoryandiswaitingforallrequeststocomplete.
ThismayoccurduringrecoveryorduringregularactivitywhenasessionbatchesmanysingleblockI/Orequeststogetherandissuesthemin
parallel.
dbfileparallelwrite
Theprocess,typicallyDBWR,hasissuedmultipleI/Orequestsinparalleltowritedirtyblocksfromthebuffercachetodiskandiswaiting
forallrequeststocomplete.
dbfilescatteredread
ThesessionhasissuedanI/Orequesttoreadaseriesofcontiguousblocksfromadatafileintothebuffercacheandiswaitingforthe
operationtocomplete.Thistypicallyhappensduringafulltablescanorfastfullindexscan.Oraclereadsupto
DB_FILE_MULTIBLOCK_READ_COUNTconsecutiveblocksatatimeandscattersthemintobuffersinthebuffercache
dbfileparallelwrite
Theprocess,typicallyDBWR,hasissuedmultipleI/Orequestsinparalleltowritedirtyblocksfromthebuffercachetodiskandiswaiting
forallrequeststocomplete.
dbfilescatteredread
ThesessionhasissuedanI/Orequesttoreadaseriesofcontiguousblocksfromadatafileintothebuffercacheandiswaitingforthe
operationtocomplete.Thistypicallyhappensduringafulltablescanorfastfullindexscan.Oraclereadsupto
DB_FILE_MULTIBLOCK_READ_COUNTconsecutiveblocksatatimeandscattersthemintobuffersinthebuffercache
dbfilesequentialread
ThesessionhasissuedanI/Orequesttoreadoneblockfromadatafileintothebuffercacheandiswaitingfortheoperationtocomplete.
ThistypicallyhappensduringanindexlookuporafetchfromatablebyROWIDwhentherequireddatablockisnotalreadyinmemory.
Thiscalldiffersfrom"dbfilescatteredread"inthatasequentialreadreadsdataintocontiguousmemory(whilstascatteredreadreads
multipleblocksandscattersthemintodifferentbuffersintheSGA).
directpathread,
directpathwrite
ThesessionhasissuedasynchronousI/Orequeststhatbypassthebuffercacheandiswaitingforthemtocomplete.Thesewaiteventsoften
involvetemporarysegments,sortingactivity,parallelqueryorhashjoins.
enqueue
Thesessioniswaitingonanenqueue(alockyoucanseeinv$lock).Thiscommonlyoccurswhenoneuseristryingtoupdatearowina
tablethatiscurrentlybeingupdatedbyanotheruser.
freebufferwaits
Thesessionneedsafreebuffersoitcanbringadatablockintothebuffercacheandiswaitingforabufferthatisnotdirtytobecome
available.ThiscanoccurifDBWRisnotwritingdirtybufferstodiskfastenough.
latchfree
Thesessioniswaitingforalatchheldbyanothersession.(Thiseventdoesnotapplytoprocessesthatarespinningwhilewaitingforalatch
whenaprocessisspinning,itisnotwaiting.)
librarycacheloadlock
Thesessioniswaitingfortheopportunitytoloadanobjectorapieceofanobjectintothelibrarycache.(Onlyoneprocesscanloadan
objectorapieceofanobjectatatime.)
librarycachepin
Thesessionwantstopinanobjectinmemoryinthelibrarycacheforexamination,ensuringnootherprocessescanupdatetheobjectatthe
sametime.ThishappenswhenyouarecompilingorparsingaPL/SQLobjectoraview.
logbufferspace
Thesessioniswaitingforspaceinthelogbuffer.(SpacebecomesavailableonlyafterLGWRhaswrittenthecurrentcontentsofthelog
buffertodisk.)ThistypicallyhappenswhenapplicationsgenerateredofasterthanLGWRcanwriteittodisk.
logfileparallelwrite
Thesessioniswaitingforblockstobewrittentoallonlineredologmembersinonegroup.LGWRistypicallytheonlyprocesstoseethis
waitevent.Itwillwaituntilallblockshavebeenwrittentoallmembers.
logfilesequentialread
Thesessioniswaitingforblockstobereadfromtheonlineredologintomemory.Thisprimarilyoccursatinstancestartupandwhenthe
ARCHprocessarchivesfilledonlineredologs.
logfileswitchcompletion
Thesessioniswaitingforalogfileswitchtocomplete,typicallysomoreredocanbegenerated.
logfilesync
ThesessioniswaitingforLGWRtofinishflushingthelogbuffertodisk.Thisoccurswhenausercommitsatransaction.(Atransactionis
notconsideredcommitteduntilalloftheredotorecoverthetransactionhasbeensuccessfullywrittentodisk).Probablyyouarecommiting
toooften
undosegmentextension
Thesessioniswaitingforanundosegmenttobeextendedorshrunk.
writecompletewaits
Thesessioniswaitingforarequestedbuffertobewrittentodiskthebuffercannotbeusedwhileitisbeingwritten.
Thereareseveralwaiteventsthatwecall"idleevents"becauseeachofthesewaiteventstypicallyoccurswhentheOracleprocesshasnothingtodoandiswaitingfor
somebodytogiveitatask.Idleeventsareusuallynotveryinterestingfromatuningstandpoint,soweusuallyoverlookthemwhenevaluatingdataextractedfromthewait
eventinterface.Thecommonidleeventsareasfollows:
IdleEvents
clientmessage
PXDeq:ExecuteReply
dispatchertimer
PXDeq:ExecutionMsg
gcsforaction
PXDeq:SignalACK
gcsremotemessage
PXDeq:TableQNormal
gesremotemessage
PXDequewait
i/oslavewait
PXIdleWait
jobqslavewait
queuemessages
lockmanagerwaitforremotemessage rdbmsipcmessage
nullevent
slavewait
parallelquerydequeue
smontimer
pipeget
SQL*Netmessagefromclient
PL/SQLlocktimer
SQL*Netmessagetoclient
pmontimer
SQL*Netmoredatafromclient
PXDeqCredit:needbuffer
virtualcircuitstatus
PXDeqCredit:sendblkd
wakeuptimemanager
WhatWaitEventInformationDoesNotProvide
IfanOracleprocesshasworktodobut,mustwaitforsomethingtohappenbeforeitcancontinue,thentheprocesswillbewaitingonanonidlewaitevent.Ifa
processhasnothingtodo,itwillbewaitingonanidlewaitevent.Sowhathappensifaprocesshasworktodoandisbusydoingit?Whenaprocessisbusy,therewill
PXDeqCredit:sendblkd
wakeuptimemanager
WhatWaitEventInformationDoesNotProvide
IfanOracleprocesshasworktodobut,mustwaitforsomethingtohappenbeforeitcancontinue,thentheprocesswillbewaitingonanonidlewaitevent.Ifa
processhasnothingtodo,itwillbewaitingonanidlewaitevent.Sowhathappensifaprocesshasworktodoandisbusydoingit?Whenaprocessisbusy,therewill
benoinformationinthewaiteventinterfacesincetheprocessisnotwaiting.
WhenwelookatthewaiteventinformationextractedfromanOracleinstance,weseedetailedinformationabouthowmanytimesandhowmuchtimewas
spentwaitingforspecificeventstooccur.ButwedonotseeanythingaboutthetimeperiodsinwhichtheprocessrequesteduseoftheCPU.Thismeansthat
thewaiteventinterfaceisnotabletoprovideinformationaboutthefollowing:
l
l
l
TimespentusingtheCPU
TimespentwaitingforaCPUtobecomeavailable
Timespentwaitingforrequestedmemorytobeswappedbackintophysicalmemory
Thisisimportanttokeepinmindbecausethereisaneasytraptofallinto.YoucouldbetroubleshootingaveryslowSELECTstatementandlearnfromthe
waiteventinterfacethattheprocessdoesnothavesignificantwaiteventswhenrunningthequery.Thiscouldleadyoutothinkthatthestatementisoptimal,
andthatitjusttakesalongtimetorun.Infact,however,thequerymightbeperforminghugenumbersoflogicalreadsandthenumberofbuffergetscouldbe
reducedbyrewritingthequery.
WhenOracleneedstoaccessadatablockandtheblockisalreadyinthebuffercache,alogicalreadoccurswithnophysicalread.Theprocessisabletoreadtheblock
withouttheoccurrenceofanywaitevents.LargeamountsofCPUtimecouldbeconsumedonsignificantnumbersoflogicalreads,andthewaiteventinterfacewillhave
nothingtoshowfortheelapsedtime.
Statementparsingandspinningwhilewaitingforalatchtobecomeavailablearetwootherexamplesofactivitiesnotaccountedforbythewaiteventinterface.AnOracle
processusesCPUtimewhileparsingastatementorspinningonabusylatchsincenowaitingisinvolved,thewaiteventinterfacedoesnothaveanythingtoreport.
BeginninginOracle9i,theportionoftheStatspackreportthatlistswaiteventinformationalsolistsCPUusage.Thisisveryhelpfulinformation.Itallowsustoeasily
comparetimespentwaitingtotimespentprocessing,soweknowwheretofocusourtuningefforts.However,itshouldbepointedoutthattheCPUusageinformationin
thissectionoftheStatspackreportdoesnotcomefromthewaiteventinterface.Instead,StatspackmergesdatacollectedfromthewaiteventinterfacewithCPUusage
informationcollectedfromthev$sysstatdynamicperformanceview.
ANoteAboutthetimed_statisticsParameter
TheOraclekerneliscapableoftimingmanyactivitiesincludingwaitevents.InOracle9iRelease2timedstatisticsarecollectedbydefault,andsometimingsarecollected
inunitsofmicroseconds.Thetimed_statisticsinstanceparameterisusedtoenableanddisabletimedstatisticscollection.Whentimed_statisticsissettoFALSE,alltimes
inthewaiteventinterfacewillappearaszero.Youmayenabletimedstatisticscollectionattheinstanceorthesessionlevelwiththefollowingcommands:
l
l
ALTERSYSTEMSETtimed_statistics=TRUE
ALTERSESSIONSETtimed_statistics=TRUE
Youmayenabletimedstatisticsattheinstancelevelonthenextandallsubsequentinstancerestartsbyaddingthefollowinglinetotheinstanceparameterfile:
timed_statistics=true
Inpractice,theoverheadofcollectingtimedstatisticsisextremelysmall.Inmostcases,thebenefityou'llgetfromhavingtiminginformationatyourdisposalwilloutweigh
theperformanceoverhead.Forseveralyears,manyDBAshavebeenrunningtheirproductionsystemswithtimedstatisticsenabledatalltimes.WithOracle9iRelease2,
thisisnowthedefaultbehavior.
CollectingWaitEventInformation
Therearetwocommonwaysofcollectingwaiteventinformation:theOracleextendedtrace(10046event)andusingV$viewswhichisthemaintopicofthisarticle.Any
userwiththeSELECTANYTABLEsystemprivilegeortheSELECT_CATALOG_ROLErolecanquerythev$views.Onlyuserswhocanconnecttothedatabaseas
SYSDBAorexecutetheDBMS_SUPPORTorDBMS_SYSTEMpackagescanactivatewaiteventtracinginotherdatabasesessions.Inatypicalenvironment,DBAs
haveaccesstothewaiteventinterfacebutgeneralusersdonot.
Oracleprovidesanumberofviewstoaidintuningafeware:
*V$SYSTEM_EVENT,whichgivesyoutotalsystemwaitsforaneventsincedatabasestartup.
*V$SESSION_EVENT,whichisatthesessionlevel.
*V$SESSION_WAITgivesyouwhatanactivesessioniswaitingonatthatpointintime.
*V$SESSION,whichgivesyousessioninformationandhasacolumnROW_WAIT_OBJ#thattellsyoutheobjectinvolvedwhenjoinedwithOBJECT_IDor
DATA_OBJECT_IDinDBA_OBJECTS.
Version10ghassomemoreV$viewsfortrackinghistorycalledV$%_HISTORY,includingV$ACTIVE_SESSION_HISTORY(ASH),whichsamplesnonidlewait
eventseverysecondandnewcolumnsinV$SESSIONtocombineinformationinV$SESSION_WAIT.
Inadditiontothefourdynamicperformanceviews,Oracleprovidesatracingfacilitywhereextremelydetailedwaiteventinformationiswrittentoatracefileforlater
evaluation.InmorerecentreleasesofOracle,aPL/SQLpackageisprovidedforactivatingthewaiteventtracingfacility.InolderreleasesofOracle,thewaiteventtracing
facilityisactivatedbysettingdebugeventnumber10046(nottobeconfusedwithawaitevent).
The V$SYSTEM_EVENT View
Thev$system_eventviewshowsonerowforeachwaiteventname,alongwiththetotalnumberoftimesaprocesshaswaitedforthisevent,thenumberoftimeouts,the
totalamountoftimewaited,andtheaveragewaittime.AllofthesefiguresarecumulativeforallOracleprocessessincetheinstancestarted.Waiteventsthathavenot
occurredatleastoncesinceinstancestartupdonotappearinthisview.Thev$system_eventviewhasthefollowingcolumns:
NameNull?Type
EVENTVARCHAR2(64)
TOTAL_WAITSNUMBER
TOTAL_TIMEOUTSNUMBER
TIME_WAITEDNUMBER
AVERAGE_WAITNUMBER
NameNull?Type
EVENTVARCHAR2(64)
TOTAL_WAITSNUMBER
TIME_WAITEDNUMBER
AVERAGE_WAITNUMBER
TIME_WAITED_MICRONUMBER
EVENTisthenameofthewaitevent.YoucanseealistofallwaiteventnamesknowntoyourOraclekernelwiththefollowingquery:
SELECTnameFROMv$event_name
TOTAL_WAITSisthetotalnumberoftimesaprocesshaswaitedforthiseventsinceinstancestartup.
TOTAL_TIMEOUTSisthetotalnumberoftimesaprocessencounteredatimeoutwhilewaitingforanevent.Whenaprocessbeginstowaitforanevent,itspecifiesa
timeoutperiodafterwhichtheoperatingsystemshouldwakeitupiftheeventhasnotyettranspired.Forexample,whenanOracleprocessissuesanI/Orequesttoreada
blockfromadatafile(thedbfilesequentialreadwaitevent),theprocesssetsatimeoutofonesecond.UsuallytheI/Orequestwillcompleteinlessthanonesecondand
notimeoutwilloccur.Butifthereadshouldtakelongerthanonesecondforwhateverreason,atimeoutwilloccurandtheprocesswillwakeup.Theprocessmightdo
someminorhousekeeping,butitwilllikelyjustbeginanothertimeoutperiodofonesecondandcontinuewaitingforthesameevent.
TIME_WAITEDandAVERAGE_WAITshowthecumulative(sum)andaveragetimespentbyprocesseswaitingforthisevent,incentiseconds.Dividethesefiguresby
100inordertogetthewaittimeinseconds.Thesetwocolumnswillshowaszeroiftimedstatisticsarenotenabled.
TIME_WAITED_MICROisthesameasTIME_WAITED,exceptthatthetimeisinmicroseconds.Dividethisfigureby1000000inordertogetthewaittimein
seconds.
Considerthefollowingqueryfromv$system_event:
SELECTevent,time_waited
FROMv$system_event
WHEREeventIN('SQL*Netmessagefromclient','smontimer',
'dbfilesequentialread','logfileparallelwrite')
EVENTTIME_WAITED(sec)
logfileparallelwrite1596.92
dbfilesequentialread286.57
smontimer1306738.37
SQL*Netmessagefromclient165289.89
Sinceinstancestartup,processesonthissystemhavewaitedatotalof286.57secondswhilereadingsingledatafileblocksfromdisk,andover1,596seconds(26minutes)
The V$SESSION_EVENT View
Thev$session_eventviewisalotlikethev$system_eventview,exceptthatitshowsseparaterowsofinformationforeachOracleprocess.Aswithv$system_event,
eventnamesdonotappearinthisviewiftheprocesshasnotwaitedforthematleastonce.Also,whenanOracleprocessterminates(asinthecaseofwhenauserlogs
offthedatabase)alloftherowsinv$session_eventforthatprocesspermanentlydisappear.Thev$session_eventviewhasthefollowingcolumns:
NameNull?Type
SIDNUMBER
EVENTVARCHAR2(64)
TOTAL_WAITSNUMBER
TIME_WAITEDNUMBER
AVERAGE_WAITNUMBER
MAX_WAITNUMBER
TIME_WAITED_MICRONUMBER
SIDindicatesthesessionIDoftheprocesswaitingfortheevent.Youcanqueryv$sessioninordertodeterminetheSIDofthesessionwhosewaiteventsyouwantto
investigate.Thenextfivecolumnsinthev$session_eventviewarethesameasinthev$system_eventview,exceptthatnowtheypertaintotheonespecificprocessinstead
ofallprocesses.
MAX_WAITindicatesthemaximumamountoftimetheprocesshadtowaitfortheevent.LikeTIME_WAITEDandAVERAGE_WAIT,theunitofmeasureis
centisecondsandwilldisplayaszeroiftimedstatisticsarenotenabled.
ConsiderthefollowingquerywhichdisplaysallwaiteventinformationforthecurrentSQL*Plussession:
SELECTevent,total_waits,time_waited_micro,max_wait
FROMv$session_event
WHERESID=
(SELECTsidFROMv$session
WHEREaudsid=USERENV('sessionid'))
EVENTTOTAL_WAITSTIME_WAITED_MICROMAX_WAIT
logfilesync1196292
dbfilesequentialread3261281
dbfilescatteredread67811546328
SQL*Netmessagetoclient763520
SQL*Netmoredatatoclient1630
SQL*Netmessagefromclient75111865627926930
SQL*Netbreak/resettoclient4222812
YoucanseethattheOracleprocessservingthissessionhasspent1.180760secondswaitingfordiskI/O.Althoughtherehavebeen76instanceswheretheprocesswaited
Ifyourunthequeryagainstv$session_eventshownaboveandgetbacknorows,itispossiblethatyouareencounteringOraclebugnumber2429929.Insomereleases
ofOracle9i,including9.2.0.1,theSIDcolumninv$session_eventactuallyshowstheSIDminusoneinsteadoftheSID.
The V$SESSION_WAIT View
Thev$session_waitviewshowsonerowforeachOracleprocess.Therowindicatesthenameofthewaiteventandadditionalparametersthatprovidefurtherinformation
aboutexactlywhattheprocessiswaitingfor(orinformationaboutthemostrecentwaiteventiftheprocessisnotcurrentlywaiting).Whilethev$system_eventand
v$session_eventviewsshowcumulativewaiteventinformation,thev$session_waitviewshowsinformationasofthepresentmomentonly.Thev$session_waitviewhas
thefollowingcolumns:
NameNull?Type
The V$SESSION_WAIT View
Thev$session_waitviewshowsonerowforeachOracleprocess.Therowindicatesthenameofthewaiteventandadditionalparametersthatprovidefurtherinformation
aboutexactlywhattheprocessiswaitingfor(orinformationaboutthemostrecentwaiteventiftheprocessisnotcurrentlywaiting).Whilethev$system_eventand
v$session_eventviewsshowcumulativewaiteventinformation,thev$session_waitviewshowsinformationasofthepresentmomentonly.Thev$session_waitviewhas
thefollowingcolumns:
NameNull?Type
SIDNUMBER
SEQ#NUMBER
EVENTVARCHAR2(64)
P1TEXTVARCHAR2(64)
P1NUMBER
P1RAWRAW(8)
P2TEXTVARCHAR2(64)
P2NUMBER
P2RAWRAW(8)
P3TEXTVARCHAR2(64)
P3NUMBER
P3RAWRAW(8)
WAIT_TIMENUMBER
SECONDS_IN_WAITNUMBER
STATEVARCHAR2(19)
SIDSessionID.
SEQ#Sequencenumberthatuniquelyidentifiesthewaitevent.Thissequenceisincrementedwitheachwaiteventcompletion.
EVENTNameofwaitevent.
P1TEXTAdescriptionofwhattheP1valueisusedfor.Notreliable,usePARAMETER(n)valuefromV$EVENT_NAMEtodetermine.
P1Thisvaluedefinitionisdependentontheevent.UsePARAMETER1valuefromV$EVENT_NAMEtodescribewhatthisvalueisusedfor.
P1RAWSamevalueasP1exceptinhexadecimalformat.
WAIT_CLASS_IDIDofthewaitclass.Columnisin10g.
WAIT_CLASS#Numberoftheclass.Columnisin10g.
WAIT_CLASSThenameofthewaitclass(Idle,Network,SystemI/O,etc.).Columnisin10g.
WAIT_TIMEThelastamountoftimethatthesessionwaitedforanevent.A0valuemeansthesessioniscurrentlywaitingontheevent.Thisvalueof0willnotincrease
untilthesessioncompletesthewaitevent.Timeisincentiseconds.
SECONDS_IN_WAITIfWAIT_TIMEequals0,thenthisisthenumberofsecondsspentwaitingontheeventsofar.Thiscangiveyoucomplicatedresultswhena
timeoutoccurslikewithenqueuesbecauseitwillresetto0,butthisisbeyondthescopeofthisarticle.
STATEStateofthewaitevent:WAIT_TIME=0WAITING,theprocessisinthestateofwaitingonanevent.WAIT_TIME=1WAITEDSHORTTIME,wait
timecompletedinlessthanacentisecond.WAIT_TIME=2WAITEDUNKOWNTIME,durationofwaitisunknownbecauseTIME_STATISTICSwassettofalse.
WAIT_TIME>0WAITEDKNOWNTIME,waitcompleted.WAIT_TIMEisthedurationofthewaiteventincentiseconds.
Thefollowingqueryshowstheparametersassociatedwiththedbfilescatteredreadwaitevent:
SELECT*
FROMv$event_name
WHEREname='dbfilescatteredread'
EVENT#NAMEPARAMETER1PARAMETER2PARAMETER3
188dbfilescatteredreadfile#block#blocks
Thistellsusthatwhenaprocessiswaitingforamultiblockreadfromdisktocomplete(asinthecaseofafulltablescanwherethedatablockswerenotalreadyinthebuff
Thefollowingquerywasrunwhileasessionwasperformingafulltablescan:
setlinesize32000
SELECTsid,seq#,substr(event,1,35)event,p1text,p1,p2text,p2,p3text,p3,wait_time,seconds_in_wait,state
FROMv$session_waitWHEREsid=442
SIDSEQ#EVENT
P1TEXTP1P1RAW
P2TEXTP2P2RAW
P3TEXTP3P3RAWWAIT_TIMESECONDS_IN_WAIT
STATE
442303dbfilescatteredread
file#1700000011
block#272100000AA1
blocks80000000810
WAITEDSHORTTIME
Youcanseethattheprocesswasnotwaitingatthemomentthisquerywasrun,butitslastwaithadbeenforanI/Orequesttoreadeightblocksfromfile17startingatbloc
Formostwaiteventsthisviewissufficient,butitishardlyarobusttuningtoolforatleasttwoimportantreasons:
*Theviewisasnapshotofthepresent.Whenthewaitsceasetoexist,thehistoryofthosewaitsexperiencedbythesessionearlierdisappearstoo,makingaftereffect
diagnosisdifficult.V$SESSION_EVENTprovidescumulativebutnotverydetaileddata.
*V$SESSION_WAITcontainsinformationonlyaboutwaiteventsforallotherrelevantinformationsuchastheuseridandterminalyouhavetojoinitwiththeview
V$SESSION.
InOracleDatabase10g,thewaitinterfacehasbeenradicallyredesignedtoprovidemoreinformationwithlessDBAintervention.Inthisarticle,wewillexplorethosenew
featuresandseehowtheyaidusinthediagnosisofperformanceproblems.Formostoftheperformanceproblems,youwillgetanextendedanalysisfromAutomatic
DatabaseDiagnosticManager(ADDM),butforimmediateproblemsnotyetcapturedbyADDM,thewaitinterfaceprovidesvaluabledatafordiagnosis.
Formostwaiteventsthisviewissufficient,butitishardlyarobusttuningtoolforatleasttwoimportantreasons:
*Theviewisasnapshotofthepresent.Whenthewaitsceasetoexist,thehistoryofthosewaitsexperiencedbythesessionearlierdisappearstoo,makingaftereffect
diagnosisdifficult.V$SESSION_EVENTprovidescumulativebutnotverydetaileddata.
*V$SESSION_WAITcontainsinformationonlyaboutwaiteventsforallotherrelevantinformationsuchastheuseridandterminalyouhavetojoinitwiththeview
V$SESSION.
InOracleDatabase10g,thewaitinterfacehasbeenradicallyredesignedtoprovidemoreinformationwithlessDBAintervention.Inthisarticle,wewillexplorethosenew
featuresandseehowtheyaidusinthediagnosisofperformanceproblems.Formostoftheperformanceproblems,youwillgetanextendedanalysisfromAutomatic
DatabaseDiagnosticManager(ADDM),butforimmediateproblemsnotyetcapturedbyADDM,thewaitinterfaceprovidesvaluabledatafordiagnosis.
TheV$SESSIONView
Sessiondetails,theonlylistingcolumnusedinthescopeofthisarticle.
ROW_WAIT_OBJ#ThisistheobjectidoftheobjectinvolvedandisjoinedtotheDBA_OBJECTS.OBJECT_IDorDBA_OBJECTS.DATA_OBJECT_IDcolumn.
Thiscolumnisonlypopulatedinversion9iandabove.Avaluethatisnot1meansthatthesessioniswaitingonalockforthisobject.Previousversionsofthedatabase
willneedtouseP(n)valuesfromV$SESSION_WAITtofindobjectsinDBA_EXTENTS.
Thev$event_nameView
Thev$event_nameviewshowsreferenceinformationratherthanuptothemomentinformationaboutinstanceperformance.Thisviewshowsonerowforeachwaitevent
knowntotheOraclekernel.Associatedwitheachwaiteventareuptothreeparametersadditionalpiecesofinformationthatprovidemoredetailaboutawaitsituation.
ThisviewdisplaysthedefinitionofP(n)columnsfromV$SESSION_WAITandcanbejoinedontheNAMEcolumn.
Thev$event_nameviewhasthefollowingcolumns:
NameNull?Type
EVENT#NUMBER
NAMEVARCHAR2(64)
PARAMETER1VARCHAR2(64)
EVENT#Numberofthewaitevent.
NAMENameoftheevent.
PARAMETER1DescriptionofthevalueforP1.
WAIT_CLASS_IDIDofthewaitclass.Columnisin10g.
WAIT_CLASS#Numberoftheclass.Columnisin10g.
WAIT_CLASSThenameofthewaitclass(Idle,Network,SystemI/O,etc.).Columnisin10g.
Thev$system_eventandv$session_eventviewsshowcumulativeinformationaboutpastwaitsinsummaryform,leavingoutparameterinformationfromeachindividual
wait.Aswewillseeinthenextsections,waiteventparameterscomeintoplayinthev$session_waitviewandwaiteventtracefiles.
TracingWaitEventActivity(SYSTEMEVENT10046)
DBAsshouldbefamiliarwiththeSQLtracefacilitybuiltintoOracle.Byusingthecommandsbelow,youcanenableSQLtraceforyoursessionoranothersession:
l
ALTERSESSIONSETsql_trace=TRUE
EXECUTESYS.dbms_system.set_sql_trace_in_session(sid,serial#,TRUE)
WhenSQLtraceisenabledforasession,theOracleprocesswritesdetailedtraceinformation(includingtimingdataiftimedstatisticsareenabled)toatracefileina
directoryspecifiedbytheuser_dump_destinstanceparameter.Thesetracefilesareplaintextfilesandhumanreadable,butrathertediousandrepetitive.Youcan
optionallyruntracefilesthroughaprocessorsuchasTKPROFinsteadoflookingatthemdirectly.
Oraclehastheabilitytodirectaprocesstoincludeadditionalinformationinthetracefile,includingwaiteventinformation.InearlierreleasesofOracle,weactivatedthe
waiteventtracingfacilitybysettingdebugevent10046.Debugevent10046stillworksinOracle9i,butrecentreleasesofOracleincludeaPL/SQLpackagebuiltinthat
givesamorefriendlymeansforactivatingwaiteventtracing.
SettingdebugeventsallowsaDBAtoinstructanOracleinstancetotakeonaspecial,atypicalbehavior.Debugeventscanbeused,forexample,tocauseOracletowrite
asystemleveldumpfilewheneveranORA0600erroroccursorskipovercorruptedblocksinatablewhenperformingafulltablescan.Mostdebugeventsshouldnever
besetunlessyouaredirectedtodosobyOracleSupport.Somedebugeventscanputyourdatabaseatrisk.
Debugevent10046affectstheamountofinformationwrittentotracefiles.Itisaverysafedebugevent,andoneofthefewthatyouareallowedtosetwithoutspecial
permissionfromOracleSupport.Debugevent10046canbesettothefollowingvalues:
DebugEventSetting
Effect
10046tracenamecontextforever,level1
EnablesordinarySQLtrace
EnablesSQLtracewithbindvariablevaluesincludedintracefile
EnablesSQLtracewithwaiteventinformationincludedintracefile
10046tracenamecontextforever,level12 Equivalentoflevel4andlevel8together
10046tracenamecontextoff
Turnsofftracing
Youcansetthe10046debugeventtotraceyoursessionandcollectwaiteventinformationinthetracefilewitheitherofthefollowingcommands:
l
EXECUTESYS.DBMS_SUPPORT.START_TRACE
ALTERSESSIONSETevents'10046tracenamecontextforever,level8'
Youcansetthe10046debugeventtotraceanothersessiononthedatabasewithanyofthefollowingcommands:
Findthesession:
EXECUTESYS.DBMS_SUPPORT.START_TRACE
Youcansetthe10046debugeventtotraceanothersessiononthedatabasewithanyofthefollowingcommands:
Findthesession:
setlinesize150
columnNameformata14
columnSIDformat9999
columnPIDformat99999
columnTERMformata15
columnOSUSERformata15
columnProgramformata15
columnStatsformata10
columnLogon_timeformata20
selecta.usernameName,a.sidSID,a.serial#,b.spidPID,
SUBSTR(A.TERMINAL,1,9)TERM,SUBSTR(A.OSUSER,1,9)OSUSER,
substr(a.program,1,10)Program,a.statusStatus,
to_char(a.logon_time,'MM/DD/YYYYhh24:mi')Logon_time
fromv$sessiona,v$processb
wherea.paddr=b.addr
anda.serial#<>'1'
anda.status='ACTIVE'
anda.usernamelikeupper('%&user%')ifyouwanttofilterbyusername
orderbya.logon_time
OnceyouhavetheSIDandSERIAL#ofthesessionyoucanusesomeOraclesuppliedpackages.
settimedstatisticsatusersessionlevelifnotsetatsystemlevel.
executesys.dbms_system.set_boo_param_in_session(&&SID,&&SERIAL,'timed_statistics',true)
setmaxdumpfilesizeifnotsetatsystemlevel.
executesys.dbms_system.set_in_param_in_session(&&SID,&&SERIAL,'max_dump_file_size',10000000)
activatelevel8extendedSQLtracing.
executesys.dbms_system.set_ev(&&SID,&&SERIAL,10046,8,'')
or
executesys.dbms_support.start_trace_in_session(&&SID,&&SERIAL)
*******runallofyourprocessinghere*******
disablesextendedSQLtracing.
executesys.dbms_system.set_ev(&&SID,&&SERIAL,10046,0,'')
or
executesys.dbms_support.stop_trace_in_session(&&SID,&&SERIAL)
CallingDBMS_SUPPORTasshownhereisequivalenttoactivatingdebugevent10046atlevel8.Youmayincludeoptionaladditionalparametersintheprocedurecall
toactivatelevel4or12.TheDBMS_SUPPORTpackageisnotinstalledinthedatabasebydefault.Youneedtorunthedbmssupp.sqlscriptfoundintherdbms/admin
directoryasSYSbeforeyoucancallDBMS_SUPPORT.
ItshouldalsobepointedoutthattheSET_EVprocedureintheDBMS_SYSTEMpackageisnotofficiallysupportedbyOracleCorporationtheypreferthatyouuse
DBMS_SUPPORTinstead.ThereasonisthatDBMS_SYSTEM.SET_EVallowsyoutosetanydebugeventinanysession.Asmentionedabovesomedebugevents
canpotentiallybedangerous,andDBMS_SYSTEM.SET_EVletsyousetsucheventsinanysession.
Unliketimedstatistics,tracingconsumesasignificantamountofsystemresources.Thereforeitisimportanttousethetracingfacilitysparingly.Traceonlythesessionsyou
needtotrace,andonlyforaslongasyouneed.Turnofftracingassoonasitisnolongerneededwithanyofthefollowingcommands:
l
EXECUTESYS.DBMS_SUPPORT.STOP_TRACE
ALTERSESSIONSETEVENTS'10046TRACENAMECONTEXTOFF'
ALTERSESSIONSETsql_trace=FALSE
EXECUTESYS.DBMS_SUPPORT.STOP_TRACE_IN_SESSION(sid,serial#)
oradebugsetorapid[OraclePIDfromv$process]
oradebugsession_event10046tracenamecontextforever,level0
EXECUTESYS.DBMS_SYSTEM.SET_EV(sid,serial#,10046,0,'')
Whenyousetdebugevent10046toalevelof8or12(orcallSTART_TRACEorSTART_TRACE_IN_SESSIONinDBMS_SUPPORT),theOracleprocesswill
writealineintothetracefileeverytimeitfinisheswaitingforanevent.Thelineinthetracefilewillcontainalmostthesameinformationthatwouldhaveappearedinthe
v$session_waitview,butperhapsinaslightlylessfriendlyformat.Youcanalsoseeinthetracefilewhichcursor(andthereforewhichSQLstatement)thewaiteventwas
associatedwith.
Hereisanexcerptfromatracefilegeneratedbysettingdebugevent10046tolevel12onanOracle8idatabase:
=====================
PARSINGINCURSOR#1len=80dep=0uid=502oct=3lid=502
tim=2293771931hv=2293373707ad='511dca20'
SELECT/*+FULL*/SUM(LENGTH(notes))
FROMcustomer_calls
WHEREstatus=:x
ENDOFSTMT
PARSE#1:c=0,e=0,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=0,tim=2293771931
BINDS#1:
bind0:dty=2mxl=22(22)mal=00scl=00pre=00oacflg=03oacfl2=0
size=24offset=0
tim=2293771931hv=2293373707ad='511dca20'
SELECT/*+FULL*/SUM(LENGTH(notes))
FROMcustomer_calls
WHEREstatus=:x
ENDOFSTMT
PARSE#1:c=0,e=0,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=0,tim=2293771931
BINDS#1:
bind0:dty=2mxl=22(22)mal=00scl=00pre=00oacflg=03oacfl2=0
size=24offset=0
bfp=09717724bln=22avl=02flg=05
value=43
EXEC#1:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=2293771931
WAIT#1:nam='SQL*Netmessagetoclient'ela=0p1=675562835p2=1p3=0
WAIT#1:nam='dbfilescatteredread'ela=3p1=17p2=923p3=8
WAIT#1:nam='dbfilesequentialread'ela=0p1=17p2=947p3=1
The"ela="figuresindicatethedurationofthewait.IntracefilesgeneratedbyOracle8iandearlier,theelapsedtimeisshownincentiseconds.BeginninginOracle9i,the
elapsedtimeisshowninmicroseconds.
Youcanseethatwhenthesessionexecutedthequery,therewasawaitshorterthanonecentisecondforamessagetobesenttotheclient,followedbyabunchofwaits
forI/Orequestsagainstfile17.MostoftheI/Orequestsweremultiblockreads,readingeightblocksatatimewhileperformingafulltablescanofthe
CUSTOMER_CALLStable.
Youcanprobablyimaginehowlargeandoverwhelmingtracefilescangetwhenanythingbutatrivialapplicationistraced.ThisiswhyOracleprovidestheTKPROF
utility.TKPROFtakesatracefileasinputandgeneratesanicelyformatted,easytoreadreport.Unfortunately,theTKPROFthatcomeswithOracle8iandearlier
releasesignoreswaiteventinformationwritteninthetracefile.ToanalyzethewaiteventinformationcollectedbyatraceinanOracle8iorearlierenvironment,youwill
needtoeitherporethroughthetracefilemanuallyorwriteyourowntracefileparserandformatter.(We'veheardofpeoplewritingperlscriptstodothis,butwehave
nevercomeacrossthem.)
TheTKPROFutilitythatcomeswithOracle9iiscapableofreportingwaiteventinformationsummarizedbydistinctstatement.BydefaultOracle9iTKPROFignores
waiteventinformationinthetracefileasearlierreleasesdid.However,ifthe"waits=yes"commandlineargumentisprovided,aseparatetableofwaiteventstatisticswill
appearinthereportforeachdistinctstatement.HereisasampleexcerptfromaTKPROFreportshowingwaiteventinformation:
********************************************************************************
SELECTA.customer_id,A.customer_name,COUNT(*)purchases,
MAX(B.transaction_date)last_purchase_date
FROMcustomersA,purchase_historyB
WHEREB.customer_id=A.customer_id
GROUPBYA.customer_id,A.customer_name
ORDERBYA.customer_id
callcountcpuelapseddiskquerycurrentrows
Parse10.060.1202700
Execute10.000.000000
Fetch146121.4125.8078015905521893
total146321.4725.9378015932521893
Missesinlibrarycacheduringparse:1
Optimizergoal:CHOOSE
Parsinguserid:17
RowsRowSourceOperation
21893SORTGROUPBY
1525494HASHJOIN
31212TABLEACCESSFULLCUSTOMERS
1525494INDEXFASTFULLSCANPURCHASE_HISTORY_PK(objectid7824)
Elapsedtimesincludewaitingonfollowingevents:
EventwaitedonTimesMax.WaitTotalWaited
Waited
SQL*Netmessagetoclient14610.000.00
dbfilesequentialread40.000.01
dbfilescatteredread7500.072.25
directpathwrite2710.832.79
directpathread2540.000.01
SQL*Netmessagefromclient14611.2822.40
********************************************************************************
Youcanseethatthequerywasparsedandexecutedonce,andtherewere1461fetchestoread21893rows.TheOracleprocessused21.47CPUsecondsand25.93
secondsofelapsedtimetoparse,execute,andfetchthequery.TheelapsedtimewasgreaterthantheCPUtimebecausetheOracleprocesshadtowaitonseveralwait
events.Thelistingatthebottomofthereportshowstheprocesswaitedonnetworkroundtripstotheclientandreadsandwritestodisk.
Allofthesewaitsseemtomakesense:Therewasonenetworkroundtriptotheclientforeachfetchcall,andittookSQL*Plusalongtime(over22seconds)todisplay
the21,893rowsofdataonmydisplay.Thedbfilesequentialreadsanddbfilescatteredreadsresultedfromthetableaccessandindexfastfullscan.Thedirectpath
writesandreadsresultedfromthehashjoinandsortoperations.IfyouaddtheCPUtimeof21.47secondsandthewaittimes(5.06secondsnotincludingtheSQL*Net
messagefromclientwaits)youcomeupwithanelapsedtimeof26.53seconds.However,thereportshowsthetotalelapsedtimeas25.93seconds.Thisisagood
exampleoftheroundofferrorthatisnotunusualwhenthousandsofveryshorttimeintervalsareaddedtogether.
Insomesituationsyoumaynotbeabletoidentifyexactlywhichsessiontotrace.Thishappensfrequentlywhenyourdatabaseissupportingawebapplicationthathas
frequentshortconnections,oranapplicationservertierthatmaintainsapoolofdatabaseconnectionsallloggedonasthesameOracleuser.Youmayhaveaspecific
queryyouwanttocollectwaiteventinformationfor,butyoumaynotknowwhichsessionwillbeexecutingthequery.Onewaytodealwiththissituationistotraceallof
thesessionsbeingstartedbyyourapplicationforabriefperiod.ThiscanbedoneeasilywiththefollowingSQL*Plusscript:
SPOOLtraceall.sql
exampleoftheroundofferrorthatisnotunusualwhenthousandsofveryshorttimeintervalsareaddedtogether.
Insomesituationsyoumaynotbeabletoidentifyexactlywhichsessiontotrace.Thishappensfrequentlywhenyourdatabaseissupportingawebapplicationthathas
frequentshortconnections,oranapplicationservertierthatmaintainsapoolofdatabaseconnectionsallloggedonasthesameOracleuser.Youmayhaveaspecific
queryyouwanttocollectwaiteventinformationfor,butyoumaynotknowwhichsessionwillbeexecutingthequery.Onewaytodealwiththissituationistotraceallof
thesessionsbeingstartedbyyourapplicationforabriefperiod.ThiscanbedoneeasilywiththefollowingSQL*Plusscript:
SPOOLtraceall.sql
SETHEADINGOFFFEEDBACKOFF
SELECT'EXECUTESYS.dbms_system.set_ev('||TO_CHAR(sid)||
','||TO_CHAR(serial#)||',10046,8,'''')'
FROMv$session
WHEREusername='WEB_USER'
SPOOLOFF
SETFEEDBACKON
@traceall.sql
ThisscriptwillstartwaiteventtracingonallsessionsconnectedtothedatabaseastheWEB_USERuser.Tostoptracingsimplychangethe8afterthe10046toa0and
runthescriptagain.Youshouldthinkabouthowmanysessionsthisscriptwilltraceandhowthatwillaffectserverloadbeforerunningonaproductionsystem.
Afteryou'vetracedanumberofsessions,youcanscanthetracefilesforoccurrencesofaspecificquerywithacommandlikethefollowing:
grepn1234567890*.trc
Replace1234567890withthehashvalueoraddressofthestatementyouareinterestedin.(Youcangetthesevaluesbyqueryingv$sql.)Theoutputoftheabovegrep
commandwilltellyouwhichtracefilesthequeryappearsinandonwhichlines.Youcanthengotothatpointinthetracefilesandgetdetailedwaiteventinformation.
Tracefilesareeasiesttoworkwithwhenyouuseadedicatedserverconnectiontothedatabase.IfyouconnecttothedatabaseusingOracle'ssharedserverarchitecture
thendifferentSQLstatementsmaybeexecutedbydifferentserverprocesses.Eachserverprocesswritestoitsowntracefile.Thusthetraceinformationforthesession
canbespreadoverseveraltracefiles.
WaitEventEnhancementsinOracle10g
Oracle10gbringsmoresignificantchangetothewaiteventinterfacethanwehaveseeninyears.Nowthereareover800waiteventsandnamesaremoredescriptive,
waiteventsarecategorizedintoclasses,severalv$viewshavebeenadded,helpfulcolumnshavebeenaddedtoexistingv$views,builtinstatisticscollectionbyActive
SessionHistoryandtheAutomaticWorkloadRepositoryhasbeenintroduced,anewtimemodelconceptforlookingathowsessionsspendtheirtimehasappeared,and
improvementshavebeenmadetothesessiontracingfacility.
Inthissection,wewillintroducewhatweseeasthetopdozenareasinwhichwaiteventsandthewaiteventinterfacehavebeenenhancedinOracle10g.Wevelisted
theseenhancementsinnoparticularorder.
MoreDescriptiveWaitEventNames
PriortoOracle10g,somewaiteventnameswerequitevagueandnotveryusefulwithoutlookingattheparametervaluesforaspecificoccurrenceoftheevent.For
example,anenqueuewaitcouldindicatevarioussituationsrangingfromcontentionforarowinatabletowaitingonauserdefinedlock.WaiteventnamesinOracle10g
aremoredescriptiveintheareasoflatches,enqueues,andbufferbusywaits.
ThereisstillalatchfreewaiteventinOracle10g,buttherearealso26morespecificlatchrelatedevents.Thesecoverthemostcommonlatchesthatexperience
contention.Inthepast,ifwesawasessionwaitingonthelatchfreeevent,wewouldhaveoutputlikethefollowing:
SQL>SELECTevent,state,p1,p2,p3
2FROMv$session_wait
3WHEREsid=162
EVENTSTATEP1P2P3
latchfreeWAITING15113593728975
Wewouldthenhavetoqueryv$event_nametodeterminethemeaningoftheparametersp1,p2,andp3:
SQL>SELECT*FROMv$event_nameWHEREname='latchfree'
3latchfreeaddressnumbertries
And,seeingthatp2isthelatchnumber,wewouldneedtoqueryv$latchtofindoutwhichlatchwasbeingwaitedupon:
SQL>SELECTname
2FROMv$latch
3WHERElatch#=97
NAME
cachebufferschains
InOracle10gwesimplysee:
SQL>SELECTevent,state
2FROMv$session_wait
3WHEREsid=162
EVENTSTATE
InOracle10gwesimplysee:
SQL>SELECTevent,state
2FROMv$session_wait
3WHEREsid=162
EVENTSTATE
latch:cachebufferschainsWAITING
Thedescriptiveeventnamesavesustwostepsindeterminingwhichlatchiscausingthewait.Themoredetaileddescriptionenablesonetomorequicklydrilldowntothe
rootcauseofthewait.
ThenamesofenqueuerelatedwaiteventshavealsobeenmademoredescriptiveinOracle10g.Thereisnolongeranenqueuewaiteventithasbeenreplacedby184
eventswithmoredetailednames.Inthepast,ifsessionswerewaitingonenqueues,wewouldhavetodecodethetypeoflockfromthep1parameter:
SQL>SELECTevent,state,seconds_in_waitsiw
2FROMv$session_wait
3WHEREsid=96
EVENTSTATESIW
enqueueWAITING24
SQL>SELECTsid,
2CHR(BITAND(p1,16777216)/16777215)||
3CHR(BITAND(p1,16711680)/65535)enq,
4DECODE(CHR(BITAND(p1,16777216)/16777215)||
5CHR(BITAND(p1,16711680)/65535),
6'TX','Transaction(RBS)',
7'TM','DMLTransaction',
8'TS','TablespaceandTempSeg',
9'TT','TemporaryTable',
10'ST','SpaceMgt(e.g.,uet$,fet$)',
11'UL','UserDefined',
12CHR(BITAND(p1, 16777216)/16777215)||
13CHR(BITAND(p1,16711680)/65535))enqueue_name,
14DECODE(BITAND(p1,65535),1,'Null',2,'Sub Share',
153,'SubExclusive',4,'Share',5,'Share/SubExclusive',
166,'Exclusive','Other')lock_mode
17FROMv$session_wait
18WHEREsid=96
SIDENQENQUEUE_NAMELOCK_MODE
96TXTransaction(RBS)Exclusive
InOracle10gwegetmoreinformationdirectlyfromtheenqueuename:
SQL>SELECTevent,state,seconds_in_waitsiw
2FROMv$session_wait
3WHEREsid=143
EVENTSTATESIW
enq:TXrowlockcontentionWAITING495
Additionally,moreinformationisavailableinsomecasesfromthep1,p2,andp3parameters,whosemeaningsvarywiththedifferentenqueuerelatedwaitevents.(See
theAppendixforalistingofallwaiteventsandtheirassociatedparametermeanings.)
WaitEventClasses
InOracle10gwaiteventsareclassifiedintocategorieswhichcanhelptheDBAtomoreeasilydeterminethelikelyrootcauseofthewait.Thecategoriesare:
Administrative Idle
Application
Network
Cluster
Scheduler
Commit
SystemI/O
Concurrency UserI/O
Configuration Other
Whilenearly70%ofthewaiteventsareintheOthercategory(557outof811inrelease10.1.0.3),themostfrequentlyencounteredonesareinwaitclasseswithhelpful
names.Letslookatthewaitclassdesignationsofsomeenqueueeventsasanexample:
SQL>SELECTwait_class,name
2FROMv$event_name
3WHEREnameLIKE'enq%'
4ANDwait_class<>'Other'
5ORDERBYwait_class
WAIT_CLASSNAME
Administrativeenq:TW contention
Administrativeenq:DB contention
Applicationenq:PW flushprewarmbuffers
2FROMv$event_name
3WHEREnameLIKE'enq%'
4ANDwait_class<>'Other'
5ORDERBYwait_class
WAIT_CLASSNAME
Administrativeenq:TW contention
Administrativeenq:DB contention
Applicationenq:PW flushprewarmbuffers
Applicationenq:RO contention
Applicationenq:RO fastobjectreuse
Applicationenq:TM contention
Applicationenq:TX rowlockcontention
Applicationenq:UL contention
Concurrencyenq:TX indexcontention
Configurationenq:ST contention
Configurationenq:TX allocateITLentry
Configurationenq:SQ contention
Configurationenq:HW contention
WeseethatTXenqueues(rowlocks)andTMenqueues(tablelocks)areintheApplicationclass,whichmakessensesincethesewaiteventsgenerallyoccurbecauseof
applicationbehavior.Meanwhile,ST(spacemanagement),HW(highwatermarkextension),andSQ(sequencenumber)enqueuesareintheConfigurationclass,asthese
canusuallybealleviatedbychangesinobjectanddatabasesettings.
TheUserI/Oclassincludesthedbfilescatteredread,dbfilesequentialread,directpathread,anddirectpathwriteeventsasonemightexpect,whiletheSystemI/O
classincludesmanywaitsrelatedtoreadingandwritingofredologsandarchivelogs.TheCommitclasshasonemember,logfilesync,asthatwaitiscausedbycommits.
AndtheIdleclassismadeupofvariouswaiteventswhichhavetraditionallybeenconsideredidleevents,suchasSQL*Netmessagefromclient.Itshouldbenotedthat
sometimessuchidleeventscanactuallybesymptomsoftherootcauseofpoorperformance,sotheyshouldnotbedisregardedwithoutconsideration.
Ingeneral,theadditionofwaitclasseshelpsdirecttheDBAmorequicklytowardtherootcauseofperformanceproblems.
v$ViewEnhancements
InOracle10gtherearequiteafewnewv$viewsthatpertaintowaitevents,andhelpfulnewcolumnshavebeenaddedtoexistingv$views.Wewilldiscussseveralofthe
enhancementsinthissection,althoughsomenewv$viewsarepartofmajornewOracle10gfunctionalityandwillbediscussedinseparatesectionslateron.
v$event_name
Threecolumnshavebeenaddedtothev$event_nameviewinOracle10g:wait_class_id,wait_class#,andwait_class.Thesecolumnsshowwhichwaitclassthewait
eventispartof.Wesawintheprevioussectionhowthisnewinformationmightbeused.Thecolumnsinv$event_namenoware:
SQL>DESCRIBEv$event_name
NameNull?Type
EVENT#NUMBER
EVENT_IDNUMBER
NAMEVARCHAR2(64)
WAIT_CLASS_IDNUMBER
WAIT_CLASS#NUMBER
WAIT_CLASSVARCHAR2(64)
v$sqlandv$sqlarea
Thev$sqlandv$sqlareaviewshavesixnewcolumnsinOracle10gthatrelatetowaitevents:
application_wait_time
concurrency_wait_time
cluster_wait_time
user_io_wait_time
plsql_exec_time
java_exec_time
ThesecolumnsaredesignedtoidentifytheamountoftimeaSQLstatementspendsinPL/SQLorJavacodeexecution,orwaitinginfourspecificwaitclasses.Wewilluse
anexampletodemonstratethebehaviorandvalueofthesenewcolumns.
Supposewecreateatablecalledtesttabwithaboutamillionrows.Wethenrunthefollowingstatementfromonesessionwithoutcommitting,andthenrunthesame
statementfromanothersession:
SQL>UPDATEtesttabSETnumcol=numcol+1WHEREROWNUM<1000
Obviouslythesecondsessionwillwaitonanenqueuewaiteventuntilthefirstsessioneithercommitsorrollsback.Afterawhilewerollbackthefirstsessionandthenthe
secondsession.Next,inathirdsession,werunthefollowingstatement:
SQL>UPDATEtesttabSETnumcol=numcol+1
Obviouslythesecondsessionwillwaitonanenqueuewaiteventuntilthefirstsessioneithercommitsorrollsback.Afterawhilewerollbackthefirstsessionandthenthe
secondsession.Next,inathirdsession,werunthefollowingstatement:
SQL>UPDATEtesttabSETnumcol=numcol+1
Theinstancewereusinghasasmallbuffercache,soquiteabitofphysicalI/Oiscausedbythestatement.AftertheUPDATEcompleteswelookatv$sqlarea:
SQL>SELECTsql_id,application_wait_timeappl,concurrency_wait_timeconcurr,
2user_io_wait_timeuser_io
3FROMv$sqlarea
4WHEREsql_textLIKE'UPDATEtesttabSETnumcol%'
SQL_IDAPPLCONCURRUSER_IO
038m56cp4am0c178500000020000
fd5mxhdbf09ny010000105040000
SQL>SELECTsql_id,sql_text
2FROMv$sqlarea
3WHEREsql_idIN('fd5mxhdbf09ny','038m56cp4am0c')
SQL_IDSQL_TEXT

038m56cp4am0cUPDATEtesttabSETnumcol=numcol+1WHEREROWNUM<1000
fd5mxhdbf09nyUPDATEtesttabSETnumcol=numcol+1
Soweseethatthefirststatement(lockingrows)spent178.5seconds(178,500,000microseconds)waitingoneventsintheApplicationwaitclassand0.02seconds
waitingoneventsintheUserI/Owaitclass.Ifwerecallfromthediscussionofwaitclasses,theTXenqueueforrowcontentionisintheApplicationwaitclass,anddata
filereadsareintheUserI/Owaitclass.Thesecondstatement,whichrequiredmorediskreads,shows105secondsofUserI/Owaitsandaverysmallamountof
concurrencywaits.
WhilethecurrentlyavailabledocumentationfromOracledoesnotprovidemuchinformationaboutthesenewcolumnsinv$sqlandv$sqlarea,theyappeartoholdmuch
promisefordiagnosingqueryperformanceproblems.
v$session_wait_history
UpthroughOracle9i,thev$viewsshowusonlythemostrecentwaiteventforeachsession.Eventhoughwaittimescanaccumulatetogreatlyslowdownaprocess,
manywaitsareveryshort(fromahumanperspective)individually.So,itsoftendifficulttograbinformationonawaiteventasitishappening.Thev$session_wait_history
view,newinOracle10g,helpsbyshowingthelasttenwaiteventseachsessionhasexperiencedAsanexample,thefollowingqueryshowsthetenmostrecentwaitevents
forsession154:
SQL>SELECTsid,seq#,event,wait_time,p1,p2,p3
2FROMv$session_wait_history
3WHEREsid=154
4ORDERBYseq#
SIDSEQ#EVENTWAIT_TIMEP1P2P3
1541dbfilesequentialread28435471
1542logbufferspace18000
Theseq#columnissupposedtoshowthechronologicalsequenceofthewaitevents,with1beingthemostrecentwaiteventinthesession.Notethatthisseq#value
differsfromtheseq#columninv$session,whichisincrementedforeachwaitexperiencedbythesessionthroughoutthelifeofthesession.
Intheabovequery,weseethatthesessionsmostrecentwaitsalternatedbetweensingleblockdiskreadsandlogbufferspace.Thismakessense,sincetheSQLthatthe
sessionwasperforminglookedlike:
INSERTINTOtable1(column1,column2)
SELECTcolumn1,column2
FROMtable2
WHERE...
Fromthislistofrecentwaits,wecanalsodrilldowntogetmoredetail.Thep1andp2valuesfordbfilesequentialreadindicatethefileandblocknumbersbeingread,so
wecanquicklydeterminewhatsegmentwasbeingread.
Thecolumnsinv$session_wait_historyare:
SQL>DESCRIBEv$session_wait_history
NameNull?Type
SIDNUMBER
Thecolumnsinv$session_wait_historyare:
SQL>DESCRIBEv$session_wait_history
NameNull?Type
SIDNUMBER
SEQ#NUMBER
EVENT#NUMBER
EVENTVARCHAR2(64)
P1TEXTVARCHAR2(64)
P1NUMBER
P2TEXTVARCHAR2(64)
P2NUMBER
P3TEXTVARCHAR2(64)
P3NUMBER
WAIT_TIMENUMBER
WAIT_COUNTNUMBER
v$session
Thev$sessionviewhasbeenenhancedinOracle10gwithseveralnewcolumnswhicharequiteuseful.Thewaiteventcolumnsfromv$session_waithavebeenaddedto
v$session.InpreviousreleasesofOracle,togetmoredetailedinformationaboutasessionexperiencingwaits(suchaswhatSQLthewaitingsessionisexecuting),wehad
tojoinv$session_waitwithv$session,asin:
SQL>SELECTs.sid,w.state,w.event,w.seconds_in_waitsiw,
2s.sql_address,s.sql_hash_valuehash_value,w.p1,w.p2,w.p3
3FROMv$sessions,v$session_waitw
4WHEREs.sid=w.sid
5ANDs.sid=154
InOracle10gwecangetallofthisinformationfromv$session:
SQL>SELECTsid,state,event,seconds_in_waitsiw,
2sql_address,sql_hash_valuehash_value,p1,p2,p3
3FROMv$session
4WHEREsid=154
SIDSTATEEVENTSIWSQL_ADDRESSHASH_VALUEP1P2P3
154WAITINGdbfilesequentialread1000000038551E8203625097388498131
Twoadditionalcolumnshavebeenaddedtov$sessionthatarehelpfulforwaiteventanalysis:blocking_sessionandblocking_session_status.Theblocking_sessioncolumn
containsthesessionid(SID)oftheholderoftheresourcethatthewaitingsessioniswaitingfor.Theblocking_session_statuscolumnindicatesthevalidityofthecontents
oftheblocking_sessioncolumn.Ifblocking_session_statusisVALID,avalidSIDispresentintheblocking_sessioncolumn.Inthepast,ifwesawthatasessionwas
waitingforaresource,wewouldhavetodoadditionalqueriestodeterminewhowasholdingtheresource.Iftheresourcewasanenqueuelock,wewouldhavetoquery
v$lock(sometimesaveryexpensivequery)todeterminewhowasholdingthelock.Nowwecanfindoutwhoisholdingthelockmuchmorequickly:
SQL>SELECTsid,blocking_session,username,
2blocking_session_statusstatus
3FROMv$session
4WHEREblocking_session_status='VALID'
SIDBLOCKING_SESSIONUSERNAMESTATUS
154157TSUTTONVALID
Ifwecombinethiswiththewaiteventinformationnowavailableinv$session,wesee:
SQL>SELECTsid,blocking_session,username,
2event,seconds_in_waitsiw
3FROMv$session
4WHEREblocking_session_status='VALID'
SIDBLOCKING_SESSIONUSERNAMEEVENTSIW
154157TSUTTONenq:TX rowlockcontention318
v$event_histogram
Thev$system_eventviewshowsthenumberofwaits,totaltimewaited,andaveragewaittimeforagivenwaiteventname(systemwidesinceinstancestartup).However,
thisaggregationcancloudthepicture,becauseasmallnumberoflongwaitscanskewthedata.Forexample,considerthefollowingqueryfromv$system_event:
SQL>SELECTevent,total_waits,time_waited,average_wait
2FROMv$system_event
3WHEREevent='enq:TXrowlockcontention'
EVENTTOTAL_WAITSTIME_WAITEDAVERAGE_WAIT
enq:TXrowlockcontention172182101966122
SQL>SELECTevent,total_waits,time_waited,average_wait
2FROMv$system_event
EVENTTOTAL_WAITSTIME_WAITEDAVERAGE_WAIT
Weseethattherehavebeen17,218waitsandthattheaveragewaittimewas1.22seconds,butwehavenoideahowthewaittimesaredistributed.Wereallofthese
waitsroughlythesamelength?Weremostofthemunderonesecondlongandafewreallylongwaitsthrewofftheaverage?Wecanttellfromv$system_event.
However,inOracle10gwecanlookatv$event_histogramforamorecompletepicture:
SQL>SELECTevent,wait_time_milli,wait_count
2FROMv$event_histogram
EVENTWAIT_TIME_MILLIWAIT_COUNT
Weseethatveryfewofthewaitswereanywherenear1.22seconds.Nearly60%ofthewaitswerelessthan0.128seconds(withmostofthosebetween16milliseconds
and64milliseconds),andmostoftheremainingwaitswerebetween2.048secondsand4.096seconds(atwhichpointsometimedoutandstartednewwaits).Inthis
way,thev$event_histogramviewgivesusamoreaccuratepictureofwaittimessummarizedbyeventname.
Thecolumnsofv$event_histogramare:
SQL>DESCRIBEv$event_histogram
NameNull?Type
EVENT#NUMBER
EVENTVARCHAR2(64)
WAIT_TIME_MILLINUMBER
WAIT_COUNTNUMBER
v$system_wait_classandv$session_wait_class
Thetwonewviewsv$system_wait_classandv$session_wait_classenableustogetsystemandsessionwaitinformationsummarizedbywaitclasses.Thisgivesusa
higherlevelviewofwhatishappeninginthesystemorsession,ratherthanfocusingonindividualevents.Theviewsareroughlyequivalenttotheviewsv$system_eventand
v$session_event,exceptthattheyrolluptheeventsbywaitclass.Thewaittimesareexpressedincentisecondssinceinstancestartupforv$system_wait_classand
centisecondssincesessionconnectionforv$session_wait_class.
Thefollowingqueriesshowhowmuchtime(incentiseconds)hasbeenspentonwaitsineachclassacrossthesystemsinceinstancestartandforonespecificsessionsince
thatsessionbegan:
SQL>SELECTwait_class,time_waited
2FROMv$system_wait_class
3ORDERBYtime_waitedDESC
WAIT_CLASSTIME_WAITED
Idle777450022
SystemI/O1261584
UserI/O116667
Configuration116481
Application72301
Other12432
Commit3496
Concurrency319
Network1
SQL>SELECTwait_class,time_waited
2FROMv$session_wait_class
3WHEREsid=154
4ORDERBYtime_waitedDESC
Idle612453
UserI/O1500
Configuration28
Commit11
Other0
Application0
Network0
Idle612453
UserI/O1500
Configuration28
Commit11
Other0
Application0
Network0
Sincethewaittimesshownintheseviewsareaggregationssincesystemorsessionstartup,theseviewsarebestusedbytakingsamplesandcomparingtheresultsto
determinewaitsoveraperiodoftime.Forinstance,youcouldgetdatafortheentireinstanceattimeT1:
DROPTABLEswc_snap
CREATETABLEswc_snap
AS
SELECTwait_class,total_waits,time_waited
FROMv$system_wait_class
Andthen,attimeT2awhilelater,getasummaryofwaitsbetweenT1andT2:
SELECTa.wait_class,(a.time_waited b.time_waited)tm_waited
FROMv$system_wait_classa,swc_snapb
WHEREa.wait_class=b.wait_class
ANDa.total_waits>NVL(b.total_waits,0)
ORDERBYtm_waitedDESC
WAIT_CLASSTM_WAITED
Idle255767
Application171
SystemI/O156
UserI/O44
Other21
Commit13
Network1
ActiveSessionHistory
InpreviousreleasesofOracle,thedetailedinformationdisplayedinv$session_waitcouldproveextremelyhelpfulindiagnosingperformanceproblemsifyouqueriedthe
viewattherighttime.Thev$session_wait_historyviewinOracle10gmakesitalittleeasiertocatchdetailedinformationbypreservingthelasttenwaitsforeachsession.
Butwhatifyouwantdetailedinformationaboutasessionswaitsforaperiodfurtherbackintime?ThisiswheretheActiveSessionHistoryfeatureofOracle10gASH
forshortcomesinhandy.ASHmakesdetailedinformationaboutasamplingofwaitsencounteredbyallsessionsavailabletousforaverylongtime.
InOracle10g,anewbackgrounddaemonprocesscalledMMNLqueriesv$sessiononceeachsecondandstoresinformationaboutallactivesessionsinacircularbuffer
inmemoryaccessiblebyanewviewcalledv$active_session_history.Howfarbackyoucanlookatsessionsinthisviewdependsonsessionactivityandhowmuch
memoryOracleallocatedforASH.Oraclesgoalistokeepatleastafewhoursofsessiondataavailableinthisview.Thev$active_session_historyviewincludesmuchof
thedetailedwaiteventinformationshowninv$session:
SQL>DESCRIBEv$active_session_history
NameNull?Type
SAMPLE_IDNUMBER
SAMPLE_TIMETIMESTAMP(3)
SESSION_IDNUMBER
SESSION_SERIAL#NUMBER
USER_IDNUMBER
SQL_IDVARCHAR2(13)
SQL_CHILD_NUMBERNUMBER
SQL_PLAN_HASH_VALUENUMBER
SQL_OPCODENUMBER
SERVICE_HASHNUMBER
SESSION_TYPEVARCHAR2(10)
SESSION_STATEVARCHAR2(7)
QC_SESSION_IDNUMBER
QC_INSTANCE_IDNUMBER
EVENTVARCHAR2(64)
EVENT_IDNUMBER
EVENT#NUMBER
SEQ#NUMBER
P1NUMBER
P2NUMBER
P3NUMBER
WAIT_TIMENUMBER
TIME_WAITEDNUMBER
CURRENT_OBJ#NUMBER
CURRENT_FILE#NUMBER
CURRENT_BLOCK#NUMBER
PROGRAMVARCHAR2(48)
MODULEVARCHAR2(48)
ACTIONVARCHAR2(32)
CLIENT_IDVARCHAR2(64)
CURRENT_FILE#NUMBER
CURRENT_BLOCK#NUMBER
PROGRAMVARCHAR2(48)
MODULEVARCHAR2(48)
ACTIONVARCHAR2(32)
CLIENT_IDVARCHAR2(64)
Asyoucansee,v$active_session_historycapturestheessentialwaitrelateddatafromv$session.ItalsocapturesusefulinformationabouttheSQLstatementcurrently
beingexecuted,aswellascurrentobjectnumber,file,andblockbeingaccessed.WhenawaitthatwassampledbyASHcompletes,Oraclefillsinthetime_waited
columnfortherowinv$active_session_historywiththeactualdurationofthewait.
TheAutomaticWorkloadRepository,whichwewilldiscussinthenextsection,writesdatafromv$active_session_historytodiskatregularintervals,preservingone
sampleeverytensecondsfromeachactivesession.So,activesessioninformationremainsaccessiblealthoughwithlessdetailevenafterthedatahasagedoutof
v$active_session_history.
BecauseASHisalwayson,youalwayshaveaccesstodetailedinformationaboutwaitsencounteredinsessionswithinthelastfewhours.Thismeansthatifauser
complainsaboutaperformanceproblem,youmaybeabletoqueryv$active_session_historyandgaininsightintotheproblemwithouthavingtoinitiateanextendedSQL
traceorstartaclosewatchofv$sessionwhiletheyreproducetheproblem.
Animportantthingtokeepinmindaboutv$active_session_history,however,isthatitispopulatedbysamplingv$sessiononceeachsecond.Asessionmayencounter
manydifferentwaitsduringaonesecondperiod,butonlytheonewaitthatwasinprogresswhenASHcollecteditssamplewillberecordedinv$active_session_history.
Forthisreason,ASHisvaluableforgeneralaggregatequeriesbutnotforprecisecountingofindividualeventsordeterminingminimumormaximumwaittimes.Statistically
speaking,thedatacollectedbyASHisprobablymoreaccurateoveralargertimeintervaland/ornumberofsessions.
Forexample,youmightqueryv$active_session_historytoseewhatpercentageoftimeoverthelasttwohoursaparticulargroupofsessionsspentwaitingondiskreads.
However,usingthisviewtodeterminehowmanydiskreadwaitsasessionencounteredinthelastminuteprobablywillnotyieldveryaccurateresults.
EventhoughASHdataisonlyasamplingofactivesessions,theinformationcanprovetobequiteuseful.Forexample,thefollowingqueryshowsthatsessionsrunningthe
ARXENVapplicationoverthelasttwohoursencounteredagreatdealofrowlevellockcontention:
SQL>SELECTDECODE(session_state,'WAITING',event,NULL)event,
2session_state,COUNT(*),SUM(time_waited)time_waited
3FROMv$active_session_history
4WHEREmodule='ARXENV'
5ANDsample_time>SYSDATE(2/24)
6GROUPBYDECODE(session_state,'WAITING',event,NULL),
7session_state
EVENTSESSION_STATECOUNT(*)TIME_WAITED
ONCPU1240
logfilesyncWAITING252686
dbfilescatteredreadWAITING228254
dbfilesequentialreadWAITING16059
controlfilesequentialreadWAITING19206
SQL*Netbreak/resettoclientWAITING19140
enq:TXrowlockcontentionWAITING922930864016
Inadditiontorunningqueriesagainstthev$active_session_historyview,youcanuseEnterpriseManagertorunreportsthatwilldisplayASHdata.
AlthoughASHrunsonallOracle10gdatabasesbydefault,youarenotallowedtoquerythev$active_session_historyview(orrunthecorrespondingreportsin
EnterpriseManager)unlessyouhavepurchasedtheDiagnosticPack.
AutomaticWorkloadRepository
Oracle10gincludesanothersignificantfacilityrelevanttothewaiteventinterface.ItsknownastheAutomaticWorkloadRepositoryorAWRforshort.AWRis
basicallyanextgenerationStatspack.Bydefault,AWRcollectsanhourlysnapshotofdatabaseperformanceinformation,storingthedataintablesintheSYSschema.
AWRisconfiguredautomaticallywhenyoucreateanOracle10gdatabase.Youcancallthedbms_workload_repositorypackagetocollectasnapshotondemand,purge
asnapshotorrangeofsnapshots,orchangethesnapshotintervalorretentionperiod.(Bydefaultsnapshotsarecollectedatthetopofeachhourandarepurgedafter
sevendays.)
AWRcollectsthesametypeofdatathatStatspackdoesincludingsystemlevelstatistics,resourceintensiveSQL,andofcourseinstancewidewaiteventinformation.
AWRalsocollectsdatathatisnewforOracle10g,suchastimemodelstatistics(whichwewilldiscussinthenextsection).Asanaside,theOracle10gversionof
Statspackalsocollectsalotofthisnewinformation,includingtimemodelstatistics.
YoucangenerateanAWRreportofdatabaseactivitybetweentwosnapshotsbyrunningtheawrrpt.sqlscriptintherdbms/admindirectoryunder$ORACLE_HOME.
ThisscriptoffersreportsformattedasplaintextorHTML.ThereportswilllookfamiliarifyouhaveusedStatspackbefore.YoucanuseEnterpriseManagertogenerate
AWRreportsaswell.
AWRoffersmanybenefitsoverStatspack.Forone,itismoretightlyintegratedintotheOraclekernel,reducingresourcerequirementsandoverheadwhencollecting
snapshots.AWRsnapshotsalsoincludeASHdatafromv$active_session_history,providingsessionlevelinformationtocomplementthesystemleveldatacollection
familiartoStatspackusers.
DatacollectedbyAWRismadeeasilyaccessibleviaviewswithnamesthatstartDBA_HIST.Thisenablesyoutowriteyourownreportsthatextractjustthedatayou
needtoaddressaspecificsituation,ifforsomereasonyoudontfindwhatyouneedinthestandardAWRreport.Forexample,thefollowingquerydisplaysthetwomost
recentsnapshotIDs:
SQL>SELECTsnap_id,begin_interval_time,end_interval_time
2FROM(
3SELECTsnap_id,begin_interval_time,end_interval_time
4FROMdba_hist_snapshot
needtoaddressaspecificsituation,ifforsomereasonyoudontfindwhatyouneedinthestandardAWRreport.Forexample,thefollowingquerydisplaysthetwomost
recentsnapshotIDs:
SQL>SELECTsnap_id,begin_interval_time,end_interval_time
2FROM(
3SELECTsnap_id,begin_interval_time,end_interval_time
4FROMdba_hist_snapshot
5ORDERBYend_interval_timeDESC
6)
7WHEREROWNUM<=2
SNAP_IDBEGIN_INTERVAL_TIMEEND_INTERVAL_TIME
36210MAR0504.00.02.018PM10MAR0505.00.36.581PM
36110MAR0503.00.25.885PM10MAR0504.00.02.018PM
JustlikeASH,AWRrunsonallOracle10gdatabasesbydefault.AlsolikeASH,youarenotallowedtoquerytheAWRviews(orrunAWRreports)unlessyouhave
licensedtheDiagnosticPack.BecauseAWRconsumessystemresourceswhencollectingsnapshotsandusesupstorageintheSYSAUXtablespace,youmaywantto
disablethecollectionofAWRsnapshotsifyouarenotlicensedtouseAWR.Thismaybedonebyusingthedbms_workload_repositorypackage.IfAWRisnot
availabletoyou,StatspackisstillagoodwaytogoinOracle10g.
TimeModelStatistics
Oracle10gintroducesanewconceptcalledTimeModelStatistics.Thisinformationprovidesyetanotherwaytoseehowtimeisspent,andwithgreaterdetailthanwas
availablepreviously.Thev$sys_time_modelviewshowstimemodelstatisticsfortheentiresystemsinceinstancestartup,whilethev$sess_time_modelviewshowstime
modelstatisticsforeachsessionsincesessionstart.Thecolumnsinthesetwoviewsareasfollows:
SQL>DESCRIBEv$sys_time_model
NameNull?Type
STAT_IDNUMBER
STAT_NAMEVARCHAR2(64)
VALUENUMBER
SQL>DESCRIBEv$sess_time_model
NameNull?Type
SIDNUMBER
STAT_IDNUMBER
STAT_NAMEVARCHAR2(64)
VALUENUMBER
Asamplequeryfromv$sys_time_modelshowsthefollowing:
SQL>SELECTstat_name,value/1000000seconds
2FROMv$sys_time_model
3ORDERBYsecondsDESC
STAT_NAMESECONDS
DBtime80970.190
sqlexecuteelapsedtime75057.271
DBCPU44448.628
backgroundelapsedtime29333.160
PL/SQLexecutionelapsedtime8824.538
backgroundcputime5170.311
parsetimeelapsed1270.147
hardparseelapsedtime838.068
PL/SQLcompilationelapsedtime176.731
sequenceloadelapsedtime112.334
connectionmanagementcallelapsedtime44.644
failedparseelapsedtime11.946
hardparse(sharingcriteria)elapsedtime5.579
hardparse(bindmismatch)elapsedtime4.610
failedparse(outofsharedmemory)elapsedtime0.000
Javaexecutionelapsedtime0.000
inboundPL/SQLrpcelapsedtime0.000
ThisqueryshowsusalotmoreinformationabouthowOraclesessionshavespenttheirtime(categorically)thanv$sysstatandv$sesstatdo.Ofcourse,wehavetoknow
howtointerpretthisinformationbeforewecanputittoworkforus.Valuesintheseviewsareshowninmicroseconds,andtheydonotincludebackgroundprocesses
unlessbackgroundappearsinthestatisticname.TheDBtimestatisticshowselapsedtimespentondatabasecalls(userprocessesonly).Thisamountstotimespent
ontheCPUorwaitingonnonidlewaitevents.Foradescriptionoftheothertimemodelstatistics,seethev$sess_time_modelviewlistingintheOracle10gDatabase
Referencemanual.
Fromthisquery,amongmanyotherusefulfacts,wecanseethatnotimehasbeenspentexecutingJava,verylittletimehasbeenspenthardparsingorcompilingPL/SQL,
backgroundprocesseshaveusedabout10%oftheCPUtime,andabout11%oftheelapsedtimeforusersessionswasspentonPL/SQLexecution.
TracingFacilityImprovements
TheextendedSQLtracefacility,alsoknownasevent10046,allowsustocaptureinatracefiledetailedinformationabouteverywaiteventencounteredbyadatabase
session.ThisfeaturehasbeenavailableinOracleforalongtime.However,Oracle10gofferssomehelpfulimprovementsinthisarea.
EnablingextendedSQLtracehasalwaysbeenabitofanuisance.InearlierreleasesofOracle,youhadtouseaclumsyALTERSESSIONSETEVENTSstatement
orworseyetcalltheundocumenteddbms_system.set_evproceduretosetthe10046eventinanotheruserssession.InOracle8ithedbms_supportpackagewas
TracingFacilityImprovements
TheextendedSQLtracefacility,alsoknownasevent10046,allowsustocaptureinatracefiledetailedinformationabouteverywaiteventencounteredbyadatabase
session.ThisfeaturehasbeenavailableinOracleforalongtime.However,Oracle10gofferssomehelpfulimprovementsinthisarea.
EnablingextendedSQLtracehasalwaysbeenabitofanuisance.InearlierreleasesofOracle,youhadtouseaclumsyALTERSESSIONSETEVENTSstatement
orworseyetcalltheundocumenteddbms_system.set_evproceduretosetthe10046eventinanotheruserssession.InOracle8ithedbms_supportpackagewas
introducedtomakethisstepeasier,butthepackagewasmissingfrommanyreleasesofOracleandusuallywasnotinstalledbydefault.
Oracle10gintroducesthenewdbms_monitorpackage.Thispackage,amongmanyotherthings,makesitveryeasytoturnextendedSQLtraceonandoffinanyOracle
session.Withoneeasytoremembercall,youcanturnextendedSQLtraceonoroff,withwaiteventsand/orbindvariablescapturedinthetracefile:
SQL>DESCRIBEdbms_monitor
...
PROCEDURESESSION_TRACE_DISABLE
ArgumentNameTypeIn/OutDefault?
SESSION_IDBINARY_INTEGERINDEFAULT
SERIAL_NUMBINARY_INTEGERINDEFAULT
PROCEDURESESSION_TRACE_ENABLE
ArgumentNameTypeIn/OutDefault?
SESSION_IDBINARY_INTEGERINDEFAULT
SERIAL_NUMBINARY_INTEGERINDEFAULT
WAITSBOOLEANINDEFAULT
BINDSBOOLEANINDEFAULT
Ifthesession_idparameterisnotspecifiedorsettoNULL,yourownsessionwillbetraced.Thus,thefollowingtwostatementsshouldbeequivalent:
l
EXECUTEdbms_monitor.session_trace_enable(waits=>TRUE,binds=>TRUE)
InOracle9iandearlier,extendedSQLtracewaseasytouseifyourapplicationconnectedtotheOracledatabaseviaadedicatedserverconnection.Ifthesharedserver
architecturewasused,eachsharedserverprocessthatservicedarequestfromthesessionbeingtracedwouldwriteitsdatatoaseparatetracefile.Furthermore,tracing
sessionsinaconnectionpoolenvironmentbecamedifficultbecauseoneOraclesessioncouldactuallyprocessrequestsformanydifferentendusersessions.
Thedbms_monitorpackageinOracle10gaddressesthisproblem.InsteadofenablingextendedSQLtraceforaspecificOraclesession,youcanenableitforaspecific
clientidentifierorcombinationofservice,module,andaction.AnytimeanyOraclesessionhasthespecifiedclientidentifierorcombinationofservice,module,andaction,
thesessionwillbetraced.Sessionscansetorcleartheirclientidentifieratwillbycallingthedbms_sessionpackage,andtheycansettheirmoduleandactionbycallingthe
dbms_application_infopackage.
Supposeawebbasedapplicationusesapoolof30databaseconnectionstoserveuserrequestsandmaintainsacurrent_sessionstabletokeeptrackofthestateofeach
endusersession.Whenauserclicksabuttonintheirbrowserwindow,theapplicationserverreceivestheHTTPrequestandhandsitofftoanapplicationserverprocess.
Theapplicationserverprocessgrabsafreedatabaseconnectionfromthepoolandaccessesthedatabaseasnecessarytoservicetherequest.Itislikelythatsubsequent
requestsfromthesameuserwillbeprocessedusingdifferentdatabaseconnections.
InOracle9iandearlieritwouldhavebeenverydifficulttotracetheoneusersdatabaseactivityinthisenvironment.Theusersdatabaseaccessesarespreadamong
multipleOracleserverprocessesintheconnectionpool.Moreover,eachofthoseOracleprocessesishandlingrequestsfrommanydifferentendusers.
Withdbms_monitorinOracle10g,theapplicationcouldbemodifiedinawaytomakeextendedSQLtraceawholeloteasier.Wementionedthattheapplicationusesthe
current_sessionstabletomaintainstateforeachendusersession.Eachtimetheapplicationservergrabsadatabaseconnectionfromthepool,itcouldsettheclient
identifierfortheOraclesessiontothesession_idfromthecurrent_sessionstablebeforedoinganydatabaseaccessforthatendusersession.Thentheapplicationcould
cleartheclientidentifierbeforereturningthedatabaseconnectiontothepool.TheOraclecallscouldlooklikethis:
EXECUTEdbms_session.set_identifier('session_id174837492748')
...dotheworkforthisendusersession...
EXECUTEdbms_session.clear_identifier
Totracethisendusersession,wecouldnowcallthedbms_monitorpackagelikethis:
SQL>EXECUTEdbms_monitor.client_id_trace_enable
>('session_id174837492748',waits=>TRUE,binds=>TRUE)
Thiscalltodbms_monitorwillcauseeachOracleprocesstowriteextendedSQLtracedatatoatracefileforallcallsthatoccurwhiletheclientidentifierforthesessionis
settothespecifiedvalue.However,eachOracleprocesswillwritetoitsowntracefile.Thiswillcausethetracedatatobesplitovermultiplefiles.Toaddressthis
problem,Oracle10gprovidesanewcommandlineutilitycalledtrcsess.Thetrcsessutilityreadsmultipletracefilesandconsolidatesentriesfromthevariousfilesthatmeet
thespecifiedcriteriaintoonetracefilethatcanbeprocessedbyTKPROF.Toconsolidatethetracedataforourcurrentexample,wecouldusethefollowingcommands:
$cd$ORACLE_BASE/admin/$ORACLE_SID/udump
$trcsessoutput=/home/rschrag/case1403/case1403trial1.trc\
clientid=session_id174837492748
Inthiswaythedbms_monitorpackageandtrcsessutilityinOracle10gmakeitaloteasiertocollectextendedSQLtracedatafromanenduserssessionwhen
connectionpoolingorothersessionaggregationtechniquesareusedbytheapplicationservertier.
$cd$ORACLE_BASE/admin/$ORACLE_SID/udump
$trcsessoutput=/home/rschrag/case1403/case1403trial1.trc\
clientid=session_id174837492748
Inthiswaythedbms_monitorpackageandtrcsessutilityinOracle10gmakeitaloteasiertocollectextendedSQLtracedatafromanenduserssessionwhen
connectionpoolingorothersessionaggregationtechniquesareusedbytheapplicationservertier.
Conclusion
Oracle10gincludesmanyenhancementstothewaiteventinterfacethatshouldmakeperformancemanagementusingwaiteventmethodologieseasierthanever.Some
enhancements,suchasdescriptivewaiteventnames,waitclasses,andthesession_trace_enableprocedureinthedbms_monitorpackage,aresimpleconveniencesthat
makeourjobseasier.Theseenhancementsdontgiveusanyinformationorpowerthatwedidnthavebefore.Otherenhancements,however,suchastimemodel
statistics,v$event_histogram,andnewcolumnsinv$sqlandv$sqlarea,provideuswithhelpfultimingandwaitinformationthatwaspreviouslyunavailable.
AlthoughasofthiswritingdocumentationonthesenewfeaturesisincompleteanddiscussiononMetalinkissurprisinglysparse,manyoftheseenhancementswilllikely
provequitevaluabletotheOracleDBAresponsibleforperformancemanagementofacomplexOraclesystem.
EventWaitAnalysiswithSTATSPACK
InSTATSPACK,youcangetasnapshotofOraclewaiteventseveryhourandplotchangesinwaitbehaviorovertime.Youcanalsosetthresholds,and
reportonlyonwaiteventsthatexceedyourpredefinedthreshold.Hereisthescriptthatiscommonlyusedforexceptionreportingofwaitevents.
prompt
prompt
prompt***********************************************************
promptExcessivewaitsonbackgroundevents
prompt***********************************************************
prompt
ttitle'Highwaitsonbackgroundevents|Rollupbyhour'
columnmydateheading'Yr.MoDyHr'formata13
columneventformata30
columntotal_waitsheading'totwaits'format999,999
columntime_waitedheading'timewait'format999,999
columntotal_timeoutsheading'timeouts'format9,999
breakonto_char(snap_time,'yyyymmdd')skip1
select
to_char(snap_time,'yyyymmddHH24')mydate,
e.event,
e.total_waitsnvl(b.total_waits,0)total_waits,
e.time_waitednvl(b.time_waited,0)time_waited,
e.total_timeoutsnvl(b.total_timeouts,0)total_timeouts
from
stats$bg_event_summaryb,
stats$bg_event_summarye,
stats$snapshotsn
wheresnap_time>sysdate&1
ande.eventnotlike'%timer'
ande.eventnotlike'%message%'
ande.eventnotlike'%slavewait%'
ande.snap_id=sn.snap_id
andb.snap_id=e.snap_id1
andb.event=e.event
ande.total_timeouts>100
and(e.total_waitsb.total_waits>100
or
e.time_waitedb.time_waited>100
)
Hereistheoutputfromthisscript.Aswecansee,wegetatime seriesresult,showingusthosedaysandhourswhenourthresholdsareexceeded.If
welookatthislisting,weseethateveryeveningbetween10:00p.m.and11:00p.m.,weexperiencehighwaitsontheredologs.
WedAug21page1
Highwaitsonbackgroundevents
Rollupbyhour
Yr.MoDyHrEVENTtotwaitstimewaittimeouts
2002081822LGWRwaitforredocopy9,3261,109286
2002081823LGWRwaitforredocopy8,5063168
2002081823bufferbusywaits21421,388206
2002081900LGWRwaitforredocopy49850
2002082100rdbmsipcreply16030,986149
2002082100rdbmsipcreply16030,986149
UsingQueriesAgainstv$event_wait
Hereisthemostcommonlyusedscriptfordisplayingsystemwideevents,calledcurrent_waits.sql.
setpages999
setlines90
columnc1heading'Event|Name'formata30
columnc2heading'Total|Waits'format999,999,999
columnc3heading'Seconds|Waiting'format999,999
columnc4heading'Total|Timeouts'format999,999,999
columnc5heading'Average|Wait|(insecs)'format99.999
ttitle'SystemwideWaitAnalysis|forcurrentwaitevents'
select
eventc1,
total_waitsc2,
time_waited/100c3,
total_timeoutsc4,
average_wait/100c5
from
sys.v_$system_event
where
eventnotin(
'dispatchertimer',
'lockelementcleanup',
'Nullevent',
'parallelquerydequeuewait',
'parallelqueryidlewaitSlaves',
'pipeget',
'PL/SQLlocktimer',
'pmontimer',
'rdbmsipcmessage',
'slavewait',
'smontimer',
'SQL*Netbreak/resettoclient',
'SQL*Netmessagefromclient',
'SQL*Netmessagetoclient',
'SQL*Netmoredatatoclient',
'virtualcircuitstatus',
'WMONgoestosleep'
)
ANDeventnotlike'DFS%'
andeventnotlike'%done%'
andeventnotlike'%Idle%'
ANDeventnotlike'KXFX%'
orderbyc2desc
Belowistheoutputfromthisscript.Aswesee,thereportshowsallcumulativewaitssinceinstancestartuptime.
TueAug20page1
System wideWaitAnalysis
forcurrentwaitevents
Average
EventTotalSecondsTotalWait
NameWaitsWaitingTimeouts(insecs)
dbfilesequentialread2,902,8503,8290.001
latchfree2,248,8644961,524,235.000
PXDeq:TableQNormal2,080,4634,469248.002
PXDeqCredit:sendblkd1,321,01952,25123,032.040
directpathread986,9516,9310.007
PXDeqCredit:needbuffer800,9701,09184.001
logfileparallelwrite415,1755,0782.012
directpathwrite321,0969,3420.029
PXDeq:ExecutionMsg198,76856,38426,020.284
logfilesequentialread118,4801640.001
PXDeq:ExecuteReply92,4875,6282,407.061
logfilesync87,2951,3277.015
dbfilescatteredread70,1121620.002
enqueue44,3351,468354.033
PXDeq:JoinACK42,503770.002
fileopen28,54580.000
PXDeq:SignalACK26,7532536,705.009
logfileswitchcompletion54185.341
controlfileparallelwrite23,7853400.014
SQL*Netmoredatafromclient19,8478780.044
PXDeq:ParseReply17,885260.001
dbfileparallelwrite17,7451,2934.073
PXDeq:MsgFragment15,612188.001
rdbmsipcreply11,01316766.015
PXDeq:TableQqref7,77820.000
LGWRwaitforredocopy7,608243.000
controlfilesequentialread4,09810.000
controlfileparallelwrite23,7853400.014
SQL*Netmoredatafromclient19,8478780.044
PXDeq:ParseReply17,885260.001
dbfileparallelwrite17,7451,2934.073
PXDeq:MsgFragment15,612188.001
rdbmsipcreply11,01316766.015
PXDeq:TableQqref7,77820.000
LGWRwaitforredocopy7,608243.000
controlfilesequentialread4,09810.000
bufferbusywaits2,97090.003
PXDeq:TableQSample2,0532527.012
librarycachepin1,35360.004
PXDeq:TableQGetKeys53010.001
localwritewait317177163.560
fileidentify31500.000
refreshcontrolfilecommand21400.002
PXqreflatch135105102.780
logfilesinglewrite8020.021
immop6700.000
processstartup6120.036
writecompletewaits3334331.032
librarycacheloadlock2562.256
logbufferspace2060.286
dbfilesinglewrite600.007
rowcachelock500.006
dbfileparallelread200.010
instancestatechange200.000
reliablemessage100.010
librarycachelock100.020
Thisshouldgivealistof"current"waitevents,butsomeeventsaremoreimportantthanothers.Thesearetheone'stowatch outfor:
l
l
l
l
l
l
l
l
l
l
asyncdiskIO
controlfileparallelwrite
controlfilesequentialread
dbfileparallelwrite
dbfilescatteredread
dbfilesequentialread
directpathread
directpathwrite
logfileparallelwrite
logfilesync
Oncewehaveasystemwidepictureofwaitevents,wecandrill downintowaiteventsthataredirectlyassociatedwithbackgroundprocesses.
setpages999
setlines90
columnc1heading'System|ID'format999
columnc2heading'Prcs'formata10
columnc3heading'Event|Name'formata30
columnc4heading'Total|Waits'format999,999
columnc5heading'Time|Waited|(insecs)'format999,999
columnc6heading'Avg|Wait|secs'format999
columnc7heading'Avg|Wait|secs'format999
columnc8heading'Max|Wait|(insecs)'format999
ttitle'SystemwideWaitAnalysis|DetailAnalysis|forcurrentwaitevents'
select
b.sidc1,
decode(b.username,NULL,c.name,b.username)c2,
eventc3,
a.total_waitsc4,
round((a.time_waited/100),2)c5,
a.total_timeoutsc6,
round((average_wait/100),2)c7,
round((a.max_wait/100),2)c8
from
sys.v_$session_eventa,
sys.v_$sessionb,
sys.v_$bgprocessc
whereeventNOTLIKE'DFS%'
andeventNOTLIKE'KXFX%'
anda.sid=b.sid
andb.paddr=c.paddr(+)
andeventNOTIN
('lockelementcleanup',
'pmontimer',
'rdbmsipcmessage',
'smontimer',
'SQL*Netmessagefromclient',
'SQL*Netbreak/resettoclient',
'SQL*Netmessagetoclient',
'SQL*Netmoredatatoclient',
'dispatchertimer',
'Nullevent',
'iodone',
'pipeget',
'PL/SQLlocktimer',
'slavewait',
'WMONgoestosleep')
'Nullevent',
'iodone',
'pipeget',
'PL/SQLlocktimer',
'slavewait',
'WMONgoestosleep')
orderby4desc
Hereistheoutputfromthisscript.HereweseeeachbackgroundprocessandthetimeeachhasspentwaitingwithinOracle.
TueAug20page1
DetailAnalysis
TimeAvgAvgMax
SystemEventTotalWaitedWaitWaitWait
IDPrcsNameWaits(insecs)secssecs(insecs)
3LGWRlogfileparallelwrite415,0825,076201
11ARC0logfilesequentialread108,834139000
4CKPTcontrolfileparallelwrite23,035327001
4CKPTfileopen20,8424000
2DBW0dbfileparallelwrite17,6841,292401
4CKPTdirectpathread9,745106006
3LGWRLGWRwaitforredocopy7,60824300
4CKPTdirectpathwrite3,7432000
5SMONdbfilescatteredread1,8247000
4CKPTcontrolfilesequentialread1,3550000
5SMONdbfilesequentialread7942000
156CPA1_OWNERdbfilesequentialread6773000
3LGWRcontrolfilesequentialread4680000
11ARC0controlfilesequentialread4330000
3LGWRcontrolfileparallelwrite2716000
2DBW0controlfilesequentialread2530000
3LGWRfileopen1900000
160CPA1_USERlogfilesync1522000
2DBW0fileopen1120000
5SMONfileopen1110000
2DBW0directpathread1080000
3LGWRdirectpathread1082000
2DBW0fileidentify1080000
3LGWRdirectpathwrite1070000
11ARC0controlfileparallelwrite1032000
11ARC0fileidentify1020000
12fileopen910000
16fileopen901000
20fileopen891000
3LGWRlogfilesinglewrite802000
3LGWRfileidentify800000
11ARC0fileopen700000
3LGWRimmop670000
3LGWRlogfilesequentialread430000
15fileopen380000
6RECOdbfilesequentialread360000
156CPA1_OWNERlatchfree3402100
160CPA1_USERlatchfree3002000
2DBW0latchfree2912900
5SMONlatchfree1701100
125SYSlatchfree140700
3LGWRlatchfree1101100
156CPA1_OWNERfileopen110000
183CPA1_USERdbfilesequentialread90000
25ARC1fileopen80000
4CKPTfileidentify40000
15latchfree30300
2DBW0processstartup30000
11ARC0enqueue24023
125SYSdbfilesequentialread20000
4CKPTlatchfree10100
12latchfree10100
11ARC0processstartup10000
3LGWRenqueue13033
6RECOfileopen10000
183CPA1_USERfileopen10000
125SYSfileopen10000
166ANANDfileopen10000
166ANANDdbfilesequentialread10000
125SYSlogfilesync10000
5SMONbufferbusywaits10000
3LGWRenqueue13033
6RECOfileopen10000
125SYSfileopen10000
25ARC1enqueue10000
3LGWRprocessstartup10000
16latchfree10100
6RECOlatchfree10100
Fromalloftheabovereports,weseethefollowingareasforimprovement:
1Eventwaitsforparallelquerydequeues.Weneedtocheckthedefaultdegreeofparallelismfordatabaseobjectsandturn offparallelismatthe
systemlevel.
2Eventwaitsfor"dbfilesequentialreads."Thisismostlikelyduetosegmentheadercontentionontheindexes,butitcouldalsobeduetodisk
contention.Wewillstartbyincreasingthenumberoffreelistsontheindexes.Ifthewaitspersist,theoffendingindexshouldbestripedacrossmultiple
diskspindles.
Theseareverygeneralwaitconditions,buttheycansometimesbefixedbychangingparametersorobjectcharacteristics.Somepossiblesolutions
include:
l
Improvetheoptimizers'propensitytouseindexesbyadjustingoptimizer_index_cost_adjtoavaluelessthan10,therebyreducingunnecessary
fulltablescans
Checkandadjustthevalueofparallel_threads_per_cputoreduceparallelquery
Checkforsegmentheadercontention/waitsonindexheaders
Createmultiplesegmentheaderblocksforstressedindexes(usingalter
indexxxxstorage(freelists1))
Distributeheavyimpacttablesandindexesontoafasterdisk,orstripetheobjectacrossmultipledisks.
SessionDetailEventWaits
Oncewehavecoveredthesystemandbackgroundwaits,wecanfindtasksthatarecurrentlywaiting.Thisscriptistrickybecauseitmustberunning
attheexactmomentofawait.SomeOracleprofessionalsrunthisscriptevery60seconds,sendinganemailwhenanimportantwaitoccurs:
wait_detail_process.sqlscript.
TueAug20page1
DetailAnalysis
TimeAvgAvgMax
SystemEventTotalWaitedWaitWaitWait
IDPrcsNameWaits(insecs)secssecs(insecs)
3LGWRlogfileparallelwrite415,0825,076201
4CKPTcontrolfileparallelwrite23,035327001
4CKPTfileopen20,8424000
2DBW0dbfileparallelwrite17,6841,292401
4CKPTdirectpathread9,745106006
3LGWRLGWRwaitforredocopy7,60824300
4CKPTdirectpathwrite3,7432000
5SMONdbfilescatteredread1,8247000
4CKPTcontrolfilesequentialread1,3550000
5SMONdbfilesequentialread7942000
156CPA1_OWNERdbfilesequentialread6773000
3LGWRcontrolfilesequentialread4680000
3LGWRcontrolfileparallelwrite2716000
2DBW0controlfilesequentialread2530000
3LGWRfileopen1900000
2DBW0fileopen1120000
5SMONfileopen1110000
2DBW0directpathread1080000
3LGWRdirectpathread1082000
2DBW0fileidentify1080000
3LGWRdirectpathwrite1070000
12fileopen910000
16fileopen901000
20fileopen891000
3LGWRimmop670000
16fileopen901000
20fileopen891000
3LGWRimmop670000
15fileopen380000
6RECOdbfilesequentialread360000
156CPA1_OWNERlatchfree3402100
2DBW0latchfree2912900
5SMONlatchfree1701100
125SYSlatchfree140700
3LGWRlatchfree1101100
156CPA1_OWNERfileopen110000
25ARC1fileopen80000
4CKPTfileidentify40000
15latchfree30300
2DBW0processstartup30000
11ARC0enqueue24023
125SYSdbfilesequentialread20000
4CKPTlatchfree10100
12latchfree10100
11ARC0processstartup10000
3LGWRenqueue13033
6RECOfileopen10000
125SYSfileopen10000
25ARC1enqueue10000
3LGWRprocessstartup10000
16latchfree10100
6RECOlatchfree10100
Hereistheoutputfromthisscript.Hereweseetwotypesofwaitconditions:
l
l
UserCPA1iswaitingtoaccessfile116,block51253Wecanusetheget_object_by_block_nbr.sqlscripttoseetheexactblock.
IUD_READ_ONLYusersarewaitingonaparallelquerydequeue
Individualprocesswaittimes
SIDNameNameWaitTimestateP1textValText
P2P3P3
ValTextVal
75CPA1_USERdbfilese.00.00WAITINGfile#116block#
quentialr
ead
51,253blocks1
52IUD1_READ_ONLYPXDeqCre.00.00WAITINGsleeptime/268,566,527passes
dit:sendsenderid
blkd
284qref0
dit:sendsenderid
blkd
284qref0
dit:sendsenderid
blkd
284qref0
dit:sendsenderid
blkd
284qref0
MethodsforUsingEventWaitData
Becausethisisashortarticle,let'sconfineourwaitdiscussiontothemostonerouswaitevent,the dbfilesequentialreadwaitevent.
Inourexamplefromthesystemwidereportswenotedthatthenumberonewaitinv$system_eventis"dbfilesequentialreads."Thiswaiteventis
usuallyforindexreads.Ifwelookatthedetaillistingforthesewaitevents,weseethatthescriptreturnsthefilenumberandblocknumberwhere
Oraclewaswaitingfortheevent.
MethodsforUsingEventWaitData
Becausethisisashortarticle,let'sconfineourwaitdiscussiontothemostonerouswaitevent,the dbfilesequentialreadwaitevent.
Inourexamplefromthesystemwidereportswenotedthatthenumberonewaitinv$system_eventis"dbfilesequentialreads."Thiswaiteventis
usuallyforindexreads.Ifwelookatthedetaillistingforthesewaitevents,weseethatthescriptreturnsthefilenumberandblocknumberwhere
Oraclewaswaitingfortheevent.
SIDNameNameWaitTimestateP1textValText
P2P3P3
ValTextVal
75CPA1_USERdbfilese.00.00WAITINGfile#116block#
quentialr
ead
51,253blocks1
HereweseethattheCPA1_USERwaswaitingonaccesstofilenumber16,blocknumber51,253.Giventhisinformation,wecandrilldowninto
dba_extentsanddba_data_filestoseethedatafilethatiscausingthewaits.
Hereisasimplescriptthatwilldisplaythefilenameforanyfilenumber:
acceptfilenbrprompt'Enterthefile#:'
selecttablespace_name,file_name
fromdba_data_files
wherefile_id=&&filenbr
Whenwerunthescriptwecanquicklyseethenameofthefilethatisexperiencingthe"dbfilesequentialreads"wait:
Enterthefile#:10
TABLESPACE_NAME
FILE_NAME

USERS
C:\ORACLE\ORADATA\DIOGENES\USERS02.DBF
ToproperlylocatethesourceofthewaitweneedmorethanjustthefilenameWecanusetheget_object_by_block_nbr.sqlscripttodisplayallobject
onthetargetblock:
setpages999
setverifyoff
setlines80
colc1heading'Segment|Name'formata40
colc2heading'Segment|Type'formata10
colc3heading'First|Block|ofSegment'format999,999,999,999
colc4heading'Last|Block|ofSegment'format999,999,999,999
acceptfilenbrprompt'Enterthefilenumber:'
acceptinblockprompt'Entertheblocknumber:'
selectsegment_namec1,
segment_typec2,
block_idc3
block_id+blocksc4
fromdba_extents
where&filenbr=file_id
and&inblock>=block_id
and&inblock<=block_id+blocks
HereweseetheexactsourceofthewaiteventisanindexcalledIDX_EVENTCASE_STATUS_OVERAGE.
Enterthefilenumber:116
Entertheblocknumber:51253
First
SegmentSegmentBlock
NameTypeofSegment
IDX_EVENTCASE_STATUS_OVERAGEINDEX51,205
So,whyaresessionswaitingonindexaccess?Themostcommoncauseisafreelistshortage.PriortoOraclebitmapfreelists(automaticextent
management),segmentheadercontentioncanhappenifmultipletaskscompetetosimultaneouslyupdatethesameindex.Thescriptcaneasilycheck
thenumberoffreelistsforthisobject:
clearcolumns
colc1heading'Table|Name'formata20
colc2heading'Table|Freelists'format99
colc3heading'Index|Name'formata20
colc4heading'Index|Freelists'format99
selectdistinctt.table_namec1,
t.freelistsc2,
index_namec3,
i.freelistsc4
fromdba_tablest,dba_indexesi
wheret.table_name=i.table_name
colc2heading'Table|Freelists'format99
colc3heading'Index|Name'formata20
colc4heading'Index|Freelists'format99
selectdistinctt.table_namec1,
t.freelistsc2,
index_namec3,
i.freelistsc4
fromdba_tablest,dba_indexesi
wheret.table_name=i.table_name
andi.index_name='IDX_EVENTCASE_STATUS_OVERAGE'
Ofcourse,therecouldbeotherproblemssuchasahighdiskenqueueonthediskthatcontainstheindex,butweshouldalwaystrytoaddfreelistsfirst.
Ifmorefreelistsdonotsolvethewaitissue,thentheindexmustbeplacedonafasterdisk,alessbusydisk,orstripedacrossmultipledisks.
Also,notethatbecausewehavetheSIDforthewaitingtask,wecanusetheSIDtojoinintov$sessionandv$sqltoseetheSQLthatwascausingthe
waitcondition.
HowWaitEventInformationCanBeUsedtoBoostSystemPerformance
Intheexamplesbelow,wewillapplythetechniquesdescribedaboveforcollectingwaiteventinformation,andwewillevaluatethedatainordertobetterunderstandhow
thesystemisfunctioninganddeterminewhatwecandotoboostsystemperformance.Hopefullytheseexampleswillgiveyouinsightsintoresearchingperformanceand
othertypesofproblemsonyoursystems.
Example#1:BufferBusyWaits
Amagazinepublisherhaddevelopedawebsitethatofferedwebaccesstomanyofthearticlesinthemagazine.Attimes,thewebsitewouldgetcompletelyboggeddown
andresponsetimewouldbecomeverypoor.Operatingsystemstatisticsonthedatabaseserverindicatedthatduringbusytimes,serveridletimewouldbenearzero.We
usedStatspacktogatherdatabasestatisticsonanhourlybasis.ExcerptsfromoneStatspackreportareasfollows:
SnapIdSnapTimeSessions
BeginSnap:6111Dec0213:00:52145
EndSnap:7111Dec0214:00:26145
Elapsed:59.57(mins)
CacheSizes
~~~~~~~~~~~
db_block_buffers:38400log_buffer:1048576
db_block_size:8192shared_pool_size:80000000
LoadProfile
~~~~~~~~~~~~PerSecondPerTransaction
Redosize:6,390.415,731.32
Logicalreads:33,962.4030,459.63
Blockchanges:44.7240.11
Physicalreads:536.77481.41
Physicalwrites:163.02146.21
Usercalls:397.75356.73
Parses:237.64213.13
Hardparses:23.8421.38
Sorts:248.38222.77
Logons:0.410.36
Executes:213.52191.50
Transactions:1.11
%BlockschangedperRead:0.13RecursiveCall%:86.07
Rollbackpertransaction%:0.23RowsperSort:30.73
InstanceEfficiencyPercentages(Target100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BufferNowait%:98.39RedoNoWait%:99.97
BufferHit%:98.42InmemorySort%:100.00
Top5WaitEvents
~~~~~~~~~~~~~~~~~Wait%Total
EventWaitsTime(cs)WtTime
bufferbusywaits1,962,3721,278,64950.03
dbfilesequentialread1,336,8701,050,87841.12
dbfilescatteredread47,71749,3261.93
latchfree38,22031,0121.21
BufferPoolStatisticsforDB:
>PoolsD:defaultpool,K:keeppool,R:recyclepool
FreeWriteBuffer
BufferConsistentPhysicalPhysicalBufferCompleteBusy
PGetsGetsReadsWritesWaitsWaitsWaits
D12,29249,863,6964703,5300032
K1,509,38520,691,9721,509,36611001,962,117
R51300000
Wenoticedthatthetwodominantwaiteventswerebufferbusywaitsanddbfilesequentialread.Thesetwowaiteventsaccountedforover23,000secondsofwaittime
duringtheonehoursampleperiod.Wedecidedtodrilldowntolearnmoreaboutthesewaitevents.Lookingatv$event_name,weseethattheparametersforthebuffer
busywaitseventareasfollows:
SELECT*FROMv$event_nameWHEREname='bufferbusywaits'
75bufferbusywaitsfile#block#id
Forthiswaitevent,theparameterinformationfromv$event_namepointsyouintherightdirectionbutdoesnotnecessarilygiveyouthefullstory.Thefirstandsecond
parameterstellyouthefileandblocknumberofthedatablockinthedatabasethatthesessionwastryingtoaccesswhenthewaiteventoccurred.Thethirdparameterisa
SELECT*FROMv$event_nameWHEREname='bufferbusywaits'
75bufferbusywaitsfile#block#id
Forthiswaitevent,theparameterinformationfromv$event_namepointsyouintherightdirectionbutdoesnotnecessarilygiveyouthefullstory.Thefirstandsecond
parameterstellyouthefileandblocknumberofthedatablockinthedatabasethatthesessionwastryingtoaccesswhenthewaiteventoccurred.Thethirdparameterisa
numericreasoncodeindicatingwhythebufferinthebuffercachewasbusyandtheprocesshadtowait.Metalinkbulletin34405.1explainswhatthedifferentreason
codesmean.
Thefollowingquerywilllistallsessionswaitingforagivenwaiteventatthesplitsecondthequeryisbegun:
SELECTsid,event,state,seconds_in_wait,wait_time,p1,p2,p3
FROMv$session_wait
WHEREevent='bufferbusywaits'
ORDERBYsid
Becauseonlyacouple(ifany)sessionsarelikelytobeencounteringthiswaitatanyonemomentintime,weusuallyneedtorunthisqueryseveraltimesinquick
succession.Oneofthetimes,wegotthefollowingoutput:
SIDEVENTSTATESECONDS_IN_WAITWAIT_TIMEP1P2P3
12bufferbusywaitsWAITE103062157130
31bufferbusywaitsWAITE103023558130
Weperformedthefollowingqueryforeachsessionwaitfound,substitutingP1forthefilenumberandP2fortheblocknumber:
SELECTowner,segment_name,segment_type
FROMdba_extents
WHEREfile_id=&absolute_file_number
AND&block_numberBETWEENblock_idANDblock_id+blocks 1
Usingthisquerywefoundthatsession12abovewaswaitingonabusybufferintheSAMPLEStableinthePRODMGRschema:
Entervalueforabsolute_file_number:30
Entervalueforblock_number:62157
OWNERSEGMENT_NAMESEGMENT_TYPE
PRODMGRSAMPLESTABLE
Inordertofigureoutwhatreasoncode130means,welookatareferencetablelistedinMetalinkbulletin34405.1.Forconveniencethistableappearsbelow.Wecansee
Reason
Code(Id)P3 Reason
>=8.1.6
0
Ablockisbeingread
100
WewanttoNEWtheblockbuttheblockiscurrentlybeingreadbyanothersession(mostlikelyforundo).
200
WewanttoNEWtheblockbutsomeoneelsehasisusingthecurrentcopysowehavetowaitforthemtofinish.
230
TryingtogetabufferinCR/CRXmode,butamodificationhasstartedonthebufferthathasnotyetbeencompleted.
AmodificationishappeningonaSCURorXCURbuffer,buthasnotyetcompleted
231
CR/CRXscanfoundtheCURRENTblock,butamodificationhasstartedonthebufferthathasnotyetbeencompleted.
130
Blockisbeingreadbyanothersessionandnoothersuitableblockimagewasfound,sowewaituntilthereadiscompleted.Thismayalsooccuraftera
buffercacheassumeddeadlock.Thekernelcan'tgetabufferinacertainamountoftimeandassumesadeadlock.ThereforitwillreadtheCRversion
oftheblock.
110
WewanttheCURRENTblockeithersharedorexclusivebuttheBlockisbeingreadintocachebyanothersession,sowehavetowaituntiltheirread()
iscompleted.
120
Wewanttogettheblockincurrentmodebutsomeoneelseiscurrentlyreadingitintothecache.Waitforthemtocompletetheread.Thisoccursduring
bufferlookup.
210
ThesessionwantstheblockinSCURorXCURmode.IfthisisabufferexchangeorthesessionisindiscreteTXmode,thesessionwaitsforthefirst
timeandthesecondtimeescalatestheblockasadeadlockandsodoesnotshowupaswaitingverylong.Inthiscasethestatistic:"exchange
deadlocks"isincrementedandweyieldtheCPUforthe"bufferdeadlock"waitevent.
220
DuringbufferlookupforaCURRENTcopyofabufferwehavefoundthebufferbutsomeoneholdsitinanincompatiblemodesowehavetowait.
NextwelookedforsessionswaitingonthedbfilesequentialreadeventsessionswaitingonindexlookupsortableaccessesbyROWID:
FROMv$session_wait
WHEREevent='dbfilesequentialread'
ORDERBYsid
Followingthesameapproachasusedwiththebufferbusywaits,wethendrilleddowntofindwhichsegmentsinthedatabasesessionswereaccessingwiththesingle
blockI/Os.InthiscaseitwasagaintheSAMPLEStable.
Fromthisweconcludedthatthebufferbusywaitsappearedrelatedtothesingleblockdiskreadsbufferblockreadswerewaitingforotherbufferblockreadsto
complete,andthoseinturnwerewaitingforsequentialreadsfromdisktocomplete.Putanotherway,sessionAwaswaitingonasingleblockdiskI/Otobringablock
intothebuffercache.Meanwhile,sessionBwantedtoreadthesameblock.Sincetheblockwasnotyetinthebuffercacheandanothersessionwasalreadyreadingit
fromdisk,sessionBwaitedonabufferbusywaitseventwhilesessionAwaitedonthedbfilesequentialreadevent.
Followingthesameapproachasusedwiththebufferbusywaits,wethendrilleddowntofindwhichsegmentsinthedatabasesessionswereaccessingwiththesingle
blockI/Os.InthiscaseitwasagaintheSAMPLEStable.
Fromthisweconcludedthatthebufferbusywaitsappearedrelatedtothesingleblockdiskreadsbufferblockreadswerewaitingforotherbufferblockreadsto
complete,andthoseinturnwerewaitingforsequentialreadsfromdisktocomplete.Putanotherway,sessionAwaswaitingonasingleblockdiskI/Otobringablock
intothebuffercache.Meanwhile,sessionBwantedtoreadthesameblock.Sincetheblockwasnotyetinthebuffercacheandanothersessionwasalreadyreadingit
fromdisk,sessionBwaitedonabufferbusywaitseventwhilesessionAwaitedonthedbfilesequentialreadevent.
WhenwelookedattheSAMPLEStablefurther,wefoundthatitcontainedaLONGcolumnandtherewasafairamountofrowchainingsomerowswerelargerthanthe
databaseblocksize.ManyqueriesagainstmultipletablesweredoingnestedloopsjoinswiththeSAMPLEStable.Thiswascausinglotsofsingleblockreadsfromthe
SAMPLEStable.
ThevaluesstoredintheLONGcolumnoftheSAMPLEStablewerequitebig.OneofthemanyweaknessesoftheLONGdatatypeisthatdataisstoredinlineinthe
tablesegmentregardlessofsize.ThisledtoaverylargeSAMPLEStablesegmenteventhoughthetabledidnothavealargenumberofrows.Whilemostqueriesagainst
theSAMPLEStablewerenotfetchingtheLONGcolumn,OraclestillhadtobringtheLONGdataintothebuffercachebecauseitwasstoredinthesameblocksasthe
othercolumns.ThisledtoalotmorephysicalI/Oandchurninthebuffercachethanshouldhavebeennecessary.
OnewaytoreducebuffercachechurnandphysicalI/OwouldhavebeentoreplacetheLONGcolumnwithaCLOBcolumn.CLOBdatacanbestoredinaseparate
segmentfromtherestofthetabledata.ThustheSAMPLEStablesegmentwouldbealotsmallerandmorelikelytofitentirelyinthebuffercache.Thiswouldreducethe
activityrateinthebuffercacheandtheamountofphysicalI/O.
Unfortunately,changingthedatabaseschemaandtheapplicationcodetoreplacetheLONGdatatypewithCLOBwasnotpossibleinaquickenoughtimeframeto
addresstheimmediateperformanceproblem.Sowehadtodevelopaquickfixthatcouldbedeployedtemporarilyuntilthedatatypecouldbechanged.Firstweenlarged
theKEEPbufferpool,reducingphysicalreadsasalloftheSAMPLEStablecouldstayinthebuffercache.Next,wecreatedanindexonallofthecolumnsinthe
SAMPLEStableexcepttheLONGcolumn.ThisallowedOracletosatisfymostqueriesagainsttheSAMPLEStablewithouthavingtoreadtheexcessivelylargetable
segment,whichgreatlyreducedtheamountofdataaccessedduringeachquery.
Thesetwoeasyandquickchangesvirtuallyeliminatedthebufferbusywaitsandreduceddbfilesequentialreadwaiteventsastotalphysicalreadsdropped90%.
ResponsetimeforthemostcommonqueriesinvolvingtheSAMPLEStabledramaticallyimproved,andoverallwebsiteresponsetimesreturnedtoanacceptablelevel.
ExcerptsfromaStatspackreportafterthetwochangesweremadeareasfollows:
SnapIdSnapTimeSessions
BeginSnap:119220Dec0213:00:49102
EndSnap:120220Dec0214:00:18102
Elapsed:59.48(mins)
CacheSizes
~~~~~~~~~~~
db_block_buffers:38400log_buffer:1048576
db_block_size:8192shared_pool_size:80000000
LoadProfile
~~~~~~~~~~~~PerSecondPerTransaction
Redosize:2,496.243,453.13
Logicalreads:30,270.8641,874.69
Blockchanges:17.0823.62
Physicalreads:36.9051.04
Physicalwrites:79.62110.14
Usercalls:313.37433.50
Parses:173.91240.57
Hardparses:17.0423.57
Sorts:169.33234.25
Logons:0.360.50
Executes:149.11206.27
Transactions:0.72
%BlockschangedperRead:0.06RecursiveCall%:84.55
Rollbackpertransaction%:0.31RowsperSort:62.86
InstanceEfficiencyPercentages(Target100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BufferNowait%:100.00RedoNoWait%:99.98
BufferHit%:99.88InmemorySort%:99.89
Top5WaitEvents
~~~~~~~~~~~~~~~~~Wait%Total
EventWaitsTime(cs)WtTime
logfilesync4,9147,49316.93
librarycachepin1,1756,09013.76
directpathread5,4883,4287.75
latchfree14,5282,9316.62
BufferPoolStatisticsforDB
>PoolsD:defaultpool,K:keeppool,R:recyclepool
FreeWriteBuffer
BufferConsistentPhysicalPhysicalBufferCompleteBusy
PGetsGetsReadsWritesWaitsWaitsWaits
D10,17839,561,0723301,1220010
K113,075,94414000
R0000000
Asyoucansee,withasimilaramountofqueryingactivity(roughly300400usercallsand30,00033,000logicalreadspersecond)wehavereducedtotalwaiteventtime
toabout442secondsduringtheonehoursampleperiod.(Directpathwritestotaled13545centisecondsandrepresented30.61%oftotalwaits135.45/0.3061=442
secondsoftotalwaittime.)Thisisquiteadifferencefromthe23,000secondsofwaittimeduringthesampleperiodwewereseeingbeforetheKEEPpoolwasenlarged
andtheextraindexwasaddedtotheSAMPLEStable.
Overtime,applicationchangesweremadesothatevenfewerqueriesagainsttheSAMPLEStablerequiredretrievingtheLONGcolumn.Eventually,theLONGcolumn
wasreplacedwithaCLOB.
Example#2:MoreBufferBusyWaits,PlusLatchContentionforGoodMeasure
toabout442secondsduringtheonehoursampleperiod.(Directpathwritestotaled13545centisecondsandrepresented30.61%oftotalwaits135.45/0.3061=442
secondsoftotalwaittime.)Thisisquiteadifferencefromthe23,000secondsofwaittimeduringthesampleperiodwewereseeingbeforetheKEEPpoolwasenlarged
andtheextraindexwasaddedtotheSAMPLEStable.
Overtime,applicationchangesweremadesothatevenfewerqueriesagainsttheSAMPLEStablerequiredretrievingtheLONGcolumn.Eventually,theLONGcolumn
wasreplacedwithaCLOB.
Example#2:MoreBufferBusyWaits,PlusLatchContentionforGoodMeasure
Themagazinepublisher'sapplicationdemonstratedbufferbusywaitswheremultipleOracleprocesseswanttoreadthesamedatablockfromdiskatthesametime.The
firstOracleprocesswouldissuetheI/Orequestandwaitonadbfilesequentialreadevent,whilesubsequentOracleprocesseswouldwaitonabufferbusywaitsevent
untilthedatablockhadmadeitswayintothebuffercache.NowwewilllookatanexamplewheremultipleOracleprocessestrytowritetodifferentcopiesofthesame
datablockinthebuffercacheatthesametime.SinceOraclehasrowlevellocking,multipleuserscanupdatedifferentrowsinthesametableatthesametimewithout
lockingeachotherout.Multipleuserscanevenupdatedifferentrowsinthesamedatablockconcurrently.But,aswewillseeinthisexample,contentionandbufferbusy
waitscansometimesoccur.
AgeneticresearchcompanyusedanOracledatabasetostoreresearchdata.Periodically,theywouldloadlargevolumesofrawdataintothedatabase.Thentheywould
runaprocessingapplicationonmultipleworkstations.Eachworkstationwouldfetchrawdatafromthedatabase,processit,andputtheprocesseddatabackinthe
database.Thecompanyhadplannedtoaddmoreworkstationstospeeduptheprocessing,butfoundthataddingmoreworkstationsactuallyseemedtoreducethe
throughput.TheOracledatabasewasbeingusedmerelyasarepository,weweretold.AllOraclehadtodowasfetchtherawdataandthenstoretheprocesseddata.
TheactualprocessingwasbeingperformedontheworkstationsoutsideoftherealmofOracle.SoitwouldseemthattheOracledatabaseshouldnotbethebottleneck.
Wefoundabout100activedatabasesessions,eachinitiatedbyaseparateworkstation.Wemodifiedtheapplicationcoderunningonworkstation30toactivatedebug
event10046forwaiteventtracing.Wedidthisbyinsertingthefollowingstatementintothecodejustafterconnectingtothedatabase:
Wealsomodifiedtheapplicationtoexitafteriteratingthroughitsmainloop500times.Weranthemodifiedapplicationontheworkstationandretrievedthetracefilethat
Oraclehadcreatedinthedirectoryspecifiedbytheuser_dump_destparameter.Theapplicationranforlessthanoneminutebeforeexiting.SincethiswasanOracle9i
database,wewereabletorunTKPROFtogenerateareportincludingwaiteventstatisticswiththefollowingcommand:
tkprofprodgen_ora_16466.trcreport_16466.prfwaits=yes
AninterestingexcerptfromtheTKPROFreportwasasfollows:
UPDATEprocessing_stations
SETstatus='ACTIVE',
status_date=SYSDATE,
data_set_id_being_processed=:b1
WHEREstation_id=30
Parse10.010.000000
Execute5000.2310.14036161010500
Fetch00.000.000000
total5010.2410.14036161010500
Parsinguserid:40
500UPDATE
500INDEXUNIQUESCANPROCESSING_STATIONS_PK(objectid9372)
Waited
bufferbusywaits260.717.87
latchfree170.572.08
logfileswitchcompletion30.090.20
TheUPDATEstatementhereseemedquitetrivialchangingthreecolumnvaluesononerow,givenaprimarykeyvalue.Oracleonlyneededtodoaboutninelogicalreads
perexecution,andnophysicalreadswererequiredatall.YetitstilltookOracleover10secondstoexecutethestatement500times.Ifwehadnotspecifiedthe
"waits=yes"TKPROFoption,wemightconcludethatthedatabaseserverwasCPUstarved.ButaquicklookatthewaiteventportionoftheTKPROFreportshows
thatmostofthe10secondsofelapsedexecutiontimewasspentwaitingonbusybuffersandlatches.
Someofthebufferbusywaitslistedinthetracefilewereasfollows:
WAIT#2:nam='bufferbusywaits'ela=527727p1=18p2=10p3=220
SincethistracefilewasgeneratedbyanOracle9idatabase,theelapsedtimesareshowninunitsofmicroseconds.Thewaitshownatthebottomofthelistabovehadan
elapsedtimeof710,696microseconds,orapproximately0.71seconds.Thiswasthelongestbufferbusywaitforthisstatement,asindicatedbythe"Max.Wait"figurein
theTKPROFreport.
Fromthepreviousexample,recallthattheparametersforthebufferbusywaitseventindicatefilenumber,blocknumber,andreasoncode.Allofthebufferbusywaits
listedaboveinvolveblock10indatafile18,withareasoncodeof220.Byqueryingthedatadictionary,wefoundthatblock10indatafile18wasallocatedtothe
PROCESSING_STATIONStable,thetablebeingupdatedbythestatementthatencounteredthebufferbusywaits.AccordingtoMetalinkbulletin34405.1,reason
code220indicatesthatthebufferisbusybecauseanothersessionisusingacopyofthebufferinanincompatiblemode.
FromtheTKPROFreportwecanseethattheapplicationupdatedthePROCESSING_STATIONStable500timesduringtheperiodtraced.Eachupdateaffectedone
theTKPROFreport.
Fromthepreviousexample,recallthattheparametersforthebufferbusywaitseventindicatefilenumber,blocknumber,andreasoncode.Allofthebufferbusywaits
listedaboveinvolveblock10indatafile18,withareasoncodeof220.Byqueryingthedatadictionary,wefoundthatblock10indatafile18wasallocatedtothe
PROCESSING_STATIONStable,thetablebeingupdatedbythestatementthatencounteredthebufferbusywaits.AccordingtoMetalinkbulletin34405.1,reason
code220indicatesthatthebufferisbusybecauseanothersessionisusingacopyofthebufferinanincompatiblemode.
FromtheTKPROFreportwecanseethattheapplicationupdatedthePROCESSING_STATIONStable500timesduringtheperiodtraced.Eachupdateaffectedone
row.Whileperformingtheupdates,therewere26timesthattheprocesshadtowaitonabufferbusywaitsevent.Fromlookingatthetracefileweknowthatallofthe
bufferbusywaitsoccurredbecausedatablock10indatafile18wasalreadyinthebuffercache,butinanincompatiblemode.These26waitstotaled7.87seconds.
AqueryagainstthedatadictionaryrevealedthefollowingstatisticsforthePROCESSING_STATIONStable:
SELECTSYSDATElast_analyzed,blocks,avg_row_len,avg_space,
chain_cnt,num_rows
FROMuser_tables
WHEREtable_name='PROCESSING_STATIONS'
SYSDATELAST_ANALYZEDBLOCKSAVG_ROW_LENAVG_SPACECHAIN_CNTNUM_ROWS
2.13211805616216860100
Wecanseethatoptimizerstatisticsweregatheredrecentlyandthetablecontains100veryshortrows,allstoredinonedatablock.Iftheapplicationsessionthatwe
tracedupdatedthePROCESSING_STATIONStable500timesinlessthanaminute,wemightsuspectthatthedatabasesessionsfromtheother99workstationsare
doingroughlythesamething.Thisaddsuptoover50,000updatestothesamedatablockperminute.Sinceeachsessionisupdatingadifferentrowinthetable,thereis
nowaitingonrowlevellocks(whichwouldshowupinthewaiteventinterfaceasenqueuewaits).However,onlyoneprocessatatimemaywritetoabufferinthebuffer
cache.Therefore,alotoftimeisbeingwastedcontendingforthisoneveryhotbuffer.
Allofthisbegsthequestion,"WhyistheapplicationupdatingarowinthePROCESSING_STATIONStable500timesinlessthanoneminute?"Itturnsoutthatthe
applicationusesthistabletocoordinateeffortsbetweentheworkstations.Thereisonerowinthetableforeachworkstation.Theapplicationrunningoneachworkstation
updatesitsrowinPROCESSING_STATIONStoindicatewhichdatasetitisprocessingsothatotherworkstationswillnotattempttoprocessthesamedataset.Also,
eachworkstationperiodicallyupdatesthestatusdatesothatifaworkstationcrashes,itwillbeeasytotellwhenitcrashedandwhatdataitwasprocessingduringthe
crash.
Processingofonedatasetcouldtakeanywherefromafractionofasecondtotwentyminutes.Theapplicationcodewaswrittentonotonlyupdatethe
PROCESSING_STATIONStablewhenanewdatasetwasprocessed,butalsotofrequentlyupdatethetimestampduringtheprocessingofonedataset.All
stakeholdersagreedthat500updatesperminutemightbeexcessive.Iftheapplicationcodecouldbemodifiedtoupdatethetableonlyoncepersecond,thenwritestothe
tablewouldreducebyroughly90%andcontentionforthehotbufferinthebuffercachewouldlikelygoaway.
However,applicationchangesneededtobetestedthoroughlybeforedeployment.Ashorttermfixwasdesireduntilthelongtermfixoftheapplicationcouldbe
implemented.Luckily,atemporaryfixwasavailablethatwassimpletoimplement.
WerebuiltthePROCESSING_STATIONStablewiththePCTFREEstorageparametersetto99andPCTUSEDsetto1.ThiscausedOracletoleave99%ofthe
availablespaceineachdatablockemptyforfuturerowexpansion.Thischangecausedthe100rowsinthetabletobespreadover100datablocksinsteadofallrows
beingneatlypackedintoonedatablock.Aqueryagainstthedatadictionaryconfirmedthis:
SELECTSYSDATElast_analyzed,blocks,avg_row_len,avg_space,
chain_cnt,num_rows
FROMuser_tables
WHEREtable_name='PROCESSING_STATIONS'
SYSDATELAST_ANALYZEDBLOCKSAVG_ROW_LENAVG_SPACECHAIN_CNTNUM_ROWS
.1308680561006280140100
Tracinganother500iterationsofthemainloopintheapplicationonworkstation30afterrebuildingthePROCESSING_STATIONStableyieldedthefollowing
TKPROFoutput:
UPDATEprocessing_stations
SETstatus='ACTIVE',
status_date=SYSDATE,
data_set_id_being_processed=:b1
WHEREstation_id=30
Parse10.000.000000
Execute5000.202.2205001009500
Fetch00.000.000000
total5010.202.2205001009500
Parsinguserid:40
500UPDATE
500INDEXUNIQUESCANPROCESSING_STATIONS_PK(objectid9379)
Waited
latchfree20.350.61
TheapplicationisstillupdatingthePROCESSING_STATIONStableveryrapidly.Butnoweachrowinthetableisstoredinaseparatedatablock,soeachworkstation
isworkingwithitsownhotbufferinthebuffercache.Thesessionsarenolongercompetingwitheachotherfortheopportunitytowritetothesamebuffer.Waitsonthe
bufferbusywaitseventhavebeeneliminated.Asasideeffect,logicalreadshavebeenreducedtoonethirdofwhattheywereoriginally.
TheoriginalTKPROFreportsalsoindicatedwaitsonthelatchfreewaitevent,indicatinglatchcontention.Aqueryagainstv$event_nameshowsthattheparameterstothe
latchfreewaiteventareasfollows:
latchfree20.350.61
TheapplicationisstillupdatingthePROCESSING_STATIONStableveryrapidly.Butnoweachrowinthetableisstoredinaseparatedatablock,soeachworkstation
isworkingwithitsownhotbufferinthebuffercache.Thesessionsarenolongercompetingwitheachotherfortheopportunitytowritetothesamebuffer.Waitsonthe
bufferbusywaitseventhavebeeneliminated.Asasideeffect,logicalreadshavebeenreducedtoonethirdofwhattheywereoriginally.
TheoriginalTKPROFreportsalsoindicatedwaitsonthelatchfreewaitevent,indicatinglatchcontention.Aqueryagainstv$event_nameshowsthattheparameterstothe
latchfreewaiteventareasfollows:
SELECT*FROMv$event_nameWHEREname='latchfree'
3latchfreeaddressnumbertries
SomeofthelatchfreewaiteventsinthetracefilegeneratedbeforethePROCESSING_STATIONStablewasrebuiltwereasfollows:
WAIT#2:nam='latchfree'ela=47004p1=15113593728p2=97p3=0
Aqueryagainstv$latchorv$latchnamewillgiveusthenameoflatchnumber97:
SELECTname
FROMv$latch
WHERElatch#=97
NAME
cachebufferschains
Sowecanseethatthefourwaiteventslistedfromthetracefileaboveamounttoawaitofapproximately0.12secondsforthecachebufferchainslatch.(Rememberthat
inOracle9itracefiles,elapsedtimesarelistedinmicrosecondswhileintracefilesfromOracle8iandearliertheelapsedtimesareincentiseconds.)Thisisnot
unexpected,sincewehad100sessionscontendingfortherighttomodifyonebufferinthebuffercache.
ComparingtheTKPROFreportbeforerebuildingthePROCESSING_STATIONStablewithafter,weseethatrebuildingthetablereducedthelatchcontentionbutdid
noteliminateit.Itislikelythatthelongtermsolutionofmodifyingtheapplicationtoreducetheupdatefrequencywilleliminateorfurtherreducethelatchcontention.
Example#3:LogFileWaits
Anapplicationperformedoccasionalloadsintoasmalldatawarehouse.Unfortunately,theloadsweretakingamuchlongertimeontheproductionsystemthantheyhadin
atestenvironment.Tofindoutwhatwashappeningontheproductiondatabaseduringadataload,weranthefollowingscriptthatdisplaysthewaiteventactivityforthe
entireinstanceovera30secondinterval.ThelistofeventsintheNOTINclausesimplyfiltersouttheidleevents,makingtheoutputalittleeasiertoread:
DROPTABLEprevious_events
CREATETABLEprevious_events
AS
SELECTSYSDATEtimestamp,v$system_event.*
FROMv$system_event
EXECUTEdbms_lock.sleep(30)
SELECTA.event,
A.total_waitsNVL(B.total_waits,0)total_waits,
A.time_waitedNVL(B.time_waited,0)time_waited
FROMv$system_eventA,previous_eventsB
WHEREB.event(+)=A.event
ANDA.eventNOTIN
(
'clientmessage','dispatchertimer','gcsforaction',
'gcsremotemessage','gesremotemessage','i/oslavewait',
'jobqslavewait','lockmanagerwaitforremotemessage',
'nullevent','Nullevent','parallelquerydequeue','pipeget',
'PL/SQLlocktimer','pmontimer','PXDeqCredit:needbuffer',
'PXDeqCredit:sendblkd','PXDeq:ExecuteReply',
'PXDeq:ExecutionMsg','PXDeq:SignalACK',
'PXDeq:TableQNormal','PXDequeWait','PXIdleWait',
'queuemessages','rdbmsipcmessage','slavewait',
'smontimer','SQL*Netmessagetoclient',
'SQL*Netmessagefromclient','SQL*Netmoredatafromclient',
'virtualcircuitstatus','wakeuptimemanager'
)
ORDERBYtime_waited
Runningthisscriptduringadataloadyieldedthefollowingoutput:
EVENTTOTAL_WAITSTIME_WAITED
LGWRwaitforredocopy00
SQL*Netbreak/resettoclient20
SQL*Netmoredatatoclient00
bufferbusywaits00
directpathread270
directpathwrite340
directpathwrite(lob)00
fileidentify60
fileopen930
librarycachepin00
localwritewait00
processstartup00
rdbmsipcreply00
refreshcontrolfilecommand00
singletaskmessage00
directpathwrite(lob)00
fileidentify60
fileopen930
librarycachepin00
localwritewait00
processstartup00
rdbmsipcreply00
refreshcontrolfilecommand00
singletaskmessage00
switchlogfilecommand00
controlfilesequentialread611
latchfree21
controlfileparallelwrite4131
logfilesinglewrite6164
dbfileparallelwrite13220
enqueue6486
logbufferspace242007
logfilesequentialread302655
logfileswitchcompletion332883
logfileparallelwrite193561
logfilesync11310249
Duringthe30secondsampleinterval,Oracleprocessesspentroughly215secondswaitingoneventsrelatedtotheonlineredolog.Thelikelyculpritinthissituationis
somecombinationofslowdisksandcontentiononthedisksholdingtheonlineredologfiles.Afteralittleresearch,wefoundthattheredologshadbeenstripedona
RAIDarrayonthefastertestsystem.Ontheproductionsystem,however,theredologswereallonasinglediskthatwassharedwithanotherveryactivedatabase.When
theonlineredologsweremovedtoseparatedisks,thedataloadsranmorerapidlyandthewaiteventspertainingtotheredologdecreasedsignificantly.
Example#4:DirectPathReadandWriteWaits
Acompanyexperienceddelayswhenqueryingsummarizedinformationabouttheirorders.Queryingindividualorderswasfast,butqueriesinvolvingtotalsorrollups
seemedtotakeunacceptablylong.WeranourDatabaseRxmonitoringtoolatregularintervalstocollectperformanceinformationfromthesystemwhiletheuserswere
performingtheirtypicaltasks.(DatabaseRxisaproactivemonitoringandalertnotificationtoolthatwedevelopedforusewithourremotedatabaseadministrationclients.
Forafreedemo,seeourwebsiteatdbrx.dbspecialists.com/guest.)
LookingattheTop5WaitEventssection,weseethattheleadingwaiteventsaredirectpathwrite,dbfilescatteredread,anddirectpathread.Thesethreewaitevents
combinedaccountedforroughly99%ofthenonidlewaittimeonthesystemduringthesampleinterval.Fromourknowledgeofthequeriesintheapplication,thedbfile
scatteredread(i.e.,fulltablescan)eventsdidn'tsurpriseus.Theabsenceofdbfilesequentialreadwaitsalsodidnotsurpriseusastherelevantblocksforindexlookups
androwfetchesbyROWIDwerelikelypresentinthebuffercache.
WefiguredwemightbeabletoreducethedbfilescatteredreadwaitswithalargerbuffercacheorjudicioususeoftheKEEPpoolintheshorttermandapplicationcode
reviewinthelongerterm.Whatseemedamiss,however,werethelargenumbersofdirectpathreadandwritewaitevents.Queryingv$session_waitwhileapplications
wererunningyieldedthefollowing:
FROMv$session_wait
WHEREevent='directpathwrite'
ORDERBYsid
SIDEVENTSTATESECONDS_IN_WAITWAIT_TIMEP1P2P3
39directpathwriteWAITI00201657
47directpathwriteWAITI0020122487
Queryingv$event_nameshowstheparametersforthedirectpathwriteevent:
SELECT*FROMv$event_nameWHEREname='directpathwrite'
115directpathwritefilenumberfirstdbablockcnt
Weseethatthefirstparameterissupposedtobethefilenumberofthedatafilecontainingtheblocksbeingwritten,butwecouldn'tfinddatafile201usingthefollowing
query:
SELECTtablespace_name,file_id"AFN",relative_fno"RFN"
FROMdba_data_files
WHEREfile_id=201
norowsselected
Thisisbecausethedatabaseisusingtempfilesinthetemporarytablespace,soweusedthefollowingquery:
SELECTtablespace_name,file_id+value"AFN",relative_fno"RFN"
FROMdba_temp_files,v$parameter
WHEREname='db_files'
TABLESPACE_NAMEAFNRFN
TEMP2011
ThisindicatedthattheTEMPtablespacewasexperiencingdirectpathwritewaits,whichisnotsurprisingasdirectpathreadsandwritesusuallyoccurinconnectionwith
temporarysegments.DirectpathI/Ooperationswereresponsibleformostofthewaitsonthesystemandappearedtobeabottleneck.Uponfurtherexamination,we
discoveredthatthesort_area_sizeparameterwassettothedefaultvalueof65536.SincethedatabaseserverhadafairamountofRAMfreeandtheregenerallyweren't
toomanysessionsconnectedsimultaneously,weincreasedthesort_area_sizeto10Mb,andthedirectpathwaitsdisappearedassortsnolongerhadtobewrittento
disk.
Ifincreasingthesort_area_sizehadnotbeenanoption,wemighthavelookedatthediskstoragewheretheTEMPtablespacetempfileswerelocatedtoseeiftherewas
I/Ocontentionwithothersystems.Wemightalsohavelookedatapplicationcodetoseeiftheamountofsortingandhashingcouldbereduced.
Thisexamplealsodemonstratessomethingthatmightseemoddatfirstglance.DuringtheDatabaseRxsampleperiod,thereweremorephysicalreadsthanlogicalreads.
Howcanthisbedon'tallphysicalreadsalsorequirelogicalreads?Well,thisiswheredirectpathreadsgettheirname.AsOraclesaysinreferencenote50415.1on
disk.
Ifincreasingthesort_area_sizehadnotbeenanoption,wemighthavelookedatthediskstoragewheretheTEMPtablespacetempfileswerelocatedtoseeiftherewas
I/Ocontentionwithothersystems.Wemightalsohavelookedatapplicationcodetoseeiftheamountofsortingandhashingcouldbereduced.
Thisexamplealsodemonstratessomethingthatmightseemoddatfirstglance.DuringtheDatabaseRxsampleperiod,thereweremorephysicalreadsthanlogicalreads.
Howcanthisbedon'tallphysicalreadsalsorequirelogicalreads?Well,thisiswheredirectpathreadsgettheirname.AsOraclesaysinreferencenote50415.1on
Metalink,"DirectpathreadsaregenerallyusedbyOraclewhenreadingdirectlyintoPGAmemory(asopposedtointothebuffercache)."Sincedirectpathreadsbypass
thebuffercache,theyaren'tcountedaslogicalreads.Thisisthereasonthatsomepeoplequestionthetraditionaldefinitionofthebuffercachehitratio,becauseitdoesn't
takeaccountofdirectpathreads.Wewon'tgetintothatdebate,butbeawarethatdirectpathreadscoulddistortyourcachehitratio.Inthisexample,thecachehitratio
wouldbenegative!
Example#5:DatabaseLinkWaitEvents
SeveraloftheSQL*Netwaiteventsareconsideredidleevents.Forexample,the"SQL*Netmessagefromclient"eventindicatesthattheprocessiswaitingforthe
applicationtosubmitaSQLstatementforprocessing.However,theSQL*Netwaiteventspertainingtodatabaselinksarenotidleeventstheycomeintoplaywhen
queriesusedatabaselinkstoaccessremotedatabases.The"SQL*Netmessagefromdblink"event,forexample,indicatesthatthelocalprocesshassubmittedarequest
toaremotedatabaseandiswaitingforaresponse.
Acompanyhaddeployedadataimportapplicationforreceivingbatchesofcustomertransactionsfromitspartners.TherewerefiveOracledatabases,oneper
geographicregion.Customertransactionsweresortedbygeographiclocation,andloadedintothedatabaseservingthatregion.Duetohumanerror,sometimesthesame
customertransactionwouldgetimportedintomultipleOracledatabases.Theyhaddevelopedareconciliationreporttoaddressthis.Thereportwassimplyaquerythat
comparedtransactionsbetweentwodatabasesandlistedrecordsfoundonboth.Unfortunately,thereporttookover30minutestorunandtheruntimewasexpectedto
getmuchworseasthecustomerbasegrew.
Lookingatthereportcode,wefoundthemainquery,whichappearedtobetheoneusingthelion'sshareofresources.WeloggedontoSQL*Plusasanapplicationuser
inonewindowandloggedontoSQL*PlusagainasaDBAinanotherwindow.FromtheDBAsessionwequeriedv$sessionandfoundtheSIDcorrespondingtoour
applicationuserSQL*Plussession.Thenwelaunchedthequeryfromtheapplicationusersessionandmonitoredv$session_eventfromtheother.Sincethewaitevent
interfacedoesnotaccountforCPUusage,wealsomonitoredv$sesstat.
Afterthereportquerycompleted31minuteslater,wequeriedv$session_eventonelasttimeandtheresultswereasfollows:
SELECTevent,total_waits,time_waited,max_wait
FROMv$session_event
WHEREsid=47
ORDERBYevent
EVENTTOTAL_WAITSTIME_WAITEDMAX_WAIT
SQL*Netmessagefromdblink152551610491931
SQL*Netmessagetodblink15255164669
latchfree4054
logfilesync122
SELECTA.name,B.value
FROMv$statnameA,v$sesstatB
WHEREA.statistic#=12
ANDB.statistic#=A.statistic#
ANDB.sid=47
NAMEVALUE
CPUusedbythissession67937
WecanignoretheSQL*NetmessagefromclientwaiteventbecausethisjustindicatestimeOraclespentwaitingforustosubmitthequeryinSQL*Plus.However,there
wereover1.5millionnetworkroundtripstoadatabaselinktotalingover1053secondsofwaittime.(The1053secondsincludesbothnetworklatencyandwaitingonthe
remotedatabaseservertorespondtorequests.)Therewereover27,000waitsonsingleblockdiskI/Ostotalingabout80secondsofwait.Therewerealsolatchfreeand
logfilesyncwaitevents,butthewaittimeswereinsignificant.Thequeryfromv$sesstatshowsthatabout679secondsofCPUtimewereused.
Knowingnothingaboutthereportqueryordatabasesize,thenumberofnetworkroundtripsseemedveryhighandthenumberofsingleblockdiskI/Osseemedlikeit
mightbehigh.Beforewecouldgoanyfurther,wereallyneededtolookatthequeryandthetablesinvolved.Thereportquerywasasfollows:
SELECTcustomer_id,batch_serial_number,batch_date,load_date,
batch_comment,control_total,control_amount
FROMcustomer_transfer_batchesA
WHEREexists
(
SELECT1
FROMcustomer_transfer_batches@prdwestB
WHEREB.customer_id=A.customer_id
ANDB.batch_serial_number=A.batch_serial_number
)
ORDERBYcustomer_id,batch_serial_number
EstimatedstatisticsontheCUSTOMER_TRANSFER_BATCHEStableshowedthatithadapproximately1,526,000rowsspanningapproximately21,825blocks.
Correlatingthewaiteventinformationwiththetablesize,itwasclearthatOraclewasreadingthelocaltableoneblockatatimeandqueryingtheremotetableonceper
localrow.Theexecutionplanforthequeryconfirmedthis:
ExecutionPlan

0SELECTSTATEMENT
10FILTER
21TABLEACCESS(BYINDEXROWID)OF'CUSTOMER_TRANSFER_BATCHES'
32INDEX(FULLSCAN)OF'CUSTOMER_TRANSFER_BATCHES_PK'(UNIQUE)
41REMOTE*PRDWEST
4SERIAL_FROM_REMOTESELECT"CUSTOMER_ID","BATCH_SERIAL_NUMBER"
FROM"CUSTOMER_TRANSFER_BATCHES""B"
WHERE"BATCH_SERIAL_NUMBER"=:1
AND"CUSTOMER_ID"=:2
10FILTER
21TABLEACCESS(BYINDEXROWID)OF'CUSTOMER_TRANSFER_BATCHES'
32INDEX(FULLSCAN)OF'CUSTOMER_TRANSFER_BATCHES_PK'(UNIQUE)
41REMOTE*PRDWEST
4SERIAL_FROM_REMOTESELECT"CUSTOMER_ID","BATCH_SERIAL_NUMBER"
FROM"CUSTOMER_TRANSFER_BATCHES""B"
WHERE"BATCH_SERIAL_NUMBER"=:1
AND"CUSTOMER_ID"=:2
Performing1.5millionnetworkroundtripsandmakingtheremotedatabaseexecute1.5millionqueriesdidnotseemtobethemostefficientwaytogo.Itwouldprobably
bemoreefficientforthelocaldatabasetopullalloftherequireddatafromtheremotedatabaseinoneshotandthenjoinittothelocaltable.Weprobablycouldhave
addedanoptimizerhinttothequerytogetOracletorewritethesubqueryasajoin,butinsteadwechosetorewritethequeryourselvesinawaythatwouldbeeasierfor
Oracle'squeryoptimizertodotherightthing.Therewrittenquerywasasfollows:
SELECTcustomer_id,batch_serial_number,batch_date,load_date,
batch_comment,control_total,control_amount
FROMcustomer_transfer_batches
WHERE(customer_id,batch_serial_number)IN
(
SELECTcustomer_id,batch_serial_number
FROMcustomer_transfer_batches
INTERSECT
SELECTcustomer_id,batch_serial_number
FROMcustomer_transfer_batches@prdwest
)
ORDERBYcustomer_id,batch_serial_number
Werantherevisedqueryandmonitoredv$session_eventandv$sesstatasbefore,withthefollowingresults:
FROMv$session_event
WHEREsid=49
ORDERBYevent
SQL*Netmoredatafromdblink5978133713
directpathread14823311
directpathwrite920357233
logfilesync122
ANDB.sid=49
NAMEVALUE
Thistimethequerycompletedin92seconds.BothelapsedtimeandCPUusagewerereducedbyabout95%.Thewaitsonthedatabaselinkwerereducedfromabout
1053secondstounder14seconds,indicatinglessnetworkoverheadandalotlessdemandbeingplacedontheremotedatabaseserver.Waitsondbfilesequentialread
anddbfilescatteredreadcombinedweresignificantlyshorterthanthewaitsondbfilesequentialreadintheoriginalquerybecauselargenumbersofsingleblockI/Os
werereplacedwithfewermultiblockI/Os.
However,therewereabout38secondsofdirectpathI/Owaits.Aswe'veseen,directpathI/Ousuallyoccursinconnectionwithtemporarysegments.Oraclehadtodo
quiteabitofsortingtoimplementtheINTERSECToperationaddedtotherewrittenquery,sothedirectpathI/Owaitswerenotunexpected.
Theperformancegainsachievedbythequeryrewriteweresignificant,andweweretemptedtoendthetuningexerciserighthere.However,thecompanyindicatedthat
thisreportwascriticaltotheiroperationandtheywantedtomakeitrunevenfasterifpossible.So,ournextstepwastospeedupthesortoperationandtherebyreduce
thedirectpathI/Owaits.
Lookingatv$parameter,wesawthatthesort_area_sizewassetto1Mb.AlargersortareawouldallowOracletoperformmoreofthesortinmemoryandlessondisk,
thusreducingthedirectpathI/O.Toseetheeffectofalargersortarea,werepeatedthetimingexercise,firstsettingsort_area_sizeto100Mbdynamicallyinthe
SQL*Plussessionwherethereportquerywasrun.Thequerycompletedin55secondsthistime,andstatisticscollectedfromv$session_eventandv$sesstatwereas
follows:
FROMv$session_event
WHEREsid=46
ORDERBYevent
SQL*Netmoredatafromdblink6050137826
logfilesync133
ANDB.sid=46
NAMEVALUE
logfilesync133
ANDB.sid=46
NAMEVALUE
YoucanseethatincreasingthesortareasizedidnotreducetheCPUtime,databaselinkwaittime,ordbfilescattered/sequentialreadtimes.However,thedirectpath
I/Owaitsdisappearedcompletely.Thisindicatesthatallsortingwasperformedinmemory.Ofcourse,itmaynotbefeasibletokeepthesortentirelyinmemoryasthe
CUSTOMER_TRANSFER_BATCHEStablegrows.However,alargersortareawillstillspeedupthesortandoverallqueryspeedeveniftheentiresortcannotbe
performedinmemory.
Thisexampledemonstratedthepracticeofiterativetuning:Curingonebottleneckoftencreatesanotherorrevealsanotherperformanceissuethatwasmaskedbythe
largerissue.Boostingperformanceofcomplexsystemsisusuallyaniterativeprocess.
Example#6:SlowBatchProcessing
Theoperationsgroupnoticedthatitsovernightbatchprocessingwastakingsignificantlylongerafteranadditionaldatafeedprogramwasaddedtothenightlyrun
schedule.Thedecisionwasmadetoaddmoreprocessorstothedatabaseservertospeedthingsup.Unfortunately,thisdidnotdothetricktheprogramsranataboutthe
sameoverallspeedasbefore.
Thev$system_eventviewcancomeinhandyinsituationslikethisinordertotakeaquickglanceattheoverallsystem.Bymakingacopyoftheview'scontentsattwo
separatepointsintime,youcangetahighlevelsummaryofwhatprocesseswerewaitingforduringthetimeinterval.HereisanextremelybasicSQL*Plusscriptthat
displaysthewaiteventactivityfortheinstanceovera30secondtimeinterval:
CREATETABLEprevious_events
AS
SELECTSYSDATEtimestamp,v$system_event.*
FROMv$system_event
EXECUTEdbms_lock.sleep(30)
SELECTA.event,
A.total_waitsNVL(B.total_waits,0)total_waits,
A.time_waitedNVL(B.time_waited,0)time_waited
FROMv$system_eventA,previous_eventsB
WHEREB.event(+)=A.event
ORDERBYA.event
Ofcourse,youcoulduseStatspack,utlbstat,oranynumberofothertoolsouttheretocapturethisinformationaswell.Runningtheabovescriptwhilethebatchprograms
wererunninggeneratedthefollowingoutput:
bufferbusywaits5326
directpathread21119
enqueue375646
fileidentify20
fileopen370
freebufferwaits11711
latchfree5244
logbufferspace28
logfilesinglewrite26
writecompletewaits6517
Theidleeventshavebeenremovedforclarity.Whatstandsouthereistheenqueuewaitevent.Duringthe30secondtimeinterval,processescollectivelyspentover56
secondswaitingforenqueues,orlocksthatyoucanseeinv$lock.Themostcommontypesoflockstoshowupinanenqueuewaitarethetablelevellock(TM)andthe
transactionenqueueorrowlevellock(TX).
Theparametersfortheenqueuewaiteventprovideinformationaboutthetypeandlocationofthelockwaitedupon,buttheinformationisfairlyobscure.Thefirsttwo
bytesofP1RAWaretheASCIIcodesofthelocktype,andthelasttwobytesaretherequestedlockmode.P2andP3correspondtotheID1andID2columnsinthe
v$lockview.
Inordertogetasenseofwherethelockingcontentionwascomingfrom,IusedtheoradebugfacilityinSQL*Plustoactivatesystemevent10046foroneofthebatch
programsasitwasrunning.Sincetracingincursalotofoverhead,thissignificantlysloweddownbatchprocessing.Butthiswasonlydoneonce,andwiththegreatergood
ofeliminatingthecontentionproblemsinmind.
Toactivateevent10046,Ilookedatv$sessionandv$processtofindtheOraclePIDofthededicatedserverprocessrunningoneofthebatchprograms.ThenIranthe
followingcommandsinSQL*Plus:
SQL>oradebugsetorapid13
Unixprocesspid:19751,image:oracle@dbserver.acme.com(TNSV1V3)
SQL>oradebugsession_event10046tracenamecontextforever,level8
Statementprocessed.
SQL>
Thiscausedthededicatedserverprocesstobeginwritingatracefileincludingwaiteventinformation.Anexcerptfromthetracefileisasfollows:
SQL>oradebugsetorapid13
Unixprocesspid:19751,image:oracle@dbserver.acme.com(TNSV1V3)
SQL>oradebugsession_event10046tracenamecontextforever,level8
Statementprocessed.
SQL>
Thiscausedthededicatedserverprocesstobeginwritingatracefileincludingwaiteventinformation.Anexcerptfromthetracefileisasfollows:
XCTENDrlbk=0,rd_only=0
WAIT#1:nam='writecompletewaits'ela=11p1=3p2=2p3=0
=====================
PARSINGINCURSOR#4len=65dep=1uid=502oct=6lid=502
tim=2313021001hv=417623354ad='55855844'
UPDATECUSTOMER_CALLSSETATTR_3=:b1WHERECUSTOMER_CALL_ID=:b2
ENDOFSTMT
WAIT#5:nam='enqueue'ela=307p1=1415053318p2=196705p3=6744
Wecanseetheenqueuewaiteventsprettyclearly.UsingthemagicaldecoderringontheP1values,wecanseethatOraclewaswaitingonTXlocksrequestedinmode6
(exclusivemode).(1415053318is54580006inhexadecimal.The54correspondstoa"T",the58toan"X",and0006isexclusivemode.Lookupthev$lockviewinthe
ServerReferenceGuideforalistoflocktypesandmodes.)
Whatisnotclearfromtheexcerptaboveiswhattableandrowtheprocesswastryingtolockwhenithadtowait.Ifwewerelookingatthedatabasewhilethelock
contentionwashappening,wecouldlookatthev$locked_objectview,ortheROW_WAIT_OBJ#andROW_WAIT_ROW#columnsofv$session.ButsinceallIhad
toworkwithwasthecontentsofthetracefile,Icouldseethatthewaitsoccurredwhileprocessingcursornumberfive.SearchingbackwardsthroughthetracefileIfound
theexactSQLstatementthatcorrespondedtocursornumberfiveatthetimeofthelockcontention.
Thisturnedouttobeenoughinformationtodiagnosethecontentionproblemanddevelopaworkaround.Byusingautonomoustransactions,wewereabletomodifythe
batchprogramsthatwerecontendingwitheachothersothattheyheldlocksoncriticalpathrowsforextremelyshortperiodsoftime.Adjustingtheschedulesofthe
programsinthenightlyrunschedulealsohelped.
Example#7:ASlowClient/ServerApplication
Aclient/serverapplicationwastakingseveralsecondstobringupacertainscreen.Theuserscomplainedconstantly,andthedeveloperswereconfoundedbecausethe
delaywasoccurringbeforetheuserhadanychancetokickoffaquery.Theonlythinghappeningintheformonstartupwassomefetchingofbasicreferencedata.Allof
theSQLhadbeentunedandwasknowntoworkveryquickly.Beingfrequentlyaccessedreferencedata,thedatablockswerelikelytobeinthebuffercacheeliminating
slowI/Oasapossiblebottleneck.
Lookingatv$session_eventforauserwhohadjustloggedontotheapplicationbutnotyetbegunanyactivitiesshowedalmost20,000waitsontheSQL*Netmessage
fromclientevent,totalingover300secondsofwaiting.Thisisanidleevent,whichwewouldtypicallyignore,butsomethingseemedwrongandwarrantedfurther
investigation.
AprocesswaitsontheSQL*Netmessagefromclienteventwhenithascompletedarequestfromauserandnowneedstowaitforthenextrequest.Youtypicallysee
highwaittimesonthiseventinfrontendapplicationsthatspendalotoftimewaitingforausertosubmitaqueryorinitiatesomeotheractionthatrequiresdatabase
activity.Sothe300secondsofwaittimeheredidnotseemunusual,butthehighnumberofwaitsdidnotseemright.
Theapplicationcodewasmodifiedinatestenvironmentsothatitwoulddisabletimedstatisticsforthesessionafterallinitializationwascomplete.Thiscausedthetime
informationinv$session_eventtoreflectwaittimesduringtheformstartupphaseonlythetimespentwaitingforausertoinitiateanactionwouldbeexcludedbecause
timedstatisticswouldbedisabledatthatpoint.
Runningthemodifiedapplicationyieldedthefollowinginformationinv$session_event:
Afetchofafewpertinentsessionstatisticsfromv$sesstatshowedthefollowing:
NAMEVALUE
sessionlogicalreads9295
physicalreads0
Thedatasuggestedthatwhentheapplicationstartedup,itwasspendingover10secondsperformingmorethan18,000networkroundtrips.Thedatabaseserverwas
usingalittleunderonesecondofCPUtimetoperform9,295logicalreadsofdatablocksthatwerealreadyinthebuffercache.
Acloserinspectionofthereferencedatabeingretrievedbytheapplicationrevealed9,245rowsofreferencedata.Theapplicationcodewasfetchingthedatafrom
cursorsonerowatatime.Thisresultedintwonetworkroundtripsandonelogicalreadforeachrowfetched.
TheapplicationwasmodifiedtouseOracle'sarrayprocessinginterfaceinordertofetch100rowsatatime.Thisspeduptheapplicationdramaticallybyreducing
networkroundtrips.AnadditionalbenefitwasthatCPUusageonthedatabaseserverwasreducedbecauseonlyaboutonelogicalreadperdatablockwasrequired,
insteadofonelogicalreadperrowretrieved.
Whenfetching100rowsatatime,thewaitsandsessionstatisticslookedlikethis:
cursorsonerowatatime.Thisresultedintwonetworkroundtripsandonelogicalreadforeachrowfetched.
TheapplicationwasmodifiedtouseOracle'sarrayprocessinginterfaceinordertofetch100rowsatatime.Thisspeduptheapplicationdramaticallybyreducing
networkroundtrips.AnadditionalbenefitwasthatCPUusageonthedatabaseserverwasreducedbecauseonlyaboutonelogicalreadperdatablockwasrequired,
insteadofonelogicalreadperrowretrieved.
Whenfetching100rowsatatime,thewaitsandsessionstatisticslookedlikethis:
NAMEVALUE
sessionlogicalreads135
physicalreads0
Byfetchingrowsupto100atatime,wewereabletodramaticallyimproveresponsetimebyreducingnetworkroundtripsandreduceCPUusageonthedatabase
serveratthesametime.
Conclusion
BecausewaiteventstouchallareasofOracleserveroperation,thewaiteventinterfacecanprovideinsightsintojustaboutanykindofperformanceproblemyoumight
experience.Othermetrics,suchascachehitratios,cangiveyouvaluableinformation,butitisthewaiteventinterfacethattieseverythingtogether.Intheexamples
discussedhere,waiteventdatadirectedustoaswiftdiagnosisofeachproblem.
Thewaiteventinterface,withitsfourdynamicperformanceviewsandtracingfacility,enablesyoutoseehowanOracleprocessisspendingitstimewhenitisnotusingthe
CPU.Whilewaiteventinformationmaynotgiveyoueverythingyoumightneedtocureeveryperformanceproblem,itdoesgiveyouaccesstoawealthofinformationthat
canguideyouonthepathtobettersystemperformance.
ThebasicconceptsbehindthewaiteventinterfacehavenotchangedsinceOracle7.ButeachreleaseofOraclebringsafewmoreenhancementsthatmakethewaitevent
interfaceeasiertouseormoreinformative.InOracle9i,TKPROFhasbeenenhancedtoparsewaiteventinformationfromtracefilesandformatitinausefulreport.
Also,sometiminginformationistrackedinmicrosecondsformoreresolution,andtheDBMS_SUPPORTpackagehasbeenincludedforeasiertraceactivation.
However,thesearejustbellsandwhistles.
Moreexampleson10g
Whatresourceiscurrentlyinhighdemand?
Whilewehavealwaysbeenabletolookatthesystemoracurrentlyrunningsessionasawholeandseetheresourcesthatareinmostusethroughthe
V$SYSTEM_EVENTandV$SESSTATviews,wereallyhavenoteverhadagoodwaytodetermineforacurrenttimeframewhattheresourcesbeingusedwere.This
querywillgiveyouforthelast30minutesthoseresourcesthatareinhighdemandonyoursystem.
selectactive_session_history.event,
sum(active_session_history.wait_time+active_session_history.time_waited)ttl_wait_time
fromv$active_session_historyactive_session_history
whereactive_session_history.sample_timebetweensysdate60/2880andsysdate
groupbyactive_session_history.event
orderby2
WaitEventTTL_WAIT_TIME
nullevent5
ksfd:asyncdiskIO476
directpathread30025
latch:cachebufferschains33258
logfilesync612660
controlfilesinglewrite1127626
readbyothersession2024242
dbfileparallelwrite2278618
enq:CFcontention4108238
rdbmsipcreply4283877
classslavewait5123780
rdbmsipcmessage11899157
jobqslavewait14732974
enq:HWcontention100841479
bufferbusywaits114070065
Whatuseriswaitingthemost?
enq:HWcontention100841479
bufferbusywaits114070065
Whatuseriswaitingthemost?
Again,thepowerinthisSQLqueryisthatwearenowabletodeterminewhatuserisconsumingthemostresourceatapointintime,independentofthetotalresources
thattheuserhasused.Youcannow,withthisquery,answerthequestionofwhoiswaitingthemostforresourcesatapointintime.Ifausercallsyouuponthephone
andsaystheyareexperiencingdelays,youcanusethisquerytoverifythattheyareactuallywaitinginthedatabaseforaresultsetforagiventimeperiod.ThisSQLis
writtenfora30minuteintervalfromcurrentsystemtimesoyoumayneedtochange.
selectsesion.sid,sesion.username,
fromv$active_session_historyactive_session_history,v$sessionsesion
andactive_session_history.session_id=sesion.sid
groupbysesion.sid,sesion.username
orderby3
SIDUserTTL_WAIT_TIME
135SCOTT91167481
149SCOTT107409491
153SCOTT110796799
WhatSQLiscurrentlyusingthemostresources?
ThisquerywillgetyoustartedintheproperdirectionofzeroinginonwhatSQLstatementisconsumingthemostresourcewaittimesonyoursystem.Nolongerdoyou
havetogototheV$SQLAREAandtrytopickoutthetop10or20SQLhogsonyoursystembydiskreadsorexecutions.NowyoureallyknowwhatSQLstatements
areconsumingresourcesandwaitingthemost.ThesearetheSQLthatyoureallyneedtotunebecausethefactthataSQLstatementperforms20,000readsdoesnot
meanthatitisabottleneckinyoursystem.
selectactive_session_history.user_id,dba_users.username,sqlarea.sql_text,
fromv$active_session_historyactive_session_history,v$sqlareasqlarea,dba_users
andactive_session_history.sql_id=sqlarea.sql_id
andactive_session_history.user_id=dba_users.user_id
groupbyactive_session_history.user_id,sqlarea.sql_text,dba_users.username
orderby4
USER_IDUserSQL_TEXTTTL_WAIT_TIME
57SCOTTinsertintosys.sourcetable(select*fromsys.source$)304169752
Whatobjectiscurrentlycausingthehighestresourcewaits?
Thisisagreatquery.NowwecanactuallyseewhichobjectsaSQLstatementishitting.Moreover,ifyoutakeafurtherlookattheV$ACTIVE_SESSION_HISTORY
viewyouwillseethatyoucantailorthisquerytoreportontheactualblocksthatarebeingaccessedwithintheobjectsfortheSQLstatement.Thisisgreathelpin
determiningifyoumayneedtoreorgyourobjectorredistributetoreducethecontentiononthatobject.
selectdba_objects.object_name,dba_objects.object_type,active_session_history.event,
fromv$active_session_historyactive_session_history,dba_objects
andactive_session_history.current_obj#=dba_objects.object_id
groupbydba_objects.object_name,dba_objects.object_type,active_session_history.event
orderby4
OBJECT_NAMEOBJECT_TYPEWaitEventTTL_WAIT_TIME
SOURCE$TABLEksfd:asyncdiskIO23
SOURCETABLETABLEksfd:asyncdiskIO453
SOURCETABLETABLElatch:cachebufferschains33258
SOURCETABLETABLEdbfilescatteredread82593
SOURCETABLETABLEdbfilesequentialread111202
SOURCETABLETABLEcontrolfileparallelwrite137237
SOURCETABLETABLEreadbyothersession165501
SOURCETABLETABLElogfilesync612660
SOURCE$TABLElogbufferspace932308
SOURCETABLETABLEcontrolfilesequentialread1428575
SOURCE$TABLElogfileswitchcompletion1856281
SOURCE$TABLEreadbyothersession1858741
SOURCE$TABLEdbfilescatteredread4275060
SOURCETABLETABLErdbmsipcreply4283877
SOURCETABLETABLElogbufferspace14152000
SOURCE$TABLEenq:HWcontention25483656
SOURCETABLETABLElogfileswitchcompletion59228080
SOURCETABLETABLEenq:HWcontention75357823
SOURCETABLETABLEbufferbusywaits114055403
SOURCETABLETABLElogbufferspace14152000
SOURCE$TABLEenq:HWcontention25483656
SOURCETABLETABLElogfileswitchcompletion59228080
SOURCETABLETABLEenq:HWcontention75357823
SOURCETABLETABLEbufferbusywaits114055403

Interpreting Wait Events

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Interpreting Wait Events

Transféré par

Droits d'auteur :

Formats disponibles

InterpretingWaitEvents

The V$SESSION_WAIT View

Vous aimerez peut-être aussi