Académique Documents
Professionnel Documents
Culture Documents
tm tm
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Chapter3:ActiveDirectoryTroubleshooting:ToolsandPractices................................................32 NarrowingDowntheProblemDomain..................................................................................................32 SeansSevenPrinciplesforBetterTroubleshooting....................................................................33 AFlowchartforADTroubleshooting..................................................................................................34 EasyStuff:NetworkIssues...........................................................................................................................35 NameResolutionIssues.................................................................................................................................36 LogSpelunking..................................................................................................................................................37 ADServiceIssues..............................................................................................................................................37 ClientDomainControllerIssues................................................................................................................39 ReplicationIssues.............................................................................................................................................40 ADDatabaseIssues..........................................................................................................................................42 GroupPolicyIssues..........................................................................................................................................43 KerberosIssues.................................................................................................................................................45 ComingUpNext.................................................................................................................................................46 DownloadAdditionaleBooksfromRealtimeNexus!........................................................................46
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Copyright Statement
2011 Realtime Publishers. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers (the Materials) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers or its web site sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at info@realtimepublishers.com.
ii
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Chapter3:ActiveDirectory Troubleshooting:ToolsandPractices
Forthemostpart,inmostorganizations,ActiveDirectory(AD)justworks.Overthepast 10yearsorso,MicrosofthasimprovedbothADsperformanceanditsstability,tothepoint wherefeworganizationswithawelldesignedADinfrastructureexperiencedaytoday issues.Thatsaid,whenthingsdogowrong,itcanbeprettyscarybecausealotofusdont havedaytodayexperienceintroubleshootingAD.Thegoalofthischapteristoprovidea structuredapproachtotroubleshootingtohelpyouputoutthosefiresfaster. Forthischapter,IllbedrawingalotonthewisdomandexperienceofSeanDeuby,afellow MicrosoftMostValuableProfessionalawardrecipientandarealADtroubleshootingguru. Youmightenjoyreadinghisinfrequentlyupdatedblogat http://www.windowsitpro.com/blogs/ActiveDirectoryTroubleshootingTipsandTricks.aspx. Althoughhedoesntpostalot,whathedoespostisworththetrip.
NarrowingDowntheProblemDomain
HowdoyoufindawolfinSiberia?ItsaquestionIandothershaveusedtokickoffany discussionontroubleshooting.Siberiais,ofcourse,ahugeplace,andfindingaparticular anythingletaloneawolfistough.Theanswertotheriddleisamaximfor troubleshooting: Buildawolfprooffencedownthecenter,andthenlookononesideofthefence. Troubleshootingconsistsmainlyoftests,designedtoseeifaparticularrootcauseis responsibleforyourproblems.Theanswertotheriddleprovidesimportantguidance: Makesureyourtests(thatis,thewolfprooffence)candefinitivelyeliminateoneormore rootcauses(thatis,onewholehalfofSiberia).Dontbotherconductingteststhatcant eliminatearootcause.Forexample,ifausercantlogin,youmightfirstchecktheir physicalnetworkconnection.Doingsodefinitivelyeliminatesapotentialproblem (networkconnectivity)sothatyoucanmoveontootherpossiblerootcauses.Ofcourse, checkingconnectivityonlyeliminatesoneortwopossiblerootcauses;abetterfirsttest wouldeliminateawholehostofthem.Forexample,checkingtoseewhetheradifferent usercouldloginmighteliminatethevastmajorityofpotentialinfrastructureproblems, makingthatabetterwolfprooffence.
32
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
SeansSevenPrinciplesforBetterTroubleshooting
HereswhereIllrepeatexcellentadviceSeanDeubyonceoffered.Followtheseseven principles(whichIllexplainthroughthefilterofmyownexperience)andyoullbeafaster, bettertroubleshooterinanycircumstance. 1. BeLogical.Payattentiontohowyoureattemptingtosolvetheproblem.Beforeyou doanything,askyourself,WhatoutcomedoIexpectfromthis?IfIgetthat outcome,whatdoesitmean?IfIdontgettheexpectedoutcome,whatdoesthat mean?Dontdoanythingunlessyouknowwhy,andunlessyoucanstatewhatthe followupstepwouldbe. 2. RememberOccamsRazor.Simplyput,thesimplestsolutionisoftenthecorrectone. Dontstartrebootingdomaincontrollersuntilyouvecheckedthattheuseristrying thecorrectpassword. 3. WhatChanged?Ifeverythingwasworkingfineanhourago,whatsdifferent?Thisis wherechangeauditingtoolscancomeinhandy.AlthoughIdontspecifically recommendit,IveusedQuestsChangeAuditorforActiveDirectoryinthepast becauseitkeepsaverydetailed,realtimelogofchanges,anditsbeenabighelpin solvingsometrickyissues.Whateverchangedrecentlyisaverylikelycandidatefor beingtherootcauseofyourcurrentwoes. 4. DontMakeAssumptions.Itseasytomakeassumptions,butstickingwithan orderlyeliminationofpossiblecauseswillgetyoutotherootcauseoftheproblem moreconsistently.Forexample,dontassumethatjustbecauseoneusercanlogon thateverythingsokaywiththeinfrastructure;theproblemusermightbehittinga differentdomaincontroller,forexample. 5. ChangeOneThingataTime,andRetest.Youwontgetanywherewithfivepeople attackingtheproblem,eachonechangingthingsastheygo.Youalsowontget anywhereifyourechangingmultiplethingsatonce.Ifthebossistearinghishairout togetthingsfixed,remindhimthatyouhavejustasmuchcapabilitytofurtherbreak thingsifyourenotmethodical. 6. Trust,butVerify,Evidence.Sometimesaninaccurateproblemdescriptioncanget yougoinginthewrongdirectionsoverifyeverything(thisgoesbacktonot makingassumptions,too).Icantlogin!ausercriesoverthephone.Loginto what?youshouldask,beforedivingintoADproblems.Maybetheuseristalking abouttheirGmailaccount. 7. DocumentEverythingYouTry.Especiallyfortoughissues,documentingeverything youtrywillhelpkeepyoufromrepeatingsteps,andwillhelpyoueliminatepossible causesmoreeasily.Itsalsocrucialintheinevitablepostmortem,whereyouand yourcolleagueswilldiscusshowtokeepthisfromhappeningagain,orhowtosolve itmorequicklythenexttime.
33
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
AFlowchartforADTroubleshooting
SeanhasfurtherhelpedbycomingupwithanADtroubleshootingflowchart,whichIll reprintinpiecesthroughoutthischapter.YoushouldcheckSeansblogorWebsite(which isshownatthebottomofthechartpages)forthelatestrevisionoftheflowchart.Seans blogalsooffersafullsizedPDFversion,whichIkeeprightnearmydeskatalltimes.The flowchartstartswiththatisshowninFigure3.1,whichisthecorestartingpointthatgets youofftothedifferentsectionsofthechart.
Figure3.1:StartingpointinADtroubleshooting. Note IstronglyrecommendthatyouheadovertoSeansblogorWebsiteto downloadthePDFversionofthisflowchartforyourself.Youmayfindalater version,whichisgreatitllstillstartoffinbasicallythissameway. Startintheupperleft,withCablepluggedintonetwork?andworkdownfromthere.The basicsthewireportionshouldbethingsyoucanquicklyeliminate,butdonteliminate themwithoutactuallytestingthem.Youmight,forexample,attempttopingaknowngood IPaddressonthenetwork(usinganIPaddresspreventspotentialDNSissuesfrom becominginvolvedatthispoint).Ifthatdoesntwork,youvegotahardwareissueofsome kindtosolve.
34
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
EasyStuff:NetworkIssues
Apingdoes,ofcourse,starttoencroachontheNetworksectionoftheflowchart.Stick withIPaddressestothispointbecausewerenotreadytoinvolveDNSyet.Ifthepingisnt successful,andyouveverifiedthenetworkadapter,cabling,router,andother infrastructurehardware,yourereadytomoveontoFigure3.2,whichistheNetwork Issuesportionoftheflowchart.
35
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
NameResolutionIssues
IfapingtoadifferentintranetsubnetworkedbyIPaddress,itstimetostartpingingby computernametotestnameresolution.Watchthepingcommandsoutputtoseeifit resolvesaserversnametothecorrectIPaddress.Ideally,usethenameofadomain controllerortwobecauseweretestingADproblems.Ifpingdoesntresolvecorrectly,or cantresolveatall,yourereadytomoveintothenameresolutionissues. TheClientDCNameResolutionIssuesflowchartisdesignedforwhenyoure troubleshootingconnectivityfromaclienttoadomaincontroller;ifyouretroubleshooting problemsonaserver,youllskipthisstepandmoveoninthecoreflowchart(Figure3.1).If youareonaclient,theflowchartthatFigure3.3showswillcomeintoplay.
36
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
LogSpelunking
Oncenameresolutionisresolved,orifitisnttheproblem,youhaveabitofcheckingtodo beforeyoumoveon.Specifically,youregoingtohavetolookintheSystemandApplication eventlogsonthedomaincontrollersintheclientslocalsite(orwhateverdomain controlleryourehavingaproblemwith,ifitsjustaspecificone).Ifyoufindanyerrors, youllhavetoresolvethemandtheymaybemorespecifictoWindowsthantoAD.Dont ignoreanything.Infact,thatdontignoreanythingisahugereasonIhatedomain controllersthatdoanythingotherthanrunAD,andperhapsDNSandDHCP.Ioncehada domaincontrollerthatwashavingrealissuestalkingtothenetwork.Therewereabunch ofIISrelatederrorsinthelog,butIignoredthosewhatdoesIIShavetodowith networkingorAD,afterall?Ishouldnthavemadeassumptions:ItturnedoutthatIISwas moreorlessjammingupthenetworkpipe.ShuttingitdownsolvedtheproblemforAD. LogExploring Havingtodigthroughtheeventlogsonmorethanonedomaincontroller heck,evendoingitononeserveristimeconsumingandfrustrating.Thisis wheresomekindoflogconsolidationandanalysistoolcanhelp tremendously.Getallyourlogsintooneplace,andhavesoftwarethatcan prefiltertheevententriestojustthosethatneedyourattention.Software likeMicrosoftSystemCenterOperationsManagercanalsohelpbecauseone ofitsjobsistoscaneventlogsandcalltoyourattentionanyeventsthat requireit. Ifyoudontseeanyerrorsspecifictothedomaincontrollerorcontrollers,youmoveon. Yourelookingfirstforerrorsrelatedtotrusts,andifyoufindany,youllneedtoresolve them.Ifyoudidfinderrorsrelatedtothedomaincontrollerorcontrollers,andyou correctedthembutthatdidntsolvetheproblem,youremovingontoADserviceissues.
ADServiceIssues
Figure3.4containstheADserviceissueportionofthetroubleshootingflowchart.Here, wevemovedintothecomplexpartofADtroubleshooting.First,ofcourse,lookintheevent logforerrorsorwarnings.Dontignoresomethingjustbecauseyoudontunderstandit; youregoingtohavetoamassknowledgeaboutobscureADeventssothatyouknowwhich onescanbesafelyignoredinagivensituation.
37
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
38
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
ClientDomainControllerIssues
Assumingyouresolvedanyclientnameresolutionissuesearlier,ifyourestillhaving problemswiththeclientcommunicatingwiththedomaincontroller,youllmovetothe ClientDCTroubleshootingchart,whichFigure3.5shows.
39
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Thisisalsothepointwhereyouregoingtowantachartofyournetworksothatyoucan confirmwhichdomaincontrollersshouldbeinwhichsites.Youllwantthatcharttoalso listeachsubnetthatbelongstoeachsite.Youhavetoverifythatrealitymatchesthe desiredconfiguration,anddontskipanysteps.Itseemsobvioustoassumethataclient wasgivenaproperaddressbyDHCPandisthereforeinthesamesite;dontevermakethat assumption.Ioncehadaclientthatseemedtobeworkingjustfinebutwasinfacthanging ontoanoutdatedIPaddress,makingtheclientbelieveitwasinadifferentsite.Theway ourLANwasconfigured,theincorrectIPaddresswasstillabletofunction(weusedalotof VLANstuffandIPaddressinggotincrediblyconfusing),buttheclientdidntseeitselfas beinginthepropersitesoitwouldnttalktotherightdomaincontroller.
ReplicationIssues
Iftheflowcharthasgottenyoutothispoint,weredealingwiththepageFigure3.6shows.
Figure3.6:Replicationissues.
40
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
TroubleshootingADreplicationisoftenperceivedasthemostdifficultandmysterious thingyoucandowithAD.Itslikemagic:eitherthetrickworksoritdoesnt,andyoull neverknowwhyeitherway.Iseemorepeoplestrugglewithreplicationissuesthanwith anythingelse,yetreplicationistheonethingthatcancomeupmostfrequently,duein largeparttoitsheavyrelianceonproperconfigurationandtheunderlyingnetwork infrastructure. Seanproposesfourreasons,whichIagreewith,thatmakereplicationtroubleshooting difficultforpeople.Inmywords,theyare: Theyvenotbeentrainedinaformaltroubleshootingmethodology.Moreadmins thanyoumightbelievetendtotroubleshootbyrote,meaningtheytrythesame thingsinthesameordereverytimewhichisgoodwithoutreallyunderstanding whattheyretestingwhichisbad. Theydontapproachtheproblemlogically.Thinkaboutwhatshappening.Doesit makesensetotestnameresolutionbetweentwodomaincontrollerswhenother communicationsbetweenthemseemunhindered? Theydontunderstandhowreplicationworks.This,Ithink,isthebiggestproblem.If youdontunderstandwhatshappeningunderthehood,youhavenomeansof isolatingindividualprocessesorcomponentstotestthem.Ifyoucantdothat,you cantfindtheproblem. Theydontunderstandwhatthetoolsdo.Thisisalsoabigproblembecauseifyou dontreallyknowwhatsbeingtested,youdontknowhowtoeliminatepotential rootcausesfromyourlistofsuspects.
Ultimately,youcantjustruntoolsintheordersomeoneelsehasprescribed.Seanproposes fourstepstohelpproceed;Iprefertolimitthelisttothree: 1. Formahypothesis.Whatdoyouthinktheproblemis?Afirewallrule?IPaddressing problem?DNSproblem?Applywhateverexperienceyouhavetojustpickaproblem thatseemslikely. 2. Predictwhatwillhappen.Inotherwords,ifyouthinkexternalcommunications mightbefailing,youmightpredictthatinternalcommunicationswillbefine. 3. Testyourprediction.Useatooltoseeifyoureright.Ifyouare,youvenarrowedthe problemdomain.Ifyourenot,youformanewhypothesis. Ifyourememberscienceclassfromelementaryschool,youmightrecognizethisasthe scientificmethod,anditworksaswellfortroubleshootingasitdoesforanyscience. Replicationtroubleshootingcannotproceedunlessyouvealreadyresolvednetworking, localonlyissues,andotherproblemsthatprecedethisstepinthecoreflowchart.Once youvedonethat,youllfindyourselfquicklylookingforOSrelatedissuesintheeventlog, thenmoveontotheDcdiagtooltheflowchartprovidesaURLwithadescriptionofthe teststorun.
41
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
ADDatabaseIssues
Next,youllmoveintotroubleshootingtheADdatabase,whichiscoveredintheflowchart thatFigure3.7shows.
Figure3.7:ADdatabasetroubleshooting.
42
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Here,youllprobablybetakingadomaincontrollerofflinesothatyoucanrebootinto DirectoryServicesRestoreMode(DSRM)makesureyouknowtheDSRMpasswordfor whateverdomaincontrolleryouredealingwith.YoulluseNTDSUTILtocheckthefile integrityoftheADdatabaseitselfbecause,atthispoint,werestartingtosuspect corruptionofsomekind.Ifyoufindit,youllbedoingadatabaserestore.Ifyoudonthavea backup,youreprobablylookingatdemotingandrepromotingthedomaincontroller,if notrebuildingtheserveentirely.Sorry. Again,thisiswherethirdpartytoolscanhelp.YoumayhavethoughtthattheADRecycle BinfeatureofWindowsServer2008R2wasagreatfeature,butitisntdesignedtodeal withatotaldatabasefailure.Thirdpartyrecoverytools(whichareavailablefrom numerousvendors)cangetyououtofajamhere.Makesureyourenotusingtooolda backup;ideally,domaincontrollerbackupsshouldntbeolderthanafewdays.Older backupswillrequirethedomaincontrollertoperformalotmorereplicationwhenitcomes backonline,andaveryoldbackupcanreintroducetombstoned(deleted)objectstothe domain,whichwouldbeaBadThing.
GroupPolicyIssues
Ifyouvemadeitthisfar,ADsmostcomplexcomponentsareworking,andyoureonto troubleshootingoneoftheeasierelements.First,recognizethattherearetwobroad classesofproblemwithGroupPolicy:nosettingsfromaGroupPolicyobjectarebeing appliedorthewrongsettingsarebeingapplied.Thischapter,asshownintheflowchartin Figure3.8,isconcernedonlywiththeformer.Ifyouregettingsettingsbutnottheright ones,youneedtodiveintotheGPOs,ResultantSetofPolicy(RSoP),andothertoolsto discoverwherethewrongsettingsarebeingdefined.
43
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Figure3.8:GroupPolicytroubleshooting. TroubleshootingGPOsisprettymuchaboutverifyingtheirconfiguration.Ifauserisnt gettingaspecificGPO,theproblemwillbeduetoreplication,inheritance,asynchronous processing(whichmeanstheyregettingtheGPO,justnotasquicklyasyouexpected),and soforth.GroupPolicyiscomplicated,andknowingallthelittletricksandgotchasiskeyto solvingproblems.IrecommendbuyingJeremyMoskowitzlatestbookonthesubject;hes prettymuchtheindustryexpertonGroupPolicyandhisbookscomeswithgreat explanationsandflowchartstohelpyoutroubleshoottheseproblems. UnravelingwhatschangedisalsotheeasiestwaytofixGPOproblems.Unfortunately, mosttoolsthattrackADconfigurationchangesdonttouchGPOsbecauseGPOsarent storedinADitself.TherearetoolsthatcanplaceGPOsunderversioncontrol,andcanhelp trackthechangesrelatedtoGPOsthatdoliveinAD(suchaswheretheGPOsarelinked). Quest,NetWrix,BlackbirdGroup,andNetIQalloffervarioussolutionsinthesespaces.
44
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
KerberosIssues
Finally,thelastareawellcoverisKerberos.Figure3.9showsthelastpageintheflowchart.
Figure3.9:Kerberosissues. Here,youllneedtoinstallresourcekittools,preferablyKerbtray.exe,sothatyoucangeta peekinsideKerberos.YoullalsoneedastrongunderstandingofhowKerberosworks. Heresabriefbreakdown: Whenyoulogon,yougetaTicketGrantingTicket(TGT)fromyourauthenticating domaincontroller.ThisenablesyoutogetKerberostickets,whichprovideaccessto aspecificserversresources.Eachserveryouaccesswillrequireyoutohaveaticket forthatserver.Soeachtimeyouaccessanewservereveryday,youllhavetofirst contactadomaincontrollertogetthatticket. Ticketvalidityiscontrolledbytimestamps.Everymachineinthedomainneedsto haveroughlythesameideaofwhattimeitis,whichiswhyWindowsautomatically synchronizestimewithinthedomain.Askewofabout5minutesisallowedby default.
45
TheDefinitiveGuidetoActiveDirectoryTroubleshooting,Auditing,andBestPractices2011Edition
Thereareafewotheruncommonissuesalsocoveredbytheflowchart.
ComingUpNext
Withthistroubleshootingguidanceunderyourbelt,itstimetomoveontoournextAD topic:security.Iveseenanincredibleamountofconfusionandmisinformationwithregard toADsecurityoverthepastfewyears,soweregoingtostartbysteppingbacktobasics andlookingatADssecurityarchitecture.WellspelloutADsrealroleinsecuringyour organizationsresources,andlookatreasonsyoumightwanttorethinkyourcurrent securitydesign.WellevenpeekatDNSsecurity.ItsallcomingupinChapter4.
DownloadAdditionaleBooksfromRealtimeNexus!
RealtimeNexusTheDigitalLibraryprovidesworldclassexpertresourcesthatIT professionalsdependontolearnaboutthenewesttechnologies.IfyoufoundthiseBookto beinformative,weencourageyoutodownloadmoreofourindustryleadingtechnology eBooksandvideoguidesatRealtimeNexus.Pleasevisit http://nexus.realtimepublishers.com.
46