Zenoss Core4 Event Management Paper

Event Management for Zenoss Core 4
January 2013 Jane Curry Skills 1st Ltd www.skills-1st.co.uk
JaneCurry Skills1stLtd 2CedarChase Taplow Maidenhead SL60EU 01628782565 jane.curry@skills1st.co.uk www.skills1st.co.uk
Synopsis
ThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsystemin ZenossCore4.TheeventarchitecturehaschangeddramaticallyinZenoss4from previousversions. ItisassumedthatthereaderisalreadyfamiliarwiththeZenossEventConsoleand withbasicnavigationaroundtheZenossGraphicalUserInterface(GUI).Itlooksin somedetailatthearchitecturebehindtheZenosseventsystemthedaemonsandhow theyareinterrelatedanditlooksatthestructureofaZenosseventandtheeventlife cycle. ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.Eventsfrom Windows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsare allexaminedindetail. TheprocessbywhichanincomingeventisconvertedintoaparticularZenosseventis knownaseventmappingandthereareanumberofdifferentpossibletechniquesfor performingthatconversion.Thesewillallbeexploredalongwiththecreationofnew eventclasses. Onceaneventhasbeenreceived,classifiedandstoredbyZenoss,automationmaybe required.Alertingtousersbyemailandpageisdiscussed,asarebackgroundactionsto runcommandsorgenerateTRAPs. LogginganddebuggingtechniquesarediscussedinsomedetailsasistheJSONAPIfor extractingdataoutofZenoss. ThispaperwaswrittenusingZenossCore4.2.3 ThepaperisacompaniontexttotheZenoss4EventManagementWorkshop.
Notations
Throughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,are highlightedbyitalics;importantpointstotakenoteofareshowninbold. Pointsofparticularnotearehighlightedbyanicon.
EventManagementforZenossCore4Skills1stLtd
23January2013
Table of Contents
1Introduction..........................................................................................................................6 2Zenosseventarchitecture....................................................................................................6 2.1EventConsole...............................................................................................................6 2.2EventManagersettings.............................................................................................10 2.3Eventdatabasetables...............................................................................................11 2.3.1Zenoss2.xand3.x...............................................................................................11 2.3.2Zenoss4................................................................................................................14 2.4Neweventdaemons....................................................................................................20 2.4.1RabbitMQ.............................................................................................................20 2.4.2zeneventserver.....................................................................................................22 2.4.3zeneventd.............................................................................................................22 2.4.4zenactiond...........................................................................................................23 2.4.5memcached...........................................................................................................23 2.5OtherdatabaserelatedchangesinZenoss4............................................................24 2.6Eventlifecycle............................................................................................................25 2.6.1Eventgeneration.................................................................................................27 2.6.2Applicationofdevicecontext..............................................................................29 2.6.3Eventclassmapping...........................................................................................29 2.6.4Applicationofeventcontext...............................................................................30 2.6.5Eventtransforms.................................................................................................30 2.6.6Databaseinsertionsanddeduplication............................................................31 2.6.7Resolution............................................................................................................32 2.6.8Ageingandarchiving..........................................................................................34 3EventsgeneratedbyZenoss..............................................................................................34 3.1zenping........................................................................................................................35 3.2zenstatus.....................................................................................................................36 3.3zenprocess...................................................................................................................36 3.4zenwin.........................................................................................................................37 3.5zenwinperf...................................................................................................................37 3.6zenperfsnmp................................................................................................................37 3.7zencommand...............................................................................................................38 4Syslogevents......................................................................................................................38 4.1Configuringsyslog.conf.............................................................................................39 4.2Zenossprocessingofsyslogmessages.......................................................................40 5ZenossprocessingofWindowseventlogs.........................................................................48 5.1ManagementusingtheWMIprotocol.......................................................................48 5.2ManagementofWindowssystemsusingsyslog.......................................................51 6EventMapping...................................................................................................................51 6.1Workingwitheventclassesandeventmappings....................................................52 6.1.1Generatingtestevents........................................................................................54 6.2Regexineventmappings...........................................................................................55 23January2013 EventManagementforZenossCore4Skills1stLtd 3
6.3Rulesineventmappings............................................................................................57 6.4Otherelementsofeventmappings...........................................................................58 7Eventtransforms...............................................................................................................58 7.1Differentwaystoapplytransforms...........................................................................59 7.2Understandingfieldsavailableforeventprocessing...............................................60 7.2.1EventProxies.......................................................................................................63 7.2.2EventDetails.......................................................................................................66 7.3Transformexamples...................................................................................................68 7.3.1CombininguserdefinedfieldsfromRegexwithtransform.............................68 7.3.2Applyingeventanddevicecontextinrelationtotransforms..........................69 8Testinganddebuggingaids..............................................................................................71 8.1Logfiles.......................................................................................................................71 8.1.1zeneventd.log.......................................................................................................71 8.1.2zeneventserver.log...............................................................................................72 8.1.3Otherlogfiles......................................................................................................75 8.2UsingzendmdtorunPythoncommands..................................................................75 8.2.1ReferencinganexistingZenosseventforuseinzendmd.................................75 8.2.2UsingzendmdtounderstandattributesforanEventSummaryProxy...........79 8.3UsingthePythondebuggerintransforms................................................................83 9ZenossandSNMP..............................................................................................................87 9.1SNMPintroduction.....................................................................................................87 9.2SNMPonLinuxsystems............................................................................................88 9.3ZenossSNMParchitecture........................................................................................91 9.3.1Thezentrapdaemon............................................................................................91 9.4InterpretingMIBs......................................................................................................93 9.4.1zenmibexample...................................................................................................94 9.4.2AfewcommentsonimportingMIBswithZenoss.............................................99 9.5TheMIBBrowserZenPack......................................................................................100 9.5.1ModifyingZenossCore4.2tomaketheMIBBrowserZenPackwork..........102 9.6MappingSNMPevents............................................................................................103 9.6.1SNMPeventmappingexample........................................................................103 10EventTriggersandNotifications.................................................................................108 10.1ZenosspriortoV4...................................................................................................108 10.2Zenoss4architecture.............................................................................................109 10.3Triggers...................................................................................................................110 10.4Notifications............................................................................................................111 10.4.1emailNotifications..........................................................................................113 10.4.2PageNotifications...........................................................................................118 10.4.3CommandNotifications..................................................................................118 10.4.4TRAPNotifications.........................................................................................120 10.5NotificationSchedules............................................................................................122 10.6Usingzenactiond.log..............................................................................................123 10.7TheeffectofdeviceProductionState....................................................................125 11AccessingeventswiththeJSONAPI...........................................................................126 4 EventManagementforZenossCore4Skills1stLtd 23January2013
11.1Definitions...............................................................................................................126 11.2UnderstandingtheJSONAPI...............................................................................127 11.3UsingtheJSONAPI..............................................................................................130 11.3.1Bashexamples.................................................................................................130 11.3.2Pythonexamples.............................................................................................134 12Conclusions.....................................................................................................................139 13AppendixA.....................................................................................................................143 13.1getevents.py............................................................................................................143 13.2zensendevent..........................................................................................................148 14References.......................................................................................................................152
23January2013
1 Introduction
ZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.There isafree,Coreoffering(whichhasmostthingsyouneed),andachargeableoffering, ZenossResourceManager,whichhasextraaddongoodiessuchashighavailability configurations,distributedmanagementservers,servicemanagementandevent correlation;italsoincludesasupportcontract. Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availability monitoring,problemmanagementandperformancemanagement.Itisdesignedaround theITILconceptofaConfigurationManagementDatabase(CMDB),theZenoss StandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserver andusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostore Pythonobjectsandtheirstates.Zenoss3usedZEO,asalayerbetweenZopeandthe ZODB;inZenoss4theZODBdataisstoredinaMySQLdatabase. TherelationalMySQLdatabaseisalsousedtoholdcurrentandhistoricalevents. PerformancedataisheldinRoundRobinDatabase(RRD)files. ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetwork Managementprotocol(SNMP),WindowsManagementInstrumentation(WMI)and collectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshand touseNagiosplugins. Zenossprovidesdocumentationat http://community.zenoss.org/community/documentation.Thereisalsoawealthof informationontheZenosswebsiteinvariousforums,FAQs,andtheWiki.Auseful bookisavailablefromPACKTPublishing,ZenossCore3.xNetworkandSystem MonitoringbyMichaelBadger,whichprovidesmuchofthesameinformationasthe ZenossAdministrationGuidebutinamuchclearerformatwithplentyofscreenshots. AlthoughthisisaZenoss3text,itstillprovidesgoodbasicinformation. ThispaperisanattempttoexpandontheeventinformationintheZenossCore4 AdministrationGuidebydrawingonmyownexperienceandthecollectedwisdomof severalZenossemployeesandcontributorsfromthecommunity.
2 Zenoss event architecture

2.1 Event Console
WhenaneventarrivesatZenoss,itisparsed,associatedwithaneventclassification andthentypically(butnotalways),itisinsertedintothe event_summarytableofthe zenoss_zepdatabase.EventscanthenbeviewedbyusersusingtheEventConsoleof theZenossGraphicalUserInterface(GUI).
23January2013
ThereareanumberwaystoaccesstheEventConsole.ThemainEventConsoleis reachedfromthetopEVENTS>EventConsolemenu.Thedefaultistoshowevents withaseverityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecent first).Eventsareassigneddifferentseverities: Name Number Colour

Critical Error Warning Info Debug Cleared
5 4 3 2 1 0
Red Orange Yellow Blue Grey Green
AlleventsalsohaveaneventStatefield.Zenoss3eventStatehadthreepossiblevalues New,AcknowledgedandSuppressed.Zenoss4hasenhancedthesedefinitionssowe nowhave: Name Number Description

New Acknowledged Suppressed Closed Cleared Dropped Aged
0 1 2 3 4 5 6
Neweventnoprevioussimilarevent Acknowledgedbyuserorrule Typicallyfrombeyondasinglepointof failure Closedbyauser Closedbyarule Discardednotsavedinthedatabase Autoclosedduetoage/severity
NotethatClosed,ClearedandAgedeventsallhavethesamestatusiconintheEvent Console. Bydefault,NewandAcknowledgedeventsareshownintheEventConsole.Anyevent whichhasbeenAcknowledgedhasatickinitsstatuscolumn.ASuppressedeventis notshownbydefaultbutcanbefilteredinifdesired;ithasasnowflakeicon.Zenoss buildsaninternaltopologyofthenetworkitismanaging(usingnmap).Ifaneventis receivedforadevicethatthetopologymapknowsisunreachable,theeventis automaticallysuppressed.ThusZenosshasabuiltinmechanismforpinpointingfailure devicesandsuppressingthefloodofeventsfrombehindsuchfailurepoints. Eventscanbesortedbyclickingonadesiredcolumnheader;clickingagainsortsinthe reverseorder.Tochangetheorderofcolumns,simplydragacolumnheader. 23January2013 EventManagementforZenossCore4Skills1stLtd 7
Thereisafilterboxaboveeachcolumnheadertohelpselectrelevantevents.Most filtersareamatchforapartialtextstring(youdon'tneedtosupplywildcards).Date fieldsprovideacalendaricontoselectanearliestdate.Thecountfieldpermitsyouto enterarange,forexampletoshoweventswithcount>10,use 10:(ifyoutype somethingillegalinthecountfilteritwillsupplyhelpfortherequiredsyntax). Toselectfieldstodisplay,hoverthemouseattheendofaheadertoseethedownarrow forsorting;thethirdoptiononthedropdownmenuistoconfigurethefieldstodisplay.
Figure1:ZenossEventConsole
FromtheEventConsole,oneormoreeventscanbeselectedbyclickingonthelinebe carefulnottoclicksomethingthatisalink(likethedevicenameoreventclass).The iconsatthetopleftcanbeusedtoAcknowledge,Close,MaptoanEventClass, UnacknowledgeorReOpen.The+iconattheendofthisrowoficonscanbeusedto generatetestevents. Doubleclickaneventtoshowthedetailsofanevent.Thisshowsbothstandardfields andanyuserdefinedfieldsorganisedunderseveralgroupingswhichcanbeexpanded andcontracted.AnyAcknowledge,CloseorReOpenwillbeshownatthebottom, includingwhoperformedtheaction.Freeformnotescanalsobeloggedhere.
23January2013
Figure2:EventdetailsshowingAcknowledgementandaddednote Thesummaryandmessagefieldsarefreeformtextfields.Thesummaryfieldallowsup
to255characters;themessagefieldallowsupto4096characters.Thesefieldsusually containsimilardata.Fordetailsofotherfields,seesection7.1.2oftheZenossCore4 Administrationguide. Bydefault,theEventConsoleisrefreshedeveryminute.Thedropdownbesidethe Refreshbuttonallowsyoutochangetheintervalortorefreshmanually.
23January2013
EventConsolesarealsoavailableatvariousplacesintheGUIwhichhavefilters alreadyapplied:

Fromadevice'sdetailpage,selectEventsinthelefthandmenu Foradeviceclass,clicktheDETAILSlinkandthenEventsinthelefthand menu ForaLocation,GrouporSystem,clicktheDETAILSlinkandthenEventsin thelefthandmenu FromanEventClass,selectEventsinthelefthandmenu
PriortoV4,ZenosseventswereeitherOpenorClosed.Openeventswerestoredin theMySQLeventsdatabaseinthestatustable.Whenaneventwasclosed,itwas movedtothehistorytableoftheeventsdatabase. WithZenoss4thereisasignificantchange.TheMySQLdatabaseforeventsiscalled zenoss_zepandithasfarmoretables,includingevent_summaryand event_archive.Openeventswillbestoredintheevents_summarytable. Beaware thattheevents_summarytablewillalsoholdclosed,clearedandagedeventsthis catchesoutmanypeoplemigratingfromolderversionsofZenosstoZenoss4.Checkthe StatusfilterintheEventConsoletoshowClosed,ClearedandAgedevents(theyall havethesamestatusicon).Closed,ClearedandAgedeventsmaybeautomatically movedtotheevent_archivetablebasedonage(after3days,bydefault).
2.2 Event Manager settings

FromtheADVANCED>Settingsmenu,chooseEventsinthelefthandmenutosetup variousparametersthatcontroltheeventssubsystem,includinghoweventsareaged andfinallypurged. Figure3onpage11showslargelydefaultsettings.EventsofseverityWarningand belowwillbeAgedafter240minutes(4hours).After4320minutes(3days)eventswith statusofClosed,ClearedorAgedwillbeArchived(movedtotheevents_archivetable). After7daysArchivedeventswillbedeletedentirely(notethislastsettingis90daysby defaultandcanresultinaverylargedatabase). Seechapter7oftheZenossCore4AdministratorsGuideformoreinformation.
10
23January2013
Figure3:EventManagerparametersforageingandarchiving
2.3 Event database tables

2.3.1 Zenoss 2.x and 3.x
Theeventsarchitecturewasthesameforversions2and3andwasrelativelysimple. Eventsweregeneratedfromsomewhere.Thezenhubdaemonprocessedthemand usuallythensavedthemeitherinthestatustableoftheMySQLeventsdatabaseor couldsendthemtothehistorytable. Thedatabasefieldsofthestatusandhistorytablesmatchedthedetailsseeninan EventConsoleandifyouwroterulesandtransformstoprocessevents,theywerebased onthesesamefieldnames. TheeventsdatabaseiscreatedautomaticallywhenZenossisinstalledandcantypically beaccessedbythezenossuserwithapasswordofzenossseeFigure4.
23January2013
11
Figure4:ZenosseventsdatabasepriortoZenoss4
TheformatofeachofthesetablesandthevalidfieldsforaZenosseventcanbeseenby examiningtheZenossdatabasesetupfilein $ZENHOME/Products/ZenEvents/db/zenevents.sql,where$ZENHOMEwillbe /opt/zenossforaCore4.2ZenossonRedHat/CentOS(theonlycurrentlysupported platform).
12
23January2013
Figure5:Definitionofstatuseventfieldsinzenevents.sqlpriortoZenoss4
zenevents.sqlalsodefinesthehistorytableinasimilarfashion. Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.Thedetail tablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthatthe Zenossadministratorrequiresforanevent.
23January2013
13
Figure6:zenevents.sqlshowingheartbeat,alert_state,loganddetailtableszenoss2and3only
IfyouareusingZenosspriortoversion4,gettheolderversionofthisZenossEvent Managementpaperfromhttp://www.skills 1st.co.uk/papers/jane/zenoss_event_management_paper.pdf.
2.3.2 Zenoss 4
WithZenoss4eventsarestillheldinaMySQLdatabasewhichisnowcalled zenoss_zepanditiscreatedwhenZenossisinstalled.Aswithearlierversions,the zenossusercanaccessthisdatabasewithapasswordofzenoss. NotethatwithZenoss4.2.3,ifinstalledwiththecoreautodeployscript,thenthe passwordfortheMySQLzenossuserischangedtoarobust,randompasswordthatis thensavedin$ZENHOME/etc/global.conf.Permissionsfor$ZENHOME/etcandits contentsareallsettofullaccessforthezenossuserandnoaccessforanyoneelse.
14
23January2013
Figure7:AccessingMySQLdatabaseswithZenoss4
Inpassing,notethatinadditiontothezenoss_zepdatabase,theirisalsoa zodbanda zodb_sessiondatabase.TheZopedatabase(ZODB)thatstoresalltheobjects(devices, deviceclasses,processes,networks,etc)isnowinMySQL. Examiningthetablesofthezenoss_zepdatabaseiswherethingsdivergesignificantly frompreviousversions.
23January2013
15
Figure8:TablesintheZenoss4zenoss_zep database
Themaintablesarenowevent_summaryandevent_archivebutthestructureis morecomplicated.Someofthedataisheldinseparatetableswithpointerstothem fromthemaintables.Theseinclude:

agent event_class event_class_key event_group event_key monitor EventManagementforZenossCore4Skills1stLtd 23January2013
16
Thedetailsoftheevent_summarytableisshownbelow.Theeventarchivetableisvery similarwithjustthetwofingerprint_hashfieldsomitted.
Figure9:Fieldsintheevent_summarytableinZenoss4
23January2013
17
Theeagleeyedwillalsospotthatsomeofthefieldnameshavechangedfromthosein Figure5.eventClassintheoldversionbecomesevent_classinV4;firstTimeinFigure5 becomesfirst_seeninthelaterversionandthereareanumberofothersimilar,subtle changes. Asmentionedabove,someofthedataisheldinseparatetablessoagent_id, event_class_id,event_class_key_id,event_group_id,event_key_idandmonitor_keyare linkstoseparatetableswiththecorrespondingdata. Somedatahaschangedfairlysubtly: Old New

evid eventState eventClassMapping severity stateChange firstTime lastTime count facility priority ntevid ownerid clearid
uuid status_id event_class_mapping_uuid severity_id status_change first_seen last_seen event_count syslog_facility syslog_priority nt_event_code current_user_uuid/current_user_name clear_fingerprint_hash/cleared_by_event_uuid
Allreferencestothedevicehavechangedsignificantly. deviceisreplacedbythefour fields,element_uuid,element_type_id,elementidentifierandelement_title whilstthecomponentfieldisreplacedbyelement_sub_uuid, element_sub_type_id,element_sub_identifierandelement_sub_title. dedupidhasbecomefingerprintandfingerprint_hash. OtherfieldswithdevicecontextsuchasprodState,DeviceClass,Location,Systems, DeviceGroups,ipAddress,monitorandDevicePrioritywillnowbefoundfromthe tags_jsonfield;theyarealsoavailableintheeventdetails. PriortoZenoss4therewasaseparatelogtablewhoseroleisnowtakenbythe notes_jsonfieldoftheevent_summarytable. Eventdetailsratherthanbeinginaseparatetable,arenowreachedfrom details_json. update_timehasbeenaddedthelasttimeaneventwasupdated.
18
23January2013
suppid(whichwasneverused)hasdisappearedintheZenoss4schema.managerhas alsodisappearedfromZenoss4. Thesetablesarecreatedbythefilesin$ZENHOME/share/zeneventserver/sql/mysql.
Figure10:Partofthe001.sqlfilethatdefinesMySQLtablesinthezenoss_zepdatabaseforZenoss4
Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwas generated:

Syslogeventspopulatethefacilityandpriorityfields Windowseventspopulatethentevidfield SNMPTRAPspopulateatleastcommunityandoidfieldsintheeventdetail. TheyalsousetheeventdetailtoprovideanyvariablespassedbyanSNMP TRAP. TheagentfielddenoteswhichZenossdaemongeneratedorprocessedthe incomingevent;forexample,zentrap,zeneventlog,zenping .
23January2013
19
FundamentallyZenossadministratorsshouldnotbeaccessingthezenoss_zepdatabase directly.Zenosshaveprovidedaninternaleventmappingsothat,largely, administratorscancontinuetousethesameeventattributenamesashavebeenused previously.Thiseventproxymappingwillbediscussedinmoredetaillater.In general,thispaperwillusetheoldnamesunlessexplicitlystatedotherwise. Ifyoudoneedtoaccesseventdatainthedatabasetables,perhapsforreportingon events,itispossiblewiththeJSONAPI(alsomoreonthislater).
2.4 New event daemons

PriortoZenoss4mostoftheworkofprocessinganeventwasperformedbythe zenhub daemonwhichalsohaslotsofotherrolestofulfil.Eventprocessingcouldbecomea severebottleneck.Zenoss4hasintroducedseveralnewsubsystemsanddaemonsto dramaticallyimprovethethroughputofeventprocessing.
2.4.1 RabbitMQ
AMessageQueueingarchitecturehasbeenimplementedtospeedupprocessingandto offeranAPIsothatZenossandotherapplicationproviderscaninteractwithevents.It isalsousedbythenewJobarchitecture.ItusestheAdvancedMessageQueueing Protocol(AMQP)standard,andtheopensourceRabbitMQimplementationin particular,fortheeventpipeline. WhenZenossisinstalledtheRabbitMQsubsystemisalsoinstalledandconfiguredwith avhostofzenoss,userzenoss,passwordzenoss.Therabbitmqctlutilitycanprovide informationaboutthestateoftheMQenvironment;notethatrabbitmqctlcommands mustberunbytherootuser.
Figure11:Usingtherabbitmqctlutilitytoshowqueuesforthe/zenossvhost
Aneasywaytoseequeuesbuildingupistotemporarilystopzeneventdandthe raweventsqueuewillthenbuildrapidly. 20 EventManagementforZenossCore4Skills1stLtd 23January2013
rabbitmqctlonitsownorwithinsufficientargumentsprovidestheusagehelp. rabbitmqctlreportgivesagoodoverallviewofthesubsystem. IftheZenossserverisrenamedthenyoumustclearandrebuildqueuesbeforethe zenhubandzenjobsdaemonswillrestart.Toresolvethis,issuethefollowing commandsastherootuser(althoughanydataqueuedatrestarttimewillbelost):

export VHOST="/zenoss" export USER="zenoss" export PASS="zenoss" rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app rabbitmqctl add_vhost "$VHOST" rabbitmqctl add_user "$USER" "$PASS" rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'
Seesection14.8oftheZenossCore4AdministratorsGuideforthisinformation. NotethatwithZenossCore4.2.3installedusingtheautodeployscript,orifthe secure_zenoss.shscripthasbeenrunstandalone,thenthepasswordinthethirdline abovewillhavebeenchanged.Examine $ZENHOME/etc/global.confforthe amqppasswordandsubstituethatvalue,ratherthanusing zenossasthepassword. ProvidedtheRabbitMQsubsystemisrunning,anymissingqueuewillautomaticallybe recreatedwhenZenossisrestarted. Tosimplyhavethequeuesrecreated,startasthezenossuser:
zenossstop su(tobecomerootuser) rabbitmqctldelete_vhost/zenoss rabbitmqctladd_vhost/zenoss rabbitmqctladd_userzenosszenoss#mightcreateanerror zenossrabbitmqctlset_permissionsp/zenosszenoss'.*''.*''.*' rabbitmqctllist_vhosts (shouldhavezenossagain) rabbitmqctlp/zenosslist_queues(shouldbenone) exit (backtozenossuser) zenossstart su rabbitmqctlp/zenosslist_queues(shouldbeseveral)
Thereisafurtherscriptavailableatgist,writtenbycluther,toresetRabbitMQ https://gist.github.com/4192854. TwoutilitiesareavailableforthezenossusertogetRabbitMQinformation:

zenqdump <queue name>
dumpstheeventsinaqueue,convertingthebinaryblobs(whichishowtheeventsare actuallystored)intohumanreadabletext. Notethatthezenqdumputilityhasparametersforuserandpasswordfor authentication,thatdefaulttozenoss/zenoss(youcanfindthiscodein $ZENHOME/lib/python/zenoss/protocols/amqpconfig.py).InZenoss4.2.3,passwords arelikelytohavebeenimprovedoninstallationsothesimplecommandshownabove 23January2013 EventManagementforZenossCore4Skills1stLtd 21
willfail.Examine$ZENHOME/etc/global.conffortheparametersamqpuserand amqppasswordandsupplythosevalues.Forexample:
zenqdumpuzenosspuy+680bEubHgdPow8Tfhzenoss.queues.zep.rawevents
Thezenqutilityhasthreedifferentoptionstomanageaqueue:
zenq count <queue name> zenq purge <queue name> zenq delete <queue name>
Thecountparametergivesacontinualoutputoftimestampandqueuelength. Thepurgeparameterpurgeseventsfromaqueue.ThiscommandissafewhenZenossis running. ThedeleteparameterdeletesthequeueandshouldnotbeusedwhenZenossisrunning. zenqdoesnothaveauthenticationparameters.
2.4.2 zeneventserver
AnewJavadaemon,zeneventserver(alsoknownaszep),hasbeencreated.Itsroleisto presenteventstotheuserinterfaceandotherclients,andtomanagetheflowofdata betweentheRabbitMQqueuesandtheMySQLdatabase.Dataispresentedtoclients viaJSONcalls.
2.4.3 zeneventd
zeneventdisanewPythondaemonwhoseresponsibilityistotakedatafromthe incomingraweventqueue,classifyit(iftheeventdoesnotalreadyhaveaclass),add devicecontextandeventcontext,andperformanytransforms.Itthenoutputstothe zeneventsqueuesothatthezeneventserverdaemoncanmanageitsprogresstothe MySQLdatabase,totheuserinterfaceandforalertingaction.
22
23January2013
Figure12:Zenoss4eventarchitecture
2.4.4 zenactiond
zenactiondhasbeencompletelyrewrittenforZenoss4.Itisresponsibleforexecuting actionsassociatedwithnotificationssuchaspaging,email,executingbackground commandsandraisingnotificationTRAPs.zenactiondwillperiodicallyinspectthe signalqueueforsignalmessages,dumpthemintoitsshareofmemcachedand subsequentlyactonthemessagesasinstructedintheassociatednotification.
2.4.5 memcached
PriortoZenoss4eachofthedaemonshaditsowncache.Thiscouldbeawasteful allocationofmemory.WithZenoss4,amemcachedsubsystemisintroducedwhich providessharedL2memorycacheforalldaemons,offeringmuchbetterperformance. memcachedisconfiguredin/etc/sysconfig/memcached.Thedefaultistoconfigure 64Mbformemcached(whichisnotpreallocated;itisonlyusedasnecessary).This shouldbeincreasedtoatleast1Gbonproductionsystemswithmorethan100devices (andrun/etc/init.d/memcachedrestart).Alsoensurethatmemcachedisenabledin $ZENHOME/etc/zope.conf.
23January2013
23
2.5 Other database-related changes in Zenoss 4

Notdirectlyrelatedtotheeventssubsystem,buttheZopedatabase(ZODB)thatusedto beheldin$ZENHOME/var/Data.fsandaccessedbythezeoctldaemon,isnowstored inthesameMySQLinstanceaszenoss_zep(andZEOhasgone). ThezodbdatabaseisthemainZopedatabaseandthereisalsoa zodb_session databasewhichholdsuserpreferencesthinkofzodb_sessionasanexpandedsetof user'scookies;ifnecessary,itcanbedeletedanditwillberecreatedautomatically. ZODBiswherealltheobjectdataisstoredrelatingtodevices,components,processes, services,networks,MIBs,etc.Theeventprocessingdaemonsneedaccesstothezodb databasetoenricheventswithdeviceandcomponentinformation. Zopeobjectsareknownaspickles,typicallyastringrepresentationofencodeddata(a blob)inotherwords,treattheZODBdatabaseasablackbox(justasData.fswas). AJSONinterfaceisprovidedtoaccessdataintheZODBandthezendmdtoolstill worksinexactlythesamewayasinpreviousversionsofZenoss,despitetheZODBnow beinginMySQL.
24
23January2013
ToprovideaccesstothethezodbMySQLdatabase,a RelStoragesubsystemisusedas ahighperformancebackendtoZODB.RelStoragemayalsousememcachedtofurther enhanceperformance. TheolderversionsofZenossdidnotdomuchbywayofindexingtheeventsdatabase. WithZenoss4holdingZODBdataaswellaseventsdatainMySQL,aneffective indexingmechanismwasrequiredsotheLucenepackageisusedfromApache.Lucene isahighperformance,fullfeaturedtextsearchenginelibrarywrittenentirelyinJava. Itisusedtoholdindexesforbothzodbandzenoss_zep.
2.6 Event life cycle

Thelifecycleofaneventhaseightphases:

Eventgeneration Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent Eventclassmappingtodistinguishonetype(class)ofeventfromanother Eventcontextadditionalinformationpertinenttoaclassofevent
23January2013
25
Eventtransformmanipulationofeventfields Databaseinsertionanddeduplication Resolution Ageingandarchiving
Figure14:Eventlifecycle,generationtodatabaseinsertion
Processingofaneventdependsontheeventclassthataneventisassignedtothe valueofitseventClassfield.Adescriptionofeachofthesephaseswillbegivenhere: subsequentsectionsofthepaperprovidemoredetailsofsomeareas. InFigure14,thefirstsixphasesoftheeventlifecycleareshown.Theblue,dashed pathshowstheprogressofaninternallygeneratedZenossevent,whichdoesnotpass throughaneventmappingphase.AneventClassfieldisproducedbythedaemonthat generatedtheevent.Itsonlywaytoapplyatransformisasaclasstransform. ThepurplepathshowstheprogressofaneventthatisgeneratedexternallytoZenoss. TheinitialparsingdaemonmustprovideaneventClassKeyfieldwhichisthenused, alongwithotherfields,inaneventclassmappingRuleand/orRegex,whichinturn providesaneventClassfield.Aftermapping,theeventmaypassthroughbothan eventclasstransformandaneventmappingtransform.
26
23January2013
AnareathathaschangedfairlysignificantlyinZenoss4isthemechanismforresolving andageingevents.PriortoVersion4,aneventwasfundamentallyopen(whichalso encompassedeventStateofAcknowledgedandSuppressedaswellasNew)andsuchan eventresidedinthestatustableoftheeventsdatabase;alternatively,aneventwas Closed,inwhichcaseitwasmovedtothehistorytableoftheeventsdatabase. WithZenoss4,thepossiblevaluesofeventStatehavebeenexpandedtoinclude: Name Number Description

New Acknowledged Suppressed Closed Cleared Dropped Aged
0 1 2 3 4 5 6
Anewevent Acknowledgedbyuserortransform Eventtypicallybeyondasinglepointoffailure Eventresolvedbyauser Eventresolvedbyanautomaticrule WouldneverreachtheMySQLdatabase

Eventautomaticallyclosedaccordingtothe
severityandlastseentimeoftheevent. Thesearewelldescribedinchapter7oftheZenossCore4AdministrationGuide.The hugedifferencehereisthatthenewevent_summarytableintheMySQLdatabasewill probablyhaveClosed/Cleared/Agedeventsinit.Theevent_archivetablehasevents thathavebeenautomaticallyagedoutbasedontheirseverityandage.
2.6.1 Event generation

Fundamentally,eventswilleitherbegeneratedbyZenossitselfintheprocessof discovery,availabilityandperformancechecking,oreventswillbegeneratedoutside ZenossandcapturedbyspecialisedZenossdaemons.
23January2013
27
Zenossdaemon zenping zendisc zenstatus zenprocess zenwin zenwinperf zencommand zenperfsnmp zenmodeler
Exampleofwheneventgenerated pingfailureoninterface newdevicediscovered TCP/UDPserviceunavailable processunavailable Windowsservicefailed WMIperformancedatacollectionfailure/threshold sshperformancedatacollectionfailure/threshold SNMPperformancedatacollectionfailure/threshold Configurationdatachangedonzenmodelerpoll
Table2.1.:EventsgeneratedbyZenossitself
Zenossdaemon zensyslog zeneventlog zentrap
Exampleofwheneventgenerated processessyslogeventsreceivedonUDP/514(default) processesWindowseventsreceivedusingWMI processesSNMPTRAPsreceivedonUDP/162
Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons
EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent. Thedaemonthatgeneratestheeventparsesthenativeinformationandassignsavalue totheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary, messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenoss daemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwill populatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface). EventsthatareinitiallygeneratedoutsideZenossarecapturedby zensyslog, zeneventlogorzentrap.Thesedaemonseachhaveaparsingmechanismtointerpret thenativeeventintotheZenosseventformat.ThePythoncodeforthezensyslogand zentrapparsingisin$ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwill be/opt/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.pydecodesSNMP TRAPs. ThedaemonsforprocessingWindowsWMIdatausedtobeastandardpartoftheCore codebutwithZenoss4thishasmovedtoaZenosssuppliedZenPack ZenPacks.zenoss.WindowsMonitor.zenwin,zenwinperfandzeneventlogcanallbe foundunderthatZenPack'sbasedirectory. Typically,theexternaleventparsingmechanismsdonotdeliveravaluefor eventClass; rathertheydeliveravaluefortheeventClassKeyfield,alongwithvaluesforsome
28
23January2013
otherfieldssuchascomponent,summary,messageandagent.Itisthenthejobofthe eventmappingphasetodistinguishtheeventclass.
2.6.2 Application of device context

Earlyintheeventprocessinglifecycle,thezeneventddaemonapplies devicecontext totheevent.Thismeansthatsevenfieldsoftheeventarepopulatedbydeterminingthe devicethatgeneratedtheeventandthenlookingupthefollowingvaluesforthedevice intheZODBdatabase:

prodState DevicePriority Location DeviceClass DeviceGroups Systems ipAddress(mayhavealreadybeenassigned)
2.6.3 Event class mapping

Eventclassmappingtendsonlytobeapplicabletoeventsthatoriginateoutsidethe Zenosssystem.Itistheprocessbywhichaneventisassignedavalueforits eventClassfieldand,potentially,otherfields. Typically,theeventgenerationphasewilldeliveraneventwithafewfieldspopulated; generallythisdoesnotincludetheeventClassfieldbutdoesincludetheeventClassKey field.OftentheZenossparsingdaemon(suchaszensyslog),willusethe same eventClassKeyforseveraldifferentnativeevents.Forexample,aneventClassKeyof dropbearisusedforseveralloginsecurityevents.Thecomponent,summary,message andagentfieldsmayalsobepopulated. Theeventclassmappingphaseexaminestheevent(suchasitis,sofar)andthenusesa numberofteststodeterminetheeventClasstoassigntothisevent: 1. AneventClassKeyfieldmustexistformappingtobesuccessful. 2. APythonRulecanbewrittentotestanyavailablefieldoftheeventorany availableattributeofthedevicefromwhichtheeventcame.Suchrulescanbe complexPythonexpressions,includinglogicalANDsandORs.Iftheruleis satisfied,theincomingevent'seventClassfieldwillbegiventheclassassociated withthatmapping.Iftheruleisnotsatisfied,thismappingisdiscarded,the classisnotassociated,andthenextmappingwillbetestedforamatch.ARule doesnothavetoexistinamappinginstance. 3. IftheRuleissatisfied(ordoesnotexist),themappingcanthenusea Regex Pythonregularexpressiontoparsetheevent'ssummaryfield,checkingfor particularstrings.TheRegexcanalsoassignpartsofthesummaryfieldtonew, 23January2013 EventManagementforZenossCore4Skills1stLtd 29
userdefineddetailfieldsoftheevent.IfaRuleexistsandissatisfied,theclass mappingwillapply,eveniftheRegexisnotsatisfied;anyuserdefinedfieldsin theRegexwillnotbecreatediftheRegexdoesnotmatch.IfaRuledoes not existthentheRegexmustbesatisfiedforthemapping(andanytransform)to apply. 4. TheGUIdialoguethatdefinesthemappingspecifiestheeventClassKey,theRule, theRegexandanyTransform.Asequencenumberisalsoavailablesothatif multipleincomingeventshavethesameeventClassKeythenthesequence numberdefinestheorderinwhichthevariousmappingswillbeapplied,lowest numberfirst.ThefirstRule/Regexmappingcombinationthatmatcheswillbe applied. Eventclassmappingisexecutedbythezeneventddaemon.
2.6.4 Application of event context

EventcontextisdefinedbytheConfigurationProperties(zProperties)ofanevent. Eventcontextcanbedefinedattheeventclasslevel,foraneventsubclass,oratthe eventmappinglevel.Aswithallobjectorientedattributes,thevaluesareinheritedby childobjectssoapplyingeventcontexttoaclassautomaticallysetsitforanysubclasses andsubclassmappings.Thethreeeventcontextattributesare:

zEventAction zEventClearClasses zEventSeverity
status|history|dropdefaultisstatus bydefaultthisisanemptyPythonlistofstrings Originalbydefault
Eventcontextisappliedintheeventlifecycle,afterRuleandRegexprocessingbut beforeanyeventtransforms.Thus,thezEventActionzPropertycanspecifyhistorybut aneventtransformcouldoverridethatactionbysettingtheevt._actionvalueto status. NotethatthestatusandhistoryvaluesreflecttheolddatabasetablespriortoZenoss4. statusnowmapstoaneventStateofNewandhistorymapstoaneventStateofClosed; bothwillbestoredintheevent_summarydatabasetable. Eventcontextisappliedbythezeneventddaemon.
2.6.5 Event transforms

Eventtransformscanbespecifiedforaneventclassmappingorforaneventclass(or subclass).AtransformiswritteninPythonandcanbeusedtomodifyanyavailable fieldsofeithertheeventorthedevicethatgeneratedtheevent.Itcanalsocreateuser definedfields. FromZenoss2.4,cascadingeventtransformsmeanthatclasstransformsareapplied fromeverylevelintheappropriateclasshierarchy,followedbyanytransformforan
30
23January2013
appliedeventmapping.PriortoZenoss2.4,eitheramappingtransformwasapplied, oraclasstransform,butnotboth.Classtransformswereonlyappliedtotheexact class,notfromtheeventclasshierarchy. AtransforminaneventmappingwillonlybeexecutedoncetheeventClassKeyhasbeen matched,andtheRulehasbeensatisfied(ifitexists).IfaRuledoesnotexist,any Regexhastobesatisfiedforthetransformtobeexecuted. Eventtransformsareexecutedbythezeneventddaemon.
2.6.6 Database insertions and de-duplication

ZenosseventsarenowstoredinaMySQLdatabasecalled zenoss_zep(usedtobe events).Themaintablesfortheeventlifecyclearethe event_summarytablefor recentevents,theevent_archivetableforoldevents. Somefieldsoftheeventareonlyassignedatdatabaseinsertiontimetheyarenot availableateventmappingoreventtransformtime.Theseinclude:

count eventState evid stateChange dedupid eventClassMapping firstTime lastTime
ItistheJavazeneventserverdaemonthatisresponsibleforgettingeventsintothedatabase. Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateevent arrives,thentherepeatcountofanexistingeventwillbeincremented.duplicateis definedashavingthefollowingfieldsthesame:

device component eventClass eventKey severity
IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalso match.Thededupidfieldiscreatedbyconcatenatingtheabovefieldstogether, separatedbythepipe(verticalbar)symbol.Thusanexamplededupidmightbe:

zenoss.skills-1st.co.uk|su|/Security/Su||5|FAILED SU (to root)jane on /dev/pts/1
wherethedeviceiszenoss.skills1st.co.uk,componentissu,eventClassis/Security/Su, theeventKeyisunset,severityis5(Critical),andthesummaryisFAILEDSU(toroot) janeon/dev/pts/1. InZenoss4,thededupidfieldisalsoknownasthe fingerprint. 23January2013 EventManagementforZenossCore4Skills1stLtd 31
Whenaneweventisreceivedbythesystem,thededupidisconstructedbythe zeneventddaemon.Transformsmaymodifyeithercomponentfieldsofthefingerprintor maydirectlymodifythededupidfield. Whenzeneventservercomestoinserttheeventinthedatabase,ifitmatchesthe dedupidforanyactiveevent,theexistingeventisupdatedwithpropertiesofthenew eventoccurrence,theevent'scountisincrementedbyone,andthelastTimefieldis updatedtobethecreatedtimeoftheneweventoccurrence. NotethatthisisasubtlebutsignificantchangefrompriorversionsofZenossasthe existingeventisupdatedwithpropertiesofthenewevent;olderversionsofZenoss simplyupdatedthecountandlastTimefields.Forexample,ifthefingerprintincludes aneventKeysodoesnotincludethesummary,theresultingeventwillnowshowthe summaryofthelatestreceivedduplicateevent. Iftheincomingeventdoesnotmatchthededupidofanyactiveevents,thenitisinserted intotheactiveeventtablewithacountof1,andthefirstTimeandlastTimefieldsare settothecreatedtimeofthenewevent.
2.6.7 Resolution
Resolutionofaproblemrepresentedbyaneventcanhappeninseveralways:

Auserclosestheevent(eventState=Closed) TheeventcontextzEventActionzPropertyforaneventclassisdrop(theeventis discarded).Forexample,eventclass/Ignore. TheeventcontextzEventActionzPropertyforaneventclassishistory (eventState=Closed).Forexample,eventclass/Archive. Atransformsetsevt._actionto'drop'(theeventisdiscarded) Atransformsetsevt._actionto'history'(eventState=Closed) Anotherclearingeventarrivesthatclearstheinitialevent(eventState=Cleared) TheEventManagersettingshaveseverityandlastSeenparametersthatdenote whicheventswillbeautomaticallyaged(eventState=Aged)
Alltheaboveeventswillstillbeintheevent_summarytableoftheMySQLdatabase. TheEventManagerparameterforEventArchiveThresholdistheonlyautomaticaction thatmoveseventsfromevent_summarytoevent_archiveanditwillmoveallevents witheventStateofClosed,ClearedandAged. Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;thereare twodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews. Thefirstclearingmechanismisthatanyeventwithaseverityof Clearwillsearchthe event_summarytableforsimilaractiveeventsandsettheireventStateto Cleared (notClosed). TheZenossCore4AdministratorsGuidedefinesthisautoclearfingerprintas: 32 EventManagementforZenossCore4Skills1stLtd 23January2013
IfcomponentUUIDexists:

componentUUID eventClass eventKey(canbeblank) device component(canbeblank) eventClass eventKey(canbeblank)
IfcomponentUUIDdoesnotexist:

Thiscanbealittleconfusing.TheEventConsoleshowsacomponentfield.Itdoesnot showacomponentUUIDfield.StrictlythecomponentfieldintheEventConsoleshows theelement_sub_identifierfieldfromtheMySQLdatabasetablethenameofthe component.SomeeventsgenerateacomponentUUID(UniversallyUniqueIdentifier) andsomedonot.InspectingtheeventinthedatabaseorusingtheJSONinterfaceis theonlywaytodeterminewhetherthisuniquecomponentidfieldexistsornot.Ifit doesexistthenitshouldalso,byimplication,denotethedevicethatthecomponent belongsto,hencethedevicefieldisunnecessary.(VersionsofZenosspriorto4didnot haveacomponentUUID;similarwasdefinedashavingthesame eventClass,device andcomponentfields.) EitherwayinCore4,theeventClassandtheeventKeyfieldsaresignificant.Ifthe componentUUIDdoesnotexistthenitistheelement_sub_identifier(componentname) thatmustmatch,alongwiththedevicename(element_identifierintheMySQLtable). Thesecondautomaticclearingmechanismextendstheautoclearfingerprintdefinition ofeventClass.TheeventcontextofaneventclassincludeszEventClearClasseswhichis alistofothereventclassesthatthisgoodnewseventwillclear,inadditiontoitsown class.Theotherconditionsoftheautoclearfingerprintremainthesame. Notethatthesameeffectcanbeachievedinatransformbyassigningalistofclass namestoevt._clearClasses. Alleventswiththesameautoclearfingerprintarecleared,notjustthemostrecent. TheclearingeventwillautomaticallyhaveitseventStatesetto Closed,providedit matchesoneormorebadnewsevents.Ifitdoesnotmatchanyeventsthenthe clearingeventisdroppedandwillnotbepersistedtothezenoss_zepdatabase.Thisis toavoidfillingupthedatabasewithredundantgoodnewsevents. Whencorrelationtakesplacesomeoftheexistingbadnewseventfieldsareupdated; stateChangebecomesthetimewhentheeventwasresolved; clearidispopulated withtheevidfieldoftheclearing,goodnewsevent. Thisautomaticresolutionofeventsisperformedbythezeneventserverdaemon. 23January2013 EventManagementforZenossCore4Skills1stLtd 33
2.6.8 Ageing and archiving

Maintenanceisrequiredonthetablesofthezenoss_zepdatabaseorthediskwillsimply fillupeventually.ThreemechanismsareprovidedbytheEventManager:
Bydefault,eventswithseveritylessthanErrorwillbeAgedafteran Event AgeingThresholdof4hours;thatis,theeventStatewillbesettoAged(strictly thevalue6). Bydefault,theEventArchiveThresholdis4320minutes(3days).Thismeans anyeventwitheventStateofClosed,ClearedorAgedwillbemovedfromthe event_summarytabletotheevent_archivetableofthezenoss_zepdatabase. TheDeleteArchivedEventsOlderThan(days)parameteris90bydefault.Thisis theonlyparameterthatautomaticallydeletesdata.Itisnotpossibletofinetune thistodelete,say,lowerseverityeventsafterdifferentintervals.
Zenosspriortoversion4providedautility, $ZENHOME/Products/ZenUtils/ZenDeleteHistory.py whichcoulddeleteeventsselectivelybasedonageandseverity.Thisutilityisnot shippedwithZenoss4andcurrentlyhasnoequivalentfunction. DeletingdatafromtheoldhistorytableinZenoss3usedtobeveryslow.InZenoss4, theevent_archivetableispartitioned,byday,ratherthanbeingonehugefile.This meansthatdeletingdataissimplyamatterofdroppingpartitionfiles.Thiscanbeseen fromthemysqlinterfacewith:

showcreatetableevent_archive;
3 Events generated by Zenoss

Inthecourseofdiscovery,availabilitymonitoringandperformancemonitoring,Zenoss maygenerateeventstorepresentachangeinthecurrentstatus.Althoughmanyevents arebadnewsitshouldberecognisedthateventscanalsobegoodnewsInterface Up,Thresholdnolongerbreached,etc. EventsgeneratedbyZenossaredependentonthevariouspollingintervalsconfigured. Toexaminethedefaultparameters,usetheADVANCED>Collectorsmenu.Clickon localhost(thecollectorontheZenosssystem).NotethatearlyversionsofZenossused thetermandmenuoptionMonitorsratherthanCollectors.
34
23January2013
Figure15:DefaultparametersforlocalhostCollector
Parameterstonoteparticularlyare:

SNMPPerformanceCycleInterval ProcessCycleInterval StatusCycleInterval WindowsServiceCycleInterval PingCycleTime ModelerCycleInterval
300secs(5mins) 180secs(3mins) 60secs(1min) 60secs(1min) 60secs(1min) 420mins(12hours)
3.1 zenping
Themostbasiclevelofavailabilitycheckingistopingpoll.The zenpingdaemonwill, bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgenerated whenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhena similarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldof theeventisincreased. Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,for exampleifasinglepointoffailurerouterisdown.WithZenoss4thisisachievedusing nmap;withearlierversions,Zenossbuiltaninternaltopologybasedonquerying routingtableswithSNMP. Ifaneventisreceivedforanisolatedelement,aneventisgeneratedwithaneventState fieldofSuppressedandthesummaryfieldreportsnotonlytheinterfaceforwhichthe pingfailed,butalsothecausaldevice;forexample: ip10.191.101.1isdown,failedatbino.skills1st.co.uk 23January2013 EventManagementforZenossCore4Skills1stLtd 35
Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghas failed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbe suspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoring eventswillnotincreaseuntilpingaccessisresumed. Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.If adevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,then ensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPand sshavailabilitymonitoringandperformancedatacollection. Thelogfileforzenpingiszenping.login$ZENHOME/log.
3.2 zenstatus
ThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDP portsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute. Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexamined fromtheINFRASTRUCTURE>IpServicesmenu.Bydefault,theonlyservice monitorsthatareactiveareforsmtpandhttp;therestaresetwithmonitoringdisabled. Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsa similarbadnewseventandthecountfieldoftheeventincreaseswhilsttheservice remainsdown. Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.
3.3 zenprocess
zenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.Ina Unixcontext,thiswouldbewhethertheprocessappearsina pseflisting;inaWindows context,theprocessmustappearintheWindowsTaskManager(andnotethatthis checkiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault. Configurationofprocessmonitoringforadeviceissimilarasforservicesthe INFRASTRUCTURE>Processesmenuprovidesawaytoconfigureprocessestobe monitored.Zenoss4comeswithdefinitionspreconfiguredforalltheZenossprocesses. ProcessmonitoringisactuallyachievedusingtheHostResourcesManagement InformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.Thismeansthat ifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation. Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsand thecountfieldincreasesonsubsequentfailedpolls. Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.
36
23January2013
3.4 zenwin
ThezenwindaemonshipswiththeZenPacks.zenoss.WindowsMonitorZenPackwith Zenoss4(itwasastandardpartoftheCorecodeinearlierversions).Itmonitors Windowsservices(notTCP/UDPservices).Thesecanbeexaminedfromthe INFRASTRUCTURE>WindowsServices.Bydefault,noneofthesemonitorsare active. zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccess servicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(or deviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepolling canbesuccessful. Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomatically clearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailed polls. Thelogfileforzenwiniszenwin.login$ZENHOME/log.
3.5 zenwinperf
zenwinperfisanewdaemonforZenoss4whichisalsopartofthe ZenPacks.zenoss.WindowsMonitorZenPack.WithearlierversionsofZenoss,many usersdeployedtheexcellentcommunityWMIDataSourceandWMIWindows PerformanceZenPackstoachievesomethingverysimilartothisnewdaemon. zenwinperfprovidesperformancemonitoringofinterfaces,filesystems,memory,CPU andpagingusingtheWMIprotocol.Defaultthresholdsareconfiguredforsomemetrics whichthengenerateeventswhenexceeded.Itcanbeextendedbytheusertomonitor otherperfmonmetricsusingtheWMIprotocol. Dataisgatheredevery5minutes. Thelogfileforzenwinperfiszenwinperf.login$ZENHOME/log.
3.6 zenperfsnmp
zenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMP performanceinformationandstatusinformationforprocesses.EvenifSNMP performancemonitoringisnotconfigured,zenperfsnmpchecksthattheSNMPagentis available. Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbe generated.Withinafurther3minutesthereshouldbeanUnabletoreadprocesseson device..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldfor individualmissingprocesseventsshouldstopincreasing.WhileSNMPaccesstothe deviceremainsbroken,thecountfieldfortheUnabletoreadprocessesondevice.. eventwillincreaseevery3minutes.
23January2013
37
Thelogfileforzenperfsnmpiszenperfsnmp.login$ZENHOME/log.
3.7 zencommand
Thezencommanddaemonperformsmonitoringbasedonrunningcommands,typically overansshconnection.Likezenperfsnmpandzenwinperfitusesperformance templatestomonitormetricsandcangenerateaneventifathresholdisbreached. Thelogfileforzencommandiszencommand.login$ZENHOME/log.
4 Syslog events
TheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linux althoughslightlydifferentversionsandformatsexist.Therearealsoopensource implementationsofsyslogforWindowssystemsandmanynetworkingdevicesalso supportthesyslogconcept. Typicallysystemmessagesareoutputtooneormorelogfilessuchas /var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslog messagestoacentralsyslogratherthanholdingfilesoneachsystem.Thewellknown defaultportforforwardingsyslogmessagesisUDP/514. Astandardsyslogsystemisconfiguredbythe syslog.conffile,typicallyin/etc.Anewer versionofsyslogisimplementedonsomesystems, syslogng,whichhasgreaterfiltering capabilities.Thesyslogngconfigurationfileistypically /etc/syslogng/syslogng.conf. AnothervariationisrsyslogdwhichistypicallyshippedwithnewerRedHat/CentOS SuSEsystems,configuredthrough/etc/rsyslog.conf. Asyslogmessageincludesapriorityandafacility.Theprioritiesare: 0 emerg 1 alert 2 crit 3 err 4 warning 5 notice 6 info 7 debug Facilitiesinclude: auth (4) cron (9) ftp(11) lpr(6) 38 authpriv(10) daemon(3) kern(0) mail(2) 23January2013
news (7) user (1)
syslog(5) uucp(8)
Thesedefinitionscanbefoundinsyslog.h(typicallyin/usr/include/sys).Bothpriority andfacilityareencodedinasingle32bitintegerwherethebottom3bitsrepresent priorityandtheremaining28bitsareusedtorepresentfacilities. Forexample,ifthefacility/prioritytagis<22>,thiswouldbe00010110inbinary,where thebottom110representsapriorityof6(info)andthetop00010representsafacilityof 2=mail.
4.1 Configuring syslog.conf

AnydevicethatisgoingtoreportsyslogeventstoZenossmusthaveitssyslog.conffile configuredwiththedestinationaddressoftheZenosssystem.Theoriginalsyslog.conf permitsfilteringbasedonpriorityandfacilityso,acatchallstatementtosendall eventstotheZenosssystem,wouldbe:
*.debug @<IP address of your Zenoss system>
Thisalsoworksforrsyslogd.SeeFigure16foranrsyslog/syslogexamplethatforwards tozen42.class.example.orgallfacilitieswithpriorityofnoticeandabovebutallcron messagesarefilteredout;authprivmessageswillbeforwardedwithseverityinfoand above.
Figure16ConfigurationfileforrsyslogsendingselectedeventstoZenossserver
23January2013
39
syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogng offerssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmay alsobepresent.
Figure17:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)
4.2 Zenoss processing of syslog messages

TocollectsyslogmessageswithZenoss,thezensyslogprocessautomaticallystartson portUDP/514andcollectsanysyslogmessagesdirectedfromothersystems.zensyslog thenparsesthesemessagesintoZenossevents.Youmustensurethatthesyslog.conf fileontheZenosssystemdoesnotenablecollectingremotesyslogsorthesyslogdand zensyslogprocesseswillclashoverwhogetsUDP/514(itispossibletoreconfigureeither daemon,ifrequired). 40 EventManagementforZenossCore4Skills1stLtd 23January2013
Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,the levelofzensyslogloggingcanbeincreased. 1. UsetheINFRASTRUCTURE>Settings>Daemonsmenu. 2. Clicktheeditconfiglinkforthezensyslogdaemon. 3. ChangethefollowingparametersandclickSave: logorig logseverity selectthis Debug
4. Inspecttheunderlyingconfigurationfilein$ZENHOME/etc/zensyslog.conf. 5. Thelogoriglinesaystologtheoriginalincomingsyslogmessage;itwillbein $ZENHOME/log/origsyslog.log.Notethatthisparameterisuniquetozensyslog andisusefulfordebugging. 6. ThelogseveritylineisagenericZenossdaemonparameter;avalueof 10isthe maximumDebuglevel. 7. Don'tforgettoSavethischange 8. UsetheRestartlinktorecyclezensyslog.Alternatively,asthezenossuser,issue thecommand:
zensyslog restart
9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log 10. Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.

host=zen241.class.example.org, ip=172.16.222.241
11. Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority. 12. Linesstartingwithtagshowthezensyslogparsingprocessasitteststhe incominglineagainstvariousPythonregularexpressions,hopefullyendingwith atagmatchline. 13. Ifamatchissuccessful,aneventClassKeymaybedetermined 14. ThelastlineforaparsedeventshouldbeaQueueingevent.
23January2013
41
Figure18:zensyslog.logshowingparsingprocess
Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablya mismatchofseverities.Thefollowingtabledemonstratesthis. Zenoss Critical(red)(5) Error(orange)(4) Warning(yellow)(3) Info(blue)(2) Debug(grey)(1) Clear(green)(0) syslogpriority emerg(0) alert(1) crit(2) err(3) warning(4) notice(5) info(6) debug(7)
Table4.1.:EventseveritiesforZenoss,syslogandWindows
Windows Error(1) Warning(2) Informational(3) Securityauditsuccess(4) Securityauditfailure(5)
NotethatthenumericvalueofZenosseventseveritydecreasesaseventsgetless criticalbutthatthepriorityofsyslogeventsincreasesaseventsgetlesscritical. DefaultmappingfromsyslogprioritytoZenosseventseverity,isperformedby $ZENHOME/Products/ZenEvents/SyslogProcessing.pysearchfordefaultSeverityMap aroundline187inCore4.2.Theresultisthat:

syslogpriority<3(emerg,alert,crit)maptoZenossseverity5(Critical) syslogpriority3(err)mapstoZenossseverity4(Error) syslogpriority4(warning)mapstoZenossseverity3(Warning) syslogpriority5or6(notice,info)maptoZenossseverity2(Info)
42
23January2013
Outofthebox,allsyslogeventsmaptotheZenosseventclassof /Unknown. SyslogProcessing.pyisthecodethatparsesanyincomingsyslogmessageandgenerates aZenossevent. ThefirstsectionhasaseriesofPythonregularexpressionstomatchagainstthe incomingsyslogline.Eachexpressionischeckedinturnuntilamatchisfound.Ifno matchisfoundthenanentrygoesto$ZENHOME/log/zensyslog.logwithparseTag failed.
Figure19:SyslogProcessing.pyregularexpressionstomatchsyslogtags
ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincoming eventtoZenosseventclassfields,asfollows:
23January2013
43
def process(self, msg, ipaddr, host, rtime): evt = dict(device=host, ipAddress=ipaddr, firstTime=rtime, lastTime=rtime, eventGroup='syslog')
Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsare bothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfield ishardcodedatthisstagetosyslog.
Figure20:SyslogProcessing.pyprocessmainroutine
parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility. ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosetthe severityfieldoftheZenossevent. 44 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure21:SyslogProcessing.pyparsingofpriority,facilityandseverity
Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefrom theincomingevent.ThedeviceandipAddressfieldsoftheZenosseventaresetatthe endofthisfunction.
23January2013
45
Figure22:SyslogProcessing.pyprocessingtheheaderinformation
TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressions atthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged. TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenoss eventsummaryfield.
46
23January2013
Figure23:SyslogProcessing.pyparsingthesyslogtag
ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdone withthebuildEventClassKeyfunction.
23January2013
47
Figure24:SyslogProcessing.pydeterminingtheEventClassKey
Notethatiftheeventhasthecomponentfieldpopulatedthenthatisusedasthe eventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.
5 Zenoss processing of Windows event logs

5.1 Management using the WMI protocol
Zenosspriortoversion4shippedWindowsmonitoringaspartoftheCorecode.Zenoss 4shipsWindowssupportwiththeZenPacks.zenoss.WindowsMonitorZenPackwhich hasaprerequisiteofZenPacks.zenoss.PySamba.TheseareZenossprovidedCore ZenPacks. IfaWindowsdevicesupportsSNMPthenitisperfectlypossibletousethatprotocol, especiallyasmostWindowsSNMPagentsalsosupporttheHostResourcesMIBsosome systeminformationisavailableinadditiontothestandardMIB2networktype information. TheZenossWindowsZenPacksintroducethe/Server/Windows/WMIdeviceclasswhich hasbothWMImodelerpluginsandWMIperformancetemplatesassociatedwithit. Targetdevicesshouldbeaddedtothisclassorsubclassesthereof.Thisallows monitoringusingtheWindowsManagementInstrumentation(WMI)protocol.Auserid andpasswordneedtobeconfiguredontargethoststopermitWMIaccessfromthe 48 EventManagementforZenossCore4Skills1stLtd 23January2013
Zenossserver;italsomeansthatfirewallsbothontheWindowsdevicesandany interveningnetworkfirewalls,mustbeconfiguredtopermitWMIaccess.TheZenoss ServermustthenbeconfiguredwithmatchingWindowszProperties(zWinUserand zWinPassword)forthetargetdevices/deviceclasses.ThereareafewotherWindows specificConfigurationPropertiesseeFigure25.ThesezPropertiescanbechangedfor adeviceclassorforaspecificdevice.
Figure25zPropertiesforWindowstargets
ZenPacks.zenoss.WindowsMonitorprovidesthreenewdaemons: zenwin monitorswindowsservicesusingWMI zenwinperf collectsperformancedatausingtheWMIprotocol zeneventlog retrievesWindowseventloginformationusingWMI ThethreezWinPerf...zPropertiesfinetunetheconfigurationofthezenwinperfdaemon; thezWinEventlogparametermustbeTruetocollectWindowseventsfromatarget device. ThezWinEventlogMinSeveritypropertydefinestheleastseriousseverityeventsthat willbeforwardedfromWindowstoZenoss.Notethatthenumericdenotationof windowseventseveritiesandtheirnamesandsupportcurrency,havechangedoverthe lifeofZenoss.SeeTable4.1onpage42forcurrentvalidseverities.Alsonotethatifyou changethisparameteryouarepresentedwithalistofZenossseverities,notWindows styleseverities;againrefertotheearliertableforatranslation.Ifyouwanttoinclude allWindowsseverities,includingsecurityauditfailure(5),youneedtoselectthe Clear severityinthedropdownmenuwhenchangingzWinEventlogMinSeverity. ThezWinEventlogClausewasintroducedduringthelifetimeofZenoss3tohelpfilter eventsfromWindowsdevices.ConsulttheZenossCore4AdministratorsGuide,chapter 23January2013 EventManagementforZenossCore4Skills1stLtd 49
6.6.6fordocumentationandexamples.Thisparameterisratherobtuse.Fundamentally aWindowsQueryLanguage(WQL)queryisconstructedtoberunbyzeneventlog:
SELECT*FROM__InstanceCreationEvent WHERETargetInstanceISA'Win32_NTLogEvent' ANDTargetInstance.EventType<=zWinEventlogMinSeverity
AnyzWinEventlogClauseislogicallyAND'edwiththisWQL;thusifyouwanttoONLY seeeventswitheventidof528and529(SuccessfullogonandLogonfailure),configure zWinEventlogClausetobe:

(TargetInstance.EventCode=529orTargetInstance.EventCode=528)
Strictly,thezeneventlogdaemonpollstargetWindowssystemsforeventsandparses themintoZenossstyleevents.Typically,theSourcefieldontheWindowseventmapsto thecomponentfieldintheZenossevent;theZenosseventClassKeyiscomposedofthe Windows<Source>_<EventID>(eg.Perflib_2003);theZenosseventGroupbecomesthe Windowslogfilename(Application,Security,etc)andtheWindowsEventIDismapped totheZenossntevidfield. Toseetheworkingsofzeneventlog,changetheloggingleveltoDebug(10),restartthe daemonandinspect$ZENHOME/log/zeneventlog.log. AgoodwaytoseetheWQLstatementbeingusedistorunzeneventlogasaoneoff commandintheforeground:
zeneventlogrunv10dwin2003.class.example.org
Figure26Partialoutputfromzeneventlogrunv10dwin2003.class.example.orgshowingWQLstatement
50
23January2013
ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbutthey mayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActionevent zPropertysettohistorysothattheydonotappearinthestatustableoftheevents database.
5.2 Management of Windows systems using syslog

ThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsulting athttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailable undertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableas chargeableproducts.ThismeansthatWindowseventlogscanalsobecollectedwith thezensyslogdaemon. NotethattheSyslogagentiscapableofbeingconfiguredtomonitorWindows applicationlogfiles,inadditiontothestandardWindowseventlogs.Whenmonitoring thestandardeventlogs,therearebetterfilteringcapabilitieswithSyslogthenwith zeneventlog.
6 Event Mapping
ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhichare definedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.Theprocessof EventClassMappingisaboutassociatinganincomingeventwithaparticularZenoss EventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthat eventbyusinganeventtransform. Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclass mapping.Theclasshierarchycanbeusefulinthatevent context,asimplementedby eventzProperties(zEventSeverity,zEventAction,zEventClearClasses),followsthe normalrulesforobjectinheritanceifzEventActionissetto dropontheevent class/Ignore,thenanysubclassesof/Ignorewillalsoinheritthatproperty. NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdrop incomingeventstotally;/Archiveclassesandsubclassesautomaticallysetthe eventStatefieldtoClosed. Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknown asinstances.Notethataneventdoesnothavetohaveanymappingsassociated,in whichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemon thatgeneratestheevent,assignstheeventclassatthattime( /Perfeventsmaywell comeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin $ZENHOME/Products/ZenModel/data/events.xml.Theycanbeinspectedfromthe ZenossGUIbyselectingtheEVENTS>EventClassesmenu.
23January2013
51
MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfield whichispopulatedbythenativeeventparsingmechanism(suchaszensyslog, zeneventlog,zentrap).Thesemechanismsmaygenerateseveraldifferenteventswith thesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetween sucheventsandpotentiallytoseparatethemintodifferenteventclasses. Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretested againsttheincomingeventlowestnumbersaretestedfirst.Dependingonwhich mappingactuallymatches(ifany)willdeterminetheresultingeventClassoftheevent.
6.1 Working with event classes and event mappings

Eventsareorganisedinanobjectorientedhierarchy;thusattributesassignedtoa parenteventclassareinheritedbyachildeventsubclass. Neweventclassescanbedefinedbynavigatingtoaneventclassandusingthe dropdownmenualongsideSubClassestoAddNewOrganizer.Thenamesuppliedisthe nameoftheneweventclass.Forexample,drilldowntothe/Securityeventclassand createanewsubclasscalledSu. Anyeventwhichdoesnotmaptoaneventclassisthegiventheclassof/ Unknown.The simplestwaytomapsuchaneventistostartfromanexistingeventintheEvent Console.Thefollowingscenarioexplainsthis,creatinganeweventclassmappingcalled suwhichmapsanincomingeventtotheeventclass/Security/Su. 1. GenerateasyslogauthenticationfailureeventattheZenosssystem. 2. OpenanEventConsolethatshowstheeventandinspectitsdetails. 3. SelecttheeventandusetheReclassifyEventiconatthetopoftheconsole.Select yournew/Security/Suclassfromthedropdownlist.Youshouldbeshownthe eventclassmappingpanel.ClickthelefthandEditmenu. 4. Youshouldfindthatthenameoftheneweventclassmappingissettos uand theEventClassKeyissettosu(notelowercasesinbothcases).The eventClassKeyfieldisactuallyderivedfromthecomponentfieldoftheincoming eventinSyslogProcessing.py(aroundline289).Thesummaryfieldoftheevent shouldhavebeencopiedintothemappingExamplebox. 5. AddatextstringtotheExplanationboxsuchasAutoaddedbyeventmapping. 6. AddatextstringtotheResolutionboxsuchasThisisadummyresolution. 7. OpenaZenossGUIwindowthatshowsallSuevents(youmayfinditusefulto haveseveralbrowsertabsopentofocusondifferentaspectsoftheZenossGUI). SelectalltheSueventsandClosethem. 8. GenerateanewSuevent. 9. CheckthedetailsoftheneweventintheEventConsole.Theeventshouldhave mappedtoeventClass/Security/Su.TheseverityshouldbeInfo(blue).The
52
23January2013
detailsoftheeventshouldshowtheeventClassMappingfieldsetto /Security/Su/su. Anyexistingeventmappingcanbemodifiedinasimilarfashion.
Figure27:Editdialogueforeventclassmapping
Wheneveryouchangeaneventmapping,itisadvisabletoclearanyexistingeventsof thatcategorybeforetestingthenewconfiguration. Whenyouareworkingwitheventmappings,don'tforgetthe Eventmenuwhichfilters anEventConsolebyEventClass. Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofa page,suchas/Events/Security/Su.
23January2013
53
6.1.1 Generating test events

TesteventscanbecreatedfromtheEventConsoleusingthe+icon.
Figure28:Dialoguetocreateatestevent
Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouare thezenossuser).Thistakesparameters: d device p component k eventClassKey s severity c eventClass y eventKey i IPaddress h help o <field>=<value>(foranyotherattribute;canhavemultipleo) monitor collectorthiseventcamefrom port=PORT defaultis8081 server=SERVER defaultislocalhost auth=AUTH defaultisadmin:zenoss Theremainderofthelineaftertheseoptionsisusedforthesummaryfield (strictlytheMessagefieldintheGUIdialoguepopulatestheevent summaryfield) ThecoreautodeployscriptdeliveredwithZenoss4.2.3hasnewfunctionalitytoincrease securityonaZenossinstallation.FormanyyearstheZenossuserofadminwitha passwordofzenosshasbeenconfiguredasstandard.Thenewinstallationscript changesthis,generatingarobustpasswordwhichisstoredinseveralconfigurationfiles in$ZENHOME/etc,includingglobal.confandhubpasswd.
54
23January2013
zensendeventisastandalonePythonutilityin $ZENHOME/binthatcommunicates withthezenhubdaemon.Noteintheusagedescriptionabove,thatthedefault auth parametervalueisadmin:zenoss;typicallythismeansthatzensendeventcommandswill failwithanUnauthorizedmessageunlessthe authparameterisaddedwiththe correctuserandpassword,foundin$ZENHOME/etc/hubpasswd. Adiscussiononmodifyingzensendeventtoautomaticallylookupthecorrect authenticationparameters,canbefoundontheZenosswikiat http://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3 ThecodeissuppliedinAppendixA.
6.2 Regex in event mappings

TheRegexelementofaneventclassmappingcanbeusedtoparsethesummaryfieldof theincomingevent,whichispresentedbytheparsingdaemon(zensyslog,zeneventlog, zentrap).TheRegexelementusesthePythonformatforregularexpressionsandcan usethePythonnamedgroupsyntaxtonotonlycheckforliteralstringsbutalsoto defineregularexpressionsforvariablepartsofastring,andassociatethatvariablepart withaname.VariablepartsofthestringarecapturedintoPythonnamedgroups thismeansthat:

Youcanhaveoneexpressionmatchlotsofsimilarbutdifferentincomingevents Thevariablepart(typicallybetweenthe(?Pand\S+))canbepassedtotherest oftheeventprocessingmechanismasanamedfieldoftheevent. Thus,intheproductshippeddropbeareventmappingfor/Security/Login/Fail, theRegexisasfollows:

exitbeforeauth$user'(?P<eventKey>\S+)',(?P<failures>\S+)fails$:Maxauthtriesreached
(?P<eventKey>\S+)willparsethecharactersafteruser'uptothenext singlequoteandplacethatstringintotheeventKeyfieldoftheevent. Similarly(?P<failures>\S+)willparsethestringthatfollowsacommaand spaceandisendedbyspaceandfails,intoaneweventattributecalled failures. Matchingtheliteralstringrepresentingabracketrequiresthebackslash escapeorthebracketwillbeinterpretedasametacharacter. TherestoftheeventsummarymustmatchtheliteraltextintheRegex; however,othertextcanappearbeyondtheendaftertriesreached. TheExampleboxshouldshowsasampleeventsummarythatismatched bytheregularexpressionintheRegexbox.IfyouattempttoSavearegex thatdoesnotmatchtheexample,theregexfieldwillbeshowninred.
FormoreinformationonPythonregularexpressions,see http://docs.python.org/2/library/re.html. 23January2013 EventManagementforZenossCore4Skills1stLtd 55
SeeFigure29foranexampleofamorespecificmapping,su_root,fortheeventclass /Security/Su.Theregexisusedtoensurethatthesummaryhasthestring pam_unix(su:auth):authenticationfailure;followedbysomefixedandsomevariable elements.

pam_unix$su:auth$:authenticationfailure;logname=(?P<logonUser>\S+) uid=(?P<uuid>\d+)euid=(?P<euid>\d+)tty=(?P<tty>\S+)ruser=(? P<fromUser>\S+)rhost=\s+user=(?P<toUser>\S+)
Figure29:EventmappingdialoguewithRegexforauthenticationfailure
Theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfortheevent whichwillbeshowninthedetailsoftheeventandcanbeusedinanysubsequentevent transforms. Additionally,theConfigurationPropertyofzEventSeverityhasbeensettoWarningfor thismapping.
Figure30Eventdetailsforauthenticationfailureeventshowingneweventfieldscreatedbytheregex
56
23January2013
TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)are satisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,user definedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,the userdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclass willfail.Noeventtransformswilltakeplace.IfaRule doesexistandissatisfiedbut theRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclass mappingwillbesuccessfulandanymappingtransformwilltakeplace.
6.3 Rules in event mappings

TheRuleelementofaneventclassmappingusesPythonexpressionstotestany instantiatedfieldoftheincomingeventagainstavalue.Expressionscanbecomplex includingPythonmethodcallsandlogicalANDsandORs.Thedefaulteventfieldsthat aredefined,aregiveninAppendixD3oftheZenossCore4AdministrationGuide. Note thatsomeofthesefieldsarenotactuallyavailableateventmappingtimenotably evid,stateChange,count,dedupid,firstTime,lastTimeand eventClassMapping.
Figure31:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes
TheRuleelementcanalsousePythonexpressionstotestforvaluesofattributesofthe devicethatgeneratedtheevent.Someofthemethodsandattributesthatare availablefordevicesaredocumentedinAppendixD2oftheZenossCore4 AdministrationGuide,underthesectiononTALESexpressions(TemplateAttribute
23January2013
57
LanguageExpressionSyntaxispartofZope.ZopeistheapplicationserverthatZenoss isbuilton). TheRuleelementwillonlybeusediftheeventClassKeyfieldinthemappinghas achievedamatchwiththeincomingevent.Afterthat,ifaRuleexists,itmustbe satisfiedbeforethismapping(andhenceclass)isapplied.
6.4 Other elements of event mappings

TheExampleelementofaneventclassmappingisasamplestringthatisusefulwhen constructingaRegex.TheRegexwillturnrediftheRegexdoesnotmatchtheExample stringwhentheSavebuttonisused. TheExplanationandResolutionelementsofaneventclassmappingarestringsthat canbeconfiguredtoprovidefurtherinformationtoZenossusers.Theyappearinthe eventdetail.Notethattheseelementscanonlybeliteralstrings;theycannotuse eitherstandardoruserdefinedfieldsfromtheevent. ThecombinationofeventClassKey,RuleandRegexdeterminetheeventclassthatwill beassociatedwiththeincomingeventandwhattransforms(ifany)willtakeplace. Theremaystillbemultiplecombinationsofthesethatsatisfyanygivenincomingevent. Ifso,theSequencemenuisusedtodecidetheprecedenceofevaluationofmatching eventmappings.Themappingswillbetestedfromthelowesttothehighestsequence number.Onceamatchisfound,anysubsequentmappings(withhighersequence numbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteria willhavealowersequencenumber. Intheexamplesaboveforthe/Security/Suclass,thegenericsumappinghassequence number1andthemorespecificsu_rootmappinghassequence0. Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclass mappingcalleddefaultmappingwhichmusthaveaneventClassKeyof defaultmapping.Thereareatleast6mappings,allcalled defaultmapping,outofthe box.Eachmapstoadifferentclass.Adefaultmappingisaspecialcasethatisusedby theeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(note thatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).In thecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluates lookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibleto createnewmappings,eitherwiththenameofdefaultmappingor,indeed,witha differentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbers ofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.
7 Event transforms
Transformscanbeusedtomodifyfieldsofanevent,createnew,userdefinedfieldsor fieldscanberetrievedfromeventsalreadyintheMySQLdatabase.
58
23January2013
7.1 Different ways to apply transforms

YoucanhavesimpleassignmentsoffieldvaluesorsetthembasedoncomplexPython programs.Thetransformmechanismcanbeappliedintwoways:

eventclasstransforms eventclassmappingtransforms
PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectly tothatexacteventclassbytheparsingmechanism(zenping,zenperfsnmp, zencommand,AddEventwithEventClassspecified,etc).Ifatransformexistedinan eventclassmappingthatwasused,theeventclasstransformwasnotused. Zenoss2.4introducedcascadingeventtransforms.Thischangedthingsintwoways. Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalready hasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1 transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseevent classisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping, t1,thentransformswillbeappliedintheorder:
Toptestclass>T1class>t1eventclassmapping
Itisperfectlypossibleforatransformtouseuserdefinedeventfieldsinstantiatedby earliertransforms;however,beveryawarethatifanystatementinatransformfails (perhapsbecauseafielddoesn'texist),thentheprocessingofthattransformwillstopat thatpointandnofurtherstatementswillbeexecuted.Anyfurthertransforms willbe executed(atleastuntilanerrorisreached). AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeen successfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,at transformtime,mostofthestandardeventfieldsareavailable, exceptthosepopulated atdatabaseinsertionstime(evid,stateChange,eventState,dedupid,count, eventClassMapping,firstTimeandlastTime).Anyuserdefinedfieldscreatedbythe Regexarealsoavailable. Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangethe classforeventsthatwouldotherwisebe/Unknown. Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist (likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored. Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue. Transformsareimplementedbythezeneventddaemonsoinspecttheendof $ZENHOME/log/zeneventd.logtoseetheerrormessagereportingtheabsenceofthe attribute. AclasstransformisconfiguredfromtheActioniconatthebottomofthelefthandmenu foraneventclass.
23January2013
59
Amappingtransformisspecifiedaspartofthesameeventmappingdialoguethat definestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,when youusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatingan error. Figure31onpage57showedaneventmappingcalledlinetestwhichincludesa transformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromthe eventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theevent summaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsand userdefinedfields.

evt.myDevId=device.id evt.mySnmpSysLoc=device.snmpLocation evt.mySnmpSysContact=device.snmpContact evt.mySnmpStatus=device.getSnmpStatusString() evt.summary="Problemis%sondevice%s.Pleasecall%s"%(evt.summary, evt.myDevId,evt.mySnmpSysContact)
Mostoftheuserdefinedfieldsareassignedtosimpleattributesofeithertheeventor thedevice;forexample,device.snmpContact.Thelinebeforetheenddemonstrates usingaPythonmethodtogetvalues;forexampledevice.getSnmpStatusString()(note the()attheendthisisthecluethatitisamethodratherthananattribute).
7.2 Understanding fields available for event processing

Sohowdoesoneworkoutwhatattributesandmethodsareavailable?TheZenoss Core4AdministrationGuidedocumentstheTALESEventAttributesinAppendixD3 butthisisonlyastartingpoint. Similarly,AppendixD2documentsTALESDeviceAttributesandmethodsbutthis informationisveryincomplete. Whenzeneventdisprocessinganevent,strictlyitisworkingonanumberofPython dictionariesthatmakeupaZepRawEventProxyobjectclass.Rememberfromthe architecturesectionthatzeneventdtakeselementsfromtheraweventsqueue,processes themandoutputstheresulttothezeneventsqueuetobefurtherprocessedbythe zeneventserverdaemon(Figure12,Zenoss4eventarchitecture).Themessagesonthe raweventqueue(likeallotherqueuemessages)areblobsofbinarydata. Thereareanumberofmodulesin$ZENHOME/lib/python/zenoss/protocolsthat manipulatethismessagedatausingGoogleprotobufsasadatainterchangeformatfor thestructuredqueuemessagedata. $ZENHOME/Products/ZenEvents/events2containsthreePythonfilesthatarecrucial forunderstandingthedetailsofhowzeneventdprocessestherawevent:

processing.py fields.py proxy.py
60
23January2013
$ZENHOME/Products/ZenEvents/zeneventd.pyhasanumberofpipelinesthatan eventpassesthrough.Theireffectcanbeseenbeanalysingzeneventd.logiftheDebug logginglevelisturnedon.
Figure32EventPipelineProcessorobjectclassinzeneventd.py
processing.pycontainsthecodetoimplementeachofthepipelinestagesexecutedby zeneventd.Therearemethodstoprocessesarawevent,adddeviceandeventcontext, processruleandregextoestablishaneventclass,andtoperformtransforms.Thereis alsoamethodtogeneratethefingerprintfield.
23January2013
61
Figure33EventFieldobjectclassin$ZENHOME/Products/ZenEvents/events2/fields.py
62
23January2013
$ZENHOME/Products/ZenEvents/events2/fields.pycontainsobjectclassdefinitions for:
EventField
TheEventFieldattributesmatchupwiththebaseMySQLdatabasefieldsin zenoss_zep. TheActor,DetailandTagfieldsaredefinedassubclassesoftheobject Hastheadditionalfieldsthatarepopulatedwhentheeventisinsertedinto thezenoss_zepdatabaseevent_summarytable.
EventSummaryField
Figure34EventSummaryFieldandZepRawEventFielddefinitions
ZepRawEventField
HasthesamefieldsasEventFieldbutalsohasclear_event_classasthatis neededbythezeneventdprocessingpipelinesasitispartoftheeventcontext.
Notethatthedefinitionsinfields.pyarenothelpfulwhendecidingwhatattributesare availabletotransforms;thesearethefieldsonefindsinthezenoss_zepdatabase.
7.2.1 Event Proxies

$ZENHOME/Products/ZenEvents/events2/proxy.pyisthekeytounderstandingwhat attributesareavailablewhenwritingrulesandtransforms.proxy.pyprovides
23January2013
63
translationsbetweenencodedformatsofeventsandahumanreadableJSON (JavaScriptObjectNotation)format. Asfaraspossible,theattributespresentedbyaproxyarethesameinZenoss4asthey wereinpreviousversions.
Figure35EventProxydefinitionin$ZENHOME/Products/ZenEvents/events2/proxy.py
64
23January2013
AnEventProxyisseveralPythondictionaries:

Themainbodyoftheeventisadictionarycalled_event Adetailsdictionary An_tagsdictionary Adictionaryfor_clearClasses Adictionaryfor_readOnlyattributes
TherearealargenumberofPython@propertydecoratorconstructswhosepurposeisto presentanattributeusingamethod,forexample:
@property defdevice(self): returnself._event.actor.element_identifier
definesanattributecalleddevicewhichisdeliveredbyamethodthatreturnsthe valueoftheevent'sactor'selement_identifier.deviceisthefieldthatwehave(have alwayshad)tomanipulateintransforms. The@propertydefinitionsattheendofFigure35showsimplerdefinitionsthatreturn thevalueofabasicfieldofanevent(usingtheEventFielddefinitionsdefinedin fields.py). WhenauserviewseventdetailsusingtheZenossGUIoraccessesdatafromfromthe event_summarytableofthezenoss_zepdatabaseusingtheJSONAPI,theeventdata presentedisanEventSummaryProxy,whichisaJSONformat.The EventSummaryProxyinheritsfromtheEventProxybutalsohasattributesthatare addedondatabaseinsertion:

evid stateChange clearid firstTime lastTime count ownerid eventState
TheEventSummaryProxywasoriginallydesignedwithanideaofkeeping allevent data,treatingduplicatesasmultipleoccurrenceswithintheEventSummaryProxy; howeverthescalabilitywasnotfeasibleso,inpractisethefieldsofaneventareinthe zero'thelementofanEventSummaryoccurrencelist.
23January2013
65
Figure36EventSummaryProxyobjectclass
proxy.pyalsodefinesaclassforZepRawEventProxywhichinheritsfromEventProxy. TheadditionalpropertiesforZepRawEventProxyarefor_ClearClasses,_actionand eventClassMapping. Itistheattributesdefinedinproxy.pyfortheZepRawEventProxyobjectclassthatare availableforuseinrulesandtransforms.
7.2.2 Event Details

Sowhathappenstoauserdefinedeventattributegenerated,say,bythevarbindsthat comeinonanSNMPTRAP? RememberthattheEventProxyhasanumberofdictionaries,includinga details dictionary.ExaminationoftheEventProxyobjectclassinproxy.pyshowsthatany 66 EventManagementforZenossCore4Skills1stLtd 23January2013
fieldsthatdon'tmatchthestandardfieldsareputin<name>,<value>pairsinthe event'sdetailsdictionary.
Figure37Processingeventdetailsinproxy.py
Theevt.detailsdictionaryisavailableasan EventDetailProxyobject(alsodefinedin proxy.py).
23January2013
67
Figure38EventDetailProxyobjectclassinproxy.py
Toaccessthesedetailsinaruleortransformtheycanbereferredtoasevt.<nameof detailfield>ifthenamedoesnotincludea.(dot);otherwisetousethesedetailsina ruleortransform,theyneedtobeaccessedthroughthe_mapdictionary.
7.3 Transform examples

7.3.1 Combining user defined fields from Regex with transform
Inthisexample,wewillreturntothe/Security/Susubclassofeventsandcombine regularexpressionsandtransforms.Theobjectiveis,forimportantdevices,toescalate theeventseverityifausertriestosutorootbuttodecreasetheseverityifthesuevent comeseitherfromanunimportantdeviceorifthesuistoaparticularuserid( student inthiscase).ImportantdevicesaredeterminedbytheeventfieldDevicePriority(note twocapitallettersinthisfieldname).Thedevicepriorityforadevicecanbechanged fromtheOverviewmenuonadevice'sdetailspage. ThisexampleisthesameasshowninFigure29butatransformhasbeenadded.
68
23January2013
Figure39:su_rooteventmappingwithtransform
NotethattheStatusmenuofamappinglosesanyPythonindentationsyouhave carefullycreated!Thetransformshouldbeenteredas:
ifevt.toUser=='root'andevt.DevicePriority>2: evt.severity=5 elifevt.toUser=='student'orevt.DevicePriority<3: evt.severity=1 evt._action='history'
TheuserdefinedfieldtoUser,createdbytheRegex,istestedagainsttheliteralstring 'root'.TheresultislogicallyAND'edwithatestofthestandardeventfield DevicePriorityfor>2.IftheresultisTruethenthestandardeventfield severityisset to5(Critical).Rememberthatthedefaultseverityforthesu_rootmappingwassetto WarningbythezEventSeverityeventcontextzProperty. Intheelifstatement,ifthisconditionisTruethentheevent'sseverityissetto 1(Debug) andthezPropertyzEventActionisoverriddenbysettingevt._action='history'inthe transform.Inthiscase,theevent'seventStateissettoClosed. NotewithanyPythontestthatincludesmultipleclauses,thetestfailsassoonasa conditionfailssointheifstatementifevt.toUserisnot'root'thenevt.DevicePrioritywill notbeevaluated.Performancecanbeimprovedbycarefulconsiderationandorderingof suchtests.
7.3.2 Applying event and device context in relation to transforms

Eventcontext(zEventSeverity,zEventAction,zEventClearClasses)isappliedthrough theConfigurationPropertiesmenuofaneventclassoreventclassmapping. DevicecontextcomprisestheeventfieldsprodState,Location,DeviceClass, DeviceGroups,SystemsandDevicePriority.ipAddressandthemonitor (collector)responsiblefortheeventalsotendtobebracketedwiththedevicecontextbut theselatterfieldsareinformationreceivedontheincomingevent,ratherthanthe devicecontextdatathatislookedupintheZopedatabase. 23January2013 EventManagementforZenossCore4Skills1stLtd 69
Thefollowingdevice_contexteventmappingexampledemonstratestheorderinwhich devicecontext,eventcontextandthemappingtransformareapplied. Createaneweventsubclass,Device_context,underthe/Skillsclass. Createamapping,device_context,forthisneweventclass.Ensurethatthe eventClassKeyisdevice_context.SettheRegextotheliteralstring:

Thisisadevicecontextevent
SetaRuleasfollows(allononeline):
getattr(evt,'Location','')=="/Kandersteg"andgetattr(evt,'_action','') =="status"and'/Skills'notinevt._clearClassesandgetattr(evt, 'severity','')>4andnotevt.component
UsingtheConfigurationPropertiesmenuforthemapping,setthezEventSeverityevent contextvaluetoError(4),zEventActiontohistoryandzEventClearClassesto/Skills. Testthemappingwithazensendevent(allononeline):

zensendeventdgroup100r1.class.example.orgsCriticalk device_contextThisisadevicecontextevent1
Thetesteventsetthedevicefieldtogroup100r1.class.example.orgwhichisincludedin theLocationcalled/Kandersteg.TheeventClassKeyshouldbesettodevice_context,the componentfieldshouldbeblankandtheeventClassshouldbeblank.
Figure40:CombiningaRule,contextandatransformforthedevice_contexteventmapping
TheRuledemonstratesthePythongetattrfunctiontotest:
Theevt.Locationfieldsetbydevicecontext,whichshouldevaluateTRUEatRule timeie.devicecontexthasbeenapplied Theevt._actionfieldthatissetbyeventcontexttohistory.Thetestshownabove actuallyevaluatesTRUEshowingthateventcontexthasnotbeenappliedat Ruletime. EventManagementforZenossCore4Skills1stLtd 23January2013
70
Similarly,theevt._clearClassesfieldtestevaluatesTRUEshowingthatevent contexthasnotbeenapplied.ThePythonsyntaxforcheckingevt._clearClassesis alittledifferentasthisattributeisdefinedasaPython listratherthanastring. Theevt.severitystartsat5inthegeneratedeventandeventcontextsetsitto4. ThistestevaluatesTRUEconfirmingthateventcontexthas notbeenapplied. Theevt.componentmustbeblank(thenullstringevaluatestothebooleanFalse) Notethatthesyntaxforthelastfieldofthegetattristwosinglequotestosupply anulldefault
Insummary,theRuleandRegexshouldevaluatesuccessfullyandthetransformwillbe applied. Thetransformdemonstrates:
Changingtheevt.severityfieldagainitwouldhavebeenmodifiedfromthe originalvalue(5)downto(4)whentheeventcontextwasappliedafterRuleand Regexprocessing.Thetransformchangesitto3. Changingtheevt.componentfieldisinteresting.Rememberthatthefingerprint dedupidfieldincludesthecomponent.Althoughtheraweventdid notincludea componentfield,thefingerprintisgeneratedafterthetransformasthededupid intheeventdoescontainthecomponent. Severaluserdefinedvariablesarecreated.Theevt.myClearClassesline demonstratesthatalluserdefinedfieldsappeartobeoftypestringbut evt._clearClassesisdefinedasaPythonlist.Youcannotassign evt.myClearClassestosomethingoftypelistunlessyouusethe joinfunctionto sticktogetherthelistelementsbackintoastringtype. Theuserdefinedfieldsdemonstratethatbothdevicecontextandeventcontext havebeenappliedbytransformtime
8 Testing and debugging aids

8.1 Log files
8.1.1 zeneventd.log
Devicecontext,eventcontext,rule,regexandtransformsareallappliedbythe zeneventddaemon.Italsoconstructsthededupidfingerprintfield.Seetheevent processingpipelinecodeforzeneventdinFigure32onpage61. Turningupthedebugflagin$ZENHOME/etc/zeneventd.confto10(Debug)providesan opportunitytotracktheprogressofeachofthestagesinthispipelinein $ZENHOME/log/zeneventd.log,notingthattheeventgainsmorefieldsasprocessing continues.
23January2013
71
zeneventd.logisalsotheplacetolookforproblemswitheventprocessing.Evenwiththe usualdebuglevelof20(Info),rule,regexandtransformproblemsarehighlighted. SearchforWARNINGinthelog. Thefollowingextractshowsatransformattemptingtochangeevt.Location(which appearsnottobeallowed).Notethatalthoughthemessageisdefinitelyhelpful,its ideasaboutlinenumbersarewayout!

2012122010:02:01,923WARNINGzen.Events:Errorprocessing transform/mappingonEventClass /Skills/Device_context/instances/device_context Problemonline475:AttributeError:can'tsetattribute Transform: 0evt.Location='/Taplow' 1evt.severity=3 2evt.myProdState=evt.prodState 3evt.myDeviceClass=evt.DeviceClass 4evt.myDeviceGroups=evt.DeviceGroups 5evt.mySystems=evt.Systems 6evt.myAction=evt._action 7evt.myClearClasses=''.join(evt._clearClasses)
WithZenoss4,youwillalsoreceiveaneventfromtheZenossserverwithsimilar information(andequallycreativelinenumbers!).WithversionsofZenosspriorto4 therewasnowarningeventandalltheeventprocessingwasperformedbyzenhubso zenhub.logwastheplacetosearchforerrors.
8.1.2 zeneventserver.log
ThezeneventserverdaemoniswritteninJava.Thismeansthaterrormessagesare difficulttocomprehendin$ZENHOME/log/zeneventserver.logwithoutanintimate knowledgeoftheJavacode. Whatisusefultohelpunderstandingofthearchitectureistoinspectthislogaround daemonstartup.
72
23January2013
Figure41:zeneventserver.logshowingdaemonstartup
InFigure41lineshighlightedinyellowshowEventManagerconfigurationparameters thatcanbecheckagainsttheADVANCED>Settings>Eventsmenu.

Maximumarchivedays:1000 Startingeventageingatinterval:60000milliseconds(s) Startingeventarchivingatinterval:60000milliseconds(s) Startingdatabasetableoptimizationatinterval:60minutes(s)
LineshighlightedingreenshowoperationsassociatedwiththeMySQLdatabaseand theassociatedLuceneindexes.
23January2013
73
Figure42:EventManagerparametersthatmatchwithzeneventserver.logstartuplog
LineshighlightedinredareinteractingwithRabbitMQAMQPsystem.Thefirst sectionshowszeneventserverconnectingtotheMQsubsystem;ifthisisunsuccessful thenmanyoftheZenossdaemonswillfail. Thesecondsectionshowsthethreadsstartinguptoconsumethevariousqueuesthat zeneventserverprocesses.

zenoss.queues.zep.zenevents zenoss.queues.zep.modelchange zenoss.queues.zep.heartbeats zenoss.queues.zep.migrated.summary zenoss.queues.zep.migrated.archive
Notethatyouwouldnotexpecttoseezeneventserverworkingon zenoss.queues.zep.raweventstheconsumerofthatqueueisthezeneventddaemon. Lineshighlightedinlightbluearesubsequent,periodicoperationsbyzeneventserver performingmaintenanceontheMySQLdatabase.Thelogshowsaneventtable partitionbeingprunedeveryhourandanewonebeingcreated,asasectionofevents areagedintotheevent_archivetable.
74
23January2013
8.1.3 Other log files

Otherlogfilesthatmayhaveabearingoneventsare:

zenhub.log event.log zenperfsnmp.log zenwinperf.log zencommand.log zensyslog.log zeneventlog.log zentrap.log
interactionsbetweendaemons )moreusefulprior problemsseenbyevent.log )toV4foreventissues issueswithperformancedataandthresholdevents issueswithperformancedataandthresholdevents issueswithperformancedataandthresholdevents daemonthatreceivessyslogevents daemonthatreceivesWindowsevents daemonthatreceivesSNMPTRAPs
8.2 Using zendmd to run Python commands

ZenossprovidesaPythoncommandlineinterface,zendmd,wherecodefortransforms canbetestedoutandtheattributesandmethodsavailablecanbeexplored. Notecarefullytheindentationofstatements.Pythonisveryparticularabout indentationtointerpretstructuresuchasforloops.Itdoesn'tmatterhowmanyspaces youindentthebodyoftheforloopbutitmustbeindentedfromtheforlineandeach lineinthemainbodyofthatforloopmusthavethesameindentation.Thebodyofafor loop,insideaforloop,wouldindentfurtherandsoon. Youshouldrunzendmdasthezenossuser.ThissectionisnotsupposedtobeaPython tutorial;thatsaid,hereareacoupleoftrickswithzendmd. Notethatthesetechniquesforaccessingeventshavechangedsubstantiallybetween previousversionsofZenossandZenoss4.
8.2.1 Referencing an existing Zenoss event for use in zendmd

Ifyouwanttoexploretheattributesandmethodsavailableforaneventorthedevice thatgeneratedtheevent,usingzendmd,youneedawaytoreferenceanevent.When executingatransform,theseobjectsaremadeavailabletoyouautomaticallyasthe evt variableandthedevicevariablebutinazendmdtestenvironmentyouneedtosupply these. WithearlierversionsofZenosstherewasamethodontheZenEventManager, getEventDetailFromStatusOrHistory,whichtookasaparameterthestringvalueofa uniqueevidanddeliveredanEventDetailobject(seeFigure43). Tofindtheevid,simplydisplayanappropriateeventintheEventConsole,bringupthe detaileddata,andcutandpastetheevidvalueintothestatementinzendmd.
23January2013
75
Figure43:UsingzendmdtosettheevtvariabletoanexistingZenosseventZenosspriortoV4
WithZenoss4,itisalittlemorecomplex.Wereallyneedtogetbacktothe ZepRawEventProxyformattotesttransformcode,butthatisnolongeravailablethe datafromtheraweventqueueisgone. WhatwedohaveaccesstoistheeventintheMySQLdatabase;howeverwedon'twant itwithdatabasestyleattributes,wewantEventProxyattributes.
76
23January2013
Figure44:UsingzendmdtoretrieveaneventfromtheMySQLdatabase,converttoan EventSummaryProxyandextractvariousfields
$ZENHOME/Products/Zuul/facades/zepfacade.pyprovidesanumberofutilitiesto accessdatafromthezenoss_zepdatabaseandmanipulateit,typicallyprovidingJSON formatdata. Figure44demonstratesusingzendmdtoaccesseventsintheMySQLdatabase,convert themtoEventSummaryProxyformatandthenprintoutvariousfields.

zep=getFacade('zep') providesaccesstothezenoss_zepdatabase evt=zep.getEventSummary('000c29d9f87b94fb11e2494936a92109') RetrievestheeventwiththespecifieduuidtheresultisinJSON rawevt=EventSummaryProxy(from_dict(EventSummary,evt)) TheEventSummaryProxyclasstakesaprotobufstyleeventasparameter, nottheJSONstyleeventwecurrentlyhave.Usefrom_dicttoconvert fromJSONtoprotobuf rawevt.device standardattribute rawevt.myLineNum attributefromdetails REMEMBERthatthisisanEventSummaryProxy,notaZepRawEventProxysoyou haveaccesstofieldsthatarenotavailableattransformtime(likecount, firstTime,...) evt theJSONformatevent(dictionary)
TheJSONstyleeventsareveryhardtoreadasshownabove.zendmdunderstandsthe pprintmethodtoprettyprintcomplexstructures.Itcanbeusefultocapturetheoutput ofpprint(evt)intoafileandthenusethevieditor%techniquetohelpmatchopening andclosingbrackets.
23January2013
77
Figure45Firstpartofzendmdpprint(evt)commanddisplayingsummaryeventinJSONformat
RememberthatFigure45andFigure46areshowingtheJSONstyleevent,notthe EventSummaryProxythatdeliverssuitableattributesfortransformmanipulation.
78
23January2013
Figure46Secondpartofzendmdpprint(evt)commanddisplayingsummaryeventinJSONformat
8.2.2 Using zendmd to understand attributes for an EventSummaryProxy

AnEventSummaryProxyisanobjectclassrepresentingaZenosseventitisaPython dictionarydatatypeadatastructureof<key>,<value>pairs.Toseewhatkeys (attributes)areavailable,usethemethodshowninthefollowingfigure.Builtin methodsstartingwithadoubleunderscorearedeliberatelyexcluded.
23January2013
79
Figure47:Usingzendmdtoprinteventattribute<key><value>pairs(partiallisting)
Thesearetheprimaryeventfieldsthatareavailabletouseinatransform (rememberingtoalsoexcludethosethatdon'texistatraweventtimeeg.count, firstTime,eventState,...). Notethatsomeofthedictionaryelementsarethemselvesdictionarieseg. details.To findoutwhatthedetailsattributesare,seeFigure48.RememberfromFigure38,that theEventDetailProxyclasshasan_mapdictionarywithname,valuepairsinit.
80
23January2013
Figure48zendmdtodisplayeventdetailsdictionaryname,valuepairs
ThegetmethodofEventDetailProxydeliversvalueswhenthatitemisa single,scalar value.Iftheitemhasmultiplevalues,alistforexample,thenthegetmethodbreaksas shownaboveonthezenoss.device.systemsattribute.Notethatitgetsawaywiththe zenoss.device.groupsattributebecause,althoughadevicemaybeinmultiplegroups,in thiscasethedeviceisonlyinasinglegroup,whereasitisamemberoftwoSystems. ThisisalsoechoedintheEventDetailsoftheZenossGUI.
23January2013
81
Figure49ZenossGUIEventDetailsshowingoneinstanceofzenoss.device.groupsand2instancesof zenoss.device.systems
Ifaneventdetailsattributeisnotascalar,usethe getAllmethodratherthantheget method.Forexample:

>>>printlist(rawevt.details.getAll('zenoss.device.systems')) [u'/Test',u'/Real'] >>>
AlsonoteinFigure48thatuserdefineddetailattributescansimplybereferredtoas rawevt.mySummaryorrawevt.mySnmpSysLocbutyoucannotrefertodetailfieldsthat containa.(dot)inthiswaythusexcludingthedefaultdetailsattributes(thosestarting withzenoss.)andexcludingSNMPTRAPvarbindfieldsthattypicallycontainadot;use thegetandgetAllmethodstoaccesssuchdetailfields.
82
23January2013
8.3 Using the Python debugger in transforms

AverypowerfulaidwhendebugginganyPythonistousethePythonDebugger, pdb. Seehttp://docs.python.org/2/library/pdb.htmlfordetaileddocumentation. pdballowsyoutobreakexecution,displaythestateofobjectsandtheirvaluesandstep throughthecode.Whenusedintransforms,thismeansrunningzeneventdinthe foregroundindebugmode(sodefinitelynotatechniqueforuseinproduction). Whenusingpdbtoexaminetransforms,itisnoteasytostepthroughthetransformcode (usingstostepornfornext)asyouendupnestedmanylayersdeepinthemethodsof thezeneventdcode;howeveritisveryusefultoexaminethestateoftheevent( evt)and alsoexplorethedevice(device). Ifyouaredoingthis,youmaywishtoreducetheZenosssystemtoaminimumsetof daemonstoavoideventsfromlotsofothersources. If$ZENHOME/etc/DAEMONS_TXT_ONLYexiststhentheonlyZenossdaemonsthat willbemanipulatedbyazenossstart/zenossstop/zenossstatuswillbethoselistedin $ZENHOME/etc/daemons.txt.Aminimumsetofdaemonswouldbe:

zeneventserver zopectl zeneventd zenhub zenjobs zenactiond
WhenyouhaverestartedZenoss,gotoADVANCED>Settings>Events,scrolltothe bottomofthepageandclickClear.Thispreventstheheartbeatfromperiodically checkingallthosedaemonsthatarenowdownandgeneratingheartbeatevents.. Toputabreakpointatthestartofatransform,addthefollowingline:

importpdb;pdb.set_trace()
Stopthezeneventddaemonandstartitintheforegroundindebugmode:
zeneventdstop zeneventdrunv10
Generateaneventthatwilltriggerthetransform;forexample:
zensendeventdzen42.class.example.orgsErrorklinetestplinetesttestline24
Inthezeneventdforegroundwindowyoushouldseeapdbprompt.Youshouldnowhave accessto:

evt device
aZepRawEventProxyobject aDeviceobject
23January2013
83
Figure50:pdbdialogueinzeneventdforegroundgeneratedbypdb.set_trace()intransform
Figure50demonstratesexploringsomeoftheattributesofbothevtanddevice.Note thatenteringasimplecarriagereturnrepeatsthepreviouspdbcommand. cinpdbcontinuesexecution. ToseelegalattributesandmethodsfortheDeviceobject,examinethe Deviceclass definitionin$ZENHOME/Products/ZenModel/Device.py. pdbdoesnothavethepprintmethodseeninzendmdbutitdoeshaveanequivalent pp utility.Forexample,toprintallprimaryeventfields,excludingbuiltinmethods,use:

pp[xforxindir(evt)ifnotx.startswith('__')]
84
23January2013
Figure51:Usingpdbtoprettyprintallprimaryeventfields
Toshowdetailfields:
ppevt.details._map.keys()
23January2013
85
Figure52:Usingpdbtodisplaydetaileventfields
Toprintascalarvalueforadetaileventfield,try:
(Pdb)printevt.details.get('zenoss.device.device_class') /Server/Linux (Pdb)printevt.details.get('mySummary') ThisisNOTagoodnews/badnewseventtestline31 (Pdb)
Toprintanonscalar(alistforexample):
(Pdb)printlist(evt.details.getAll('zenoss.device.systems')) [u'/Test',u'/Real']
Anattempttoprintalldetailfieldnamesandvaluesmightbe:
pp[(v,evt.details.get(v))forvinevt.details._map.keys()] ***Exception:Exception(u'Detailzenoss.device.systemshasmorethanone valuebuttheoldeventsystemexpectsonlyone: <google.protobuf.internal.cpp_message.RepeatedScalarContainerobjectat 0x5e01ef0>',) (Pdb)
Thiscomesupagainsttheproblemdescribedinthezendmdsectionwherethe get methodfailswithnonscalarvalues.Apartialcircumvention,giventheknowledgethat noneoftheuserdefinedvariablesarenonscalar,wouldbe:

(Pdb)pp[(v,evt.details.get(v))forvinevt.details._map.keys()ifnot v.startswith('zenoss.device')] [('mySummary',u'ThisisNOTagoodnews/badnewseventtestline31'), ('eventClassMapping',u'/Skills/linetest'), ('line_num',u'31')] (Pdb)
PerhapsabettersolutionistoacceptallvaluesaslistsandusethegetAllmethod, whichthenworksforalleventdetailsname,valuepairs.
(Pdb)pp[(v,list(evt.details.getAll(v)))forvinevt.details._map.keys()]
[('mySummary',[u'ThisisNOTagoodnews/badnewseventtestline31']), ('eventClassMapping',[u'/Skills/linetest']), ('zenoss.device.location',[u'/Taplow']), ('line_num',[u'31']), (u'zenoss.device.ip_address',[u'192.168.10.42']), ('zenoss.device.groups',[u'/Skills1st']), ('zenoss.device.device_class',[u'/Server/Linux']),
86
23January2013
('zenoss.device.production_state',[u'1000']), ('zenoss.device.priority',[u'3']), ('zenoss.device.systems',[u'/Test',u'/Real'])] (Pdb)
9 Zenoss and SNMP

9.1 SNMP introduction
TheSimpleNetworkManagementProtocol(SNMP)definesManagementInformation Base(MIB)variablesthatcanbepolledtoprovideperformanceandconfiguration information.TheSNMPstandardalsoprovidesforagentstosendeventstoamanager. Version1ofSNMPdefinestheseasTRAPs;versions2and3ofthestandardcallsthem NOTIFICATIONs(ZenosssupportsallthreeversionsofSNMP).BothMIBvariables andTRAPs/NOTIFICATIONsuseObjectIdentifiers(OIDs)todenotedifferent variablesandevents. SNMPTRAPsaredistinguishedbytheirEnterpriseObjectId(OID),thegenericTRAP numberandthespecificTRAPnumber. Natively,OIDsaredefinedasstringsofdotteddecimalsthatrepresentapaththrougha treebasedhierarchy,wheretherootofthetreeis1andrepresentstheISO organisation;ithasasubbranch,3,whichrepresentsorganisations(org);ithasasub branch,6,whichrepresentstheUSDepartmentofDefense(dod);ithasasubbranch,1, whichrepresentsinternet,andsoon.Thus,allOIDsstartwith1.3.6.1. Thereisastandard,MIB2,whichdefinesanumberofvariablesthateverySNMP capabledevicemustsupport;thesearelargelysimple,networkrelatedvariables,such asinterfaceInOctets.InadditiontoMIB2,therearealargenumberofstandardised MIBsdefinedinRequestForComment(RFC)documents;anexamplewouldbeRFC 1493definingthebridgeMIB. ThethirdcategoryofMIBsareknownasEnterpriseSpecific,whicharespecifictoa particularvendor'sparticularagentforexample,theCiscoFirewallMIB.Enterprise specificMIBsoftenincludedefinitionsofEnterpriseSpecificTRAPs,inadditiontoMIB variables. MIBsourcefilestranslatedotteddecimalOIDsintomoremeaningfultext.MIBfiles areavailableformanystandards(liketheHOSTRESOURCESMIB)and,typically,any supplierwhogeneratestheirownenterprisespecificMIBvariablesandTRAPs,should makeavailableasourceMIBfiletoaidthistranslation. SNMPagentstypicallycomeaspartofthebaseOperatingSystem(Windows,Unix, Linux,CiscoIOS);howevertheymaynotbeactivatedautomaticallyandwillrequire someconfiguration.SomeagentssupportlittlemorethanMIB2;otherssupportawide rangeofstandardMIBsandenterprisespecificMIBs. TheSNMPcommunicationprotocolvariesdependingontheversionofSNMP.Versions 1and2(strictly2c)useacommunitynamestringasanauthenticationmechanism 23January2013 EventManagementforZenossCore4Skills1stLtd 87
betweenSNMPmanagerandagent.Managersmustbeconfiguredwiththecorrect communitynamestouseforanagent;SNMPagentsmustbeconfiguredforwhich manager(s)areallowedaccesstothem,andwhichSNMPmanager(s)tosendTRAPsto. SNMPV3ismorecomplextoconfigurebutprovidesfacilitiesforstrongauthentication onSNMPpacketsandforencryptionofdataifsodesired. InadditiontorequestingMIB2variables,ZenosswilltrytoaccessthestandardHost ResourcesMIBtogetprocessinformationforservermachines.Itwillalsoattemptto accesstheWindowsInformantMIBforallWindowsserversystems,inordertogetCPU andfilesysteminformation.TheInformantMIBisafreeextensionsubagentandMIB availablefromInformantathttp://www.wtcs.org/informant/index.htm.Notethatthe baseWindowsSNMPagentshouldbeinstalledandconfiguredbeforeinstallingthe Informantextension. OnceSNMPagentsareconfiguredwithcommunitynameandTRAPdestination,a simplewaytotestthemissimplytorecycletheSNMPagent(indeedtheywillneed recyclingafteranyconfigurationchanges).OnaWindowssystem,usetheServices utilitytostopandstartSNMP;onaLinuxsystem,/etc/init.d/snmpdrestartwill usuallysuffice.IneithercaseyoushouldeitherseeacoldstartTRAP(genericTRAP 0)orawarmstartTRAP(genericTRAP1)intheZenossEventConsole.Theevent detailsshouldshowthecommunitynamefromtheTRAPpacket. AnothergoodwayofgeneratingTRAPsistoforceanauthenticationTRAP(generic TRAP4).Aneasywaytodothisistousethesnmpwalkcommandwithabad communityname.Ifthecommunityispublic,forahostsystemcalledzenoss,try:
snmpwalk -v 1 -c public zenoss system snmpwalk -v 1 -c fred zenoss system test with good community to generate several TRAP 4's
9.2 SNMP on Linux systems

MostLinuxsystemscomewithsomeflavourofthenetsnmpagent(formerlytheUCD agent).ManyLinuxdefaultconfigurationsforthisagentprovideverylimitedSNMP access.Thesnmpagentconfigurationistypicallycalledsnmpd.conf;thelocationofthis filevariesbetweendifferentLinuximplementationsbut/etc/snmpisacommonchoice. YouwillneedrootauthoritytomanipulatetheSNMPconfigurationanddaemon.
88
23January2013
Figure53:snmpd.conffornetsnmpagent
Figure53showsansnmpd.confthatconfiguresforSNMPV1andSNMPV2c,providing accesstotheentireMIB(theallview).TRAPs,includingAuthenticationTRAPs,are senttothezen42host.ThesysContactandsysLocationvariablesareset(theseare retrievedasstandardbyaZenossmodelerpoll). Thesnmpdagentshouldbestoppedandrestartedafteranychangestosnmpd.conf.

/etc/init.d/snmpdstop /etc/init.d/snmpdstart
23January2013
89
AsimplewaytotestthatTRAPsareconfiguredistogenerateanAuthenticationTRAP.
snmpwalkv1cpubliczen42system snmpwalkv1cfredzen42system testwithgoodcommunity togenerateseveralTRAP4's
Whereavailable,theV3oftheSNMPstandardshouldreallybeusedasitprovides strongauthentication(notjustacommunitynamethatpassesoverthenetworkinclear) anditalsoprovidesdataencryptionifdesired.Althoughslightlyhardertosetup,itis nottooonerous.Ontheagent,auseridmustbegeneratedwithparametersfor authenticationandencryption(privacy),specifyingtheencryptionalgorithmandthe encryptionpasswordtobeused.

#ForSNMPV3 #Uncommentnext5lines com2secsnmpv3testlocalhostdummycontext com2secsnmpv3testzen42dummycontext groupsnmpv3groupusmsnmpv3test #accesssnmpv3group""usmauthexactallallall
accesssnmpv3group""usmprivexactallallall
rwuserjane
# #rwuserjanecreatedbySTOPPINGSNMPDandrunning #netsnmpconfigcreatesnmpv3userafraclmyeaxfraclmyexXDESAMD5jane #/var/lib/netsnmp/snmpd.confismodifiedwith(hidden)encryptionkeyand #rwuserjaneisaddedtothisfile(/etc/snmp/snmpd.conf) #testwithfollowingifnoprivacy(dataencryption) #snmpwalkv3aMD5AfraclmyealauthNoPrivujanezen42system #or,withencryption #snmpwalkv3aMD5AfraclmyeaXfraclmyexlauthPrivujanezen42system # #Restartthesnmpddaemon #NotethatonCentOSnetsnmpdevelmustbeinstalledtoprovide #netsnmpconfig
ZenossmustalsobeconfiguredtohavematchingSNMPV3parametersforthisagent.
Figure54:ConfigurationPropertiesforagentwithSNMPV3
90
23January2013
NotethatthestandardsnmpwalkcommandfromtheCommandicondoesnotworkfor SNMPV3butitisrelativelyeasytocreateanewcommandfrom ADVANCED> Settings>CommandswhichrunsanappropriatesnmpwalkwiththeSNMPV3 parameterssubstituted.
Figure55:CreatinganewCommandoptiontorunsnmpwalkV3
NotethatdifferentimplementationsofnetsnmpondifferentOperatingSystemsmay workslightlydifferently.Forexample,OpenSuSEdoesnotneedthe netsnmpdevel packageandtherwuseriscreatedinaseparatesnmpd.confunder/usr/share/snmp (whichiscreatedautomaticallyifitdoesn'texist).
9.3 Zenoss SNMP architecture

9.3.1 The zentrap daemon
zentrapistheZenossdaemonthatprocessesincomingSNMPTRAPs.Bydefault, zentrapwillsitonthewellknowSNMPTRAPportofUDP/162thiscanbe reconfigured,ifrequired.BothSNMPversion1TRAPsandSNMPversion2 NOTIFICATIONsaresupported. zentrapprocessingisimplementedbythePythonprogram $ZENHOME/Products/ZenEvents/zentrap.py.
23January2013
91
Figure56:zentrap.pypart1checkingforextra0andprocessingofgenericTRAPs
zentrap.pyparsestheincomingSNMPProtocolDataUnit(PDU)toretrievethe enterpriseOID,thegenericTRAPnumberandthespecificTRAPnumber. ThealgorithmforinterpretingincomingTRAPEnterprisefieldshaschangedseveral timesovertheyearsbecausesomeagentshaveanextra0definedintheirMIBwhich theydonotsendonanactualTRAP(seethecommentsinthecodeinFigure56).In Zenoss4.2,thealgorithmfirsttriestofindaMIBintheZODBdatabasethat correspondswiththeincomingTRAP,withtheextra0;ifthisfails,thenapartial matchisattemptedwithouttheextra0(notethatthecommentinthecodeis inaccurate).Eitherway,theoidfieldoftheeventissettotheconcatenationofthe enterpriseandthespecifictrapnumber,withorwithoutthe0inthemiddle,depending ontheoutcomeoftheoid2namelookupfunction. ThegenericTRAPs(0through5)aretranslatedtostringssuchassnmp_coldStart. usingtheeventTypedictionary.ForspecificTRAPs(genericTRAP6),eventTypedelivers theconcatenationoftheenterpriseOIDandthespecificTRAPnumber;forexample, 1.3.6.1.4.1.123istheenterprise,thespecifictrapnumberis1234,so eventTypedelivers
92
23January2013
1.3.6.1.4.1.123.1234.AnyvariablesoftheTRAP(varbinds)arealsoparsedoutintoOID /valuepairsiftheMIBprovidesthistranslation. Theoid2namefunctionlooksupintheZODBdatabasetoseeiftranslationsare availablefortheenterpriseOID,thespecificTRAPnumberandthevarbindidentifiers, totranslatefromdotteddecimalnotationtotextualstrings.
Figure57:zentrap.pypart2eventfieldsettings
Thefollowingeventfieldsarethenset:

component eventClassKey eventGroup severity summary community firstTime lastTime monitor
leftblank settoeventType trap 3 snmptrapfollowedbyeventType settocommunitynamestring(thisisauserdefinedfield) settotimestamp settotimestamp settoCollectorthatreceivedtheTRAP
9.4 Interpreting MIBs

TohelpdecodeSNMPTRAPenterpriseOIDsfromdotteddecimal(suchas. 1.3.6.1.4.1.8072.4.0.2 )intoslightlymoremeaningfultext(likensNotifyShutdown)the zenmibcommandcanbeusedtoimportbothstandardMIBsourcefiles(suchas SNMPv2SMIwhichdefinesstandardOIDs)andvendorspecificMIBs.Thebase directoryforMIBsinlaterversionsofZenossis$ZENHOME/share/mibs. 23January2013 EventManagementforZenossCore4Skills1stLtd 93
ThezenmibcommandwithoutparameterswilltrytoimportallMIBfilesthatarein $ZENHOME/share/mibs/site.AspecificMIBfilecanbeprovidedasaparameter;the commandshouldeitherberunfromthe$ZENHOME/share/mibs/sitedirectory(in whichcaseafullpathnameisnotrequiredandthefileisexpectedtobeinthat directory)orafullyqualifiedpathnamecanbespecified.
9.4.1 zenmib example

Tohelpunderstandthezenmibcommand,hereisaworkedexample.Itusestheagent fornetsnmpwhichistheagenttypicallyshippedwithaLinuxsystem.Theenterprise OIDfornetsnmpis.1.3.6.1.4.1.8072. 1. Recycleanetsnmpagentwith/etc/init.d/snmpdrestart.Inadditiontothe genericcoldstartTRAP,youshouldalsoseeTRAP.1.3.6.1.4.1.8072.4.2.This comesfromthenetsnmpenterprise(.1.3.6.1.4.1.8072). 2. TheactualTRAPisdefinedinthefileNETSNMPAGENTMIB.txtwhichshould beshippedaspartoftheOperatingSystemnetsnmppackage.Typicallythis MIBfilecanbefoundunder/usr/share/snmp/mibs.FindandexamineNET SNMPAGENTMIB.txt.Strictly,theMIBfileisdefiningSNMPV2 NOTIFICATIONs,ratherthanSNMPV1TRAPssearchinthefileforthe stringNOTIFItofindtherelevantlines.AlsonotetheIMPORTSsectionatthe topoftheMIBfile,especiallytheimportfromNETSNMPMIB.Thisindicates thatNETSNMPAGENTMIBisdependentonalsoloadingNETSNMPMIBin additiontosomestandardSNMPv2MIBs.
Figure58:MIBfileforNET_SNMP_AGENTMIBshowingIMPORTSsection
94
23January2013
Figure59:MIBfileforNETSNMPAGENTMIBshowingnotifications
3. InspecttheNETSNMPMIB.txtfileandsearchforthestring Notifications.You shouldseethatthenetSnmpNotificationPrefixisdefinedasbranch4beneath netSnmpandthatnetSnmpNotificationsisbranch0under netSnmpNotificationPrefix.
Figure60:MIBfileforNETSNMPMIBshowingOIDsfornotificationhierarchy
4. AtthetopofthefileyoushouldfindthelinesthatdefinetheenterpriseOIDfor netSnmp.
23January2013
95
Figure61:MIBfileforNETSNMPMIBshowingOIDfornetSnmp
5. Betweenthem,thesefilesgiveus(almost)theOIDfortheunknownTRAPwe received1.3.6.1.4.1.8072.4.0.2.
1.3.6.1.4.1isthestandardiso.org.dod.internet.private.enterprisesOID whichisdefinedintheIMPORTfromSNMPv2SMI netSnmpis{enterprises8072} netSnmpNotificationPrefixisbranch4undernetSnmp netSnmpNotificationsisbranch0undernetSnmpNotificationPrefix nsNotifyShutdownisNOTIFICATION2undernetSnmpNotifications
6. NotethatsomeSNMPagents(includingthenetsnmpagent)areknowntoomit the0fromtheTRAPthattheyactuallygenerate,whichiswhythe oidfieldinthe detailsoftheeventdoesnotquitematchtheOIDspecifiedintheMIBfile. 7. $ZENHOME/share/mibscontainsfivesubdirectoriesfourofwhichcontain sourceMIBfilesprovidedwithZenoss(iana,ietf,irtf,tubs).Thefifthdirectory, site,iswhereotherMIBstobeimported,shouldbeplaced. 8. ThesitedirectoryshouldcontainZENOSSMIB.txtwhichisprovidedasstandard todefineTRAPsthataresentbytheNotificationfunction(thiswillbediscussed later). 9. CopyNETSNMPAGENTMIB.txttothesitedirectory.Atthispointdonotcopy NETSNMPMIB.txt;wewilldemonstratetheerrormessagewhencorequisite MIBsarenotavailable.
96
23January2013
10. ToimportintoZenossuse:
zenmib run -v10 NET-SNMP-AGENT-MIB.txt
11. YoushouldseethattheNETSNMPAGENTMIB.txtfileisimportedbutwith errors;thereshouldbeaWARNINGmessagesayingtheNETSNMPMIBcould notbefound.
Figure62:ImportingNETSNMPAGENTbeforeprerequisitesinplace
12. NoteintheRunningsmidumplinethatthestandardSNMPv2prerequisitefiles thatwerelistedasIMPORTsinFigure58haveautomaticallybeenlocatedin $ZENHOME/share/mibs/ietf;howeverultimately0nodesand0notifications wereloaded. 13. FromtheZenossGUI,usetheADVANCED>MIBsmenu.TheNETSNMP AGENTMIBislistedbut,assuggested,ithasnoOIDMappingsandnoTRAPs.
23January2013
97
Figure63:MIBGUIwithimportedNETSNMPAGENTMIBbutnoOIDsorTRAPs
14. CopyNETSNMPMIB.txtto$ZENHOME/share/mibs/siteandrerunthe zenmibcommand.
Figure64:SuccessfulimportofNETSNMPAGENTgivencorrectprerequisites
15. ThereisaDEBUGlinenotingthattheNETSNMPAGENTMIBisalready imported;thisisnotanissue.ThisimportwilloverwriteanyexistingMIBofthat name.
98
23January2013
16. NotethattheRunningsmidumplinealsolooksinthesitedirectoryandfindsthe prerequisiteNETSNMPMIB.txtinadditiontofindingthestandardSNMPv2 MIBsintheietfdirectory.45nodesand3notificationshavebeenloaded. 17. ReturntotheZenossGUIandrefreshtheMIBsmenu.ClickingontheNET SNMPAGENTMIBshouldnowdisplay45OIDMappingsandthreeTRAPs, includingnsNotifyShutdown. 18. RestartthesnmpagentontheZenosssystemwith/etc/init.d/snmpdrestart.You shouldseeaneventintheEventConsolethatnowcontains snmptrap nsNotifyShutdowninthesummaryfield,ratherthansnmptrap 1.3.6.1.4.1.8072.4.2 .Ifthisdoesnotwork,youmayneedtorecyclethezentrap daemon.YoucandothiswiththeGUIfromtheADVANCED>Settings> Daemonsmenuor,asthezenossuserfromacommandline,usezentraprestart. 19. ZenosshasimplementedanumberofchangesinthewayMIBsareinterpreted. RememberfromFigure60thatnetSnmpNotificationsisbranch0under netSnmpNotificationPrefix;however,someagentsomitthis0whentheyactually generateTRAPs.Zenoss4.2hasprocessingin $ZENHOME/Products/ZenEvent/zentrap.pytotryandinterpretactualTRAPs bothwithandwithouttheextra0.TheeventconsoleshowedaneventwithOID 1.3.6.1.4.1.8072.4.2fortheoriginalevent;comparetheeventdetailsofthe originaleventwiththenewonethatcontainsnsNotifyShutdowninthesummary field.Youshouldfindthattheneweventhasan oidfieldof1.3.6.1.4.1.8072.4. 0.2. 20. Examine$ZENHOME/Products/ZenEvent/zentrap.py(aroundline580inZenoss Core4.2)toseethecodethathandlesthisextra0digitprocessing.
9.4.2 A few comments on importing MIBs with Zenoss

ThereareafewquirkstodowithimportingMIBsintoZenossandthequirkshave changedsubtlyoverseveralversionsofZenoss.
NotethatMIBsimportedintoZenossareonlyusedforinterpretingSNMPV1 TRAPsandSNMPV2NOTIFICATIONsforuseintheEventsubsystem. AlthoughtheOIDsareimportedfromMIBs,theycannotbeusedforMIB browsingorwhenworkingwithOIDsforperformancesampling,thresholdingand graphing. AlwaysensureyoudoMIBworkasthezenossuser. Bydefault.zenmibrunv10willtryandimporteverythingunder $ZENHOME/share/mibs/site.Thev10simplyaddsmoreverboseoutput. zenmibshouldcheckintheotherdirectoriesforprerequisites. WheneveryouhaveimportedaMIB,checkattheGUIonthe MIBspage.You shouldseethenameoftheMIBandyoushouldusuallyseenonzerocounts undertheOIDMappingsandTRAPdropdownmenus. EventManagementforZenossCore4Skills1stLtd 99
23January2013
TherearesomeMIBsthatwillresultinzerocounts,forexampleiftheMIB sourcefileonlydefinesSNMPstructureanddoesnotincludethedefinitionfor anyOIDsorTRAPs. Checktheoutputofthezenmibcommandcarefullyforerrormessages. IfOIDtranslationsdonotappeartobeworkingineventsafterimportingaMIB, recyclethezentrapdaemonfromtheADVANCED>Settings>Daemonsmenu or,asthezenossuser,runzentraprestart. IfeventmappingsandtransformsarebuiltassumingthataMIB hasbeen imported,forexample,testingtheeventClassKeyfieldforenterprises.8072.4.2, andthatMIBisthenremovedfromtheZopedatabase,thenthemappingand/or transformwillfail.EspecialcareshouldbetakenwithanyZenPackthatimports MIBsastheremovaloftheZenPackislikelytoremovethoseMIBs. Zenoss4.2(and3.2.1)appeartohaveatimingbugthataffectssomeinstallations. Thesymptomisthatzenmibapparentlysatisfiesitschecksbutthenreports Loaded0MIBfile(s).TheonlysolutionIhavefound(whichappearstowork perfectly)istouseazenmib.pyfromaZenoss3.1installation.Thisfilebelongsin $ZENHOME/Products/ZenModel.
Figure65:OccasionaltimingbugwithZenoss4.2.Replacezenmib.pywithaZenoss3.1version.
9.5 The MIB Browser ZenPack

ThereisanexcellentcommunityZenPackavailabletoperformMIBBrowsing.Thisis notdirectlyrelevanttoTRAP/NOTIFICATIONprocessing,butitisusefulfor investigatingMIBswithaviewtobuildingSNMPperformancetemplates. Itcanbedownloadedfromhttp://wiki.zenoss.org/ZenPack:MIB_Browser.Unfortunately thisZenPackkeepsgettingbrokenbynewversionsofZenoss.Ifyoufollowthelinkto 100 EventManagementforZenossCore4Skills1stLtd 23January2013
DownloadforZenossCore3.1,thisdoesindeedworkforCore3.1;thisversionshouldbe downloadedandmodifiedforCore3.2;forZenoss4.2,followthe DownloadforZenoss Core4.2linkandperformthesamemodificationswhicharedocumentedinthe commentsifyoufollowthedocumentationlinkhttp://community.zenoss.org/docs/DOC 10321.BasicallyyourevertthelaterCorefilesbacktothe3.1levelofcode. ItprovidesaMIBbrowsertoexploreanyOIDthathasbeenloadedintoZenoss,along withatestfacilitytosnmpwalkaconfigurabledevicetoretrievevaluesforanyselected partoftheMIBtree.NotethatitonlysupportsSNMPV1. TheMIBBrowserZenPackchangestheADVANCED>MIBsmenuandcreatesaMIB Browserlefthandmenu.SelectingtheMIBBrowsermenuoffersasimilarlayouttothe OverviewmenubutitintroducesnewiconsalongsidethenameofaMIB.Clickingthe iconstartstheMIBBrowseragainsttheselectedMIB.
Figure66:StartingtheMIBBrowserclickagainstthemagnifiericonforagivenMIB
Inordertoperformansnmpwalk,youneedtoprovideatargetdeviceandanSNMPv1 communitynameundertheTestSettingstab.Arighthandmouseclickthenprovides thesnmpwalkmenuagainsttheleveloftheMIBtreethatyouarepositionedon. TheOIDDetailswindowgivesthesameinformationyouwouldseeifyouinspectedthe MIBsourcefile.UsethiswindowtocutandpasteintoOIDfieldsinperformance templates.
23January2013
101
Figure67:UsingtheMIBBrowserZenPack
9.5.1 Modifying Zenoss Core 4.2 to make the MIB Browser ZenPack work
1. Downloadtheeggfileandinstallinthenormalway.Itshouldinstallwithno errors.
zenpackinstallZenPacks.community.mib_browser1.2py2.7.egg zenhubrestart zopectlrestart
2. Changeto$ZENHOME/Products/ZenUI3/browser.Backupbackcompat.py, navigation.zcmlandbackcompat.zcml. 3. Inbackcompat.py,commentoutthelinesattheenddefiningMibClass.Ifthere arealsosimilarlinesforMibNodeandMibNotification,commentthemouttoo.

#defMibClass(ob): #id='/'.join(ob.getPhysicalPath()) #return'/zport/dmd/mibs#mibtree:'+id
4. Innavigation.zcml,aroundline233,changetheurllinetobe url="/zport/dmd/Mibs/mibOrganizerOverview".Notecarefullythecasesensitivity onmibs/Mibs.

url="/zport/dmd/mibs" +url="/zport/dmd/Mibs/mibOrganizerOverview"
5. Inbackcompat.zcml,aroundline260commentoutlinesfortheadapterfor Products.ZenModel.MibOrganizer.MibOrganizer.Ifadapterstanzasalsoexistfor MibNode,MibNotificationandMibModule,commentthemouttoo. 6. Changedirectoryto$ZENHOME/Products/ZenModel/skins/zenmodeland backupviewMibModule.pt.
102
23January2013
7. ModifyviewMibModule.py.Changethetemplateinthefirstline.
<tal:blockmetal:usemacro="here/templates/macros/page2"> +<tal:blockmetal:usemacro="here/page_macros/oldnew">
8. YouwillneedtocompletelyrestartZenossandmakesureyourbrowsercache iscleared.
9.6 Mapping SNMP events

ZenossprovidessomeeventmappingsforSNMPTRAPsoutofthebox.Asdiscussedin anearliersection,thefile$ZENHOME/Products/ZenModel/data/events.xml configuresallthestandardmappingssosearchingthisfileforSNMPprovidesinsight fordefaultcustomisation. MostSNMPTRAPsmaptotheZenoss/Unknowneventclass.Thereareoneortwo exceptionsforsomegenericTRAPssuchasLinkUp(3),LinkDown(2)andthe AuthenticationTRAP(4).Eventfieldsthatareautomaticallypopulatedbythe zentrap processingincludesummary,eventClassKeyandagent.Theeventdetailsshowsthe communityandoidName/Valuepairs.Notethatthevalueoftheoidfieldisalways innumericformat,nottranslatedthroughanimportedMIB. Thismeansthat,typically,theeventonlymapsontheEventClassKey,whichis interpretedbyzentrap.pyasenterprises.<enterprisenumber>.<specifictrap>ifthe SNMPv2SMIhasbeenimportedor1.3.6.1.4.1.<enterprisenumber>.<specifictrap> otherwise.Thesummaryfieldwillbesnmptrap<enterpriseOID><specifictrap>and theagentfieldwillbesettozentrap.Thesetranslationsassumethattheenterprise specificMIBhasnotbeenimported. TRAPsandNOTIFICATIONsmayhaveoneormoreTRAPvariables(varbinds).These varbindsappearintheeventdetailswherethefield nameisthevarbindOID(possibly translatedthroughaMIBlookup)andthecorrespondingfield valueisthevalueofthat varbind. EventclassmappingscanbedevisedwithvariousRule,RegexandTransformelements, toparseouttheintelligencefromSNMPTRAPsandeithercreatenewuserdefined eventfieldsormodifyexistingfields(suchasevt.summary). NotethateventmappingsthatparseoutSNMPOIDsandvarbindsmustbeawareof whethertherelevantMIBshavebeenimported,ornot.IfaMIBisimported,OID mappingbasedonmatchingdotteddecimalnotationwillfailastheMIBOID translationshappenbeforeeventmapping.
9.6.1 SNMP event mapping example

InordertointerpretenterprisespecificTRAPs,mappingsareusuallyrequired.Often anactionormodificationisrequired,effectivelybasedonwhatenterprisetheTRAP camefrom(Cisco,netsnmp,...),soasubclassofeventsarerequiredthatinheritsome
23January2013
103
commoncharacteristicsbutsomeeventdetailsvarydependingontheexactenterprise specificTRAPnumber. ManyenterpriseTRAPsalsoincludeseveralvarbindsthatneedtobeinterpretedand processed. Inthemappingexampleshownhere,threesmallscriptsareusedtogenerateTRAPs fromthe1.3.6.1.4.1.123enterpriseoneforeachofspecificTRAPs1234,1235and1236. ThefirsttwohaveasinglevarbindwhosestringtypevalueisHelloworld4,wherethe endnumberis4or5;thethirdscriptgeneratesaTRAPwith2varbinds.Notethateach ofthevarbindsexhibittheextra0behaviour,ie.thevarbindfieldwillbe 1.3.6.1.4.1.123.0.1234.
#!/bin/bash # #Generateasampletrap #Sendtrapusingthesnmptrapsuppliedwithnetsnmp #TraphereisEnterprise1.3.6.1.4.1.123,trap1236 #EnsureyouchangethelineforMANAGERtobeyourZenossServer # #Uncommentnextlineforextradebugging #setx MANAGER=zen42.class.example.org HOST=zen42.class.example.org ENTERPRISE=.1.3.6.1.4.1.123 GENTRAP=6 SPECTRAP=1236 TRAPVAR1=.1.3.6.1.4.1.123.0.12361 TRAPVAR2=.1.3.6.1.4.1.123.0.12362 VARBIND1="Helloworldvarbind161" VARBIND2="Helloworldvarbind262" TIMESTAMP=1 # /usr/bin/snmptrapv1cpublic$MANAGER$ENTERPRISE$HOST$GENTRAP $SPECTRAP$TIMESTAMP\ $TRAPVAR1s"$VARBIND1"\ $TRAPVAR2s"$VARBIND2" #
1. Withoutanymapping,whengen_mytrap_1234.shisrun,itwillmaptothe /Unknowneventclass. 2. CreateaneweventsubclassSnmpundertheclass/Skills. 3. Mapthe1234eventbyselectingitandusingtheReclassifyanEventicon. Choose/Skills/Snmpfromthedropdownselectionbox.Leavetherestofthe EventClassMappingparametersasdefaultsfornow.Thismeansthattheevent onlymapsontheeventClassKey,whichtranslatesto<enterpriseOID>.<specific trap>.Themappingnameisautomaticallyassignedthenameofthe eventClassKey(1.3.6.1.4.1.123.1234 ifSNMPv2SMIisnotimported; enterprises.123.1234ifitis).Referbacktothesnippetofthezentrapcodein Figure57formoreinformationontheparsingoftheTRAPintoeventfields. Checkthatyoureventclassmappingworks.
104
23January2013
Fromhere,ensurethattheSNMPv2SMIMIBisimported;thusanyTRAPenterprise field(andhenceeventClassKey)willstartwithenterprises,not1.3.6.1.4.1.Inmost cases,thesamewillapplytothenamefieldofaTRAPvarbind. Thenextstepistointerpretthevarbind.EachoftheTRAPsgeneratedbythetest scriptscomefromtheEnterprise1.3.6.1.4.1.123andthenameofeachofthevarbinds alsostartswith1.3.6.1.4.1.123thus,inthedetailoftheinterpretedevent,thevarbind namefieldswillstartwithenterprises.AtransformwillextractthatpartoftheOID afterenterprises.Itwillalsosubstitutethevalueofthevarbindintotheevent summary. Attransformtime,strictlytheeventisaZepRawEventProxyobject,whichhasadetails dictionary(anEventDetailProxyobject)aspartofit(referbacktoFigure35,Figure37 andFigure38).Alsorememberthatalthoughonecanrefertodetaileventfieldsby name(eg.evt.line_num)iftheyaresimplenames,you cannotusethismethodifthe detailnamehasadotinit. Ifoneisinterestedinthevaluesofsuchfields,thegetorgetAllmethodsareneeded. Sincethegetmethodfailswithanattributeerrorifthevalueisnonscalar,itissaferto assumethatallvaluesmaybenonscalarandusethegetAllmethod. InversionsofZenosspriorto4,atransformtointerpretTRAPvarbindswouldlooklike this:
for attr in dir(evt): if attr.startswith('enterprises.123.'): evt.myRestOfOID=attr.replace('enterprises.123.','') evt.myFieldValue=getattr(evt,attr) evt.summary=(evt.summary + + evt.myFieldValue)
ThiswillfailwithZenoss4astheneweventstructuredoesnotdeliverdetailevent fieldsasaresultofdir(evt).AZenoss4versionwouldbe:
forattrinevt.details._map.keys(): ifattr.startswith('enterprises.123'): evt.myRestOfOID=attr.replace('enterprises.123.','') evt.myFieldValue=''.join(list(evt.details.getAll(attr))) evt.summary=(evt.summary++evt.myFieldValue)
1. Thefirstlinecyclesthroughtheeventdetailsattributenames. 2. Thestartswithlineensuresthattransformsonlytakeplaceforattributesthat startwithenterprises.123ie.varbindattributefields. 3. NotethatthereplacelineisreplacingtheOIDspecified,withthenullstring thesyntaxafterthecommaissinglequotesinglequote.Therestoftheattribute (ie.the0.1234bit)iskeptandbecomesthevalueoftheuserfieldmyRestOfOID. 4. Theevt.myFieldValuelineusesthegetAllmethodincasethevarbindvalueis nonscalar.Toconcatenatetheresultinglistwiththe evt.summarystring,the listisconvertedintoastringwiththejoinfunction.
23January2013
105
5. Runningthescripttogeneratea1234TRAPshouldnowgenerateanevent with:

Theeventmappedtothe/Skills/Snmpclass Thesummaryfieldshouldsaysnmptrapenterprises.123.1234Helloworld 4. TheEventDetailsshouldshowvaluesforcommunity,oid,myFieldValue andmyRestOfOID,inadditiontothedefaultvarbindname/valuepairof enterprises.123.0.1234/Helloworld4
6. Runningthescripttogeneratea1235TRAPwillstillgenerateaneventwith the/UnknownclassastheeventclassmappingisbasedontheeventClassKeyof enterprises.123.1234. Sofar,weareonlymatchingasingleSNMPTRAPwiththeeventClassKeyfield.The objectiveistomapalleventsfromtheenterprise1.3.6.1.4.1.123.WithSNMP,you oftenwanttoapplyatransformtoseveralsimilareventswhichareonlydistinguished bythelaterpartsoftheOIDfield.Thetestscriptsallgenerateeventswhose eventClassKeystartwith1.3.6.1.4.1.123.buttheydifferinthelastnumber. ARulewillbeusedtomatchallappropriateevents.However,aRuleisonlyinspected iftheeventClassKeyhasalreadymatchedsuccessfullyandwehavenocontroloverthe eventClassKeythatissetbyzentrap.py.Thus,thedefaultmappingconceptwillbe used. 1. ClearallSNMPeventsforyourZenosssystem. 2. Edittheenterprises.123.1234mapping.

IntheRuleboxputevt.eventClassKey.startswith('enterprises.123.') ChangetheNameofthemappingtoenterprises.123 IntheTransformboxput:
forattrinevt.details._map.keys(): ifattr.startswith('enterprises.123'): evt.myRestOfOID=attr.replace('enterprises.123.','') evt.myFieldValue=''.join(list(evt.details.getAll(attr))) evt.summary=evt.summary+"defaultmapping"+evt.myFieldValue
Savethemappingaway
3. Runthegen_mytrap_1234.shscriptandthegen_mytrap_1235.shscript. 4. ChecktheeventsintheEventConsole 5. Youshouldfindthatthe1234TRAPmapssuccessfullybutthe1235TRAP doesn't.Thisisbecausetheinitialtestforeventclassmappingchecksthe eventClassKeythatisstillsettoenterprises.123.1234sotheprocessingnever evengetsasfarascheckingourRule!Notethatwehavenocontroloverhowthe eventClassKeyfieldispopulatedbytheeventprocessingmechanismitisparsed outforusbyzentrap.py(seeFigure57again). 106 EventManagementforZenossCore4Skills1stLtd 23January2013
6. ThisiswherethemagicstringofdefaultmappingcanbeusedintheEventClass Keyfield.SettheEventClassKeytodefaultmapping(Noteitmustbealllower case).IftheprocessofmappinganeventcannotfindamatchfortheEventClassKey thenitwillrerunthemappingprocesswithanEventClassKeyofdefaultmapping. 7. Savethemapping. 8. ChecktheSequencemenu.ThereareseveralmappingsthatallmaponanEvent ClassKeyofdefaultmapping.Chooseasuitablesequencenumberforthenew defaultmapping.Savethemapping. 9. Clearexistingevents.Rerunbothscripts.Checkthatbotheventsnowmap correctly.
Figure68:MappingforSNMPTRAPwithrule,transformandeventClassKeyofdefaultmapping
Thetesteventsusedsofar,onlyhaveonevarbind.WhatifyourTRAPhasseveral varbindsandyouwanttouseinformationfromeachofthem?Thescript gen_mytrap_1236.shgeneratesaspecificTRAP1236,withtwovarbinds:

varbind1 varbind2
1.3.6.1.4.1.123.0.12361 1.3.6.1.4.1.123.0.12362
Helloworldvarbind161 Helloworldvarbind162
Runningthescriptgen_mytrap_1236.shshouldresultinaneventthatmapstothe /Skills/Snmpclass,withthemyFieldValueandmyRestOfOIDfieldsmatchingthedatain thelastvarbindthatwasprocessed,andthesummaryreflectingthedatafromallvarbinds. Toprovideamoreeleganttransformsolutionwhereyoudonotknowifadetailvalueis scalarornot,thePythontry/exceptconstructcouldbeused:

forattrinevt.details._map.keys(): ifattr.startswith('enterprises.123'): evt.myRestOfOID=attr.replace('enterprises.123.','') try: evt.myFieldValue=evt.details.get(attr) except: evt.myFieldValue=''.join(list(evt.details.getAll(attr))) evt.summary=evt.summary+"defaultmapping"+evt.myFieldValue
Checktheendof$ZENHOME/log/zeneventd.logfordebugginghelp. 23January2013 EventManagementforZenossCore4Skills1stLtd 107
10 Event Triggers and Notifications

10.1 Zenoss prior to V4
PriortoZenoss4,thereweretwowaysofautomatingresponsestoevents.
UserAlertingRules

Emailtousers Pagingtousers Scriptsruninthebackground
EventCommands
Theuseractionswereconfiguredonaperuserorperusergroupbasis.Thismeantthat similaremails/pagesformanyusersorgroupshadtobecreatedindividually;therewas noeasywaytocopyanAlertingRulefromoneusertoanother. EventCommandsusedaverysimilarmethodtodefinewhenacommandshouldbe automaticallyruninthebackground. AlertingRulesandEventCommandswereexecutedbythezenactionddaemonwhich processedanyrequestsevery60seconds.Duplicateeventsdidnotcreatemultiple actionsandthiswashandledbythealert_statetableoftheMySQLeventsdatabase. ThisisprobablytheareathathaschangedmostforusersofZenoss4.
108
23January2013
10.2 Zenoss 4 architecture

Zenoss4hascompletelychangedthearchitectureoftheMySQLeventsdatabase.There isnoalert_statetableinthezenoss_zepdatabase.zenactiondisstillresponsiblefor executingactionsbutithasbeencompletelyrewrittenandtakesinputfromaRabbitMQ queuecalledsignalwhichisfedbythezeneventserverdaemon.Thismakesalerting muchmoreresponsive.
Figure69:Zenosseventarchitectureactionprocessinginbottomright
AlertingRuleshavegoneinZenoss4andarereplacedbytheconceptsof:

Triggers Notifications
Triggersdefinewhatcausesaresponse.ANotificationistheresponse.Thisisbetter inseveralways.Bothmechanismsaredecoupledfromusersandfromeachother. Notificationsnowincludeeventcommandsaswellasthetraditionalemailandpaging, andSNMPTRAPshavealsobeenaddedasanotificationaction. TriggerandNotificationSubscriptionsobjectsaredefinedintheZopedatabase(though theTriggerisastubobjectthatisusedformanagingpermissionsanddoesnotcontain theactualtriggerrules). ThereisanewEVENTS>TriggersmenufordefiningbothTriggersandNotifications.
23January2013
109
10.3 Triggers
Triggersdefineunderwhatconditionssomeactionshouldtakeplace.Theyaredefined fromtheEVENTS>Triggersmenu.Usethe+icontoaddanewtrigger;doubleclick anexistingtriggertomodifyit.
Figure70:CreatinganewTrigger
Notethatbydefault,anewtriggeriscreatedas Enabledbutwithanillegalrule! DevicePriorityequalswithoutavaluewillcauselotsoferrorsinzeneventserver.log. WhencreatingtheTriggerrule,combinationsorlogicalANDsandORscanbeused(the allandanyoptions).Usethe+icontoaddfurtherconditions.Allthestandardevent attributesareavailabletoselectfromthedropdownboxes.Userdefinedeventfieldsare notavailableherealthoughitispossibleinZenPackstoprovideforuserdefinedevent fields. UnlikeearlierversionsofZenoss,itisalsopossibletonestcriteriatobuildupthe overallrule.Usetherightmosticontoaddanestedclause.
Figure71:ATriggerrulewithnestedclause
TheUserstaboftheTriggerdefinitionistocontrolwhocanmanipulatethisTrigger. Bothglobalandspecificrolescanbeallocated.Userswhohaveeithertheglobal ManagerorZenManagerrolewillautomaticallyhavemanageaccesstotriggers,aswill thetriggerowner(creator). 110 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure72:TriggerUserstabforglobalanduserspecificroles
NotethatthisUserstabhasnoeffectonwhoreceivesanyrelatedNotifications.
10.4 Notifications
NotificationsarecreatedfromthesamemenupathasTriggers.Anameanda notificationtypearetheinitialrequirements. NotethatacarefulnamingconventionforTriggersandNotificationsmakesthe environmentmucheasiertoworkwith.
Figure73:CreatinganemailNotification
TheNotificationiscreatednotEnabledbydefault.Youcanchoosewhethertosend goodnewsClearnotificationsandwhethertodelayaNotification(usefulforless criticaleventsthatmayselfclear).Eventscanbesentrepeatedlyoronlyontheinitial occurrence.
23January2013
111
Figure74:Notificationdetails
AkeyfieldforaNotificationistheTriggerthatcausestheNotification.Configured Triggerswillbeofferedinthedropdownbox.MakesureyouselectaTriggerandclick AddifyousimplyselecttheTriggerandthenSUBMITtheentireNotification,the Triggerwillnotbesaved. DependingontheNotificationtypeselectedwhentheNotificationiscreated,the Contenttabwillvary;theothersremainthesame,thoughforCommandandTrap notificationstheSubscribertabisnotrelevanttowhethertheactiontakesplaceas thesearebackgroundactionsnotuserrelatedactions. ThedifferentNotificationactionsareencodedin $ZENHOME/Products/ZenModel/actions.py.
112
23January2013
Figure75:$ZENHOME/Products/ZenModel/actions.pyimplementsNotificationactions
10.4.1 email Notifications

TheContenttabforemailallowsyoutocustomisetheemailsubjectandbody,using standardfieldsfromtheevent,usingTALESexpressions(TemplateAttribute LanguageExpressionSyntax,fromZope)toreferencefieldsoftheevent,evt.See AppendixDoftheZenossAdministrationGuideformoredetails. Notethatyoumust useTALEStheevt.<eventfield>syntaxusedinmappingrulesandtransformsdoesnot workineventcommands.TALESsyntaxtakestheform:
${evt/<event field>}
Alsoseesection2.6oftheZenossCore4AdministratorsGuide.
23January2013
113
Figure76:TheContenttabofaNotificationpart1
AlsonotethatpreviousversionsofZenossprovidedaccesstothedevvariabletoaccess attributesofthedevicethatcausedtheevent.Thedevvariableisnolongerlegalforuse inNotificationcontent. SeparatedefinitionscanbeprovidedfortheproblemandclearingNotifications. ThebottomoftheNotificationconfigurationpanelallowsyoutooverridedefault configurationsformailhostparameters.
114
23January2013
Figure77:NotificationContentwithmailserverparameters
TheseparametersarespecifiedgloballyfromtheADVANCED>Settings>Settings menu.
23January2013
115
Figure78:Defaultsettingsformailserverandpaging
DoensurethattheFromAddressforEmailssettingsarelegalformailservers.A difficultscenariotodebugiswhereemailnotificationsneverarrivebecausetheyare discardedbyamailserverbecauseoftheFromaddress. Thethirdtab,Subscribers,ontheNotificationdefinitionpaneldefineswhoreceivesthe notification.Inaddition,thispanelalsoserversasimilarpurposetothe Userstabfor TriggersinthatitdefineswhoisallowedtomanagetheNotificationdefinition.Unlike Triggers,ifnosubscriber(userorusergroup)isspecified(andexplicitlyAdded)thenno emailwillbereceived.Itisnotnecessarytospecifyanymanagementrolesthough.
116
23January2013
Figure79:SubscriberstoNotifications
23January2013
117
10.4.2 Page Notifications
Figure80:Usersettingsshowingemailandpageparameters
APagenotificationisverysimilartoemail,simplyprovidingaContenttabtospecifya MessageformatandaClearMessageformat.Aswithemail,the evtvariableisavailable forparametersubstitution.Thecommandusedtosendpagemessagesisthatspecified globallyfromADVANCED>Settings>Settings(seeFigure78).Theindividual recipientcomesfromthoseusers/groupsspecifiedintheSubscriberstabwhomusthave theirpagerdetailsconfiguredonthatusershomepage(thisisalsowhereauser'semail addressisspecified).
10.4.3 Command Notifications

TheContenttabforaCommandNotificationspecifiesabadnewsandagoodnews command,atimeparameterforhowlongthecommandmayrununtilitisdeemedto havefailed,andenvironmentvariablescanalsobespecifiedas <variable>=<value>. ThelatterisusefulasinpastversionsofZenossacommonissuewastocreateanEvent Commandbutforgettosourceanynecessaryenvironmentvariablesinthescript.Since thescriptisrunbyzenactiond,ithasverylittledefaultcontextinwhichtorunsothings like$ZENHOME,$PATHwerenotautomaticallyset.
118
23January2013
Figure81:ACommandNotification
Notethattousetheseenvironmentvariablesinascriptyouneedtoescapethedollar withadollareg.$$ZENHOME.Multipleenvironmentvariablearesemicolonseparated andyoudonotincludethedollarwhenyouspecifythenameoftheenvironment variable. Alsonotethat,althoughasubscriberisnottypicallyrequiredastheCommand notificationisabackgroundscript,duetoabugInCore4.2,environmentvariableswill beignoredunlessthereisasubscriber.Itisnotoneroustosetupadummyuser subscriberasacircumventiontothisissue. CommandNotificationsmaybesimplebuiltinshellcommandsasshownaboveorthey canbecomplexscriptsinotherlanguages,providedtheycanbeexecutedfromashell environment.Again,standardfieldsfromtheeventcanbesubstitutedusingTALES expressions.Noteinthefigureabovetheuseofbackticsaroundthe datecommandto runthedatecommandbeforeaddingtheoutputoftheenvironmentvariablesandthe goodnews/badnewsmessage.
23January2013
119
10.4.4 TRAP Notifications

SNMPTRAPnotificationsarenewwithZenoss4.ItwaspossibletocreateTRAP forwardingscenariosusingEventCommandsinthepastbutthisabilityisnow standard.TheContenttabinthiscaseconfigurestrapdestination.
Figure82:Trapnotification
ThetrapdestinationmayeitherbearesolvablenameoranIPaddress. NotethatwithZenossCore4.2thereisabugthatmeansselectingSNMPv1resultsin noTRAPbeingissued,eventhoughzenactiond.logreportsthataTRAPhasbeen successfullysent. TheTRAPisdefinedin$ZENHOME/share/mibs/site/ZENOSSMIB.txt.Itisasingle TRAPwithmanyvarbindsthatarepopulatedwiththefieldsoftheoriginalevent.It wouldbegoodpractisetoimportthisMIBintoaZenossserverthatisreceivingsuch notificationTRAPs.
120
23January2013
Figure83:TrapresultingfromaNotificationTRAPwithouttheZENOSSMIB.txtimported
Thusthevarbindnameswillbetranslatedtosomethingmorehelpful.
Figure84:TrapresultingfromaNotificationTRAPwiththeZenossMIBimported
CarefulinspectionoftheTRAPwiththeZenossMIBimportedrevealsanomissionin theMIB;varbind8forthemessagefieldisnotdefinedsoitshowsintheeventdetails withthenamezenTrapDef.8. 23January2013 EventManagementforZenossCore4Skills1stLtd 121
NotethattheversionofZENOSSMIB.txtshippedwithCore4.2.3hasbeenmodified fromthe4.2versioninsuchawaythatitdoesnotimportcleanly(therearenon printingcharactersinthefile).Foradescriptionoftheproblemandaworkingfile,see http://jira.zenoss.com/jira/browse/ZEN5060.
10.5 Notification Schedules

AnyNotificationtypemayhaveoneormoreschedulesassociatedwithit.Theseare effectivelyMaintenanceWindows(andareindeedimplementedbythesamecodeas MaintenanceWindows).Theyallowdifferentresponsestotakeplaceatdifferenttimes. IfnoNotificationScheduleexiststhentheNotificationisalwaysactive.
Figure85:Notificationschedule
ThescheduleiscreatedasnotEnabledbydefault.Typicallytheschedulewillrepeat overcertainperiodsseeFigure85. Withdebugloggingturnedonforthezenactionddaemon,thestartofaNotification schedulecanbeclearlyseen. AnInfoseverityeventiscreatedwhenanyMaintenanceWindowstartsanditiscleared bytheClearseverityeventgeneratedwhentheMaintenanceWindowends.
122
23January2013
Figure86:zenactiond.logshowingthestartofaNotificationSchedule
Figure87:EventsforMaintenanceWindowsstarting/stopping
10.6 Using zenactiond.log

AllNotificationsareprocessedbythezenactionddaemon.Todebugissuesandalsoasa learningaid,itishelpfultosetthedebugginglevelto Debug(logseverity10), rememberingtorecyclezenactiond. Inspectingzenactiond.logprovidesagoodinsightintohowzenactiondprocessesevents fromtheRabbitMQsignalqueueandthenteststhemagainsttheconfigured Notifications. 23January2013 EventManagementforZenossCore4Skills1stLtd 123
TheTriggersareprocessedbythezeneventserverdaemontodecidewhattoplaceonthe signalqueue.ThereareobviouslydifferentsignalsforeachNotificationtype. Aprocessingcyclestartswithaprocessingmessageentry(highlightedingreen)in Figure88. Notificationsarecheckedastowhethertheyareenabledornot(highlightedinblue).
Figure88:zenactiond.logprocessingasignalagainstvariousNotifications
Theeventthatgeneratedthissignalwasa/Security/Sueventandshouldtriggerboth thezen42_email_traps_suNotificationandthezen42_trapNotification.InFigure88the logshowszen42_email_traps_subeingdiscarded(highlightedinyellow);thisisbecause thesignalmessageiskeyedtoaTRAPNotificationtype,notanemailone (unfortunatelyzenactiond.logdoesnotshowthisdetail). Thematchwithzen42_trapishighlightedinredwherethecheckingforanotification schedulewindowcanalsobeseen.Thestartofthenotificationactiontogeneratethe TRAPisalsohighlighted. Oncetheactioniscompleted,zenactiond.logshowssimilariterationsthroughthe Notificationslistwithaseparatesignalmessage,wherethe zen42_email_traps_su Notificationisselectedandactionedandthezen42_trapNotificationisdiscarded.
124
23January2013
10.7 The effect of device Production State

TheProductionStateofadevicecanbeusedtocontroldifferentmanagementaspects ofasystem.ProductionStateforadeviceisconfiguredonthedevice'shomepage OverviewandmaybemodifiedbyMaintenanceWindowsconfiguredforadevice,device class,Group,SystemorLocation. WhenconfiguringaMaintenanceWindow,theproductionstateisdefinedbothfor duringthewindowandthestatetoreturnto,wherethelatteristypically Original.
Figure89:MaintenanceWindowfordeviceclass/Server/LinuxforfirstSundayinthemonth
Chapter8oftheZenossCore4AdministrationGuidedescribesthedifferentProduction Statesandtheeffectthatthesehave.Threedifferenttypesofmanagementare defined:

Monitoring Alerting Dashboard
pingpollingandeventgeneration generatingalerts(emails,pagers,commands,traps) whethertoincludeintheDeviceIssuesportlet
Inpractise,anythingtodowithNotificationsiscontrolledbythefiltersintheTrigger. IfnoProductionStatefilterisconfiguredthentheNotificationwillrun,bydefault. AdeviceProductionStateofProductionwillresultineventscontributingtotheDevice IssuesportletoftheZenossDashboardandallmonitoringwilltakeplace. AProductionStateofDecommissionedshouldresultinallmonitoringceasing;hence,all eventsgeneratedbyZenosswillceaseandnorelatedNotificationswillbegenerated; however,externallygeneratedevents(fromsyslog,externalTRAPs,Windowsevent logs)willcontinuetobereceivedandrelatedNotificationswillbegeneratedunlessa triggerfilterexcludingonProductionStateexists.Thedevicewillnotberecordedinthe DashboardDeviceIssuesportlet.NotethattheoverallStatusicononadevice'sStatus pagewillturngreen!
23January2013
125
AnyProductionStateotherthanProductionwillresultinthedevicenotbeingincluded ontheDashboardDeviceIssuesportlet. TheonlyProductionStatethatautomaticallystopsallmonitoringis Decommissioned; however,thezPropertyofzProdStateThresholdcanbesetaspartofthe ConfigurationPropertiesofadeviceordeviceclass.Thisvariablecontrolsthe ProductionStatevaluebeneathwhichallmonitoringceases.Bydefaultthisvalueis 300whichmeansthatsettingaProductionStateofMaintenancedoesnotpreventping andsnmpmonitoring.IfyouwanttopreventallmonitoringforMaintenancestate devices,changethezProdStateThresholdvalueatthetopdeviceclasslevelto301.
11 Accessing events with the JSON API

DuringthelifeofZenoss3,theJSONAPIwasintroducedasameansofaccessingdata withinZenoss.Insomeways,itissimilartousingthezendmdPythonenvironmentand inmanycasesitreflectsthesamecallsavailableinzendmd,butagreatadvantageof theJSONAPIisthatitcanbeusedremotelyfromtheZenossserveranditrequiresno intimateknowledgeofPython.
11.1 Definitions
Forthosewhoarenotfromadevelopmentbackground(andpossiblywithapologiesto thosewhoare),herearesomedefinitions. AnApplicationProgrammingInterface(API)isawayofaccessingstuff. StuffinthecontextofZenossmeansobjectsthatrepresentrealthings.Forexample, Pythonobjectsthatrepresentdevices,networkinterfaces,filesystems,processesand users;databaseobjectsintheMySQLdatabasethatrepresentevents. JavaScriptObjectNotation(JSON)isalightweightdatainterchangeformat.Itiseasy forhumanstoreadandwritebeingatextformatthatiscompletelylanguage independentbutusesconventionsthatarefamiliartoprogrammersoftheCfamilyof languages,includingC,C++,C#,Java,JavaScript,Perl,Python,andmanyothers. ThustheJSONAPIprovidesadocumentedwayofaccessingdifferentsortsofdata withinZenoss,usingacommoninterface.Whateverstuffisbeingaccessed,wepresent requestsinatextformatandtheresultsaretranslatedbackintotextformatforus. Inordertopresentourrequestsfordata,aURLisrequiredplusauseridandpassword thathasauthoritytoaccesstheZenossdatarequested.Asusers,wecanconstruct requestsinexactlythesamewayastheZenossGUIdoes;theZenossGUIitselfusesthe JSONAPItopresentdatatous. AnotherbenefitofusingtheJSONAPIratherthanusingPythondirectly,isthatZenoss DevelopmentmaychangetheunderlyingPythonintheZenossCorecodebut,provided theymaintaintheJSONAPIinterface,anyaccessfunctionalitybuiltontopoftheAPI
126
23January2013
canremainunchanged.ForthisreasonthereisarecommendationthattheAPIbeused inpreferencetowritingPythoncodetoaccessdatadirectly.
11.2 Understanding the JSON API

TheJSONAPIisshippedasstandardwithZenossCore.Thedocumentationcanbe foundat http://community.zenoss.org/community/documentation/official_documentation/api;this isactuallyazippedbundlecontainingdocumentationinhtmlformat,apdfguideand bothPythonandJavasamplesforusingtheAPI. TherearealsosomesamplesofusingtheJSONAPIwithbashandcurlat https://gist.github.com/1901884. TheJSONAPIexposesthemethodsthatcanbefoundintheZenosscodeunder $ZENHOME/Products/Zuul/routers. Theeasiestwaytoviewthedocumentationistodownloadthezipbundle,unzipitand pointabrowserattheapidoc/html/index.htmlfile.
Figure90:JSONAPIdocumentationinhtmlformat
Thelefthandmenusshowthemodules,effectivelythefilesthatcanbefoundunder $ZENHOME/Zuul/Products/routers.Typicallythesefileseachdefineoneclassthough thenetworkfilehasaclassforeachofNetworkRouterandNetwork6Router. Clickonamoduletoseeanoverviewofwhatitcontains.Notethe Availableatlinethat helpsindicatetheurlthatreachesthisdata. ClickonthelinktotheClass,EventsRouter,toseeallthemethodsforthisclass. 23January2013 EventManagementforZenossCore4Skills1stLtd 127
Figure91:JSONAPIdetailsofthezepmodule
Figure92:JSONAPImethodsfortheEventsRouterclass
Clickonamethodtogetamoredetailedoverviewwithdescriptionsoftheinput parametersandthevaluesreturned.
128
23January2013
Figure93:JSONAPIdetailsforthequerymethodintheEventsRouterclass
Atalllevelsofthedocumentationtherearelinkstothe sourcecode.Thisshouldbevery closetothecodethatyouseeifyouinspectthefile$ZENHOME/Products/Zuul/routers thoughthelinenumbersmaynotmatchexactlydependingontheexactlevelofcodeyou arerunning.
Figure94:JSONAPIsourcecodeforthequerymethod
23January2013
129
Ifyouinspectthe__init__methodsourcecodefortheEventsRouterclass,youcansee thatthezepattributeissetto:
self.zep = Zuul.getFacade('zep', context)
Eachofthefilesin$ZENHOME/Products/Zuul/routershasmethodsthatcallthe matchingfacadefoundunder$ZENHOME/Products/Zuul/facades. Thinkoftheroutersasawaytoreachtherightbasicareaofdatadevice,mibs, triggers,zepwithsometoplevelmethodslikequery,_buildFilter;andthinkofthe facadesasmoredetailedaccessmethods;so,havinggainedaccesstotheevents throughthezeprouter,thefacadeprovidescreateEventFilter,getEventSummaries, acknowledgeEventSummaries,andsoon.
11.3 Using the JSON API

ThedocumentationbundleincludessamplecodeforusingtheJSONAPIfromPython programsandJavaprograms.Furthersamplesareavailableat https://gist.github.com/1901884/thatdemonstrateabashshellharnessfordrivingthe APIusingthecurlutility. NotethatthePythonsamplesbothrequireslightbugfixestodevice.pyandzep.py respectivelyin$ZENHOME/Products/Zuul/routersforthebaseZenossCore4.2code seeadiscussionandsolutionsontheZenossUser'sforumat http://community.zenoss.org/message/70052#70052.Theseissuesappeartobefixed withCore4.2.3.
11.3.1 Bash examples

Getthebashexamplesfromhttps://gist.github.com/1901884/(usetheDownloadGist link)andunpackthebundletogetzenoss_curlExamples.sh.Editthisfiletoreflectyour Zenossserverparameters,ifrequired,thoughthecodealreadyhasadefaultserverof localhost,port8080,userofadminandpasswordofzenosssoitwillprobablyworkasis ifyouhavenotchangedinstalldefaults. AllthecodetodowithservicesreferstotheenterpriseZenossResourceManager chargeableproductsotheycanberemoved.Tocutthefiledowntoabasicsamplethat justaddsadevice,n7k1,tothe/Network/Router/Ciscodeviceclass,alsoremovethe helperfunctionsforUCSandVCSobjectssothatyouendupwithashellscriptasshown inFigure95.Notethatthedeviceclasshasalsobeenchangedfromtheoriginalscript astheclassmustexist. Thesingleremainingbodylineofthescriptis:
zenoss_add_devicen7k1"/Network/Router/Cisco"
callingthehelperfunction:
zenoss_add_device()
130
23January2013
Figure95:Modifiedzenoss_curlExample.shtoaddasingle/Network/Router/Ciscodevice
Thisfunctiontakes2parameterswhere$1isthehostnameand$2isthedeviceclass.It thencallsthezenoss_apifunction:
zenoss_apidevice_routerDeviceRouteraddDevice "{\"deviceName\":\"$DEVICE_HOSTNAME\",\"deviceClass\":\"$DEVICE_CLASS\",\"c ollector\":\"localhost\",\"model\":true,\"title\":\"\",\"productionState\": \"1000\",\"priority\":\"3\",\"snmpCommunity\":\"\",\"snmpPort\":161,\"tag\" :\"\",\"rackSlot\":\"\",\"serialNumber\":\"\",\"hwManufacturer\":\"\",\"hwP roductName\":\"\",\"osManufacturer\":\"\",\"osProductName\":\"\",\"comments \":\"\"}"
zenoss_apirequiresfourparameters:
zenoss_api(){ ROUTER_ENDPOINT=$1 ROUTER_ACTION=$2 ROUTER_METHOD=$3 DATA=$4
wheretheROUTER_ENDPOINTvalueofdevice_routerisfoundfromtheJSONAPI documentationbylookingattheAvailableat:/zport/dmd/device_routerlineforthe moduleProducts.Zuul.routers.device.TheROUTER_ACTIONisDeviceRouterthe Classshowninthedocumentation;theROUTER_METHODisaddDevicethemethod foundbyexploringtheDeviceRouterclass;andtheDATAparametercontains
23January2013
131
<parametername>:<parametervalue>stringpairs,commaseparated,withdouble quotescarefullyescapedbybackslashes.
Figure96:addDevicemethodfortheDeviceRouterclassdetailinginputparameters
Ensurethattheshellscriptisexecutableandrunit.Checkthatthedeviceisadded. Thesetxlineatthetopofthescriptcanbeuncommentedtoprovidedebugging. Hereisasecondexamplethatexploresthecapabilitiesofthetriggersinterface. ExploringthetriggersmodulewiththeAPIdocumentationshowsthatsomemethods needadataparameterandsomedon't.Thisiswhytherearetwohelperfunctionsin Figure97.
132
23January2013
Figure97:zenoss_JSONAPI_curl_triggers.shpart1showing2helperfunctions
Figure98:zenoss_JSONAPI_curl_triggers.shpart2callingthehelperfunctionswithdifferentmethods
Themainbodyofzenoss_JSONAPI_curl_triggers.shhastwocallstozenoss_api_triggers (withnodataparameter)toproducealistoftriggersandthedetailforeachtrigger, respectively;thethirdcallusesthesecondhelperfunctionwiththe getTriggermethod andprovidesauuidparametertojustgetthedetailofaspecifictrigger.Theuuidwas 23January2013 EventManagementforZenossCore4Skills1stLtd 133
determinedfromthegetTriggerListoutputandthenhardcodedbackintothescriptasan example. OutputlookslikeFigure99.
Figure99:Outputfromzenoss_JSONAPI_curl_triggers.sh
Notethatusingthebash/curlinterfacewiththeEventsRouterclassinthezeprouter module,ismuchharderasmanyofthemethodsrequireadictionaryasaninput parameter.Forthisreason,itiseasiertodrivetheeventspartoftheJSONAPIfroma Pythonharness.
11.3.2 Python examples

TheJSONAPIdocumentationbundledeliversapythonsubdirectorywithexamples.Be suretocheckhttp://community.zenoss.org/message/70052#70052ifyouareseeing unexplainableerrors. api_example.pyprovidesagenericclass,ZenossAPIExample(),whichconnectstothe Zenossserver.
134
23January2013
Figure100:api_example.pypart1withconnectionlogicandroutersdefined
Theclasshasa_router_requestmethodthathasparametersfortherouterclassto connectto,themethodtoexecuteandadatalistthatpassesparameterstothemethod, performingthetranslationbetweenPythonobjectsandJSON,asrequired. Fourhelperfunctionsarealsoprovidedinapi_example.py,eachofwhichutilisesthe _router_requestmethod.
23January2013
135
Figure101:api_example.pypart2with_router_requestmethod
defget_devices(self,deviceClass='/zport/dmd/Devices'): defget_events(self,device=None,component=None,eventClass=None): defadd_device(self,deviceName,deviceClass): defcreate_event_on_device(self,device,severity,summary):
Figure102:api_example.pypart3withhelpermethodstoaccessdeviceandeventsobjects
136
23January2013
event_curses.pyisanexamplescriptthatimportsapi_exampleandusestheget_events methodtoaccesseventsintheMySQLdatabase.Theonlyotherdependencyisthe importoftexttablewhichisalsoincludedinthesamedirectory(see JSONAPIQuickstart.txtinthetopleveldirectoryofthedocumentation).
Figure103:event_curses.pyhighlightingcallstotheapi_examplefunctionality
Whenevent_curses.pyisrunwithpythonevent_curses.py,alistofeventsisoutputtothe screenwithDevice,Component,SummaryandEventClassfields,eachlinebeing colourcodedbyseverity.Asshipped,allNewandAcknowledgedstatuseventsof severity5,4,3and2,areretrievedfromtheMySQLdatabase.
23January2013
137
Figure104:Outputofpythonevent_curses.py
Notethatifevent_curses.pydoesnotrunthenopenanewcommandterminalwitha defaultscreensizeandtryagain. Tobemoreselectiveontheeventcursesoutput,lookcloselyatthecommentedout rawEvents=lineinFigure103.Thelinerestrictsoutputtojusteventsfrom zen42.class.example.org. ForanextensionofusingthequerymethodoftheEventsRouterclass,seeget_events.py inAppendixA.Ittakesparameterstoselectthefiltercriteriaforactiveeventsandthen outputsalargenumberoffields.pythonget_events.pyhelpprovidestheusage.
138
23January2013
Figure105:get_events.pyoutputtoselectactiveeventsandoutputtotheconsole
12 Conclusions
ZenosshasanextensiveeventsystemcapableofreceivingeventsfromWindows,syslogs andSNMPTRAPs,inadditiontoreceivingtheeventsgeneratedinternallybyZenoss's owndiscovery,availabilityandperformancemonitoring. AlargenumberofeventclassesaredefinedandconfiguredwhenZenossisinstalled. Thesecanbemodified,removedoraddedto. Aneventfollowsafairlycomplexeventlifecycleprocesswherebyitismappedtoan eventclassandthen,optionally,itistransformedsuchthatdefaultfieldsoftheevent canbechangedanduserdefinedfieldscanbecreated. EventmappingforeventsfromWindows,syslogsorSNMP,dependsontheinitial ZenossparsingdaemondeliveringaneventClassKeyfieldwhichmustcorrespondtoa definedmapping.Subsequently,aPythonRuleand/oraPythonRegexcanbeusedto furtherdistinguishbetweenincomingeventsandmaptodifferenteventclasses.
23January2013
139
Figure106:Eventattributesthroughtheeventlifecycle(part1)
DevicecontextisappliedtoanincomingeventfromtheZODBdatabase;devicecontext includestheprodState,DevicePriority,Location,DeviceClass,DeviceGroupsand Systemsfieldvalues.Devicecontextprovidestheabilityfortransformstotakeaccount ofthedeviceordeviceclasshierarchy. AneventclassincludeseventcontextzEventAction,zEventSeverityand zEventClearClasseswhichcanbeappliedtoindividualsubclassesofeventsortoclass hierarchies.Thismeanstransformscanbeaffectedbyeventtype. Eventtransformscanbesimpleassignmentofeventfieldsorcanincludecomplex Pythonprograms.AgoodenvironmentfortestingPythonisthezendmdcommandline utility.Transformsand/ortheeventcontextcanbeusedtohelpcleareventsthathave beenresolved.AnyeventwithaseverityofClearedwillautomaticallyclearother similarevents;zEventClearClassescanbeusedtolistextraclassesthatareclearedin addition.
140
23January2013
EventsaresavedintheMySQLzenoss_zepdatabaseintheevent_summarytable. EventscanbeClosedbyusersorClearedbyotherevents;theycanalsobeAgedbased onseverityandlengthoftimethattheeventhaspersisted.Afteraconfigurable interval,nonactiveevents(witheventStateofClosed,ClearedandAged)aremovedto theevent_archivetableofthedatabase.Eventually,archivedeventscanbedeleted.
23January2013
141
Wheneventsoccur,actionscanbegeneratedeithertoalertusersbyusingemailora pagingsystem;alternatively,backgroundactionscanbeconfiguredtorunacommand ontheZenossserverortogenerateanSNMPTRAP. TheJSONAPIprovidesagenericinterfaceforaccessingdataintheZenosssystem. Aswithanyenterprisemanagementsystem,Zenosshasthetoolstoconfigurealmost anyresponsetoanyevent.
142
23January2013
13 Appendix A
13.1 getevents.py
get_events.pytoselectactiveevents.
#Zenoss4.xJSONAPIExample(python) # #Toquicklyexplore,execute'pythoniget_events.py # #>>>z=getEventsWithJSON() #>>>events=z.get_events() #etc. importjson importurllib importurllib2 fromoptparseimportOptionParser importpprint #ZENOSS_INSTANCE='http://ZENOSSSERVER:8080' #Changethenextline(s)tosuityourenvironment # ZENOSS_INSTANCE='http://zen42.class.example.org:8080' ZENOSS_USERNAME='admin' ZENOSS_PASSWORD='zenoss' ROUTERS={'MessagingRouter':'messaging', 'EventsRouter':'evconsole', 'ProcessRouter':'process', 'ServiceRouter':'service', 'DeviceRouter':'device', 'NetworkRouter':'network', 'TemplateRouter':'template', 'DetailNavRouter':'detailnav', 'ReportRouter':'report', 'MibRouter':'mib', 'ZenPackRouter':'zenpack'} classgetEventsWithJSON(): def__init__(self,debug=False): """ InitializetheAPIconnection,login,andstoreauthentication cookie """ #UsetheHTTPCookieProcessorasurllib2doesnotsavecookiesby default self.urlOpener= urllib2.build_opener(urllib2.HTTPCookieProcessor()) ifdebug: self.urlOpener.add_handler(urllib2.HTTPHandler(debuglevel=1)) self.reqCount=1 #ConstructPOSTparamsandsubmitlogin. loginParams=urllib.urlencode(dict( __ac_name=ZENOSS_USERNAME, __ac_password=ZENOSS_PASSWORD,
23January2013
143
submitted='true', came_from=ZENOSS_INSTANCE+'/zport/dmd')) self.urlOpener.open(ZENOSS_INSTANCE+ '/zport/acl_users/cookieAuthHelper/login', loginParams) def_router_request(self,router,method,data=[]): ifrouternotinROUTERS: raiseException('Router"'+router+'"notavailable.') #ConstructastandardURLrequestforAPIcalls req=urllib2.Request(ZENOSS_INSTANCE+'/zport/dmd/'+ ROUTERS[router]+'_router') #NOTE:ContenttypeMUSTbesetto'application/json'forthese requests req.add_header('Contenttype','application/json;charset=utf8') #ConverttherequestparametersintoJSON reqData=json.dumps([dict( action=router, method=method, data=data, type='rpc', tid=self.reqCount)]) #Incrementtherequestcount('tid').Moreimportantifsending multiple #callsinasinglerequest self.reqCount+=1 #SubmittherequestandconvertthereturnedJSONtoobjects returnjson.loads(self.urlOpener.open(req,reqData).read()) defget_events(self,filter={},sort='severity',dir='DESC'): """UseEventsRouteraction(Class)andquerymethodfound inJSONAPIdocsonZenosswebsite: query(self,limit=0,start=0,sort='lastTime',dir='desc', params=None, archive=False,uid=None,detailFormat=False) Parameters: limit(integer)(optional)Maxindexofeventstoretrieve (default:0) start(integer)(optional)Minindexofeventstoretrieve (default:0) sort(string)(optional)Keyonwhichtosortthereturnresults (default:'lastTime') dir(string)(optional)Sortorder;canbeeither'ASC'or'DESC' (default:'DESC') params(dictionary)(optional)Keyvaluepairoffiltersforthis search.(default:None) paramsarethefilterstothequerymethodandcanbefoundinthe _buildFiltermethod. severity=params.get('severity'), status=[iforiinparams.get('eventState',[])], event_class=filter(None,[params.get('eventClass')]),
144
23January2013
Notethatthetimevaluescanberangeswhereavalidrange wouldbe '2012090707:57:33/2012112217:57:33' first_seen=params.get('firstTime')and self._timeRange(params.get('firstTime')), last_seen=params.get('lastTime')and self._timeRange(params.get('lastTime')), status_change=params.get('stateChange')and self._timeRange(params.get('stateChange')), uuid=filterEventUuids, count_range=params.get('count'), element_title=params.get('device'), element_sub_title=params.get('component'), event_summary=params.get('summary'), current_user_name=params.get('ownerid'), agent=params.get('agent'), monitor=params.get('monitor'), fingerprint=params.get('dedupid'), tags=params.get('tags'), details=details, archive(boolean)(optional)Truetosearchtheeventhistory tableinsteadofactiveevents(default:False) uid(string)(optional)Contextforthequery(default:None) Returns:dictionary Properties: events:([dictionary])Listofobjectsrepresentingevents totalCount:(integer)Totalcountofeventsreturned asof:(float)Currenttime """ data=dict(start=0,limit=1000) ifsort:data['sort']=sort ifdir:data['dir']=dir data['params']=filter #print'data[params]is%s\n'%(data['params']) #print'datais%s\n'%(data) returnself._router_request('EventsRouter','query',[data]) ['result'] if__name__=="__main__": usage='python%progseverity=severityeventState=eventState device=deviceeventClass=eventClasscomponent=componentagent=agent monitor=monitorcount=countlastTime=lastTimefirstTime=firstTime stateChange=stateChangesort=lastTimedir=DESC' parser=OptionParser(usage) parser.add_option("severity",dest='severity', help='severitycommaseparatednumericvalueseg. severity=5,4forCriticalandError') parser.add_option("eventState",dest='eventState',default='0,1', help='eventStatecommaseparatednumericvalues eg.eventState=0,1forNewandAck') parser.add_option("device",dest='device', help='eg.device=\'zen42.class.example.org\'') parser.add_option("eventClass",dest='eventClass',
23January2013
145
help='eg.eventClass=\'/Skills\'') parser.add_option("component",dest='component', help='eg.component=\'TestComponent\'') parser.add_option("agent",dest='agent', help='eg.agent=\'zensyslog\'') parser.add_option("monitor",dest='monitor', help='eg.monitor=\'localhost\'') parser.add_option("count",dest='count', help='numericvalueeg.count=3orrangecount 3,30') parser.add_option("lastTime",dest='lastTime', help='eg.forarangeseparatestart&endwith/ lastTime=\'2012090707:57:33/2012112217:57:33\'') parser.add_option("firstTime",dest='firstTime', help='eg.firstTime=\'2012112217:57:33\'') parser.add_option("stateChange",dest='stateChange', help='eg.stateChange=\'2012112217:57:33\'') parser.add_option("sort",dest='sort',default='lastTime', help='thekeytosortoneg.sort=\'lastTime\'') parser.add_option("dir",dest='dir',default='DESC', help='thedirectiontosorteg.dir=\'ASC\'or dir=\'DESC\'') (options,args)=parser.parse_args() #optionsisanobjectwewantthedictionaryvalueofit #Someoftheoptionsneedalittlemunging... option_dict=vars(options) ifoption_dict['severity']: option_dict['severity']=option_dict['severity'].split(',') ifoption_dict['eventState']: option_dict['eventState']=option_dict['eventState'].split(',') #countcaneitherbeanumberorarange(ineitherlistortuple format) #(see$ZENHOME/Products/Zuul/facades/zepfacade.py createEventFiltermethod) #butifthismethodgetsalistitassumesthereare2elementsto thelist. #Wemaygetalistwithasinglevaluesoconvertittoanumber andthe #createEventFiltermethodcancope ifoption_dict['count']: option_dict['count']=option_dict['count'].split(',') iflen(option_dict['count'])==1: option_dict['count']=int(option_dict['count'][0]) #option_dictincludesthesortanddirkeys(aswehavedefaultedthem inoptparse) #Thesearenotpartofthefilterstringsoweneedtopopthemoutof thedictionary #touseseparately. s=option_dict.pop('sort') d=option_dict.pop('dir') #Needtocheckthesekeysforsanity #andprovidesensibledefaultsotherwise dirlist=['ASC','DESC'] ifnotdindirlist: d='DESC'
146
23January2013
sortlist=['severity','eventState','eventClass','firstTime', 'lastTime', 'stateChange','count','device','component','agent', 'monitor'] ifnotsinsortlist: s='lastTime' #print'optionsis%s\n'%(options) #print'option_dictis%s\n'%(option_dict) events=getEventsWithJSON() #filter['evid']='000c29d9f87b838911e2347cddf7a720' pp=pprint.PrettyPrinter(indent=4) fields=['eventState','DeviceClass','count','device','Location', 'Systems','severity','firstTime','lastTime','summary'] #fields=['eventState','DeviceClass','count','device','Location', 'severity','firstTime','lastTime','summary'] print'eventState,DeviceClass,count,device,Location,Systems, severity,firstTime,lastTime,summary' #print'eventState,DeviceClass,count,device,Location,severity, firstTime,lastTime,summary' out=events.get_events(filter=option_dict,sort=s,dir=d) foreinout['events']: #pp.pprint(e) outState=e['eventState'] ife['DeviceClass']: outDeviceClass=e['DeviceClass'][0]['name'] else:outDeviceClass=[] outcount=e['count'] outdevice=e['device']['text'] ife['Location']: outLocation=e['Location'][0]['name'] else:outLocation=[] outSystems=[] forpos,valinenumerate(e['Systems']): sy=str(e['Systems'][pos]['name']) outSystems.append(sy) outseverity=e['severity'] outfirstTime=e['firstTime'] outlastTime=e['lastTime'] outsummary=e['summary'] print'%s,%s,%s,%s,%s,%s,%s,%s,%s,%s'%(outState, outDeviceClass,outcount,outdevice,outLocation,outSystems,outseverity, outfirstTime,outlastTime,outsummary) #print'%s,%s,%s,%s,%s,%s,%s,%s,%s'%(outState, outDeviceClass,outcount,outdevice,outLocation,outseverity, outfirstTime,outlastTime,outsummary) #print'\ntotalCountis%dandasofis%s'%(out['totalCount'], out['asof'])
23January2013
147
13.2 zensendevent
Modifiedzensendeventtoautomaticallyretrievelocalauthenticationparameters. ZenossCore4.2.3changedsomesecuritypoliciesatinstallationtimewhichresultsin zensendeventfailingunlessauthparametersaredeterminedandsuppliedexplicitly.
#!/opt/zenoss/bin/python __doc__="""zensendevent SendeventsonacommandlineviaXMLRPCorfromaXMLfile. ThiscommandcanbeputonanymachinewithPythoninstalled,and doesnotneedZopeorZenoss. """ importsocket fromxmlrpclibimportServerProxy fromoptparseimportOptionParser fromxml.saximportmake_parser,saxutils fromxml.sax.handlerimportContentHandler XML_RPC_PORT=8081 sevconvert={ "critical":5, "error":4, "warn":3, "info":2, "debug":1, "clear":0 } classImportEventXML(ContentHandler): ignoredElements=set([ 'ZenossEvents','url','SourceComponent', 'ReporterComponent','EventId', 'clearid','eventClassMapping', 'eventState','lastTime','firstTime','prodState', 'EventSpecific','stateChange', ]) evt={} property='' value='' def__init__(self,serv): ContentHandler.__init__(self) self.sent=0 self.total=0 self.serv=serv defstartElement(self,name,attrs): self.value='' ifname=='ZenossEvent': self.evt={} elifname=='property':
148
23January2013
self.property=attrs['name'] defcharacters(self,content): self.value+=content defendElement(self,name): name=str(name) value=str(self.value) ifnameinself.ignoredElements: return elifname=='property'andvalueandvalue!='|': self.evt[self.property]=value elifnamein['Systems','DeviceGroups']: ifvalueandvalue!='|': self.evt[name]=value elifnamein['eventClassKey','eventKey']: ifvalue: self.evt[name]=value elifname=='severity': self.evt[name]=int(value) elifname=='ZenossEvent': self.total+=1 try: self.serv.sendEvent(self.evt) self.sent+=1 exceptException,ex: printstr(ex) printevt elifvalue: self.evt[name]=value defsendXMLEvents(serv,xmlfile): infile=open(xmlfile) parser=make_parser() CH=ImportEventXML(serv) parser.setContentHandler(CH) try: parser.parse(infile) finally: infile.close() print"Sent%sof%sevents"%(CH.sent,CH.total) device=socket.getfqdn() ifdevice.endswith('.'):device=device[:1] parser=OptionParser(usage="usage:%prog[options]summary") parser.add_option("d","device",dest="device",default=device, help="devicefromwhichthiseventissent,default:%default") parser.add_option("i","ipAddress",dest="ipAddress",default="", help="Ipfromwhichthiseventwassent,default:%default") parser.add_option("y","eventkey",dest="eventkey",default="", help="eventKeytobeused,default:%default") parser.add_option("p","component",dest="component",default="", help="componentfromwhichthiseventissent,default:''")
23January2013
149
parser.add_option("k","eventclasskey",dest="eventClassKey", default="", help="eventClassKeyforthisevent,default:''") parser.add_option("s","severity",dest="severity",default="Warn", help="severityofthisevent:Critical,Error,Warn,Info,Debug, Clear") parser.add_option("c","eventclass",dest="eventClass",default=None, help="eventclassforthisevent,default:''") parser.add_option("monitor",dest="monitor",default="localhost", help="monitorfromwhichthiseventcame") parser.add_option("port",dest="port",default=XML_RPC_PORT, help="xmlrpcserverport,default:%default") parser.add_option("server",dest="server",default="localhost", help="xmlrpcserver,default:%default") parser.add_option("auth",dest="auth",default="admin:zenoss", help="xmlrpcserverauth,default:%default") parser.add_option("o","other",dest="other",default=[], action='append', help="Specifyotherevent_field=valuearguments.Canbespecified" "morethanonce.") parser.add_option('f',"file",dest="input_file",default="", help="ImporteventsfromXMLfile.") parser.add_option('v',dest="show_event",default=False, action='store_true', help="ShowtheeventdatasenttoZenoss.") opts,args=parser.parse_args() #HackbyJCtogethubpasswdauthenticationintoauthoption #Passwordisheldin$ZENHOME/etc/hubpasswdin(almost)correctformat <user>:<password>\n importos #ifauthisthedefault ifopts.auth=='admin:zenoss': zenhome=os.environ['ZENHOME'] #Trytoaccess$ZENHOME/etc/hubpasswdandstriptrailingnewline try: pwfile=open(os.path.join(zenhome,'etc','hubpasswd'),'r') opts.auth=pwfile.read().rstrip() pwfile.close() print'Extractingnecessaryuser:passwordautomatically\n' #Ifthisfailsthenfallbacktodefaultandprintmessage except: print'Attempttodetecthubpasswdfailed\n' #EndofJChack url="http://%s@%s:%s"%(opts.auth,opts.server,opts.port) serv=ServerProxy(url) ifopts.input_file: sendXMLEvents(serv,opts.input_file) importsys sys.exit(0) evt={} ifopts.severity.lower()insevconvert: evt['severity']=sevconvert[opts.severity.lower()] else:
150
23January2013
parser.error('Unknownseverity') evt['summary']="".join(args) ifnotevt['summary']: parser.error('nosummarysupplied') evt['device']=opts.device evt['component']=opts.component evt['ipAddress']=opts.ipAddress ifopts.eventkey: evt['eventKey']=opts.eventkey ifopts.eventClassKey: evt['eventClassKey']=opts.eventClassKey ifopts.eventClass: evt['eventClass']=opts.eventClass evt['monitor']=opts.monitor forlineinopts.other: try: field,value=line.split('=',1) evt[field]=value except: pass ifopts.show_event: frompprintimportpprint pprint(evt) serv.sendEvent(evt)
23January2013
151
14 References
1. ZenossCommunitysitehttp://community.zenoss.org 2. Zenossnetwork,systemsandapplicationmonitoringcommercialsite http://www.zenoss.com/ 3. Zenossdocumentationmainpage http://community.zenoss.org/community/documentation 4. ZenossCore4AdministrationGuide http://community.zenoss.org/community/documentation/official_documentation/ze nossguide 5. ZenossDeveloper'sGuide http://community.zenoss.org/community/documentation/official_documentation/ze nossdevguide 6. Zenoss4.2JSONAPIdocumentation
http://community.zenoss.org/community/documentation/official_documentation/api
7. SamplesofusingtheJSONAPIwithbashandcurlcanbefoundat https://gist.github.com/1901884. 8. InformationonRelStorageandmemcachedhttp://wiki.zenoss.org/RelStorage 9. InformationonRabbitMQhttp://wiki.zenoss.org/Working_with_Queues 10. ScripttoresetRabbitMQhttps://gist.github.com/4192854 11. InformationonAMQPhttp://www.amqp.org/ 12. InformationonLuceneindexinghttp://lucene.apache.org/core/ 13. InformationonJSONhttp://www.json.org/ 14. DiscussiononmodifyingzensendeventutilityonZenosswiki http://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3 15. ReferenceforWin32_NTLogEventclasseventlogseverities http://msdn.microsoft.com/en gb/library/windows/desktop/aa394226%28v=vs.85%29.aspx 16. InformationonPythonregularexpressions http://docs.python.org/2/library/re.html, http://www.python.org/doc/2.5.2/lib/re-syntax.html and http://docs.python.org/dev/howto/regex.html 17. Informationonprotobufshttp://code.google.com/p/protobuf/ 18. InformationonthePythondebugger(pdb) http://docs.python.org/2/library/pdb.html 19. AsageneralPythonreference,tryLearningPythonbyMarkLutz,publishedby O'Reilly
152
23January2013
20. TheMIBBrowserZenPack.Documentationandcommentsat http://community.zenoss.org/docs/DOC10321;codefrom http://wiki.zenoss.org/ZenPack:MIB_Browser. 21. SNMPRequestsForComment(RFCs)http://www.ietf.org/rfc.html

V1RFCs1155,1157,1212,1213,1215 V2RFCs2578,2579,2580,3416,3417,3418 V3RFCs25782580,341618,3411,3412,3413,3414,3415
22. SNMPHostResourcesMIB,RFCs1514and2790http://www.ietf.org/rfc.html 23. FortheextensionSNMPMIBfromInformant,goto http://www.wtcs.org/informant/index.htm 24. ForinformationonZopeTALESexpressions,see http://docs.zope.org/zope2/zope2book/AppendixC.html 25. DatagramSyslogClienthttp://syslogserver.comforsyslogWindowssystems. 26. Raddlenetworkemulationopensourcepackagehttp://raddle.sourceforge.net/ 27. Zenoss4EventManagementWorkshopavailablefromSkills1stLtd, http://www.skills1st.co.uk/products/courses/
23January2013
153
Acknowledgements
AnumberofpeoplehavecontributedinformationandadvicetothisprojectandIwould liketothankthem.

GeorgesReichsfortheoriginalamazingarchitecturedesigndiagram ChetLutherforhisawesomeknowledgeofZenossandhiswillingnesstoshare thatknowledge AndrewKirchforinitialproofreadingandsomeusefulcomments AndrewFindlayofSkills1stforhelpwithtypesetting
Abouttheauthor
JaneCurryhasbeenanetworkandsystemsmanagementtechnicalconsultantand trainerfor25years.Duringher11yearsworkingforIBMshefulfilledbothpresales andconsultancyrolesspanningthefullrangeofIBM'sSystemViewproductspriorto 1996andthen,whenIBMboughtTivoli,shespecialisedinthesystemsmanagement productsofDistributedMonitoring&IBMTivoliMonitoring(ITM),thenetwork managementproduct,TivoliNetViewandtheproblemmanagementproductTivoli EnterpriseConsole(TEC).AllarebasedaroundtheTivoliFrameworkarchitecture. Since1997Janehasbeenanindependentbusinesswomanworkingwithmany companies,bothlargeandsmall,commercialandpublicsector,deliveringTivoli consultancyandtraining.Overthelast5yearsherworkhasbeenmoreinvolvedwith OpenSourceofferings,especiallyZenoss. ShehasdevelopedanumberofZenPackaddonstoZenossCoreandhasalargenumber oflocalandremoteconsultancyclientsforZenosscustomisationanddevelopment.She hasalsocreatedseveralworkshopofferingstoaugmentZenoss'sowneducational offerings.SheisafrequentcontributortotheZenossforumsandIRCchat conversationsandwasmadeaZenossMasterbyZenossinFebruary2009
154
23January2013

Zenoss Core4 Event Management Paper

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Zenoss Core4 Event Management Paper

Transféré par

Droits d'auteur :

Formats disponibles

Event Management for Zenoss Core 4

January 2013 Jane Curry Skills 1st Ltd www.skills-1st.co.uk

JaneCurry Skills1stLtd 2CedarChase Taplow Maidenhead SL60EU 01628782565 jane.curry@skills1st.co.uk www.skills1st.co.uk

2 Zenoss event architecture

ThereareanumberwaystoaccesstheEventConsole.ThemainEventConsoleis reachedfromthetopEVENTS>EventConsolemenu.Thedefaultistoshowevents withaseverityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecent first).Eventsareassigneddifferentseverities: Name Number Colour

Critical Error Warning Info Debug Cleared

Red Orange Yellow Blue Grey Green

AlleventsalsohaveaneventStatefield.Zenoss3eventStatehadthreepossiblevalues New,AcknowledgedandSuppressed.Zenoss4hasenhancedthesedefinitionssowe nowhave: Name Number Description

New Acknowledged Suppressed Closed Cleared Dropped Aged

Neweventnoprevioussimilarevent Acknowledgedbyuserorrule Typicallyfrombeyondasinglepointof failure Closedbyauser Closedbyarule Discardednotsavedinthedatabase Autoclosedduetoage/severity

Fromadevice'sdetailpage,selectEventsinthelefthandmenu Foradeviceclass,clicktheDETAILSlinkandthenEventsinthelefthand menu ForaLocation,GrouporSystem,clicktheDETAILSlinkandthenEventsin thelefthandmenu FromanEventClass,selectEventsinthelefthandmenu

2.2 Event Manager settings

2.3 Event database tables

TheformatofeachofthesetablesandthevalidfieldsforaZenosseventcanbeseenby examiningtheZenossdatabasesetupfilein $ZENHOME/Products/ZenEvents/db/zenevents.sql,where$ZENHOMEwillbe /opt/zenossforaCore4.2ZenossonRedHat/CentOS(theonlycurrentlysupported platform).

zenevents.sqlalsodefinesthehistorytableinasimilarfashion. Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.Thedetail tablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthatthe Zenossadministratorrequiresforanevent.

IfyouareusingZenosspriortoversion4,gettheolderversionofthisZenossEvent Managementpaperfromhttp://www.skills 1st.co.uk/papers/jane/zenoss_event_management_paper.pdf.

Inpassing,notethatinadditiontothezenoss_zepdatabase,theirisalsoa zodbanda zodb_sessiondatabase.TheZopedatabase(ZODB)thatstoresalltheobjects(devices, deviceclasses,processes,networks,etc)isnowinMySQL. Examiningthetablesofthezenoss_zepdatabaseiswherethingsdivergesignificantly frompreviousversions.

Themaintablesarenowevent_summaryandevent_archivebutthestructureis morecomplicated.Someofthedataisheldinseparatetableswithpointerstothem fromthemaintables.Theseinclude:

agent event_class event_class_key event_group event_key monitor EventManagementforZenossCore4Skills1stLtd 23January2013

suppid(whichwasneverused)hasdisappearedintheZenoss4schema.managerhas alsodisappearedfromZenoss4. Thesetablesarecreatedbythefilesin$ZENHOME/share/zeneventserver/sql/mysql.

2.4 New event daemons

Aneasywaytoseequeuesbuildingupistotemporarilystopzeneventdandthe raweventsqueuewillthenbuildrapidly. 20 EventManagementforZenossCore4Skills1stLtd 23January2013

Thereisafurtherscriptavailableatgist,writtenbycluther,toresetRabbitMQ https://gist.github.com/4192854. TwoutilitiesareavailableforthezenossusertogetRabbitMQinformation:

Thecountparametergivesacontinualoutputoftimestampandqueuelength. Thepurgeparameterpurgeseventsfromaqueue.ThiscommandissafewhenZenossis running. ThedeleteparameterdeletesthequeueandshouldnotbeusedwhenZenossisrunning. zenqdoesnothaveauthenticationparameters.

2.5 Other database-related changes in Zenoss 4

2.6 Event life cycle

Eventgeneration Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent Eventclassmappingtodistinguishonetype(class)ofeventfromanother Eventcontextadditionalinformationpertinenttoaclassofevent

Eventtransformmanipulationofeventfields Databaseinsertionanddeduplication Resolution Ageingandarchiving

New Acknowledged Suppressed Closed Cleared Dropped Aged

Anewevent Acknowledgedbyuserortransform Eventtypicallybeyondasinglepointoffailure Eventresolvedbyauser Eventresolvedbyanautomaticrule WouldneverreachtheMySQLdatabase

2.6.1 Event generation

Zenossdaemon zensyslog zeneventlog zentrap

Exampleofwheneventgenerated processessyslogeventsreceivedonUDP/514(default) processesWindowseventsreceivedusingWMI processesSNMPTRAPsreceivedonUDP/162

2.6.2 Application of device context

prodState DevicePriority Location DeviceClass DeviceGroups Systems ipAddress(mayhavealreadybeenassigned)

2.6.3 Event class mapping

2.6.4 Application of event context

zEventAction zEventClearClasses zEventSeverity

status|history|dropdefaultisstatus bydefaultthisisanemptyPythonlistofstrings Originalbydefault

2.6.5 Event transforms

2.6.6 Database insertions and de-duplication

count eventState evid stateChange dedupid eventClassMapping firstTime lastTime

ItistheJavazeneventserverdaemonthatisresponsibleforgettingeventsintothedatabase. Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateevent arrives,thentherepeatcountofanexistingeventwillbeincremented.duplicateis definedashavingthefollowingfieldsthesame:

device component eventClass eventKey severity

IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalso match.Thededupidfieldiscreatedbyconcatenatingtheabovefieldstogether, separatedbythepipe(verticalbar)symbol.Thusanexamplededupidmightbe:

wherethedeviceiszenoss.skills1st.co.uk,componentissu,eventClassis/Security/Su, theeventKeyisunset,severityis5(Critical),andthesummaryisFAILEDSU(toroot) janeon/dev/pts/1. InZenoss4,thededupidfieldisalsoknownasthe fingerprint. 23January2013 EventManagementforZenossCore4Skills1stLtd 31

componentUUID eventClass eventKey(canbeblank) device component(canbeblank) eventClass eventKey(canbeblank)

2.6.8 Ageing and archiving

3 Events generated by Zenoss

SNMPPerformanceCycleInterval ProcessCycleInterval StatusCycleInterval WindowsServiceCycleInterval PingCycleTime ModelerCycleInterval

300secs(5mins) 180secs(3mins) 60secs(1min) 60secs(1min) 60secs(1min) 420mins(12hours)

news (7) user (1)

4.1 Configuring syslog.conf

Thisalsoworksforrsyslogd.SeeFigure16foranrsyslog/syslogexamplethatforwards tozen42.class.example.orgallfacilitieswithpriorityofnoticeandabovebutallcron messagesarefilteredout;authprivmessageswillbeforwardedwithseverityinfoand above.

syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogng offerssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmay alsobepresent.

4.2 Zenoss processing of syslog messages

Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,the levelofzensyslogloggingcanbeincreased. 1. UsetheINFRASTRUCTURE>Settings>Daemonsmenu. 2. Clicktheeditconfiglinkforthezensyslogdaemon. 3. ChangethefollowingparametersandclickSave: logorig logseverity selectthis Debug

9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log 10. Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.

Windows Error(1) Warning(2) Informational(3) Securityauditsuccess(4) Securityauditfailure(5)

syslogpriority<3(emerg,alert,crit)maptoZenossseverity5(Critical) syslogpriority3(err)mapstoZenossseverity4(Error) syslogpriority4(warning)mapstoZenossseverity3(Warning) syslogpriority5or6(notice,info)maptoZenossseverity2(Info)

Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsare bothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfield ishardcodedatthisstagetosyslog.

parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility. ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosetthe severityfieldoftheZenossevent. 44 EventManagementforZenossCore4Skills1stLtd 23January2013

Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefrom theincomingevent.ThedeviceandipAddressfieldsoftheZenosseventaresetatthe endofthisfunction.

TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressions atthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged. TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenoss eventsummaryfield.