Zenoss Core4 Event Management Paper

Event Management for Zenoss Core 4

January 2013

Jane Curry

Skills 1st Ltd

www.skills-1st.co.uk

JaneCurrySkills1stLtd2CedarChaseTaplowMaidenheadSL60EU01628782565

[email protected]

www.skills1st.co.uk

SynopsisThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsysteminZenossCore4.TheeventarchitecturehaschangeddramaticallyinZenoss4frompreviousversions.

ItisassumedthatthereaderisalreadyfamiliarwiththeZenossEventConsoleandwithbasicnavigationaroundtheZenossGraphicalUserInterface(GUI).ItlooksinsomedetailatthearchitecturebehindtheZenosseventsystemthedaemonsandhowtheyareinterrelatedanditlooksatthestructureofaZenosseventandtheeventlifecycle.

ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.EventsfromWindows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsareallexaminedindetail.

TheprocessbywhichanincomingeventisconvertedintoaparticularZenosseventisknownaseventmappingandthereareanumberofdifferentpossibletechniquesforperformingthatconversion.Thesewillallbeexploredalongwiththecreationofneweventclasses.

Onceaneventhasbeenreceived,classifiedandstoredbyZenoss,automationmayberequired.Alertingtousersbyemailandpageisdiscussed,asarebackgroundactionstoruncommandsorgenerateTRAPs.

LogginganddebuggingtechniquesarediscussedinsomedetailsasistheJSONAPIforextractingdataoutofZenoss.

ThispaperwaswrittenusingZenossCore4.2.3

ThepaperisacompaniontexttotheZenoss4EventManagementWorkshop.

NotationsThroughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,arehighlightedbyitalics;importantpointstotakenoteofareshowninbold.

Pointsofparticularnotearehighlightedbyanicon.

2 EventManagementforZenossCore4Skills1stLtd 23January2013

Table of Contents1Introduction..........................................................................................................................62Zenosseventarchitecture....................................................................................................6

2.1EventConsole...............................................................................................................62.2EventManagersettings.............................................................................................102.3Eventdatabasetables...............................................................................................11

2.3.1Zenoss2.xand3.x...............................................................................................112.3.2Zenoss4................................................................................................................14

2.4Neweventdaemons....................................................................................................202.4.1RabbitMQ.............................................................................................................202.4.2zeneventserver.....................................................................................................222.4.3zeneventd.............................................................................................................222.4.4zenactiond...........................................................................................................232.4.5memcached...........................................................................................................23

2.5OtherdatabaserelatedchangesinZenoss4............................................................242.6Eventlifecycle............................................................................................................25

2.6.1Eventgeneration.................................................................................................272.6.2Applicationofdevicecontext..............................................................................292.6.3Eventclassmapping...........................................................................................292.6.4Applicationofeventcontext...............................................................................302.6.5Eventtransforms.................................................................................................302.6.6Databaseinsertionsanddeduplication............................................................312.6.7Resolution............................................................................................................322.6.8Ageingandarchiving..........................................................................................34

3EventsgeneratedbyZenoss..............................................................................................343.1zenping........................................................................................................................353.2zenstatus.....................................................................................................................363.3zenprocess...................................................................................................................363.4zenwin.........................................................................................................................373.5zenwinperf...................................................................................................................373.6zenperfsnmp................................................................................................................373.7zencommand...............................................................................................................38

4Syslogevents......................................................................................................................384.1Configuringsyslog.conf.............................................................................................394.2Zenossprocessingofsyslogmessages.......................................................................40

5ZenossprocessingofWindowseventlogs.........................................................................485.1ManagementusingtheWMIprotocol.......................................................................485.2ManagementofWindowssystemsusingsyslog.......................................................51

6EventMapping...................................................................................................................516.1Workingwitheventclassesandeventmappings....................................................52

6.1.1Generatingtestevents........................................................................................546.2Regexineventmappings...........................................................................................55

23January2013 EventManagementforZenossCore4Skills1stLtd 3

6.3Rulesineventmappings............................................................................................576.4Otherelementsofeventmappings...........................................................................58

7Eventtransforms...............................................................................................................587.1Differentwaystoapplytransforms...........................................................................597.2Understandingfieldsavailableforeventprocessing...............................................60

7.2.1EventProxies.......................................................................................................637.2.2EventDetails.......................................................................................................66

7.3Transformexamples...................................................................................................687.3.1CombininguserdefinedfieldsfromRegexwithtransform.............................687.3.2Applyingeventanddevicecontextinrelationtotransforms..........................69

8Testinganddebuggingaids..............................................................................................718.1Logfiles.......................................................................................................................71

8.1.1zeneventd.log.......................................................................................................718.1.2zeneventserver.log...............................................................................................728.1.3Otherlogfiles......................................................................................................75

8.2UsingzendmdtorunPythoncommands..................................................................758.2.1ReferencinganexistingZenosseventforuseinzendmd.................................758.2.2UsingzendmdtounderstandattributesforanEventSummaryProxy...........79

8.3UsingthePythondebuggerintransforms................................................................839ZenossandSNMP..............................................................................................................87

9.1SNMPintroduction.....................................................................................................879.2SNMPonLinuxsystems............................................................................................889.3ZenossSNMParchitecture........................................................................................91

9.3.1Thezentrapdaemon............................................................................................919.4InterpretingMIBs......................................................................................................93

9.4.1zenmibexample...................................................................................................949.4.2AfewcommentsonimportingMIBswithZenoss.............................................99

9.5TheMIBBrowserZenPack......................................................................................1009.5.1ModifyingZenossCore4.2tomaketheMIBBrowserZenPackwork..........102

9.6MappingSNMPevents............................................................................................1039.6.1SNMPeventmappingexample........................................................................103

10EventTriggersandNotifications.................................................................................10810.1ZenosspriortoV4...................................................................................................10810.2Zenoss4architecture.............................................................................................10910.3Triggers...................................................................................................................11010.4Notifications............................................................................................................111

10.4.1emailNotifications..........................................................................................11310.4.2PageNotifications...........................................................................................11810.4.3CommandNotifications..................................................................................11810.4.4TRAPNotifications.........................................................................................120

10.5NotificationSchedules............................................................................................12210.6Usingzenactiond.log..............................................................................................12310.7TheeffectofdeviceProductionState....................................................................125

11AccessingeventswiththeJSONAPI...........................................................................126


11.1Definitions...............................................................................................................12611.2UnderstandingtheJSONAPI...............................................................................12711.3UsingtheJSONAPI..............................................................................................130

11.3.1Bashexamples.................................................................................................13011.3.2Pythonexamples.............................................................................................134

12Conclusions.....................................................................................................................13913AppendixA.....................................................................................................................143

13.1getevents.py............................................................................................................14313.2zensendevent..........................................................................................................148

14References.......................................................................................................................152


1 IntroductionZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.Thereisafree,Coreoffering(whichhasmostthingsyouneed),andachargeableoffering,ZenossResourceManager,whichhasextraaddongoodiessuchashighavailabilityconfigurations,distributedmanagementservers,servicemanagementandeventcorrelation;italsoincludesasupportcontract.

Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availabilitymonitoring,problemmanagementandperformancemanagement.ItisdesignedaroundtheITILconceptofaConfigurationManagementDatabase(CMDB),theZenossStandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserverandusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostorePythonobjectsandtheirstates.Zenoss3usedZEO,asalayerbetweenZopeandtheZODB;inZenoss4theZODBdataisstoredinaMySQLdatabase.

TherelationalMySQLdatabaseisalsousedtoholdcurrentandhistoricalevents.PerformancedataisheldinRoundRobinDatabase(RRD)files.

ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetworkManagementprotocol(SNMP),WindowsManagementInstrumentation(WMI)andcollectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshandtouseNagiosplugins.

Zenossprovidesdocumentationathttp://community.zenoss.org/community/documentation.ThereisalsoawealthofinformationontheZenosswebsiteinvariousforums,FAQs,andtheWiki.AusefulbookisavailablefromPACKTPublishing,ZenossCore3.xNetworkandSystemMonitoringbyMichaelBadger,whichprovidesmuchofthesameinformationastheZenossAdministrationGuidebutinamuchclearerformatwithplentyofscreenshots.AlthoughthisisaZenoss3text,itstillprovidesgoodbasicinformation.

ThispaperisanattempttoexpandontheeventinformationintheZenossCore4AdministrationGuidebydrawingonmyownexperienceandthecollectedwisdomofseveralZenossemployeesandcontributorsfromthecommunity.

2 Zenoss event architecture2.1 Event ConsoleWhenaneventarrivesatZenoss,itisparsed,associatedwithaneventclassificationandthentypically(butnotalways),itisinsertedintotheevent_summarytableofthezenoss_zepdatabase.EventscanthenbeviewedbyusersusingtheEventConsoleoftheZenossGraphicalUserInterface(GUI).


ThereareanumberwaystoaccesstheEventConsole.ThemainEventConsoleisreachedfromthetopEVENTS>EventConsolemenu.ThedefaultistoshoweventswithaseverityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecentfirst).Eventsareassigneddifferentseverities:

Name Number Colour

Critical 5 Red

Error 4 Orange

Warning 3 Yellow

Info 2 Blue

Debug 1 Grey

Cleared 0 Green

AlleventsalsohaveaneventStatefield.Zenoss3eventStatehadthreepossiblevaluesNew,AcknowledgedandSuppressed.Zenoss4hasenhancedthesedefinitionssowenowhave:

Name Number Description

New 0 Neweventnoprevioussimilarevent

Acknowledged 1 Acknowledgedbyuserorrule

Suppressed 2 Typicallyfrombeyondasinglepointoffailure

Closed 3 Closedbyauser

Cleared 4 Closedbyarule

Dropped 5 Discardednotsavedinthedatabase

Aged 6 Autoclosedduetoage/severity

NotethatClosed,ClearedandAgedeventsallhavethesamestatusiconintheEventConsole.

Bydefault,NewandAcknowledgedeventsareshownintheEventConsole.AnyeventwhichhasbeenAcknowledgedhasatickinitsstatuscolumn.ASuppressedeventisnotshownbydefaultbutcanbefilteredinifdesired;ithasasnowflakeicon.Zenossbuildsaninternaltopologyofthenetworkitismanaging(usingnmap).Ifaneventisreceivedforadevicethatthetopologymapknowsisunreachable,theeventisautomaticallysuppressed.ThusZenosshasabuiltinmechanismforpinpointingfailuredevicesandsuppressingthefloodofeventsfrombehindsuchfailurepoints.

Eventscanbesortedbyclickingonadesiredcolumnheader;clickingagainsortsinthereverseorder.Tochangetheorderofcolumns,simplydragacolumnheader.


Thereisafilterboxaboveeachcolumnheadertohelpselectrelevantevents.Mostfiltersareamatchforapartialtextstring(youdon'tneedtosupplywildcards).Datefieldsprovideacalendaricontoselectanearliestdate.Thecountfieldpermitsyoutoenterarange,forexampletoshoweventswithcount>10,use10:(ifyoutypesomethingillegalinthecountfilteritwillsupplyhelpfortherequiredsyntax).

Toselectfieldstodisplay,hoverthemouseattheendofaheadertoseethedownarrowforsorting;thethirdoptiononthedropdownmenuistoconfigurethefieldstodisplay.

FromtheEventConsole,oneormoreeventscanbeselectedbyclickingonthelinebecarefulnottoclicksomethingthatisalink(likethedevicenameoreventclass).TheiconsatthetopleftcanbeusedtoAcknowledge,Close,MaptoanEventClass,UnacknowledgeorReOpen.The+iconattheendofthisrowoficonscanbeusedtogeneratetestevents.

Doubleclickaneventtoshowthedetailsofanevent.Thisshowsbothstandardfieldsandanyuserdefinedfieldsorganisedunderseveralgroupingswhichcanbeexpandedandcontracted.AnyAcknowledge,CloseorReOpenwillbeshownatthebottom,includingwhoperformedtheaction.Freeformnotescanalsobeloggedhere.


Figure1:ZenossEventConsole

Thesummaryandmessagefieldsarefreeformtextfields.Thesummaryfieldallowsupto255characters;themessagefieldallowsupto4096characters.Thesefieldsusuallycontainsimilardata.Fordetailsofotherfields,seesection7.1.2oftheZenossCore4Administrationguide.

Bydefault,theEventConsoleisrefreshedeveryminute.ThedropdownbesidetheRefreshbuttonallowsyoutochangetheintervalortorefreshmanually.


Figure2:EventdetailsshowingAcknowledgementandaddednote

EventConsolesarealsoavailableatvariousplacesintheGUIwhichhavefiltersalreadyapplied:

Fromadevice'sdetailpage,selectEventsinthelefthandmenu

Foradeviceclass,clicktheDETAILSlinkandthenEventsinthelefthandmenu

ForaLocation,GrouporSystem,clicktheDETAILSlinkandthenEventsinthelefthandmenu

FromanEventClass,selectEventsinthelefthandmenu

PriortoV4,ZenosseventswereeitherOpenorClosed.OpeneventswerestoredintheMySQLeventsdatabaseinthestatustable.Whenaneventwasclosed,itwasmovedtothehistorytableoftheeventsdatabase.

WithZenoss4thereisasignificantchange.TheMySQLdatabaseforeventsiscalledzenoss_zepandithasfarmoretables,includingevent_summaryandevent_archive.Openeventswillbestoredintheevents_summarytable.Beawarethattheevents_summarytablewillalsoholdclosed,clearedandagedeventsthiscatchesoutmanypeoplemigratingfromolderversionsofZenosstoZenoss4.ChecktheStatusfilterintheEventConsoletoshowClosed,ClearedandAgedevents(theyallhavethesamestatusicon).Closed,ClearedandAgedeventsmaybeautomaticallymovedtotheevent_archivetablebasedonage(after3days,bydefault).

2.2 Event Manager settingsFromtheADVANCED>Settingsmenu,chooseEventsinthelefthandmenutosetupvariousparametersthatcontroltheeventssubsystem,includinghoweventsareagedandfinallypurged.

Figure3onpage11showslargelydefaultsettings.EventsofseverityWarningandbelowwillbeAgedafter240minutes(4hours).After4320minutes(3days)eventswithstatusofClosed,ClearedorAgedwillbeArchived(movedtotheevents_archivetable).After7daysArchivedeventswillbedeletedentirely(notethislastsettingis90daysbydefaultandcanresultinaverylargedatabase).

Seechapter7oftheZenossCore4AdministratorsGuideformoreinformation.


2.3 Event database tables 2.3.1 Zenoss 2.x and 3.xTheeventsarchitecturewasthesameforversions2and3andwasrelativelysimple.Eventsweregeneratedfromsomewhere.ThezenhubdaemonprocessedthemandusuallythensavedthemeitherinthestatustableoftheMySQLeventsdatabaseorcouldsendthemtothehistorytable.

ThedatabasefieldsofthestatusandhistorytablesmatchedthedetailsseeninanEventConsoleandifyouwroterulesandtransformstoprocessevents,theywerebasedonthesesamefieldnames.

TheeventsdatabaseiscreatedautomaticallywhenZenossisinstalledandcantypicallybeaccessedbythezenossuserwithapasswordofzenossseeFigure4.


Figure3:EventManagerparametersforageingandarchiving

TheformatofeachofthesetablesandthevalidfieldsforaZenosseventcanbeseenbyexaminingtheZenossdatabasesetupfilein$ZENHOME/Products/ZenEvents/db/zenevents.sql,where$ZENHOMEwillbe/opt/zenossforaCore4.2ZenossonRedHat/CentOS(theonlycurrentlysupportedplatform).


Figure4:ZenosseventsdatabasepriortoZenoss4

zenevents.sqlalsodefinesthehistorytableinasimilarfashion.

Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.ThedetailtablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthattheZenossadministratorrequiresforanevent.


Figure5:Definitionofstatuseventfieldsinzenevents.sqlpriortoZenoss4

IfyouareusingZenosspriortoversion4,gettheolderversionofthisZenossEventManagementpaperfromhttp://www.skills1st.co.uk/papers/jane/zenoss_event_management_paper.pdf.

2.3.2 Zenoss 4WithZenoss4eventsarestillheldinaMySQLdatabasewhichisnowcalledzenoss_zepanditiscreatedwhenZenossisinstalled.Aswithearlierversions,thezenossusercanaccessthisdatabasewithapasswordofzenoss.

NotethatwithZenoss4.2.3,ifinstalledwiththecoreautodeployscript,thenthepasswordfortheMySQLzenossuserischangedtoarobust,randompasswordthatisthensavedin$ZENHOME/etc/global.conf.Permissionsfor$ZENHOME/etcanditscontentsareallsettofullaccessforthezenossuserandnoaccessforanyoneelse.


Figure6:zenevents.sqlshowingheartbeat,alert_state,loganddetailtableszenoss2and3only

Inpassing,notethatinadditiontothezenoss_zepdatabase,theirisalsoazodbandazodb_sessiondatabase.TheZopedatabase(ZODB)thatstoresalltheobjects(devices,deviceclasses,processes,networks,etc)isnowinMySQL.

Examiningthetablesofthezenoss_zepdatabaseiswherethingsdivergesignificantlyfrompreviousversions.


Figure7:AccessingMySQLdatabaseswithZenoss4

Themaintablesarenowevent_summaryandevent_archivebutthestructureismorecomplicated.Someofthedataisheldinseparatetableswithpointerstothemfromthemaintables.Theseinclude:

agent event_class event_class_key event_group event_key monitor


Figure8:TablesintheZenoss4zenoss_zepdatabase

Thedetailsoftheevent_summarytableisshownbelow.Theeventarchivetableisverysimilarwithjustthetwofingerprint_hashfieldsomitted.


Figure9:Fieldsintheevent_summarytableinZenoss4

TheeagleeyedwillalsospotthatsomeofthefieldnameshavechangedfromthoseinFigure5.eventClassintheoldversionbecomesevent_classinV4;firstTimeinFigure5becomesfirst_seeninthelaterversionandthereareanumberofothersimilar,subtlechanges.

Asmentionedabove,someofthedataisheldinseparatetablessoagent_id,event_class_id,event_class_key_id,event_group_id,event_key_idandmonitor_keyarelinkstoseparatetableswiththecorrespondingdata.

Somedatahaschangedfairlysubtly:

Old New

evid uuid

eventState status_id

eventClassMapping event_class_mapping_uuid

severity severity_id

stateChange status_change

firstTime first_seen

lastTime last_seen

count event_count

facility syslog_facility

priority syslog_priority

ntevid nt_event_code

ownerid current_user_uuid/current_user_name

clearid clear_fingerprint_hash/cleared_by_event_uuid

Allreferencestothedevicehavechangedsignificantly.deviceisreplacedbythefourfields,element_uuid,element_type_id,elementidentifierandelement_titlewhilstthecomponentfieldisreplacedbyelement_sub_uuid,element_sub_type_id,element_sub_identifierandelement_sub_title.

dedupidhasbecomefingerprintandfingerprint_hash.

OtherfieldswithdevicecontextsuchasprodState,DeviceClass,Location,Systems,DeviceGroups,ipAddress,monitorandDevicePrioritywillnowbefoundfromthetags_jsonfield;theyarealsoavailableintheeventdetails.

PriortoZenoss4therewasaseparatelogtablewhoseroleisnowtakenbythenotes_jsonfieldoftheevent_summarytable.

Eventdetailsratherthanbeinginaseparatetable,arenowreachedfromdetails_json.

update_timehasbeenaddedthelasttimeaneventwasupdated.


suppid(whichwasneverused)hasdisappearedintheZenoss4schema.managerhasalsodisappearedfromZenoss4.

Thesetablesarecreatedbythefilesin$ZENHOME/share/zeneventserver/sql/mysql.

Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwasgenerated:

Syslogeventspopulatethefacilityandpriorityfields

Windowseventspopulatethentevidfield

SNMPTRAPspopulateatleastcommunityandoidfieldsintheeventdetail.TheyalsousetheeventdetailtoprovideanyvariablespassedbyanSNMPTRAP.

TheagentfielddenoteswhichZenossdaemongeneratedorprocessedtheincomingevent;forexample,zentrap,zeneventlog,zenping.


Figure10:Partofthe001.sqlfilethatdefinesMySQLtablesinthezenoss_zepdatabaseforZenoss4

FundamentallyZenossadministratorsshouldnotbeaccessingthezenoss_zepdatabasedirectly.Zenosshaveprovidedaninternaleventmappingsothat,largely,administratorscancontinuetousethesameeventattributenamesashavebeenusedpreviously.Thiseventproxymappingwillbediscussedinmoredetaillater.Ingeneral,thispaperwillusetheoldnamesunlessexplicitlystatedotherwise.

Ifyoudoneedtoaccesseventdatainthedatabasetables,perhapsforreportingonevents,itispossiblewiththeJSONAPI(alsomoreonthislater).

2.4 New event daemonsPriortoZenoss4mostoftheworkofprocessinganeventwasperformedbythezenhubdaemonwhichalsohaslotsofotherrolestofulfil.Eventprocessingcouldbecomeaseverebottleneck.Zenoss4hasintroducedseveralnewsubsystemsanddaemonstodramaticallyimprovethethroughputofeventprocessing.

2.4.1 RabbitMQAMessageQueueingarchitecturehasbeenimplementedtospeedupprocessingandtoofferanAPIsothatZenossandotherapplicationproviderscaninteractwithevents.ItisalsousedbythenewJobarchitecture.ItusestheAdvancedMessageQueueingProtocol(AMQP)standard,andtheopensourceRabbitMQimplementationinparticular,fortheeventpipeline.

WhenZenossisinstalledtheRabbitMQsubsystemisalsoinstalledandconfiguredwithavhostofzenoss,userzenoss,passwordzenoss.TherabbitmqctlutilitycanprovideinformationaboutthestateoftheMQenvironment;notethatrabbitmqctlcommandsmustberunbytherootuser.

Aneasywaytoseequeuesbuildingupistotemporarilystopzeneventdandtheraweventsqueuewillthenbuildrapidly.


Figure11:Usingtherabbitmqctlutilitytoshowqueuesforthe/zenossvhost

rabbitmqctlonitsownorwithinsufficientargumentsprovidestheusagehelp.rabbitmqctlreportgivesagoodoverallviewofthesubsystem.

IftheZenossserverisrenamedthenyoumustclearandrebuildqueuesbeforethezenhubandzenjobsdaemonswillrestart.Toresolvethis,issuethefollowingcommandsastherootuser(althoughanydataqueuedatrestarttimewillbelost):export VHOST="/zenoss"export USER="zenoss"export PASS="zenoss"rabbitmqctl stop_apprabbitmqctl resetrabbitmqctl start_apprabbitmqctl add_vhost "$VHOST"rabbitmqctl add_user "$USER" "$PASS"rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'

Seesection14.8oftheZenossCore4AdministratorsGuideforthisinformation.

NotethatwithZenossCore4.2.3installedusingtheautodeployscript,orifthesecure_zenoss.shscripthasbeenrunstandalone,thenthepasswordinthethirdlineabovewillhavebeenchanged.Examine$ZENHOME/etc/global.conffortheamqppasswordandsubstituethatvalue,ratherthanusingzenossasthepassword.

ProvidedtheRabbitMQsubsystemisrunning,anymissingqueuewillautomaticallyberecreatedwhenZenossisrestarted.

Tosimplyhavethequeuesrecreated,startasthezenossuser:zenossstopsu(tobecomerootuser)rabbitmqctldelete_vhost/zenossrabbitmqctladd_vhost/zenossrabbitmqctladd_userzenosszenoss#mightcreateanerrorzenossrabbitmqctlset_permissionsp/zenosszenoss'.*''.*''.*'rabbitmqctllist_vhosts (shouldhavezenossagain)rabbitmqctlp/zenosslist_queues(shouldbenone)exit (backtozenossuser)zenossstartsurabbitmqctlp/zenosslist_queues(shouldbeseveral)

Thereisafurtherscriptavailableatgist,writtenbycluther,toresetRabbitMQhttps://gist.github.com/4192854.

TwoutilitiesareavailableforthezenossusertogetRabbitMQinformation:zenqdump

dumpstheeventsinaqueue,convertingthebinaryblobs(whichishowtheeventsareactuallystored)intohumanreadabletext.

Notethatthezenqdumputilityhasparametersforuserandpasswordforauthentication,thatdefaulttozenoss/zenoss(youcanfindthiscodein$ZENHOME/lib/python/zenoss/protocols/amqpconfig.py).InZenoss4.2.3,passwordsarelikelytohavebeenimprovedoninstallationsothesimplecommandshownabove


willfail.Examine$ZENHOME/etc/global.conffortheparametersamqpuserandamqppasswordandsupplythosevalues.Forexample:

zenqdumpuzenosspuy+680bEubHgdPow8Tfhzenoss.queues.zep.rawevents

Thezenqutilityhasthreedifferentoptionstomanageaqueue:zenq count zenq purge zenq delete

Thecountparametergivesacontinualoutputoftimestampandqueuelength.

Thepurgeparameterpurgeseventsfromaqueue.ThiscommandissafewhenZenossisrunning.

ThedeleteparameterdeletesthequeueandshouldnotbeusedwhenZenossisrunning.

zenqdoesnothaveauthenticationparameters.

2.4.2 zeneventserverAnewJavadaemon,zeneventserver(alsoknownaszep),hasbeencreated.Itsroleistopresenteventstotheuserinterfaceandotherclients,andtomanagetheflowofdatabetweentheRabbitMQqueuesandtheMySQLdatabase.DataispresentedtoclientsviaJSONcalls.

2.4.3 zeneventdzeneventdisanewPythondaemonwhoseresponsibilityistotakedatafromtheincomingraweventqueue,classifyit(iftheeventdoesnotalreadyhaveaclass),adddevicecontextandeventcontext,andperformanytransforms.ItthenoutputstothezeneventsqueuesothatthezeneventserverdaemoncanmanageitsprogresstotheMySQLdatabase,totheuserinterfaceandforalertingaction.


2.4.4 zenactiond zenactiondhasbeencompletelyrewrittenforZenoss4.Itisresponsibleforexecutingactionsassociatedwithnotificationssuchaspaging,email,executingbackgroundcommandsandraisingnotificationTRAPs.zenactiondwillperiodicallyinspectthesignalqueueforsignalmessages,dumpthemintoitsshareofmemcachedandsubsequentlyactonthemessagesasinstructedintheassociatednotification.

2.4.5 memcachedPriortoZenoss4eachofthedaemonshaditsowncache.Thiscouldbeawastefulallocationofmemory.WithZenoss4,amemcachedsubsystemisintroducedwhichprovidessharedL2memorycacheforalldaemons,offeringmuchbetterperformance.

memcachedisconfiguredin/etc/sysconfig/memcached.Thedefaultistoconfigure64Mbformemcached(whichisnotpreallocated;itisonlyusedasnecessary).Thisshouldbeincreasedtoatleast1Gbonproductionsystemswithmorethan100devices(andrun/etc/init.d/memcachedrestart).Alsoensurethatmemcachedisenabledin$ZENHOME/etc/zope.conf.


Figure12:Zenoss4eventarchitecture

2.5 Other database-related changes in Zenoss 4Notdirectlyrelatedtotheeventssubsystem,buttheZopedatabase(ZODB)thatusedtobeheldin$ZENHOME/var/Data.fsandaccessedbythezeoctldaemon,isnowstoredinthesameMySQLinstanceaszenoss_zep(andZEOhasgone).

ThezodbdatabaseisthemainZopedatabaseandthereisalsoazodb_sessiondatabasewhichholdsuserpreferencesthinkofzodb_sessionasanexpandedsetofuser'scookies;ifnecessary,itcanbedeletedanditwillberecreatedautomatically.

ZODBiswherealltheobjectdataisstoredrelatingtodevices,components,processes,services,networks,MIBs,etc.Theeventprocessingdaemonsneedaccesstothezodbdatabasetoenricheventswithdeviceandcomponentinformation.

Zopeobjectsareknownaspickles,typicallyastringrepresentationofencodeddata(ablob)inotherwords,treattheZODBdatabaseasablackbox(justasData.fswas).AJSONinterfaceisprovidedtoaccessdataintheZODBandthezendmdtoolstillworksinexactlythesamewayasinpreviousversionsofZenoss,despitetheZODBnowbeinginMySQL.


ToprovideaccesstothethezodbMySQLdatabase,aRelStoragesubsystemisusedasahighperformancebackendtoZODB.RelStoragemayalsousememcachedtofurtherenhanceperformance.

TheolderversionsofZenossdidnotdomuchbywayofindexingtheeventsdatabase.WithZenoss4holdingZODBdataaswellaseventsdatainMySQL,aneffectiveindexingmechanismwasrequiredsotheLucenepackageisusedfromApache.Luceneisahighperformance,fullfeaturedtextsearchenginelibrarywrittenentirelyinJava.Itisusedtoholdindexesforbothzodbandzenoss_zep.

2.6 Event life cycleThelifecycleofaneventhaseightphases:

Eventgeneration

Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent

Eventclassmappingtodistinguishonetype(class)ofeventfromanother

Eventcontextadditionalinformationpertinenttoaclassofevent


Eventtransformmanipulationofeventfields

Databaseinsertionanddeduplication

Resolution

Ageingandarchiving

ProcessingofaneventdependsontheeventclassthataneventisassignedtothevalueofitseventClassfield.Adescriptionofeachofthesephaseswillbegivenhere:subsequentsectionsofthepaperprovidemoredetailsofsomeareas.

InFigure14,thefirstsixphasesoftheeventlifecycleareshown.Theblue,dashedpathshowstheprogressofaninternallygeneratedZenossevent,whichdoesnotpassthroughaneventmappingphase.AneventClassfieldisproducedbythedaemonthatgeneratedtheevent.Itsonlywaytoapplyatransformisasaclasstransform.

ThepurplepathshowstheprogressofaneventthatisgeneratedexternallytoZenoss.TheinitialparsingdaemonmustprovideaneventClassKeyfieldwhichisthenused,alongwithotherfields,inaneventclassmappingRuleand/orRegex,whichinturnprovidesaneventClassfield.Aftermapping,theeventmaypassthroughbothaneventclasstransformandaneventmappingtransform.


Figure14:Eventlifecycle,generationtodatabaseinsertion

AnareathathaschangedfairlysignificantlyinZenoss4isthemechanismforresolvingandageingevents.PriortoVersion4,aneventwasfundamentallyopen(whichalsoencompassedeventStateofAcknowledgedandSuppressedaswellasNew)andsuchaneventresidedinthestatustableoftheeventsdatabase;alternatively,aneventwasClosed,inwhichcaseitwasmovedtothehistorytableoftheeventsdatabase.

WithZenoss4,thepossiblevaluesofeventStatehavebeenexpandedtoinclude:

Name Number Description

New 0 Anewevent

Acknowledged 1 Acknowledgedbyuserortransform

Suppressed 2 Eventtypicallybeyondasinglepointoffailure

Closed 3 Eventresolvedbyauser

Cleared 4 Eventresolvedbyanautomaticrule

Dropped 5 WouldneverreachtheMySQLdatabase

Aged 6 Eventautomaticallyclosedaccordingtotheseverityandlastseentimeoftheevent.

Thesearewelldescribedinchapter7oftheZenossCore4AdministrationGuide.Thehugedifferencehereisthatthenewevent_summarytableintheMySQLdatabasewillprobablyhaveClosed/Cleared/Agedeventsinit.Theevent_archivetablehaseventsthathavebeenautomaticallyagedoutbasedontheirseverityandage.

2.6.1 Event generationFundamentally,eventswilleitherbegeneratedbyZenossitselfintheprocessofdiscovery,availabilityandperformancechecking,oreventswillbegeneratedoutsideZenossandcapturedbyspecialisedZenossdaemons.


Zenossdaemon Exampleofwheneventgenerated

zenping pingfailureoninterface

zendisc newdevicediscovered

zenstatus TCP/UDPserviceunavailable

zenprocess processunavailable

zenwin Windowsservicefailed

zenwinperf WMIperformancedatacollectionfailure/threshold

zencommand sshperformancedatacollectionfailure/threshold

zenperfsnmp SNMPperformancedatacollectionfailure/threshold

zenmodeler Configurationdatachangedonzenmodelerpoll

Table2.1.:EventsgeneratedbyZenossitself

Zenossdaemon Exampleofwheneventgenerated

zensyslog processessyslogeventsreceivedonUDP/514(default)

zeneventlog processesWindowseventsreceivedusingWMI

zentrap processesSNMPTRAPsreceivedonUDP/162

Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons

EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent.ThedaemonthatgeneratestheeventparsesthenativeinformationandassignsavaluetotheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary,messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenossdaemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwillpopulatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface).

EventsthatareinitiallygeneratedoutsideZenossarecapturedbyzensyslog,zeneventlogorzentrap.ThesedaemonseachhaveaparsingmechanismtointerpretthenativeeventintotheZenosseventformat.ThePythoncodeforthezensyslogandzentrapparsingisin$ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwillbe/opt/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.pydecodesSNMPTRAPs.

ThedaemonsforprocessingWindowsWMIdatausedtobeastandardpartoftheCorecodebutwithZenoss4thishasmovedtoaZenosssuppliedZenPackZenPacks.zenoss.WindowsMonitor.zenwin,zenwinperfandzeneventlogcanallbefoundunderthatZenPack'sbasedirectory.

Typically,theexternaleventparsingmechanismsdonotdeliveravalueforeventClass;rathertheydeliveravaluefortheeventClassKeyfield,alongwithvaluesforsome


otherfieldssuchascomponent,summary,messageandagent.Itisthenthejoboftheeventmappingphasetodistinguishtheeventclass.

2.6.2 Application of device contextEarlyintheeventprocessinglifecycle,thezeneventddaemonappliesdevicecontexttotheevent.ThismeansthatsevenfieldsoftheeventarepopulatedbydeterminingthedevicethatgeneratedtheeventandthenlookingupthefollowingvaluesforthedeviceintheZODBdatabase:

prodState DevicePriority Location DeviceClass DeviceGroups Systems ipAddress(mayhavealreadybeenassigned)

2.6.3 Event class mappingEventclassmappingtendsonlytobeapplicabletoeventsthatoriginateoutsidetheZenosssystem.ItistheprocessbywhichaneventisassignedavalueforitseventClassfieldand,potentially,otherfields.

Typically,theeventgenerationphasewilldeliveraneventwithafewfieldspopulated;generallythisdoesnotincludetheeventClassfieldbutdoesincludetheeventClassKeyfield.OftentheZenossparsingdaemon(suchaszensyslog),willusethesameeventClassKeyforseveraldifferentnativeevents.Forexample,aneventClassKeyofdropbearisusedforseveralloginsecurityevents.Thecomponent,summary,messageandagentfieldsmayalsobepopulated.

Theeventclassmappingphaseexaminestheevent(suchasitis,sofar)andthenusesanumberofteststodeterminetheeventClasstoassigntothisevent:

1. AneventClassKeyfieldmustexistformappingtobesuccessful.

2. APythonRulecanbewrittentotestanyavailablefieldoftheeventoranyavailableattributeofthedevicefromwhichtheeventcame.SuchrulescanbecomplexPythonexpressions,includinglogicalANDsandORs.Iftheruleissatisfied,theincomingevent'seventClassfieldwillbegiventheclassassociatedwiththatmapping.Iftheruleisnotsatisfied,thismappingisdiscarded,theclassisnotassociated,andthenextmappingwillbetestedforamatch.ARuledoesnothavetoexistinamappinginstance.

3. IftheRuleissatisfied(ordoesnotexist),themappingcanthenuseaRegexPythonregularexpressiontoparsetheevent'ssummaryfield,checkingforparticularstrings.TheRegexcanalsoassignpartsofthesummaryfieldtonew,


userdefineddetailfieldsoftheevent.IfaRuleexistsandissatisfied,theclassmappingwillapply,eveniftheRegexisnotsatisfied;anyuserdefinedfieldsintheRegexwillnotbecreatediftheRegexdoesnotmatch.IfaRuledoesnotexistthentheRegexmustbesatisfiedforthemapping(andanytransform)toapply.

4. TheGUIdialoguethatdefinesthemappingspecifiestheeventClassKey,theRule,theRegexandanyTransform.AsequencenumberisalsoavailablesothatifmultipleincomingeventshavethesameeventClassKeythenthesequencenumberdefinestheorderinwhichthevariousmappingswillbeapplied,lowestnumberfirst.ThefirstRule/Regexmappingcombinationthatmatcheswillbeapplied.

Eventclassmappingisexecutedbythezeneventddaemon.

2.6.4 Application of event contextEventcontextisdefinedbytheConfigurationProperties(zProperties)ofanevent.Eventcontextcanbedefinedattheeventclasslevel,foraneventsubclass,orattheeventmappinglevel.Aswithallobjectorientedattributes,thevaluesareinheritedbychildobjectssoapplyingeventcontexttoaclassautomaticallysetsitforanysubclassesandsubclassmappings.Thethreeeventcontextattributesare:

zEventAction status|history|dropdefaultisstatus

zEventClearClasses bydefaultthisisanemptyPythonlistofstrings

zEventSeverity Originalbydefault

Eventcontextisappliedintheeventlifecycle,afterRuleandRegexprocessingbutbeforeanyeventtransforms.Thus,thezEventActionzPropertycanspecifyhistorybutaneventtransformcouldoverridethatactionbysettingtheevt._actionvaluetostatus.

NotethatthestatusandhistoryvaluesreflecttheolddatabasetablespriortoZenoss4.statusnowmapstoaneventStateofNewandhistorymapstoaneventStateofClosed;bothwillbestoredintheevent_summarydatabasetable.

Eventcontextisappliedbythezeneventddaemon.

2.6.5 Event transformsEventtransformscanbespecifiedforaneventclassmappingorforaneventclass(orsubclass).AtransformiswritteninPythonandcanbeusedtomodifyanyavailablefieldsofeithertheeventorthedevicethatgeneratedtheevent.Itcanalsocreateuserdefinedfields.

FromZenoss2.4,cascadingeventtransformsmeanthatclasstransformsareappliedfromeverylevelintheappropriateclasshierarchy,followedbyanytransformforan


appliedeventmapping.PriortoZenoss2.4,eitheramappingtransformwasapplied,oraclasstransform,butnotboth.Classtransformswereonlyappliedtotheexactclass,notfromtheeventclasshierarchy.

AtransforminaneventmappingwillonlybeexecutedoncetheeventClassKeyhasbeenmatched,andtheRulehasbeensatisfied(ifitexists).IfaRuledoesnotexist,anyRegexhastobesatisfiedforthetransformtobeexecuted.

Eventtransformsareexecutedbythezeneventddaemon.

2.6.6 Database insertions and de-duplicationZenosseventsarenowstoredinaMySQLdatabasecalledzenoss_zep(usedtobeevents).Themaintablesfortheeventlifecyclearetheevent_summarytableforrecentevents,theevent_archivetableforoldevents.

Somefieldsoftheeventareonlyassignedatdatabaseinsertiontimetheyarenotavailableateventmappingoreventtransformtime.Theseinclude:

count eventState evid stateChange dedupid eventClassMapping firstTime lastTime

ItistheJavazeneventserverdaemonthatisresponsibleforgettingeventsintothedatabase.

Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateeventarrives,thentherepeatcountofanexistingeventwillbeincremented.duplicateisdefinedashavingthefollowingfieldsthesame:

device component eventClass eventKey severity

IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalsomatch.Thededupidfieldiscreatedbyconcatenatingtheabovefieldstogether,separatedbythepipe(verticalbar)symbol.Thusanexamplededupidmightbe:zenoss.skills-1st.co.uk|su|/Security/Su||5|FAILED SU (to root)jane on /dev/pts/1

wherethedeviceiszenoss.skills1st.co.uk,componentissu,eventClassis/Security/Su,theeventKeyisunset,severityis5(Critical),andthesummaryisFAILEDSU(toroot)janeon/dev/pts/1.

InZenoss4,thededupidfieldisalsoknownasthefingerprint.


Whenaneweventisreceivedbythesystem,thededupidisconstructedbythezeneventddaemon.Transformsmaymodifyeithercomponentfieldsofthefingerprintormaydirectlymodifythededupidfield.

Whenzeneventservercomestoinserttheeventinthedatabase,ifitmatchesthededupidforanyactiveevent,theexistingeventisupdatedwithpropertiesoftheneweventoccurrence,theevent'scountisincrementedbyone,andthelastTimefieldisupdatedtobethecreatedtimeoftheneweventoccurrence.

NotethatthisisasubtlebutsignificantchangefrompriorversionsofZenossastheexistingeventisupdatedwithpropertiesofthenewevent;olderversionsofZenosssimplyupdatedthecountandlastTimefields.Forexample,ifthefingerprintincludesaneventKeysodoesnotincludethesummary,theresultingeventwillnowshowthesummaryofthelatestreceivedduplicateevent.

Iftheincomingeventdoesnotmatchthededupidofanyactiveevents,thenitisinsertedintotheactiveeventtablewithacountof1,andthefirstTimeandlastTimefieldsaresettothecreatedtimeofthenewevent.

2.6.7 ResolutionResolutionofaproblemrepresentedbyaneventcanhappeninseveralways:

Auserclosestheevent(eventState=Closed)

TheeventcontextzEventActionzPropertyforaneventclassisdrop(theeventisdiscarded).Forexample,eventclass/Ignore.

TheeventcontextzEventActionzPropertyforaneventclassishistory(eventState=Closed).Forexample,eventclass/Archive.

Atransformsetsevt._actionto'drop'(theeventisdiscarded)

Atransformsetsevt._actionto'history'(eventState=Closed)

Anotherclearingeventarrivesthatclearstheinitialevent(eventState=Cleared)

TheEventManagersettingshaveseverityandlastSeenparametersthatdenotewhicheventswillbeautomaticallyaged(eventState=Aged)

Alltheaboveeventswillstillbeintheevent_summarytableoftheMySQLdatabase.TheEventManagerparameterforEventArchiveThresholdistheonlyautomaticactionthatmoveseventsfromevent_summarytoevent_archiveanditwillmovealleventswitheventStateofClosed,ClearedandAged.

Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;therearetwodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews.

ThefirstclearingmechanismisthatanyeventwithaseverityofClearwillsearchtheevent_summarytableforsimilaractiveeventsandsettheireventStatetoCleared(notClosed).

TheZenossCore4AdministratorsGuidedefinesthisautoclearfingerprintas:


IfcomponentUUIDexists:

componentUUID

eventClass

eventKey(canbeblank)

IfcomponentUUIDdoesnotexist:

device

component(canbeblank)

eventClass

eventKey(canbeblank)

Thiscanbealittleconfusing.TheEventConsoleshowsacomponentfield.ItdoesnotshowacomponentUUIDfield.StrictlythecomponentfieldintheEventConsoleshowstheelement_sub_identifierfieldfromtheMySQLdatabasetablethenameofthecomponent.SomeeventsgenerateacomponentUUID(UniversallyUniqueIdentifier)andsomedonot.InspectingtheeventinthedatabaseorusingtheJSONinterfaceistheonlywaytodeterminewhetherthisuniquecomponentidfieldexistsornot.Ifitdoesexistthenitshouldalso,byimplication,denotethedevicethatthecomponentbelongsto,hencethedevicefieldisunnecessary.(VersionsofZenosspriorto4didnothaveacomponentUUID;similarwasdefinedashavingthesameeventClass,deviceandcomponentfields.)

EitherwayinCore4,theeventClassandtheeventKeyfieldsaresignificant.IfthecomponentUUIDdoesnotexistthenitistheelement_sub_identifier(componentname)thatmustmatch,alongwiththedevicename(element_identifierintheMySQLtable).

ThesecondautomaticclearingmechanismextendstheautoclearfingerprintdefinitionofeventClass.TheeventcontextofaneventclassincludeszEventClearClasseswhichisalistofothereventclassesthatthisgoodnewseventwillclear,inadditiontoitsownclass.Theotherconditionsoftheautoclearfingerprintremainthesame.

Notethatthesameeffectcanbeachievedinatransformbyassigningalistofclassnamestoevt._clearClasses.

Alleventswiththesameautoclearfingerprintarecleared,notjustthemostrecent.

TheclearingeventwillautomaticallyhaveitseventStatesettoClosed,provideditmatchesoneormorebadnewsevents.Ifitdoesnotmatchanyeventsthentheclearingeventisdroppedandwillnotbepersistedtothezenoss_zepdatabase.Thisistoavoidfillingupthedatabasewithredundantgoodnewsevents.

Whencorrelationtakesplacesomeoftheexistingbadnewseventfieldsareupdated;stateChangebecomesthetimewhentheeventwasresolved;clearidispopulatedwiththeevidfieldoftheclearing,goodnewsevent.

Thisautomaticresolutionofeventsisperformedbythezeneventserverdaemon.


2.6.8 Ageing and archivingMaintenanceisrequiredonthetablesofthezenoss_zepdatabaseorthediskwillsimplyfillupeventually.ThreemechanismsareprovidedbytheEventManager:

Bydefault,eventswithseveritylessthanErrorwillbeAgedafteranEventAgeingThresholdof4hours;thatis,theeventStatewillbesettoAged(strictlythevalue6).

Bydefault,theEventArchiveThresholdis4320minutes(3days).ThismeansanyeventwitheventStateofClosed,ClearedorAgedwillbemovedfromtheevent_summarytabletotheevent_archivetableofthezenoss_zepdatabase.

TheDeleteArchivedEventsOlderThan(days)parameteris90bydefault.Thisistheonlyparameterthatautomaticallydeletesdata.Itisnotpossibletofinetunethistodelete,say,lowerseverityeventsafterdifferentintervals.

Zenosspriortoversion4providedautility,$ZENHOME/Products/ZenUtils/ZenDeleteHistory.pywhichcoulddeleteeventsselectivelybasedonageandseverity.ThisutilityisnotshippedwithZenoss4andcurrentlyhasnoequivalentfunction.

DeletingdatafromtheoldhistorytableinZenoss3usedtobeveryslow.InZenoss4,theevent_archivetableispartitioned,byday,ratherthanbeingonehugefile.Thismeansthatdeletingdataissimplyamatterofdroppingpartitionfiles.Thiscanbeseenfromthemysqlinterfacewith:

showcreatetableevent_archive;

3 Events generated by ZenossInthecourseofdiscovery,availabilitymonitoringandperformancemonitoring,Zenossmaygenerateeventstorepresentachangeinthecurrentstatus.AlthoughmanyeventsarebadnewsitshouldberecognisedthateventscanalsobegoodnewsInterfaceUp,Thresholdnolongerbreached,etc.

EventsgeneratedbyZenossaredependentonthevariouspollingintervalsconfigured.Toexaminethedefaultparameters,usetheADVANCED>Collectorsmenu.Clickonlocalhost(thecollectorontheZenosssystem).NotethatearlyversionsofZenossusedthetermandmenuoptionMonitorsratherthanCollectors.


Parameterstonoteparticularlyare:

SNMPPerformanceCycleInterval 300secs(5mins)

ProcessCycleInterval 180secs(3mins)

StatusCycleInterval 60secs(1min)

WindowsServiceCycleInterval 60secs(1min)

PingCycleTime 60secs(1min)

ModelerCycleInterval 420mins(12hours)

3.1 zenpingThemostbasiclevelofavailabilitycheckingistopingpoll.Thezenpingdaemonwill,bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgeneratedwhenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhenasimilarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldoftheeventisincreased.

Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,forexampleifasinglepointoffailurerouterisdown.WithZenoss4thisisachievedusingnmap;withearlierversions,ZenossbuiltaninternaltopologybasedonqueryingroutingtableswithSNMP.

Ifaneventisreceivedforanisolatedelement,aneventisgeneratedwithaneventStatefieldofSuppressedandthesummaryfieldreportsnotonlytheinterfaceforwhichthepingfailed,butalsothecausaldevice;forexample:

ip10.191.101.1isdown,failedatbino.skills1st.co.uk


Figure15:DefaultparametersforlocalhostCollector

Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghasfailed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbesuspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoringeventswillnotincreaseuntilpingaccessisresumed.

Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.Ifadevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,thenensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPandsshavailabilitymonitoringandperformancedatacollection.

Thelogfileforzenpingiszenping.login$ZENHOME/log.

3.2 zenstatusThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDPportsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute.Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexaminedfromtheINFRASTRUCTURE>IpServicesmenu.Bydefault,theonlyservicemonitorsthatareactiveareforsmtpandhttp;therestaresetwithmonitoringdisabled.

Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsasimilarbadnewseventandthecountfieldoftheeventincreaseswhilsttheserviceremainsdown.

Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.

3.3 zenprocesszenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.InaUnixcontext,thiswouldbewhethertheprocessappearsinapseflisting;inaWindowscontext,theprocessmustappearintheWindowsTaskManager(andnotethatthischeckiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault.

ConfigurationofprocessmonitoringforadeviceissimilarasforservicestheINFRASTRUCTURE>Processesmenuprovidesawaytoconfigureprocessestobemonitored.Zenoss4comeswithdefinitionspreconfiguredforalltheZenossprocesses.

ProcessmonitoringisactuallyachievedusingtheHostResourcesManagementInformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.ThismeansthatifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation.

Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsandthecountfieldincreasesonsubsequentfailedpolls.

Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.


3.4 zenwinThezenwindaemonshipswiththeZenPacks.zenoss.WindowsMonitorZenPackwithZenoss4(itwasastandardpartoftheCorecodeinearlierversions).ItmonitorsWindowsservices(notTCP/UDPservices).ThesecanbeexaminedfromtheINFRASTRUCTURE>WindowsServices.Bydefault,noneofthesemonitorsareactive.

zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccessservicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(ordeviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepollingcanbesuccessful.

Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomaticallyclearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailedpolls.

Thelogfileforzenwiniszenwin.login$ZENHOME/log.

3.5 zenwinperfzenwinperfisanewdaemonforZenoss4whichisalsopartoftheZenPacks.zenoss.WindowsMonitorZenPack.WithearlierversionsofZenoss,manyusersdeployedtheexcellentcommunityWMIDataSourceandWMIWindowsPerformanceZenPackstoachievesomethingverysimilartothisnewdaemon.

zenwinperfprovidesperformancemonitoringofinterfaces,filesystems,memory,CPUandpagingusingtheWMIprotocol.Defaultthresholdsareconfiguredforsomemetricswhichthengenerateeventswhenexceeded.ItcanbeextendedbytheusertomonitorotherperfmonmetricsusingtheWMIprotocol.

Dataisgatheredevery5minutes.

Thelogfileforzenwinperfiszenwinperf.login$ZENHOME/log.

3.6 zenperfsnmpzenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMPperformanceinformationandstatusinformationforprocesses.EvenifSNMPperformancemonitoringisnotconfigured,zenperfsnmpchecksthattheSNMPagentisavailable.

Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbegenerated.Withinafurther3minutesthereshouldbeanUnabletoreadprocessesondevice..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldforindividualmissingprocesseventsshouldstopincreasing.WhileSNMPaccesstothedeviceremainsbroken,thecountfieldfortheUnabletoreadprocessesondevice..eventwillincreaseevery3minutes.


Thelogfileforzenperfsnmpiszenperfsnmp.login$ZENHOME/log.

3.7 zencommandThezencommanddaemonperformsmonitoringbasedonrunningcommands,typicallyoveransshconnection.Likezenperfsnmpandzenwinperfitusesperformancetemplatestomonitormetricsandcangenerateaneventifathresholdisbreached.

Thelogfileforzencommandiszencommand.login$ZENHOME/log.

4 Syslog eventsTheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linuxalthoughslightlydifferentversionsandformatsexist.TherearealsoopensourceimplementationsofsyslogforWindowssystemsandmanynetworkingdevicesalsosupportthesyslogconcept.

Typicallysystemmessagesareoutputtooneormorelogfilessuchas/var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslogmessagestoacentralsyslogratherthanholdingfilesoneachsystem.ThewellknowndefaultportforforwardingsyslogmessagesisUDP/514.

Astandardsyslogsystemisconfiguredbythesyslog.conffile,typicallyin/etc.Anewerversionofsyslogisimplementedonsomesystems,syslogng,whichhasgreaterfilteringcapabilities.Thesyslogngconfigurationfileistypically/etc/syslogng/syslogng.conf.

AnothervariationisrsyslogdwhichistypicallyshippedwithnewerRedHat/CentOSSuSEsystems,configuredthrough/etc/rsyslog.conf.

Asyslogmessageincludesapriorityandafacility.Theprioritiesare:

0 emerg1 alert2 crit3 err4 warning5 notice6 info7 debugFacilitiesinclude:

auth (4) authpriv(10)

cron (9) daemon(3)

ftp(11) kern(0)

lpr(6) mail(2)


news (7) syslog(5)

user (1) uucp(8)

Thesedefinitionscanbefoundinsyslog.h(typicallyin/usr/include/sys).Bothpriorityandfacilityareencodedinasingle32bitintegerwherethebottom3bitsrepresentpriorityandtheremaining28bitsareusedtorepresentfacilities.

Forexample,ifthefacility/prioritytagis,thiswouldbe00010110inbinary,wherethebottom110representsapriorityof6(info)andthetop00010representsafacilityof2=mail.

4.1 Configuring syslog.conf AnydevicethatisgoingtoreportsyslogeventstoZenossmusthaveitssyslog.conffileconfiguredwiththedestinationaddressoftheZenosssystem.Theoriginalsyslog.confpermitsfilteringbasedonpriorityandfacilityso,acatchallstatementtosendalleventstotheZenosssystem,wouldbe: *.debug @

Thisalsoworksforrsyslogd.SeeFigure16foranrsyslog/syslogexamplethatforwardstozen42.class.example.orgallfacilitieswithpriorityofnoticeandabovebutallcronmessagesarefilteredout;authprivmessageswillbeforwardedwithseverityinfoandabove.


Figure16ConfigurationfileforrsyslogsendingselectedeventstoZenossserver

syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogngofferssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmayalsobepresent.

4.2 Zenoss processing of syslog messagesTocollectsyslogmessageswithZenoss,thezensyslogprocessautomaticallystartsonportUDP/514andcollectsanysyslogmessagesdirectedfromothersystems.zensyslogthenparsesthesemessagesintoZenossevents.Youmustensurethatthesyslog.conffileontheZenosssystemdoesnotenablecollectingremotesyslogsorthesyslogdandzensyslogprocesseswillclashoverwhogetsUDP/514(itispossibletoreconfigureeitherdaemon,ifrequired).


Figure17:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)

Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,thelevelofzensyslogloggingcanbeincreased.

1. UsetheINFRASTRUCTURE>Settings>Daemonsmenu.

2. Clicktheeditconfiglinkforthezensyslogdaemon.

3. ChangethefollowingparametersandclickSave:

logorig selectthis

logseverity Debug

4. Inspecttheunderlyingconfigurationfilein$ZENHOME/etc/zensyslog.conf.

5. Thelogoriglinesaystologtheoriginalincomingsyslogmessage;itwillbein$ZENHOME/log/origsyslog.log.Notethatthisparameterisuniquetozensyslogandisusefulfordebugging.

6. ThelogseveritylineisagenericZenossdaemonparameter;avalueof10isthemaximumDebuglevel.

7. Don'tforgettoSavethischange

8. UsetheRestartlinktorecyclezensyslog.Alternatively,asthezenossuser,issuethecommand:

zensyslog restart9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log

10.Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.host=zen241.class.example.org, ip=172.16.222.241

11.Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority.

12.LinesstartingwithtagshowthezensyslogparsingprocessasitteststheincominglineagainstvariousPythonregularexpressions,hopefullyendingwithatagmatchline.

13.Ifamatchissuccessful,aneventClassKeymaybedetermined

14.ThelastlineforaparsedeventshouldbeaQueueingevent.


Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablyamismatchofseverities.Thefollowingtabledemonstratesthis.

Zenoss syslogpriority Windows

Critical(red)(5) emerg(0) Error(1)

Error(orange)(4) alert(1) Warning(2)

Warning(yellow)(3) crit(2) Informational(3)

Info(blue)(2) err(3) Securityauditsuccess(4)

Debug(grey)(1) warning(4) Securityauditfailure(5)

Clear(green)(0) notice(5)

info(6)

debug(7)

Table4.1.:EventseveritiesforZenoss,syslogandWindows

NotethatthenumericvalueofZenosseventseveritydecreasesaseventsgetlesscriticalbutthatthepriorityofsyslogeventsincreasesaseventsgetlesscritical.

DefaultmappingfromsyslogprioritytoZenosseventseverity,isperformedby$ZENHOME/Products/ZenEvents/SyslogProcessing.pysearchfordefaultSeverityMaparoundline187inCore4.2.Theresultisthat:

syslogpriority

Outofthebox,allsyslogeventsmaptotheZenosseventclassof/Unknown.

SyslogProcessing.pyisthecodethatparsesanyincomingsyslogmessageandgeneratesaZenossevent.

ThefirstsectionhasaseriesofPythonregularexpressionstomatchagainsttheincomingsyslogline.Eachexpressionischeckedinturnuntilamatchisfound.Ifnomatchisfoundthenanentrygoesto$ZENHOME/log/zensyslog.logwithparseTagfailed.

ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincomingeventtoZenosseventclassfields,asfollows:


Figure19:SyslogProcessing.pyregularexpressionstomatchsyslogtags

def process(self, msg, ipaddr, host, rtime): evt = dict(device=host, ipAddress=ipaddr, firstTime=rtime, lastTime=rtime, eventGroup='syslog')

Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsarebothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfieldishardcodedatthisstagetosyslog.

parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility.

ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosettheseverityfieldoftheZenossevent.


Figure20:SyslogProcessing.pyprocessmainroutine

Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefromtheincomingevent.ThedeviceandipAddressfieldsoftheZenosseventaresetattheendofthisfunction.


Figure21:SyslogProcessing.pyparsingofpriority,facilityandseverity

TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressionsatthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged.TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenosseventsummaryfield.


Figure22:SyslogProcessing.pyprocessingtheheaderinformation

ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdonewiththebuildEventClassKeyfunction.


Figure23:SyslogProcessing.pyparsingthesyslogtag

NotethatiftheeventhasthecomponentfieldpopulatedthenthatisusedastheeventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.

5 Zenoss processing of Windows event logs5.1 Management using the WMI protocol

Zenosspriortoversion4shippedWindowsmonitoringaspartoftheCorecode.Zenoss4shipsWindowssupportwiththeZenPacks.zenoss.WindowsMonitorZenPackwhichhasaprerequisiteofZenPacks.zenoss.PySamba.TheseareZenossprovidedCoreZenPacks.

IfaWindowsdevicesupportsSNMPthenitisperfectlypossibletousethatprotocol,especiallyasmostWindowsSNMPagentsalsosupporttheHostResourcesMIBsosomesysteminformationisavailableinadditiontothestandardMIB2networktypeinformation.

TheZenossWindowsZenPacksintroducethe/Server/Windows/WMIdeviceclasswhichhasbothWMImodelerpluginsandWMIperformancetemplatesassociatedwithit.Targetdevicesshouldbeaddedtothisclassorsubclassesthereof.ThisallowsmonitoringusingtheWindowsManagementInstrumentation(WMI)protocol.AuseridandpasswordneedtobeconfiguredontargethoststopermitWMIaccessfromthe


Figure24:SyslogProcessing.pydeterminingtheEventClassKey

Zenossserver;italsomeansthatfirewallsbothontheWindowsdevicesandanyinterveningnetworkfirewalls,mustbeconfiguredtopermitWMIaccess.TheZenossServermustthenbeconfiguredwithmatchingWindowszProperties(zWinUserandzWinPassword)forthetargetdevices/deviceclasses.ThereareafewotherWindowsspecificConfigurationPropertiesseeFigure25.ThesezPropertiescanbechangedforadeviceclassorforaspecificdevice.

ZenPacks.zenoss.WindowsMonitorprovidesthreenewdaemons: zenwin monitorswindowsservicesusingWMI zenwinperf collectsperformancedatausingtheWMIprotocol zeneventlog retrievesWindowseventloginformationusingWMI

ThethreezWinPerf...zPropertiesfinetunetheconfigurationofthezenwinperfdaemon;thezWinEventlogparametermustbeTruetocollectWindowseventsfromatargetdevice.

ThezWinEventlogMinSeveritypropertydefinestheleastseriousseverityeventsthatwillbeforwardedfromWindowstoZenoss.Notethatthenumericdenotationofwindowseventseveritiesandtheirnamesandsupportcurrency,havechangedoverthelifeofZenoss.SeeTable4.1onpage42forcurrentvalidseverities.AlsonotethatifyouchangethisparameteryouarepresentedwithalistofZenossseverities,notWindowsstyleseverities;againrefertotheearliertableforatranslation.IfyouwanttoincludeallWindowsseverities,includingsecurityauditfailure(5),youneedtoselecttheClearseverityinthedropdownmenuwhenchangingzWinEventlogMinSeverity.

ThezWinEventlogClausewasintroducedduringthelifetimeofZenoss3tohelpfiltereventsfromWindowsdevices.ConsulttheZenossCore4AdministratorsGuide,chapter


Figure25zPropertiesforWindowstargets

6.6.6fordocumentationandexamples.Thisparameterisratherobtuse.FundamentallyaWindowsQueryLanguage(WQL)queryisconstructedtoberunbyzeneventlog:

SELECT*FROM__InstanceCreationEventWHERETargetInstanceISA'Win32_NTLogEvent'ANDTargetInstance.EventType

ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbuttheymayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActioneventzPropertysettohistorysothattheydonotappearinthestatustableoftheeventsdatabase.

5.2 Management of Windows systems using syslogThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsultingathttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailableundertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableaschargeableproducts.ThismeansthatWindowseventlogscanalsobecollectedwiththezensyslogdaemon.

NotethattheSyslogagentiscapableofbeingconfiguredtomonitorWindowsapplicationlogfiles,inadditiontothestandardWindowseventlogs.Whenmonitoringthestandardeventlogs,therearebetterfilteringcapabilitieswithSyslogthenwithzeneventlog.

6 Event Mapping

ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhicharedefinedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.TheprocessofEventClassMappingisaboutassociatinganincomingeventwithaparticularZenossEventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthateventbyusinganeventtransform.

Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclassmapping.Theclasshierarchycanbeusefulinthateventcontext,asimplementedbyeventzProperties(zEventSeverity,zEventAction,zEventClearClasses),followsthenormalrulesforobjectinheritanceifzEventActionissettodropontheeventclass/Ignore,thenanysubclassesof/Ignorewillalsoinheritthatproperty.

NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdropincomingeventstotally;/ArchiveclassesandsubclassesautomaticallysettheeventStatefieldtoClosed.

Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknownasinstances.Notethataneventdoesnothavetohaveanymappingsassociated,inwhichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemonthatgeneratestheevent,assignstheeventclassatthattime(/Perfeventsmaywellcomeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin$ZENHOME/Products/ZenModel/data/events.xml.TheycanbeinspectedfromtheZenossGUIbyselectingtheEVENTS>EventClassesmenu.


MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfieldwhichispopulatedbythenativeeventparsingmechanism(suchaszensyslog,zeneventlog,zentrap).ThesemechanismsmaygenerateseveraldifferenteventswiththesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetweensucheventsandpotentiallytoseparatethemintodifferenteventclasses.

Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretestedagainsttheincomingeventlowestnumbersaretestedfirst.Dependingonwhichmappingactuallymatches(ifany)willdeterminetheresultingeventClassoftheevent.

6.1 Working with event classes and event mappingsEventsareorganisedinanobjectorientedhierarchy;thusattributesassignedtoaparenteventclassareinheritedbyachildeventsubclass.

NeweventclassescanbedefinedbynavigatingtoaneventclassandusingthedropdownmenualongsideSubClassestoAddNewOrganizer.Thenamesuppliedisthenameoftheneweventclass.Forexample,drilldowntothe/SecurityeventclassandcreateanewsubclasscalledSu.

Anyeventwhichdoesnotmaptoaneventclassisthegiventheclassof/Unknown.ThesimplestwaytomapsuchaneventistostartfromanexistingeventintheEventConsole.Thefollowingscenarioexplainsthis,creatinganeweventclassmappingcalledsuwhichmapsanincomingeventtotheeventclass/Security/Su.

1. GenerateasyslogauthenticationfailureeventattheZenosssystem.

2. OpenanEventConsolethatshowstheeventandinspectitsdetails.

3. SelecttheeventandusetheReclassifyEventiconatthetopoftheconsole.Selectyournew/Security/Suclassfromthedropdownlist.Youshouldbeshowntheeventclassmappingpanel.ClickthelefthandEditmenu.

4. YoushouldfindthatthenameoftheneweventclassmappingissettosuandtheEventClassKeyissettosu(notelowercasesinbothcases).TheeventClassKeyfieldisactuallyderivedfromthecomponentfieldoftheincomingeventinSyslogProcessing.py(aroundline289).ThesummaryfieldoftheeventshouldhavebeencopiedintothemappingExamplebox.

5. AddatextstringtotheExplanationboxsuchasAutoaddedbyeventmapping.

6. AddatextstringtotheResolutionboxsuchasThisisadummyresolution.

7. OpenaZenossGUIwindowthatshowsallSuevents(youmayfinditusefultohaveseveralbrowsertabsopentofocusondifferentaspectsoftheZenossGUI).SelectalltheSueventsandClosethem.

8. GenerateanewSuevent.

9. CheckthedetailsoftheneweventintheEventConsole.TheeventshouldhavemappedtoeventClass/Security/Su.TheseverityshouldbeInfo(blue).The


detailsoftheeventshouldshowtheeventClassMappingfieldsetto/Security/Su/su.

Anyexistingeventmappingcanbemodifiedinasimilarfashion.

Wheneveryouchangeaneventmapping,itisadvisabletoclearanyexistingeventsofthatcategorybeforetestingthenewconfiguration.

Whenyouareworkingwitheventmappings,don'tforgettheEventmenuwhichfiltersanEventConsolebyEventClass.

Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofapage,suchas/Events/Security/Su.


Figure27:Editdialogueforeventclassmapping

6.1.1 Generating test eventsTesteventscanbecreatedfromtheEventConsoleusingthe+icon.

Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouarethezenossuser).Thistakesparameters:

d device p component k eventClassKey s severity c eventClass y eventKey i IPaddress h help o =(foranyotherattribute;canhavemultipleo) monitor collectorthiseventcamefrom port=PORT defaultis8081 server=SERVER defaultislocalhost auth=AUTH defaultisadmin:zenoss Theremainderofthelineaftertheseoptionsisusedforthesummaryfield

(strictlytheMessagefieldintheGUIdialoguepopulatestheeventsummaryfield)

ThecoreautodeployscriptdeliveredwithZenoss4.2.3hasnewfunctionalitytoincreasesecurityonaZenossinstallation.FormanyyearstheZenossuserofadminwithapasswordofzenosshasbeenconfiguredasstandard.Thenewinstallationscriptchangesthis,generatingarobustpasswordwhichisstoredinseveralconfigurationfilesin$ZENHOME/etc,includingglobal.confandhubpasswd.


Figure28:Dialoguetocreateatestevent

zensendeventisastandalonePythonutilityin$ZENHOME/binthatcommunicateswiththezenhubdaemon.Noteintheusagedescriptionabove,thatthedefaultauthparametervalueisadmin:zenoss;typicallythismeansthatzensendeventcommandswillfailwithanUnauthorizedmessageunlesstheauthparameterisaddedwiththecorrectuserandpassword,foundin$ZENHOME/etc/hubpasswd.

Adiscussiononmodifyingzensendeventtoautomaticallylookupthecorrectauthenticationparameters,canbefoundontheZenosswikiathttp://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3ThecodeissuppliedinAppendixA.

6.2 Regex in event mappingsTheRegexelementofaneventclassmappingcanbeusedtoparsethesummaryfieldoftheincomingevent,whichispresentedbytheparsingdaemon(zensyslog,zeneventlog,zentrap).TheRegexelementusesthePythonformatforregularexpressionsandcanusethePythonnamedgroupsyntaxtonotonlycheckforliteralstringsbutalsotodefineregularexpressionsforvariablepartsofastring,andassociatethatvariablepartwithaname.VariablepartsofthestringarecapturedintoPythonnamedgroupsthismeansthat:

Youcanhaveoneexpressionmatchlotsofsimilarbutdifferentincomingevents

Thevariablepart(typicallybetweenthe(?Pand\S+))canbepassedtotherestoftheeventprocessingmechanismasanamedfieldoftheevent.

Thus,intheproductshippeddropbeareventmappingfor/Security/Login/Fail,theRegexisasfollows:

exitbeforeauth$user'(?P\S+)',(?P\S+)fails$:Maxauthtriesreached

(?P\S+)willparsethecharactersafteruser'uptothenextsinglequoteandplacethatstringintotheeventKeyfieldoftheevent.Similarly(?P\S+)willparsethestringthatfollowsacommaandspaceandisendedbyspaceandfails,intoaneweventattributecalledfailures.

Matchingtheliteralstringrepresentingabracketrequiresthebackslashescapeorthebracketwillbeinterpretedasametacharacter.

TherestoftheeventsummarymustmatchtheliteraltextintheRegex;however,othertextcanappearbeyondtheendaftertriesreached.

TheExampleboxshouldshowsasampleeventsummarythatismatchedbytheregularexpressionintheRegexbox.IfyouattempttoSavearegexthatdoesnotmatchtheexample,theregexfieldwillbeshowninred.

FormoreinformationonPythonregularexpressions,seehttp://docs.python.org/2/library/re.html.


SeeFigure29foranexampleofamorespecificmapping,su_root,fortheeventclass/Security/Su.Theregexisusedtoensurethatthesummaryhasthestringpam_unix(su:auth):authenticationfailure;followedbysomefixedandsomevariableelements.

pam_unix$su:auth$:authenticationfailure;logname=(?P\S+)uid=(?P\d+)euid=(?P\d+)tty=(?P\S+)ruser=(?P\S+)rhost=\s+user=(?P\S+)

Theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfortheeventwhichwillbeshowninthedetailsoftheeventandcanbeusedinanysubsequenteventtransforms.

Additionally,theConfigurationPropertyofzEventSeverityhasbeensettoWarningforthismapping.


Figure29:EventmappingdialoguewithRegexforauthenticationfailure

Figure30Eventdetailsforauthenticationfailureeventshowingneweventfieldscreatedbytheregex

TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)aresatisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,userdefinedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,theuserdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclasswillfail.Noeventtransformswilltakeplace.IfaRuledoesexistandissatisfiedbuttheRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclassmappingwillbesuccessfulandanymappingtransformwilltakeplace.

6.3 Rules in event mappingsTheRuleelementofaneventclassmappingusesPythonexpressionstotestanyinstantiatedfieldoftheincomingeventagainstavalue.ExpressionscanbecomplexincludingPythonmethodcallsandlogicalANDsandORs.Thedefaulteventfieldsthataredefined,aregiveninAppendixD3oftheZenossCore4AdministrationGuide.Notethatsomeofthesefieldsarenotactuallyavailableateventmappingtimenotablyevid,stateChange,count,dedupid,firstTime,lastTimeandeventClassMapping.

TheRuleelementcanalsousePythonexpressionstotestforvaluesofattributesofthedevicethatgeneratedtheevent.SomeofthemethodsandattributesthatareavailablefordevicesaredocumentedinAppendixD2oftheZenossCore4AdministrationGuide,underthesectiononTALESexpressions(TemplateAttribute


Figure31:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes

LanguageExpressionSyntaxispartofZope.ZopeistheapplicationserverthatZenossisbuilton).

TheRuleelementwillonlybeusediftheeventClassKeyfieldinthemappinghasachievedamatchwiththeincomingevent.Afterthat,ifaRuleexists,itmustbesatisfiedbeforethismapping(andhenceclass)isapplied.

6.4 Other elements of event mappings TheExampleelementofaneventclassmappingisasamplestringthatisusefulwhenconstructingaRegex.TheRegexwillturnrediftheRegexdoesnotmatchtheExamplestringwhentheSavebuttonisused.

TheExplanationandResolutionelementsofaneventclassmappingarestringsthatcanbeconfiguredtoprovidefurtherinformationtoZenossusers.Theyappearintheeventdetail.Notethattheseelementscanonlybeliteralstrings;theycannotuseeitherstandardoruserdefinedfieldsfromtheevent.

ThecombinationofeventClassKey,RuleandRegexdeterminetheeventclassthatwillbeassociatedwiththeincomingeventandwhattransforms(ifany)willtakeplace.Theremaystillbemultiplecombinationsofthesethatsatisfyanygivenincomingevent.Ifso,theSequencemenuisusedtodecidetheprecedenceofevaluationofmatchingeventmappings.Themappingswillbetestedfromthelowesttothehighestsequencenumber.Onceamatchisfound,anysubsequentmappings(withhighersequencenumbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteriawillhavealowersequencenumber.

Intheexamplesaboveforthe/Security/Suclass,thegenericsumappinghassequencenumber1andthemorespecificsu_rootmappinghassequence0.

Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclassmappingcalleddefaultmappingwhichmusthaveaneventClassKeyofdefaultmapping.Thereareatleast6mappings,allcalleddefaultmapping,outofthebox.Eachmapstoadifferentclass.AdefaultmappingisaspecialcasethatisusedbytheeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(notethatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).InthecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluateslookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibletocreatenewmappings,eitherwiththenameofdefaultmappingor,indeed,withadifferentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbersofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.

7 Event transformsTransformscanbeusedtomodifyfieldsofanevent,createnew,userdefinedfieldsorfieldscanberetrievedfromeventsalreadyintheMySQLdatabase.


7.1 Different ways to apply transformsYoucanhavesimpleassignmentsoffieldvaluesorsetthembasedoncomplexPythonprograms.Thetransformmechanismcanbeappliedintwoways:

eventclasstransforms

eventclassmappingtransforms

PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectlytothatexacteventclassbytheparsingmechanism(zenping,zenperfsnmp,zencommand,AddEventwithEventClassspecified,etc).Ifatransformexistedinaneventclassmappingthatwasused,theeventclasstransformwasnotused.

Zenoss2.4introducedcascadingeventtransforms.Thischangedthingsintwoways.Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalreadyhasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseeventclassisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping,t1,thentransformswillbeappliedintheorder:

Toptestclass>T1class>t1eventclassmapping

Itisperfectlypossibleforatransformtouseuserdefinedeventfieldsinstantiatedbyearliertransforms;however,beveryawarethatifanystatementinatransformfails(perhapsbecauseafielddoesn'texist),thentheprocessingofthattransformwillstopatthatpointandnofurtherstatementswillbeexecuted.Anyfurthertransformswillbeexecuted(atleastuntilanerrorisreached).

AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeensuccessfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,attransformtime,mostofthestandardeventfieldsareavailable,exceptthosepopulatedatdatabaseinsertionstime(evid,stateChange,eventState,dedupid,count,eventClassMapping,firstTimeandlastTime).AnyuserdefinedfieldscreatedbytheRegexarealsoavailable.

Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangetheclassforeventsthatwouldotherwisebe/Unknown.

Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist(likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored.Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue.Transformsareimplementedbythezeneventddaemonsoinspecttheendof$ZENHOME/log/zeneventd.logtoseetheerrormessagereportingtheabsenceoftheattribute.

AclasstransformisconfiguredfromtheActioniconatthebottomofthelefthandmenuforaneventclass.


AmappingtransformisspecifiedaspartofthesameeventmappingdialoguethatdefinestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,whenyouusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatinganerror.

Figure31onpage57showedaneventmappingcalledlinetestwhichincludesatransformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromtheeventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theeventsummaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsanduserdefinedfields.

evt.myDevId=device.idevt.mySnmpSysLoc=device.snmpLocationevt.mySnmpSysContact=device.snmpContactevt.mySnmpStatus=device.getSnmpStatusString()evt.summary="Problemis%sondevice%s.Pleasecall%s"%(evt.summary,

evt.myDevId,evt.mySnmpSysContact)Mostoftheuserdefinedfieldsareassignedtosimpleattributesofeithertheeventorthedevice;forexample,device.snmpContact.ThelinebeforetheenddemonstratesusingaPythonmethodtogetvalues;forexampledevice.getSnmpStatusString()(notethe()attheendthisisthecluethatitisamethodratherthananattribute).

7.2 Understanding fields available for event processingSohowdoesoneworkoutwhatattributesandmethodsareavailable?TheZenossCore4AdministrationGuidedocumentstheTALESEventAttributesinAppendixD3butthisisonlyastartingpoint.

Similarly,AppendixD2documentsTALESDeviceAttributesandmethodsbutthisinformationisveryincomplete.

Whenzeneventdisprocessinganevent,strictlyitisworkingonanumberofPythondictionariesthatmakeupaZepRawEventProxyobjectclass.Rememberfromthearchitecturesectionthatzeneventdtakeselementsfromtheraweventsqueue,processesthemandoutputstheresulttothezeneventsqueuetobefurtherprocessedbythezeneventserverdaemon(Figure12,Zenoss4eventarchitecture).Themessagesontheraweventqueue(likeallotherqueuemessages)areblobsofbinarydata.

Thereareanumberofmodulesin$ZENHOME/lib/python/zenoss/protocolsthatmanipulatethismessagedatausingGoogleprotobufsasadatainterchangeformatforthestructuredqueuemessagedata.

$ZENHOME/Products/ZenEvents/events2containsthreePythonfilesthatarecrucialforunderstandingthedetailsofhowzeneventdprocessestherawevent:

processing.py

fields.py

proxy.py


$ZENHOME/Products/ZenEvents/zeneventd.pyhasanumberofpipelinesthataneventpassesthrough.Theireffectcanbeseenbeanalysingzeneventd.logiftheDebuglogginglevelisturnedon.

processing.pycontainsthecodetoimplementeachofthepipelinestagesexecutedbyzeneventd.Therearemethodstoprocessesarawevent,adddeviceandeventcontext,processruleandregextoestablishaneventclass,andtoperformtransforms.Thereisalsoamethodtogeneratethefingerprintfield.


Figure32EventPipelineProcessorobjectclassinzeneventd.py


Figure33EventFieldobjectclassin$ZENHOME/Products/ZenEvents/events2/fields.py

$ZENHOME/Products/ZenEvents/events2/fields.pycontainsobjectclassdefinitionsfor:

EventField

TheEventFieldattributesmatchupwiththebaseMySQLdatabasefieldsinzenoss_zep.

TheActor,DetailandTagfieldsaredefinedassubclassesoftheobject

EventSummaryField

Hastheadditionalfieldsthatarepopulatedwhentheeventisinsertedintothezenoss_zepdatabaseevent_summarytable.

ZepRawEventField

HasthesamefieldsasEventFieldbutalsohasclear_event_classasthatisneededbythezeneventdprocessingpipelinesasitispartoftheeventcontext.

Notethatthedefinitionsinfields.pyarenothelpfulwhendecidingwhatattributesareavailabletotransforms;thesearethefieldsonefindsinthezenoss_zepdatabase.

7.2.1 Event Proxies$ZENHOME/Products/ZenEvents/events2/proxy.pyisthekeytounderstandingwhatattributesareavailablewhenwritingrulesandtransforms.proxy.pyprovides


Figure34EventSummaryFieldandZepRawEventFielddefinitions

translationsbetweenencodedformatsofeventsandahumanreadableJSON(JavaScriptObjectNotation)format.

Asfaraspossible,theattributespresentedbyaproxyarethesameinZenoss4astheywereinpreviousversions.


Figure35EventProxydefinitionin$ZENHOME/Products/ZenEvents/events2/proxy.py

AnEventProxyisseveralPythondictionaries:

Themainbodyoftheeventisadictionarycalled_event

Adetailsdictionary

An_tagsdictionary

Adictionaryfor_clearClasses

Adictionaryfor_readOnlyattributes

TherearealargenumberofPython@propertydecoratorconstructswhosepurposeistopresentanattributeusingamethod,forexample:

@propertydefdevice(self):returnself._event.actor.element_identifier

definesanattributecalleddevicewhichisdeliveredbyamethodthatreturnsthevalueoftheevent'sactor'selement_identifier.deviceisthefieldthatwehave(havealwayshad)tomanipulateintransforms.

The@propertydefinitionsattheendofFigure35showsimplerdefinitionsthatreturnthevalueofabasicfieldofanevent(usingtheEventFielddefinitionsdefinedinfields.py).

WhenauserviewseventdetailsusingtheZenossGUIoraccessesdatafromfromtheevent_summarytableofthezenoss_zepdatabaseusingtheJSONAPI,theeventdatapresentedisanEventSummaryProxy,whichisaJSONformat.TheEventSummaryProxyinheritsfromtheEventProxybutalsohasattributesthatareaddedondatabaseinsertion:

evid

stateChange

clearid

firstTime

lastTime

count

ownerid

eventState

TheEventSummaryProxywasoriginallydesignedwithanideaofkeepingalleventdata,treatingduplicatesasmultipleoccurrenceswithintheEventSummaryProxy;howeverthescalabilitywasnotfeasibleso,inpractisethefieldsofaneventareinthezero'thelementofanEventSummaryoccurrencelist.


proxy.pyalsodefinesaclassforZepRawEventProxywhichinheritsfromEventProxy.TheadditionalpropertiesforZepRawEventProxyarefor_ClearClasses,_actionandeventClassMapping.

Itistheattributesdefinedinproxy.pyfortheZepRawEventProxyobjectclassthatareavailableforuseinrulesandtransforms.

7.2.2 Event DetailsSowhathappenstoauserdefinedeventattributegenerated,say,bythevarbindsthatcomeinonanSNMPTRAP?

RememberthattheEventProxyhasanumberofdictionaries,includingadetailsdict

Documents

Zenoss Core4 Event Management Paper