Monitoring (with) PuppetMonitoring (with) Puppet

PuppetCamp Dusseldorf

@KrisBuytaert

KrisKris BuytaertBuytaert

● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu

● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Organising too many confs , #devopsdays, Organising too many confs , #devopsdays, #loadays, ... #loadays, ...

● Evangelizing devopsEvangelizing devops

Do you know what your children do at 5 am in Do you know what your children do at 5 am in the morning ?the morning ?

● Sleeping ?Sleeping ?

● Crashing a Party ?Crashing a Party ?

● Why are the cops at your front door ?Why are the cops at your front door ?

● What happened ?What happened ?

● How long have they been gone ?How long have they been gone ?

devops = clamsdevops = clams● CultureCulture

● (Lean)(Lean)

● Automate all the things ... Automate all the things ...

• Build Automation Build Automation

• Test Automation Test Automation

• IACIAC

● Monitoring , Metrics ... Monitoring , Metrics ...

● SharingSharing

Monitoring is usually an Monitoring is usually an aftertoughtaftertought

ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME

#monitoringsucks#monitoringsucks● John Vincent (@lusis)John Vincent (@lusis)

● A sub movement A sub movement

● https://github.com/monitoringsucks/https://github.com/monitoringsucks/

#monitoringlove#monitoringlove• #monitoringlove hacksessions#monitoringlove hacksessions

• #monitorama#monitorama

For years we've tolerated humans to to For years we've tolerated humans to to make structural manual changes to the make structural manual changes to the infrastructure our critical infrastructure our critical applications are running on.applications are running on.

Whilst at the same time demanding Whilst at the same time demanding those critical applications to go those critical applications to go

trough rigid test scenarios.trough rigid test scenarios.

Who let this happen ?Who let this happen ?

Infrastructure as CodeInfrastructure as Code● Model our infrastructure Model our infrastructure

● A fast reproducable platformA fast reproducable platform

● Disaster discovery for free“ ”Disaster discovery for free“ ”

Infrastructure as CodeInfrastructure as Code● Code = Code Code = Code

● Version Control Version Control

● Quality ChecksQuality Checks

● TestingTesting

● Continuous IntegrationContinuous Integration

● Continous DeliveryContinous Delivery

Infrastructure as CodeInfrastructure as Code● Core Infrastructure Core Infrastructure

● Middleware deployment and integrationMiddleware deployment and integration

● Automated continuous application Automated continuous application deploymentdeployment

● Integrated Security enforcementIntegrated Security enforcement

● Host, Service and Application Host, Service and Application Monitoring configuredMonitoring configured

Why #monitoringsucksWhy #monitoringsucks

● Manual config (gui)Manual config (gui)

● Not in sync with realityNot in sync with reality

● Hosts onlyHosts only

● Services sometimesServices sometimes

● Appliccation neverAppliccation never

● ChaosChaos

• Really ?Really ?

• Service,Service,

• FunctionalitiesFunctionalities

• eg. vhosts etceg. vhosts etc

• Single Source of TruthSingle Source of Truth

But tools do AutodetectionBut tools do Autodetection

Monitoring 101Monitoring 101● Deploy a host,Deploy a host,

● Add it to the monitoringAdd it to the monitoring

● Add collection toolsAdd collection tools

● Add check definitionsAdd check definitions

● Update the monitoring tool configUpdate the monitoring tool config

● RestartRestart

Collect Metrics 101Collect Metrics 101

CollectdCollectd● CollectsCollects

● Zillion PluginsZillion Plugins

• Nginx,apache, mysql, diskNginx,apache, mysql, disk

● Graphite Carbon PluginGraphite Carbon Plugin

● Send metrics to graphiteSend metrics to graphite

● https://github.com/KrisBuytaert/puppehttps://github.com/KrisBuytaert/puppet-collectdt-collectd

Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/

● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !

● git clonegit clone

● git submodule init git submodule init

● git submodule update git submodule update

● vagrant up vagrant up

2014 style dashboards2014 style dashboards

Alerting 101Alerting 101

Alert all the thingsAlert all the thingsAnd get alertfatigueAnd get alertfatigue

● We are in Germany, EuropeWe are in Germany, Europe

● It needs to be configurable It needs to be configurable

=> Icinga=> Icinga

But Monitoring config is But Monitoring config is Data ?Data ?● If it can be generated it's not user If it can be generated it's not user generated content anymoregenerated content anymore

● Your computer can generate your Your computer can generate your config !config !

Stored ConfigsStored Configs

Collection and ExportCollection and Export

Export :Export :

@@resource { @@resource {

... }... }

Collect:Collect:

Resource <<| Resource <<| query |>>query |>>

Clean out nodes that dissapearClean out nodes that dissapear

puppet node clean puppet node clean

Exporting and Collecting Exporting and Collecting

Default Puppet TypesDefault Puppet Types

Puppet-icinga modulePuppet-icinga module● https://github.com/inuits/puppet-https://github.com/inuits/puppet-icingaicinga

Monitoring a VhostMonitoring a Vhost

Alternative ApproachesAlternative Approaches

● https://gist.github.com/jfryman/5808537https://gist.github.com/jfryman/5808537

● https://github.com/favoretti/puppetdbhttps://github.com/favoretti/puppetdb-external-naginator-external-naginator

● Deploy a new appDeploy a new app

● Add monitoringAdd monitoring

● Add Real application monitoringAdd Real application monitoring

● Both on infra and on app levelBoth on infra and on app level

Monitoring Puppet & FriendsMonitoring Puppet & Friends

Puppet RunsPuppet Runs

PuppetMasterPuppetMaster

PuppetMasterPuppetMaster @@nagios_service{"check_socket_8140_puppet_${::fqdn}":@@nagios_service{"check_socket_8140_puppet_${::fqdn}":

check_command => 'check_tcp!8140',check_command => 'check_tcp!8140',

service_description => 'TCP puppet on port 8140',service_description => 'TCP puppet on port 8140',

host_name => $::fqdn,host_name => $::fqdn,

use => 'generic-service',use => 'generic-service',

contact_groups => $::environment,contact_groups => $::environment,

notification_period => $::icinga::notification_period,notification_period => $::icinga::notification_period,

notifications_enabled => $::icinga::notifications_enabled,notifications_enabled => $::icinga::notifications_enabled,

target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",

}}

PuppetDBPuppetDB

PuppetDB(2)PuppetDB(2)● check_puppetdb_memorycheck_puppetdb_memory

• Java heap memoryJava heap memory

● check_puppetdb_processedcheck_puppetdb_processed

• Nr of reports processedNr of reports processed

● check_puppetdb_populationscheck_puppetdb_populations

• Resources, nodes, resources per nodeResources, nodes, resources per node

● check_puppetdb_queuecheck_puppetdb_queue

• Is pgsql down ? :)Is pgsql down ? :)

Puppet DashboardPuppet Dashboard @@nagios_service{"check_http_puppet_dashboard_${::fqdn}":@@nagios_service{"check_http_puppet_dashboard_${::fqdn}":

check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",

service_description => 'HTTP PuppetDashboard on port 3000',service_description => 'HTTP PuppetDashboard on port 3000',

host_name => $::fqdn,host_name => $::fqdn,

use => 'generic-service',use => 'generic-service',

contact_groups => $::environment,contact_groups => $::environment,

notification_period => $::icinga::notification_period,notification_period => $::icinga::notification_period,

notifications_enabled => $::icinga::notifications_enabled,notifications_enabled => $::icinga::notifications_enabled,

target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",

}}

Puppet Dashboard(2)Puppet Dashboard(2)

/usr/share/puppet-dashboard/spool/usr/share/puppet-dashboard/spool

PuppetServerPuppetServer

#MonitoringSucks#MonitoringSucks● Puppetruns break our Icinga boxenPuppetruns break our Icinga boxen

● BadlyBadly

● FrequentlyFrequently

It ain't borkenIt ain't borken● Successful puppet runSuccessful puppet run

● Successful Icinga reconfigureSuccessful Icinga reconfigure

● Disk usage growsDisk usage grows

● FastFast

Be aware of bucketsBe aware of buckets

A Puppet BugA Puppet Bug

Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics

● JMXTransJMXTrans

● Export JMXConfigsExport JMXConfigs

● Configure NRPE CheckConfigure NRPE Check

● Export NagiosCheckExport NagiosCheck

● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode

● Graph EmGraph Em

Collect Nagios Configs Collect Nagios Configs on Nagios Serveron Nagios Server

Triggers on GraphsTriggers on Graphs

Triggers on GraphsTriggers on Graphs

SummarySummary● Honour your parentsHonour your parents

● Don't manually do what machines can Don't manually do what machines can do for youdo for you

● Monitor your puppet infrastructure Monitor your puppet infrastructure too !too !

● Send Pull RequestsSend Pull Requests

● Icinga2 module in the worksIcinga2 module in the works

ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221