Puppet Testing For The Win

Phil Zimmerman phil.zimmerman@twcable.comhttps://twitter.com/phil_zimmerman

Puppet

Software defined infrastructure – perfect for VCS

Configuration Management for servers

Declarative language written in Ruby DSL

Uses manifests to define server configurations

Brings servers into a desired state and keeps them there

Eliminates “snowflake” environments

Puppet Review - Modules Puppet Modules

Self-contained bundles of code Develop your own Download from the Puppet Forge (

https://forge.puppetlabs.com/) Contain manifests, files, templates and,

ahem… tests

Puppet Review – Manifests Puppet Manifests

End in the .pp file extension Each manifest in a puppet module should

contain one class or defined type Define the set of resources (packages,

files, services) that the module represents Can contain logic (conditionals,

collections, functions, etc) Are the source for the compiled catalog

Puppet Review - Catalog

The catalog Represents the DAG (directed acyclic graph)

of resources and the desired system state for a given node

Is compiled from the set of modules’ manifests defined for a given node

In master/agent puppet, compiled by the master and applied on the agent node

Masterless puppet, compiled locally on node Represented on disk as a YAML document

How Puppet Compiles and Applies a Catalog

Hypothetical Workflow

Need to upgrade Java version on tomcat6 vms

Get latest puppet code from vcs Make the version change in my

manifest Simple change, it looks good to me Commit my changes

Please work, please work…

Face Palm

Oh no – Java was updated on my tomcat7 vms too…. Wait, wat?!

Face Palm

FAIL!!

Daily Life Without Tests

Why Test Puppet Code?

Puppet manifests are code Improve consistency and predictability

of server provisioning Well-defined tools (rspec-puppet,

puppet parser, puppet-lint, serverspec, vagrant, etc.)

Automatable Complex, data-driven server

configuration Think of others and future you!

Testing Tools

Syntax Checking Static Analysis Unit Tests (rspec-puppet) Configure Jenkins to Run These Vagrant Server-spec Packer

Syntax Checking

puppet parser validate-make sure the manifests will

generate a catalog

Static Analysis

puppet-lint-make sure we adhere to the

puppet style guide

“Unit” Tests

rspec-puppet (http://rspec-puppet.com/) Written by Tim Sharpe (https://github.com/rodjek) rspec, extended to work with puppet “unit tests” for puppet code Designed to test the catalog▪ Tests at the module level, not system level▪ Verify resources are present and dependencies are met▪ Verify resources are configured as expected▪ Verify file content (even when using templates and hiera

– yes!) puppetlabs-spec_helper (Rakefile, .fixtures.yml)

rspec-puppet

rspec-puppet ruby gem rspec-puppet-init▪ Rakefile▪ spec/spec_helper.rb▪ spec/{classes,defines,functions,hosts,fixtures}

puppetlabs_spec_helper ruby gem .fixtures.yml Ideal for testing manifests referencing forge

modules Both gems work together to ease the burden

of boilerplate setup and configuration

An Example – sshd

sshd intall_spec.rb file

Test that the sshd package is installed

sshd intall_spec.rb file

Make sure sshd_config file is present with desired attributes:

sshd intall_spec.rb file

Ensure sshd_config has certain entries:

sshd intall_spec.rb file

Verify sshd service is enabled and running with proper resource dependencies in place:

Testing Behavior

Parameterized class let(:params) { {:foo => ‘abc’, :bar =>

‘xyz’} }

Specify values for facter facts let(:facts) { {:operatingsystem =>

‘CentOS’, :ipaddress => ‘192.168.33.10’} }

Specify fqdn for a node let(:node) { ‘puppet-test-

01.lab.webapps.rr.com’ }

Mocking (Sort of , But Not Really)

Testing With Hiera Data

hiera-puppet-helper gem

Run the Tests

What This Looks Like in Jenkins

Puppet Testing In Jenkins

Lint Warnings

rspec-puppet Test Results

Automated Tests and Trending in Jenkins

We Can Test Modules – WIN!

This is awesome, but we’re not done Next level of testing is to perform a

puppet run on a test vm and verify all is good

We are ready for a server test – enter serverspec

How Do I Know My Server Is Configured Correctly?

Server Spec (http://serverspec.org/)

Designed to validate that a server is configured appropriately after it’s been provisioned

Independent of Puppet, Chef, CFEngine, SaltStack, etc.

Tests your servers’ actual state directly via ssh▪ No server-side software or agents required!

Serverspec Should Feel Familiar

serverspec ruby gem similar dsl as rspec, rspec-puppet serverspec-init

spec dir sample spec file spec_helper.rb Rakefile

spec_helper.rb

serverspec - sshd_spec.rb

serverspec resource types

Some Examples

describe iptables do it { should have_rule(‘-P INPUT ACCEPT’).with_table(‘mangle’).with_chain(‘INPUT’) }end

describe port(2003) do it { should be_listening.with(‘udp’) }end

describe package(‘httpd’) do it { should be_installed }end

describe service(‘sshd’) do it { should be_monitored_by(‘monit’) }end

Group Spec Files According to Host Name

Automated Post-Provisioning Validation

Vagrant Disposable Testing VMs We use Puppet Enterprise at TWC Vagrantfile that auto installs and configures

Puppet Enterprise master and agent(s) https://github.com/adrienthebo/vagrant-pe_build

Personal replica of production Puppet Enterprise setup

Can apply any role to the agent and test the server config

Destroy the agent vm when done

Packer.io

“Create identical machine images for multiple platforms from a single source configuration”

Supports all the main provisioners including Puppet

Can optionally create a vagrant box from the same source configuration

Automatable and Testable Extendable plugin architecture Powerful option for any vm architecture,

especially cloud-based (internal and external) Full of awesome

Helpful Links

Miscellaneous Links http://www.slideshare.net/PuppetLabs/stephen-connolly http://www.slideshare.net/PuppetLabs/automated-puppet-testing-puppe

tcamp-chicago-12-scott-nottingham

https://github.com/adrienthebo/vagrant-pe_build https://github.com/puppetlabs/rspec-system

Vim Tools Syntastic (https://github.com/scrooloose/syntastic)

▪ Checks syntax and displays errors to the user

Vim-puppet (https://github.com/rodjek/vim-puppet)▪ Syntax highlighting▪ Style checking

Cool Tool Links Vagrant - http://www.vagrantup.com Packer - http://www.packer.io Stackhammer - http://www.cloudsmith.com