Upload
yury-tsarev
View
147
Download
2
Embed Size (px)
Citation preview
Test Driven Infrastructure with Docker, Test Kitchen and Serverspec Yury Tsarev
Agenda
▸ Goal▸ Test kitchen▸ Implementation at GoodData▸ Create test-driven infrastructure change▸ Wrap-Up
Goal
▸ Infrastructure code should be treated as any other code▸ Apply TDD for puppet▸ Grow regression test suite
Test Kitchen
▸ http://kitchen.ci/ ▸ Test orchestrator▸ Originated in Chef community▸ Very pluggable on all levels▸ "Your infrastructure deserves tests too."▸ Book - Test-Driven Infrastructure with Chef
Test Kitchen - High Level Process I
1. Create VM/Container2. Run configuration management code there 3. Run the test suite
Test Kitchen - High Level Process II
Test Kitchen
Drivercreate
Provisionerconverge
Verifierverify
Instance Under Test
Configuration File - .kitchen.yml
▸ Driver: what type of VM/containerization/cloud to use▸ Amazon EC2, Blue Box, CloudStack, Digital Ocean, Rackspace, OpenStack,
Vagrant, Docker, LXC containers
▸ Provisioner: which configuration management tool to apply▸ Chef, Puppet, Ansible, SaltStack
▸ Verifier: test automation type to verify with▸ Bats, shUnit2, RSpec, Serverspec
▸ Transport: mechanism to upload files into instance under test
▸ Platform: os/environment to run tests on▸ Suite: test suite to execute
Implementation at GoodData
▸ Docker driver▸ Puppet provisioner▸ SFTP Transport▸ Serverspec verifier
Infrastructure Test Pipeline in GD
Puppet Catalog
CompilationTest Kitchen
Docker Container
Puppet Serverspec
VM Instance in Dev/QA/Staging Env
Puppet Serverspec
Shellmocking
Docker Container
Puppet Serverspec
Shellmocking
Docker Driver
▸ https://github.com/portertech/kitchen-docker▸ kitchen driver to work with docker containers as machines
under testdriver:
name: docker
image: docker-registry.example.com:80/gdc:R23
platform: rhel
use_sudo: false
provision_command:
yum clean all && yum makecache
Puppet Provisioner
▸ https://github.com/neillturner/kitchen-puppet▸ Uploads puppet code into instance▸ Runs puppet there - converge▸ Provides facts customization facility
Puppet Provisioner - Example
provisioner:
name: puppet_apply
require_chef_for_busser: false
modules_path: puppet/modules
manifests_path: puppet/manifests
hiera_data_path: puppet/hieradata
facterlib: /etc/puppet/facter
install_custom_facts: true
custom_facts:
docker: 1
ec2data_freeipa_otp: test
ec2data_nopuppet_cron: 1
ec2_public_ipv4: 127.0.0.1
# our custom fact for use in serverspec call
serverspec_check_type: role
Transport
▸ https://github.com/coderanger/kitchen-sync▸ Replaced default scp with sftp transport▸ Reduced 1.5m to 5 sec for puppet and test suite upload
transport:
name: sftp
remote_ruby_path: 'ruby193-ruby'
username: kitchen
Serverspec Test Suite
▸ http://serverspec.org/ - RSpec based framework▸ http://serverspec.org/resource_types.html ▸ Independent of kitchen - can be used standalone▸ We keep test suite together with puppet code under
spec directory to have ability to create consistent PRs
Serverspec DSL Examples
describe file('/var/log/httpd') do
it { should be_directory }
end
describe command('apachectl -M') do
its(:stdout) { should contain
('proxy_module')}
end
describe default_gateway do
its(:ipaddress) { should eq '192.168.10.1'
}
its(:interface) { should eq 'br0'
}
end
describe cgroup('group1') do
its('cpuset.cpus') { should eq 1 }
end
describe docker_container('focused_curie')
do
its(:HostConfig_NetworkMode) { should eq
'bridge' }
its(:Path) { should eq '/bin/sh' }
end
describe port(53) do
it { should be_listening.with('udp') }
end
Serverspec Verifier: Busser or Shell
▸ Simple
https://github.com/test-kitchen/busser-serverspec
▸ Advanced
https://github.com/vincentbernat/serverspec-example
Serverspec Verifier in GD
▸ Based on shell verifier and serverspec test suite▸ Runs a serverspec test suite against the configured
(converged) instance▸ Reports in shell and junit - jenkins ready
verifier:
name: shell
remote_exec: true
command: |
sudo -s <<SERVERSPEC
export SERVERSPEC_ENV=$EC2DATA_ENVIRONMENT
export SERVERSPEC_BACKEND=exec
serverspec junit=true tag=~skip_in_kitchen check:role:$EC2DATA_TYPE
SERVERSPEC
Shellmocking
▸ To bypass external dependencies and docker specific limitations
▸ Simple ruby script▸ Wraps package manager invocation▸ Mock is defined in simple yaml format:
package:
/path/to/executable: contents
Shellmock ExampleConfiguration Resulting mock
ipa-client:
/usr/sbin/ipa-client-install: |
echo 'server = freeipa.example.
com' > /etc/ipa/default.conf
echo 'fake' > /etc/krb5.keytab
$ cat /usr/sbin/ipa-client-install
echo 'server = freeipa01.intgdc.
com' > /etc/ipa/default.conf
echo 'fake' > /etc/krb5.keytab
sssd:
/usr/sbin/sssd:
$ cat /usr/sbin/sssd
#!/bin/bash
echo I am a fake /usr/sbin/sssd
Linux Kernel Capabilities
▸ Sometimes it is necessary to enable kernel capabilities to make things working in container
driver:
cap_add:
- SYS_RESOURCE
▸ In the above example we are allowing container to set NPROC resource limit
▸ See `man capabilities` for an actual list for you kernel version
Defining Puppet Type in Kitchen
▸ Declare in platforms section of .kitchen.ymlplatforms:
- name: zuul
provisioner:
custom_facts:
ec2data_type: zuul
▸ kitchen converge▸ shellmock▸ repeat
Create Test-Driven Infrastructure Change
▸ Write serverspec expectation for new code▸ kitchen verify <type>▸ Observe related test is red▸ Write puppet code▸ kitchen converge <type>▸ kitchen verify <type>▸ Observe related test is green▸ Commit the changes and create PR to puppet repo▸ DEMO
WTF? Why should I write the same thing two times both in puppet and serverspec?
▸ It pays off at scale▸ It creates Red-Green-Refactor TDD flow
▸ Testable code▸ Regression test suite
▸ Ideally you are not only repeating matching puppet declarations in serverspec but testing the outcomeunder_kerberos_auth = ['', 'status', 'status.json', 'status/payload', 'images/grey.png']
under_kerberos_auth.each do |path|
describe command("curl -k -X POST http://127.0.0.1/#{path}") do
its(:stdout) { should match(/301/) }
end
describe command("curl -k -X POST https://127.0.0.1/#{path}") do
its(:stdout) { should match(/401/) }
its(:stdout) { should match(/Authorization Required/) }
end
end
Wrap-Up: Benefits
▸ Scratch environment▸ Test in isolation▸ Easy to test permutations▸ Resource efficiency▸ Test-first/test-driven approach for infrastructure code▸ Fast feedback - even before a commit▸ Naturally growing regression test suite▸ Easily pluggable into CI/CD pipeline
Wrap-Up: OSS Side of Things
▸ Multiple open source projects combined▸ We contributed a lot into related kitchen projects and
serverspec▸ Nominated as kitchen-puppet core contributor
Wrap-Up: What is Next ?
▸ Even with containers it is still resource costly operation▸ We use automation based on puppet-catalog-diff to
determine which puppet types are affected by Pull Request▸ Then only affected types are tested by kitchen
Wrap-Up: A Note on Agnostic Approach
▸ As the kitchen driver is agnostic to virtualization solution/cloud provider
▸ As the kitchen provisioner is agnostic to configuration management solution
▸ As the serverspec is agnostic to configuration type at all▸ We are capable to make brave movements in future!
THANKS! Questions?
▸ http://kitchen.ci/ ▸ https://github.com/portertech/kitchen-docker ▸ https://github.com/neillturner/kitchen-puppet ▸ http://serverspec.org/