Groovy DevOps in the Cloud for GR8Conf US 2015

01

About me02

Andrey AdamovichBio: Developer, coach, speaker, author

Company: Aestas/IT (http://aestasit.com)

E­mail: andrey@aestasit.com

Linkedin: http://www.linkedin.com/in/andreyadamovich

Lanyrd: http://lanyrd.com/profile/andrey­adamovich

GitHub: https://github.com/aadamovich

SO: http://stackoverflow.com/users/162792/andrey­adamovich

Twitter: @codingandrey, @aestasit

••••••••

03

What's this presentation about?DevOps

Intrastructure Provisioning

Continuous Integration

Continuous Delivery

••••

04

TechnologiesGroovy ­ http://groovy.codehaus.org

Gradle ­ http://gradle.org

Jenkins ­ http://jenkins­ci.org

Puppet ­ http://puppetlabs.com

AWS ­ http://aws.amazon.com

•••••

05

Developers +Operations =

?06

First Blood07

Ant + Gradleant.taskdef(

name: 'scp',

classname: 'o.a.t.a.t.o.ssh.Scp',

classpath: configurations.secureShell.asPath)

ant.taskdef(

name: 'sshexec',

classname: 'o.a.t.a.t.o.ssh.SSHExec',

classpath: configurations.secureShell.asPath)

01.

02.

03.

04.

05.06.

07.

08.

09.

08

Simple callant.sshexec(

host: host,

username: user,

password: password,

command: command,

trust: 'true',

failonerror: failOnError)

01.

02.

03.

04.

05.

06.

07.

09

Next step: wrapper functiondef ssh(String command,

Properties props,

boolean failOnError = false,

String suCommandQuoteChar = "'",

String outputProperty = null)

...

01.

02.

03.

04.

05.

06.

07.

10

Next step: wrapper functiondef scp(String file,

String remoteDir,

Properties props)

...

01.

02.

03.

04.

05.

11

Task example Itask installFonts <<

forAllServers props ‐>

ssh('yes | yum install *font*', props)

01.

02.

03.

04.

05.

12

Task example IItask uninstallNginx <<

forAllServers props ‐>

ssh('/etc/init.d/nginx stop', props)

ssh('yes | yum remove nginx', props, true)

ssh('rm ‐rf /etc/yum.repos.d/nginx.repo', props)

ssh('rm ‐rf /var/log/nginx', props)

ssh('rm ‐rf /etc/nginx /var/nginx', props)

01.

02.

03.

04.

05.

06.

07.

08.

09. 13

DrawbacksNew connection each time

Excplicit repeating parameters

Complex scripts are hard to maintain

Tasks are not idempotent

••••

14

Sshoogr

15

Sshoogr featuresGroovy­based SSH DSL for:

Remote command execution

File uploading/downloading

Tunneling

•••

16

Sshoogr usage (import)@Grab(

group='com.aestasit.infrastructure.sshoogr',

module='sshoogr',

version='0.9.16')

import static com.aestasit.ssh.DefaultSsh.*

01.

02.

03.

04.

05.

17

Sshoogr usage (defaults)defaultUser = 'root'

defaultKeyFile = new File('secret.pem')

execOptions

verbose = true

showCommand = true

01.

02.

03.

04.

05.

06.

18

Sshoogr usage (connection)remoteSession

url = 'user2:654321@localhost:2222'

exec 'rm ‐rf /tmp/*'

exec 'touch /var/lock/my.pid'

remoteFile('/var/my.conf').text = "enabled=true"

01.

02.

03.

04.

05.

06.

19

Sshoogr usage (multi­line content)remoteFile('/etc/yum.repos.d/puppet.repo').text = '''

[puppet]

name=Puppet Labs Packages

baseurl=http://yum.puppetlabs.com/el/

enabled=0

gpgcheck=0

'''

01.

02.

03.

04.

05.

06.

07.

20

Sshoogr usage (file copying)remoteSession

scp

from localDir "$buildDir/application"

into remoteDir '/var/bea/domain/application'

01.

02.

03.

04.

05.

06.

21

Sshoogr usage (command result)def result = exec(command: '/usr/bin/mycmd',

failOnError: false, showOutput: false)

if (result.exitStatus == 1)

result.output.eachLine line ‐>

if (line.contains('WARNING'))

throw new RuntimeException("Warning!!!")

01.

02.

03.

04.

05.

06.

07.

08.

09. 22

Sshoogr usage (shortcuts)if (ok('/usr/bin/mycmd'))

...

if (fail('/usr/bin/othercmd'))

...

01.

02.

03.

04.

05.

06.

23

Sshoogr usage (tunnels)tunnel('1.2.3.4', 8080) int localPort ‐>

def url = "http://localhost:$localPort/flushCache"

def result = new URL(url).text

if (result == 'OK')

println "Cache is flushed!"

else

throw new RuntimeException(result)

01.

02.

03.

04.

05.

06.

07.

08.

09. 24

Sshoogr usage (prefix/suffix)prefix('sudo ')

exec 'rm ‐rf /var/log/abc.log'

exec 'service abc restart'

suffix(' >> output.log')

exec 'yum ‐y install nginx'

exec 'yum ‐y install mc'

exec 'yum ‐y install links'

01.

02.

03.

04.

05.

06.

07.

08.

09. 25

Still problemsComplex scripts are still not easy to maintain

Scripts are usually not idempotent••

26

Puppet27

Why Puppet?More mature than competition

Large community

Readable DSL

Good acceptance from DEVs and OPs

No need to learn Ruby ;)

•••••

28

Puppet example

29

Puppet provisioning

30

Puppet provisioning

31

Puppet provisioning

32

Puppet provisioning

33

Puppet state management

34

Puppet state management

35

Puppet state management

36

Puppet modules

37

Puppet modules

38

Puppet modules

39

Deploymentvs.

Infrastructuremanagement40

Push vs. Pull41

Pull

42

Push

43

Sshoogr +Gradle +Puppet44

Upload modulestask uploadModules <<

remoteSession

exec 'rm ‐rf /tmp/repo.zip'

scp

from localFile "$buildDir/repo.zip"

into remoteDir "/root"

...

01.

02.

03.

04.

05.

06.

07.

08.

45

Upload modules ...

exec 'rm ‐rf /etc/puppet/modules'

exec 'unzip /tmp/repo.zip ‐d /etc/puppet/modules'

01.

02.

03.

04.

05.

46

Apply manifeststask puppetApply(dependsOn: uploadModules) <<

remoteSession

scp

from localFile "$buildDir/setup.pp"

into remoteDir "/tmp"

exec 'puppet apply /tmp/setup.pp'

01.

02.

03.

04.

05.

06.

07.

08.

09. 47

What we solved?Separated infrastructure state description and operations tasks

Scripts became more maintainable and idempotent••

48

In the meanwhile...We started developing complex/generic Puppet modules

Modules need proper testing

...on different platforms

•••

49

Do you test, right?How to test this stuff?

How to reuse a JUnit approach to testing?

We wanted things to be SIMPLE!

•••

50

PUnit

51

PUnitSimple testing tool for verifying remote server state

Uses Sshoogr and JUnit

Reuse reporting features of JUnit

As simple as ...

••••

52

PUnit example (derby)class DerbyInstallTest

extends BasePuppetIntegrationTest

@Before

void installDerby()

apply("include derby")

...

01.

02.

03.

04.

05.

06.

07.

08.

53

PUnit example (derby)@Test

void ensureDerbyRunning()

command('service derby status > derbystatus.log')

assertTrue fileText("/root/derbystatus.log")

.contains('Derby')

assertTrue fileText("/root/derbystatus.log")

.contains('is running.')

01.

02.

03.

04.

05.

06.

07.

08.

54

PUnit example (derby)@Test

void ensureCanConnect()

Thread.sleep(10000)

uploadScript()

command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +

'testDataScript.sql > derbytest.log')

...

01.

02.

03.

04.

05.

06.

07.

55

Continuous integration

56

Jenkins build

57

Nextproblem?

58

ScalabilityHow do we test on different OS?

How do we run parallel tests on multiple architectures?

How do we avoid selling our houses?

•••

59

Amazon WebServices

60

Elastic Compute CloudMature

Great API

Virtual hardware variety

OS variety

••••

61

Gramazon

62

GramazonGroovy­based API for interacting with EC2

Integration with Gradle••

63

Gramazon example Itask startInstance(type: StartInstance)

keyName 'cloud‐do'

securityGroup 'cloud‐do'

instanceName 'gramazon/cloud‐do'

stateFileName 'cloud‐do.json'

ami 'ami‐6f07e418'

instanceType 't1.micro'

waitForStart true

01.

02.

03.

04.

05.

06.

07.

08.

09. 64

Gramazon example IItask terminateInstance(type: TerminateInstance)

stateFileName 'cloud‐do.json'

01.

02.

03.

65

The flowStart instance(s)

Upload manifests

Run tests

Generate report

Terminate instance(s)

1.

2.

3.

4.

5.

66

Next issue?67

Imgr

68

ImgrA tool for building images

Inspired by Packer••

69

SupportsShell

Puppet••

70

Configuration example

71

Summary72

Images, manifests, tasks

73

The big picture

74

ConclusionsGroovy/Gradle is an ultimate automation glue!

Reuse your existing Java knowledge

...to build a bridge between DEVs and OPs

Reuse development best practices for OPs

Infrastructure configuration is outsourcable now!

Automate!

••••••

75

Source codeSshoogr: https://github.com/aestasit/sshoogr

Sshoogr Gradle: https://github.com/aestasit/sshoogr­gradle

Groowin: https://github.com/aestasit/groowin

Groowin Gradle: https://github.com/aestasit/groowin­gradle

PUnit: https://github.com/aestasit/puppet­unit

Gramazon: https://github.com/aestasit/gramazon

Imgr: https://github.com/aestasit/imgr

•••••••

76

Questions?77

Thank you!78