Aws + Puppet = Dynamic Scale

©2015,  Amazon  Web  Services,  Inc.  or  its  affiliates.   All  rights  reserved

Shiva  NSolutions  Architect,  AWS

@shivamaan

AWS  +  Puppet  =  Dynamic  Scale

Application  Rate  of  Change  Is  Increasing

Source  2013  State  of  DevOps  Report  Puppet  Labs,  IT  Revolution  Press

Top  Performers30x  more  frequent  code  deployments

Of  those  with  DevOps  implemented  >  12  month:

More  than  25%  can  deploy on  demand

~11.6sMean  timebetween  deployments  (weekday)

~1,079Max  number  of  deployments  in  a  single  hour

~10,000Mean  number  of  hosts  simultaneously  receiving  a  deployment  

~30,000Max  number  of  hosts  simultaneously  receiving  a  deployment  

DEPLOYMENTS  AT  AMAZON.COM

75%Reduction  in  outages  triggered  by  software  deployments  since  2006

90%Reduction  in  outage  minutes  triggered  by  software  deployments

Business  Value  of  Frequent  Deployments

~0.001%Software  deployments  cause  anoutage

example.com – Online  auction

AWS Sydney Region

Public subnet App subnetHAProxy Vertx

Glassfish

Data subnet

Puppet  Stack

ELKStackMgmt subnetCloudFront S3 SES SNS

Route 53

Jenkins

Let’s  build  

[  ]  OS  [  ]  Patches[  ]  OS  Configuration[  ]  VM  Templates  (Amazon  Machine  Image)[  ]  VPC[  ]  Networking  (Subnets,  ELBs)[  ]  Network  Security  (NACLs  and  SGs)[  ]  Software[  ]  Application  (Code)[  ]  Services

Start  with  a  Base  Image

Machine  Images  can  come  from  several  sources..- AWS  Provided  (Quick  Start)- AWS  Marketplace- Community  AMIs- Build  your  own  AMIs- Migrate  On  Premise  Image We  need  a  server  to  

build  an  image  to  build  a  server

Let’s  build  

[X]  OS  [  ]  Patches[  ]  OS  Configuration[  ]  VM  Templates  (Amazon  Machine  Image)[  ]  VPC[  ]  Networking  (Subnets,  ELBs)[  ]  Network  Security  (NACLs  and  SGs)[  ]  Software[  ]  Application  (Code)[  ]  Services

Layers  to  Golden  Image

CI-­‐CD  Pipeline- git push  puppetmanifests- Packer/AMINator- puppet  apply- Test  image- Register  AMI,  meta-­‐data  in  CMDB

Let’s  build  

[X]  OS  [X]  Patches[X]  OS  Configuration[X]  VM  Templates  (Amazon  Machine  Image)[  ]  VPC[  ]  Networking  (Subnets,  ELBs)[  ]  Network  Security  (NACLs  and  SGs)[  ]  Software[  ]  Application  (Code)[  ]  Services

Cloudformation

Let’s  build  

[X]  OS  [X]  Patches[X]  OS  Configuration[X]  VM  Templates  (Amazon  Machine  Image)[X]  VPC[X]  Networking  (Subnets,  ELBs)[X]  Network  Security  (NACLs  and  SGs)[  ]  Software[  ]  Application  (Code)[  ]  Services

Build  Puppet  Stack

- EC2  Metadata  service$  curl  http://169.254.169.254/latest/user-­datahiera_role=PuppetMasteraws s3  cp s3://puppetmanifests /etc/puppet/modules

- Self  awareness  by  hiera magic- Self  manifestation  with  puppet  apply

Build  Software  Stack

- CloudFormation- RDS- Elasticache- S3- CloudFront- Route  53- ELB- SNS- Autoscaling groups

- PuppetMaster- Build  ELK  Stack  (PuppetForge)- Build  Glassfish  layer  (Custom  types  and  providers)- Build  Vertx layer  (Custom  types  and  providers)- Build  HAProxy layer  (PuppetForge)

Let’s  build  

[X]  OS  [X]  Patches[X]  OS  Configuration[X]  VM  Templates  (Amazon  Machine  Image)[X]  VPC[X]  Networking  (Subnets,  ELBs)[X]  Network  Security  (NACLs  and  SGs)[X]  Software[  ]  Application  (Code)[  ]  Services

Application  Deployment

define  glassfish::app  ($user   =  'glassfish',$action =  'deploy',$bindaddr =  '127.0.0.1',$mgmtport =  '8080',$deploy_source =  '${glassfish::real_instance_basedir/deployments/${name}.war',)  {require glassfishif ($action ==  'deploy')  {app {  "${name}":ensure   =>  present,mgmtport =>  $mgmtport,user   =>  $user,source   =>  $deploy_source,

}}

}

Let’s  build  

[X]  OS  [X]  Patches[X]  OS  Configuration[X]  VM  Templates  (Amazon  Machine  Image)[X]  VPC[X]  Networking  (Subnets,  ELBs)[X]  Network  Security  (NACLs  and  SGs)[X]  Software[X]  Application  (Code)[X]  Services

example.com – Online  auction

AWS Sydney Region

Public subnet App subnetHAProxy Vertx

Glassfish

Data subnet

Puppet  Stack

ELKStackMgmt subnetCloudFront S3 SES SNS

Route 53

Jenkins

Let’s  run

[  ]  Tune[  ]  Release[ ]  Orchestrate  and  Automate

Decision,  Decisions…

- In  place  deployments  (Puppet  run)- MCollective- CodeDeploy

- Immutable  deployments  - Blue-­‐Green  deployment- AMI  baking

AWS  CodeDeploy

Bake  an  AMI Puppet  +

Sweet  spot  

To  AMI  or  not  to  AMI

Start  an  instance

Configure  the  instance

Create  an  AMI  from  your  instance

Start  new  ones  from  the  AMI

Launch  an  instance

Use  metadata  service  to  get  hiera roles

Puppet  run  performs  post  launch  config

CI  Pipeline  

• Vagrant• Jenkins

– Minor  release• CodeDeploy release  to  environment• Packer  build  Golden  AMI*  -­‐>  Update  DynamoDB -­‐>  Schedule  ASG  refresh

– Major  release• Packer  build  Golden  AMI*  -­‐>  Update  DynamoDB -­‐>  Refresh  ASG

*  Golden  AMI  – Glassfish  +  HAProxy +  Vertx +  Applications  with  all  services  turned  off

Thank  You