* as CodeÁrni Birgisson

Infrastructure Engineer @arnibirgisson

What is QuizUp?

What is QuizUp?

• The biggest trivia game in the world!

• Record setting 1M in first week

• 1000+ curated topics in 6 languages

• MyQuizUp topics

• Millions of questions

QuizUp

Infrastructure

• QuizUp-Admin

• Codebase python

• Created AMI’s along

with ASG, ELB AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]{ASG

ELB

Datastores

• Multiple services

• AMI + ASG per service

• ELB for service-to-

serviceAMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]{ASG

ELB

AMI #2

[Java App]

ELB

AMI #2

[Java App]AMI #2

[Java App]

Datastores

• Increasing number of services

• SOA / Micro Service architecture

• Routers (nginx)

• ZooKeeper

• ZooRunnerAMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI#1

[Python App]

ASG

Routers

AMI #2

[Java App]AMI #2

[Java App]AMI #2

[Java App]

Datastores

RoutersRoutersRouters

AMI #2

[Java App]{ AMI #2

[Java App]AMI #3

[Java App]

• Docker

• Python, Node, Go, Java, Scala,

C++

• More generic AMIs

AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Docker App]

ASG

Routers

AMI #2

[Java App]AMI #2

[Java App]AMI #2

[Docker App]

Datastores

RoutersRoutersRouters

AMI #2

[Java App]{ AMI #2

[Java App]AMI #3

[Docker App]

• ~50 different services

• Mostly JVM

• All services run in docker

• Single AMI

• Datastores too*AMI

[Python App]AMI

[Python App]AMI

[Python App]AMI

[Python App]

Single AMI

[App]

[Dependancies]

ASG

Routers

AMI #2

[Java App]AMI #2

[Java App]

Single AMI

[App]

[Dependancies]

Datastores

RoutersRoutersRouters

AMI #2

[Java App]

{

AMI #2

[Java App]

Single AMI

[App]

[Dependancies]

… x 50

Why IaC?

Why IaC?• EC2 Classic - VPC

• QuizUp-Admin outdated

• Increasing complexity

• Iteration speed

• Clean setup, not a VPC version of the old

stuff.

Experiments

Experiments

• CloudFormation, Troposphere, Terraform

• VPC with one service

Experiments• VPC

• Subnets

• Routing Table

• Auto-Scale Group

• Launch Configuration

• Security Group (+ rules)

• Support for variables

Experiments

• Decided to use Terraform

• Plan / Apply (preview changes)

• Why is that important?

Infrastructure as Code

What’s Included

• Repository

• Terraform

• CI

• Module for a Service

• Modules for DynamoDB, ELBs, etc.

deploy

staging

iterate

Test

deploy

production

PR/Revie

w

merge

CI JOB checks syntax and

outputs plan

Branch with

proposed change

Iteration

Single AMI

• Can become any service

• Packer + Puppet

• QuizUp Docker Manager

• Logging & Metrics

Single AMI

Quizup Docker Manager

statsd log shipper

Instance metadata ZooKeeper

application dependancy dependancy

Docker registry

Configuration as Code

Configuration

• Servers fetch config from ZooKeeper

• ZooKeeper manually updated

• Can we use a similar flow?

Yes!

• In house git repository

• git/filesystem to ZooKeeper sync tool

• Jenkins CI

• Configuration through same workflow

Branches, PR

What have we gained?

What have we gained?• Production infrastructure changes worry-

free non-events

• Code = Documentation

• Peer reviews, PRs, Comments

• Disaster recovery

• History (+ reverting!)

What have we gained?

• Anyone can propose infrastructure changes

• Faster and more consistent changes

• Staging is a small copy of production

What have we gained?

Infrastructure as Code

helps maintain a devops culture

by using the same workflow

to modify applications, infrastructure and

configuration

What’s not great?

State

• State file must be kept safe!

• Hard to “inject” resources into state

• (Currently) no way to create Terraform

description of current resources

Speed

• Must choose between:

• Apply everything always (slower)

Infrastructure = Code

• Apply in parts (speedy)

Requires plumbing. Can diverge.

No sidelining

• No manual changes

• They are reset during next apply

• Pro or Con ?

Thank You