Cloud Reliability Patterns

Gustavo Barrancos (@gbarrancos) Rafael Ferreira (@rafaeldff)

Cloud Reliability Patterns @ Nubank

“Nu Minimal Keynote Template” © Nu Bank - 05.01.2014 2

Complex systems usually operate in failure mode

John Gall

Growth

3

Microservices

4

5

Asynchronous Messaging

Asynchronous Messaging

6

• Decoupling • Pub/Sub • Fault Isolation • Load Distribution • Operability

Asynchronous Messaging

7

• Concerns • Complexity • Harder to test • Single Point of Failure

Asynchronous Messaging

8

Apache Kafka

9

Blue - Green Deployments

Blue-Green Deployments

10

“A release technique that reduces downtime and risk by running two identical production environments: Blue and Green”

https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html

Blue-Green Deployments - AWS Autoscaling Groups

11

Elastic Load Balancer

Blue-Green Deployments - AWS Elastic Network Interface

12

13

Pervasive Monitoring

Pervasive monitoring - Systems Health Metrics

“Nu Minimal Keynote Template” © Nu Bank - 05.01.2014 14

15

Riemannhttp://www.riemann.io

Pervasive monitoring - Riemann

16

17

Pervasive Monitoring - Business Metrics

Pervasive Monitoring - Errors

18

19

Traceability

Traceability - log all I/O

“Nu Minimal Keynote Template” © Nu Bank - 05.01.2014 20

2016-03-28T17:50:23.991Z [ACQUISITION] INFO - {:cid "DEFAULT.6DTVD.PUQPE.EQ36K.4XQGO", :log :out-request, :method :get, :uri “https://prod-customers.nubank.com.br/api/customers/21ede21-1a6b-44dc-b731-f58279edd421"}

2016-03-28T17:50:23.993Z [CUSTOMERS] INFO - {:cid "DEFAULT.6DTVD.PUQPE.EQ36K.4XQGO.WHBVH", :log :in-request, :method :get, :path "/api/customers/:id"}

Traceability - Correlation Id

“Nu Minimal Keynote Template” © Nu Bank - 05.01.2014 21

22

Healthchecks

Healthchecks

23

Elastic Load Balancer

Healthchecks - Heartbeats

24

Elastic Load Balancer

25

Deadletters

Deadletters

26

Elastic Load Balancer

27

Circuit Breakers

Circuit Breakers

28

Elastic Load Balancer

29

Questions?

(we are hiring! http://nubank.workable.com)