Kubernetes @ Nanit

Kubernetes @ nanitChen Fisher, VP Engineering

nanitSmart baby monitor

API

Clipper

Focus

Gateway

Grant

Mailman

RTMPRabbitMQ

S3

Stampa

State Department

Timeline

Taper

Algorithms

Communication

MicroservicesDocker

docker-compose up -d

Orchestration

Orchestration

- Service Discovery

- High Availability

- Resource Management

- Port Management

API

Clipper

Focus

Gateway

Grant

Mailman

RTMPRabbitMQ

S3

Stampa

State Department

Timeline

Taper

Algorithms

ECS vs Kubernetes

https://railsadventures.wordpress.com/2015/12/06/why-we-chose-kubernetes-over-ecs/-- Erez Rabih

Service Discovery / ECS

- Offers no service discovery

- One can use external tools like Consul

(https://aws.amazon.com/blogs/compute/service-discovery-via-consul-with-amazon-ecs/)

Service Discovery / Kubernetes

- Has built-in service discovery (http://kubernetes.io/docs/user-guide/services/)

- One can simply refer to a service by its name

- Can group and isolate services by namespaces

High Availability / ECS

- Uses ELB health checks

- Limited to HTTP/TCP health checks

- If not using ELB then only recovers on container exit

- Must create ELB for every service to get health checks

High Availability / Kubernetes

- Offers exec health check in addition to http

- One can perform application level health check

Port Management / ECS

- Maps an ELB to a service which exposes a container port on host

- Cannot have more than one container with same exposed port on same host

- Must manually manage container ports

- Assigns random port for every container

- Maps ELB to the container’s random port

Port Management / Kubernetes

Kubernetes

- AWS

- Kube 1.2.2

- Two clusters: Staging & Production

- 20+ m4-large instances on production (pre-launch)

- 100+ pods

- RDS, DynamoDB outside of cluster

From Dev to Prod

Dev and test locallyclojure, ruby, go, c/c++, elixir

Push to Githubpull requests

Jenkins picks up and testsmarks commit on Github

Manual merge to master

Deployment via Slack

Deployment / directory structure

● nanit/

○ api/■ app/

■ Dockerfile■ code...

■ kube/■ dep.yml

■ Makefile

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: {{APP_NAME}}spec: replicas: {{REPLICAS}} template: metadata: labels: app: {{APP_NAME}} spec: volumes: - name: some-secret secret: secretName: some-secret containers: - name: {{APP_NAME}} image: {{IMAGE_NAME}}...

Deployment / kube yaml file

Deployment / Makefile

APP_NAME=apiIMAGE_TAG=$(shell git log -n 1 --pretty=format:%h app)IMAGE_NAME=nanit/$(APP_NAME):$(IMAGE_TAG)

define generate-dep sed -e 's/{{IMAGE_NAME}}/$(IMAGE_NAME)/g' kube/dep.ymlendef

docker: sudo docker pull $(IMAGE_NAME) || (sudo docker build -t $(IMAGE_NAME) app && sudo docker push $(IMAGE_NAME))

deploy: docker $(call generate-dep) | kubectl apply -f -

migrate: …

test:...

Deployment / config

from Makefile:REPLICAS?=$(shell curl -s config/$(NANIT_ENV)/api/replicas)

from kube yaml file:spec: replicas: {{REPLICAS}}

Post deployment(Monitoring & Alerts)

Post deployment (Monitoring & Alerting)

- Currently using built in Kibana & Grafana

- Alerting to Slack using proprietary service

Thank You!