PDXPortland - Dockerize Django

  • View

  • Download

Embed Size (px)


  • Dockerizing Django

    PDX PortlandOctober 27th, 2016

  • Who are we?

    Michael Dougherty @maackle

    Senior Front-end Engineer CrowdStreet, Inc.

    Hannes Hapke@hanneshapke

    Software Engineer Talentpair, Inc.

  • Our pre-Docker World Single instance world

    (e.g. celery ran on the web server)

    Outdated Amazon machine image

    No documentation about the setup, consultancy work

    Live data monkey patching

    Scaling/Recovery time > 8 hours

    Clunky QA setup > bottleneck

  • Our post-Docker World Single instance world

    (e.g. celery ran on the web server)

    Outdated Amazon machine image

    No documentation about the setup, consultancy work

    Live data monkey patching

    Scaling/Recovery time > 8 hours

    Clunky QA setup > bottleneck

    One service per container, redundancy of instances

    One common base image shared across all instances

    Explicit, declarative server setup

    Immutable infrastructure (mostly)

    Scaling/Recovery time ~ 20min

    As many QA instances as we want

  • What is Docker?

    Docker Compose Machine Swarm

  • Docker containers wrap a piece of software in a complete filesystem that

    contains everything needed to run: code, runtime, system tools, system libraries anything that can be installed on a

    server. This guarantees that the software will always run the same, regardless of its environment. *

    Basically a virtual env for your operating system.

    * from https://www.docker.com/what-docker


  • Docker vs. Vagrant

  • Where is the difference?

    Images from https://www.docker.com/what-docker

    VM includes OS No OS needed


  • How does Docker work?

  • Create a Dockerfile

    Build the Docker image and push it to the docker registry

    Plain Docker

    FROM ubuntu:16.04

    RUN apt-get update && apt-get upgrade -y

    RUN pip install Django

    COPY requirements.txt .

    WORKDIR /dev

    $ docker build -t your_project/your-whale . $ docker images $ docker tag {image_hash} your_project/your-whale:latest$ docker push

  • Plain Docker Run a shell in a docker container -i start an interactive container -t creates Pseudo interface with stdin and stdout

    Run the Django server in a container -d run container in detached mode -P maps all ports to the host machine

    $ docker run -i -t ubuntu /bin/bash

    $ docker run -d -P my-container python manage.py run server

  • What if we need multiple services?

    Docker Compose Machine Swarm

  • Docker Compose

    Compose is a tool for orchestrating the building, running, and intercommunication of

    multi-container Docker applications.

  • How does it work?

    1) Define a Dockerfile for every service 2) Define a Docker compose description of the environment

    3) Use docker-compose build/up to start all services

    version: '2'

    services: web: build: . ports: - "5000:5000" volumes: - .:/code redis: image: redis

  • How can I easily provision a server with

    the containers?Docker Compose Machine Swarm

  • Docker Machine

    is a great tool which creates Docker hosts anywhere.Yes, anywhere.

    Locally, AWS EC2, Digital Ocean, MS Azure, you name it.

    No Ansible, Puppet, Chef, fabric, etc. required.

  • What if I need multiple instances with multiple


    Docker Compose Machine Swarm

  • Docker Swarm

  • Dockerize for real

  • If you start from scratch Docker documentation includes a great Django


    Too much work? The Django Cookie Cutter template includes a great Docker setup

    Other projects:

    django-docker on github

  • If you convert a project like us

  • Reorganize your folder structure

  • Normalize folders Create folders for every service

    docker-compose-{env}.yml go into the project root

    Dockerfiles go into every service folder

    startup.sh scripts go into the service folders

    Keep your local folder structure similar to the folder structure within the container(s) - for sanity

  • Reorganized foldersProject Root|-apps|-settings|-static|-templates|-manage.py|-fabfile.py|-urls.py\ requirements.txt

    Project Root|-django| |-apps| |- | |-Dockerfile| |-startup-django.sh | -manage.py|-nginx|-webpack|-docker-compose.yml |-urls.py\ requirements.txt

  • Build a base image

  • Base Image Create one (or more) base Dockerfile(s) with all

    common packages

    Service containers can use this base image - this will increase build speed

    If you store the base image(s) in a separate git repo, the docker registry will build them automatically for you

  • Base ImageFROM ubuntu:16.04

    RUN apt-get update && apt-get upgrade -y

    RUN apt-get install -y vim # Install some useful editor

    RUN apt-get install -y build-essential git software-properties-common

    RUN apt-get install -y python python-dev \\ python-setuptools build-essential

    RUN apt-get install -y nodejs npm RUN npm install -g n # upgrading the npm version RUN n stable


  • Base Image

    Add image of the Docker registry

  • Set up Docker compose for the different environments

  • Docker Compose

    For every environment, local, QA, staging, production, define a docker-compose-{env}.yml file

    The files describe the environment stack

    Each service within the docker-compose file can have its own Dockerfile

  • Docker Composeversion: '2'

    volumes: postgres_data_dev: {} redisdata: {} webpack_data: {}

    services: postgres: image: postgres:9.5 volumes: - postgres_data_dev:/var/lib/postgresql/data restart: always environment: - POSTGRES_USER=postgres_user - POSTGRES_DB=my_fancy_db - POSTGRES_PASSWORD=

    webpack: image: crowdstreet/crowdstreet-whale:latest command: npm run watch environment: - NODE_PATH=/node_modules volumes: - ./webpack/frontend-src:/frontend-src - ./django:/crowdstreet-src - webpack_data:/webpack_data/ ports: - "3000:3000" restart: always

  • Docker Compose django: build: context: . dockerfile: ./django/Dockerfile-dev command: python /crowdstreet-src/manage.py runserver --settings=settings.dev depends_on: - postgres environment: - ENV=dev - DJANGO_SETTINGS_MODULE=settings volumes: - ./django:/crowdstreet-src - ./webpack/frontend-src:/frontend-src - webpack_data:/webpack_data/ ports: - "8000:8000" - "80:8000" links: - postgres - redis - webpack - memcached

    redis: restart: always image: redis:latest volumes: - redisdata:/data restart: always

  • Docker Compose Build your service stack with

    Start the container stack with

    Access a single container with

    $ docker-compose -f docker-compose-{env}.yml build

    $ docker-compose -f docker-compose-{env}.yml up

    $ docker-compose -f docker-compose-{env}.yml run django bash

    $ docker-compose -f docker-compose-{env}.yml run container name command

  • Set up Docker machine and deploy to the world

  • Docker machine is awesome!

  • Docker Machine Withwill provision you an AWS instance

    Activate the instance with

    Afterwards, any docker-compose command will be executed on the active machine

    Easy to start/stop/terminate machines

    $ docker-machine create --driver amazonec2 --amazonec2-region [e.g. us-west-2] --amazonec2-vpc-id [YOUR_VPC_ID vpc-xxxxxx] --amazonec2-instance-type [e.g. t2.small] [INSTANCE_NAME]

    $ docker-machine env [INSTANCE_NAME]

  • Lessons Learned

  • Or... how to cowboy code with Docker

    Sometimes you just need to manually change something

    Docker provides ways to get a shell inside a running instance and copy files back and forth

    Your changes will of course be lost next time you spin up a new container

  • The Disciplined Way:

    The Cowboy Way:

    $ docker-compose run django bash

    $ docker exec -it {container_id} bash

  • How does QA work with Docker?

    No QA bottleneck anymore

    No database gridlock anymore

    Each feature branch gets its own instance

    Once feature is tested, instance gets terminated

  • How can I access the manage.py shell/migrate?

    Access the bash of the django container with

    Continue as usual withSome for migrations, make_migrations, etc.

    Or run it from outside of the container stack with

    $ docker-compose -f docker-compose-{env}.yml run django bash

    # ./manage.py shell

    docker-compose -f run django python manage.py migrate

  • Help, ipdb doesnt work anymore

    Start the Django container with the service ports enabled

    If no command is specified, then Docker will default to the command in the docker-compose.yml file

    $ docker-compose -f dev.yml run --service-ports django

  • How to run tests?

    Start the Django container with your test command$ docker-compose -f run django manage.py test

  • CI Testing is convenient Setup for Circle CI

    machine: pre: - curl -sSL