MURORAN INSTITUTE OF TECHNOLOGY
Business Con*nuity with Docker
2015/06/04
Yoshitaka Kuwata Muroran Ins*tute of Technology
CloudOpen Japan 2015
Copyright(C) 2015 by Yoshitaka Kuwata
MURORAN INSTITUTE OF TECHNOLOGY
Overview of Talk 1. Who is Talking 2. Mo*va*on of Disaster Recovery 3. Exis*ng Solu*ons 4. Requirement of DR 5. Architecture for DR with Docker 6. Evalua*ons 7. Discussions 8. Conclusions
Copyright(C) 2015 by Yoshitaka Kuwata 2
MURORAN INSTITUTE OF TECHNOLOGY
Yoshitaka Kuwata Associate Professor Center for Mul*media Aided Educa*on Muroran Ins*tute of Technology, Hokkaido • Educa*on of Basic IT course • Design and opera*on of campus network & computer systems • Research on Cloud Architecture and OSS Project Analysis
1. Who am I
MURORAN: 1.5 hours from
Sapporo, One hour from Chitose
Airport.
Copyright(C) 2015 by Yoshitaka Kuwata 3
MURORAN INSTITUTE OF TECHNOLOGY
2. Mo*va*on of Disaster Recovery
Copyright(C) 2015 by Yoshitaka Kuwata 4
Awareness of Disaster
Preven*on
Business Con*nuity
Management
Design of Exis*ng Systems
Compliance
Lessons of Great
Earthquakes
Interna*onal Standard
・Func*onal Design ・Non Func*onal Design Capacity Extensibility Operability Maintainability Security Fault Tolerance Reliability Availability
・Analysis Iden*fy Cri*cal Factors ・Management System Business Impact Analysis Establish BCP Documenta*on ・Implementa*on Apply BCP Training ・Opera*on
Need to apply BCP to exis*ng systems.
The design should also be used for next systems.
MURORAN INSTITUTE OF TECHNOLOGY
3. Requirement form light-‐weight DR
Pre Condi*ons • Primary systems are located on local site. (not all of systems are cloud ready) • Cloud services are used only for backup data and backup system. • Make a set of daily backup copy of data to the clouds. • When the primary systems downs, backup systems are started manually. • When recovered, manually copy data from backup systems to primary
systems, then stop backup systems.
Copyright(C) 2015 by Yoshitaka Kuwata 5
Requirements (1) Data must be kept in safe in remote loca*ons (2) Keep the service running in remote site, when primary site is down
(3) Switch loca*on of service manually. (4) The architecture should be the same with current design.
MURORAN INSTITUTE OF TECHNOLOGY
4. Exis*ng Solu*ons in Cloud
No Name Method
1 Mul*-‐Datacenter In order to avoid failure of a datacenter, use mul*ple servers in different datacenters located in different regions.
2 Sorry Page When main site is down, redirect requests to backup servers; sorry server
3 Cross-‐Region Replica*on
Make a set of backup copy to different loca*on.
Copyright(C) 2015 by Yoshitaka Kuwata 6
AWS Cloud Design Padern(AWS-‐CDP) for Disaster Recovery
Reference: hdp://aws.clouddesignpadern.org/index.php/
MURORAN INSTITUTE OF TECHNOLOGY
4. Mul*-‐datacenter in AWS-‐CDP
7
Servers A
EC2 instances EC2 instances
dispatch Region A Region B
Copyright(C) 2015 by Yoshitaka Kuwata
Load balancer
Servers B
Reference: hdp://aws.clouddesignpadern.org/index.php/
MURORAN INSTITUTE OF TECHNOLOGY
4. Sorry page in AWS-‐CDP
8
Primary Servers
S3
EC2 instances Object Storage System
• Health check • Change DNS entry
Copyright(C) 2015 by Yoshitaka Kuwata
DNS Server(Route 53)
Backup Server
Reference: hdp://aws.clouddesignpadern.org/index.php/
MURORAN INSTITUTE OF TECHNOLOGY
4. Cross-‐Region Replica*on in AWS-‐CDP
9
Primary Server Secondary Server
EC2 instance EC2instance
DB DB Backup Restore
Remote Copy
Region A Region B
Copyright(C) 2015 by Yoshitaka Kuwata
Note: This padern is one of AWS-‐CDP candidates.
Reference: hdp://aws.clouddesignpadern.org/index.php/
MURORAN INSTITUTE OF TECHNOLOGY
5. DR with Docker Useful Features of Docker
(1) Light-weight Virtualization with LXC Partitioning of OS(2) Differential File System AUFS, LVM thin provisioning, Overlayfs. (3) Portable Image Docker Image (4) Repository Management Docker-HUB, Local Repository(5) Building Automation of Docker Image Dockerfile (6) Optimal Runtime Environment Minimum set of programs are deployed in images
Copyright(C) 2015 by Yoshitaka Kuwata 10
MURORAN INSTITUTE OF TECHNOLOGY
5. How we make use of Docker for DR
We focused on two aspects of Docker • Differential File System
– We can reduce the size of daily backup volume
• Portability of Images – We can build private repository on academic cloud service, and commercial service as well.
– We can run secondary site on cloud services.
• Key Technologies – Design paderns of backup systems – Implementa*on and use of private repository – Security
11 Copyright(C) 2015 by Yoshitaka Kuwata
MURORAN INSTITUTE OF TECHNOLOGY
Architecture for DR
LMS
Data
Storage System
Data
①Make Backup
LMS
② run alterna*ve server
Data Container (Docker)
LMS:Learning Management System
12
University
Academic Cloud AWS
Storage System
Data
LMS
Data
AWS can be used as alterna*ve of
Academic Cloud
Campus LAN
SINET
③Recovery
Copyright(C) 2015 by Yoshitaka Kuwata
MURORAN INSTITUTE OF TECHNOLOGY
5. Architecture for Disaster Recovery with Docker
13
Primary System
A
A
Docker Repository
Servers on site
Servers on Cloud Service
pull & run
push
Docker Engine
Docker Engine
B
B
Linux
App1
App2
A
B
Copyright(C) 2015 by Yoshitaka Kuwata
1. Build Private Docker Repository on Cloud Service • The Repository can be built with backend
2. Copy Backup data of Primary System to a Docker Repository on Cloud Services
Overview
Secondary system
MURORAN INSTITUTE OF TECHNOLOGY
5. Make a set of backup copy to repo.
14 Copyright(C) 2015 by Yoshitaka Kuwata
1. Push images to local repository on Cloud Services.
Daily Opera*ons Primary System
A
Docker Repository
Servers on site
Servers on Cloud Service
Docker Engine
B
Linux
App1
App2
A
B
MURORAN INSTITUTE OF TECHNOLOGY
5. Switching to Secondary system
15 Copyright(C) 2015 by Yoshitaka Kuwata
1. Launch secondary system from image in Docker Repository on Cloud Service
2. Redirect access to Secondary system
Switching to Secondary system Primary System
Secondary system
A
A
Docker Repository
Servers on site
Servers on Cloud Service
pull & run
Docker Engine
Docker Engine
B
B
Linux
App1
App2
A
B
OFFLINE
MURORAN INSTITUTE OF TECHNOLOGY
5. Recovery Opera*on
16 Copyright(C) 2015 by Yoshitaka Kuwata
1. Push data of Secondary System to Docker Repository 2. Stop Secondary System 3. Launch Primary system with data from Docker
Repository 4. Redirect access to Primary System
Primary System
A
A
Docker Repository
Servers on site
Servers on Cloud Service
push
Docker Engine
Docker Engine
B
B
Linux
App1
App2
A
B
Secondary system
MURORAN INSTITUTE OF TECHNOLOGY
6. Evalua*on Environment Item Specifications Application Moodle 2.7 with Language Pack (Japanese) Number of Course 0〜20 Number of Sessions 15 / course Repository docker-‐registry on Docker in a local machine CPU Intel Core2 Quad Q8400 2.66GHz Memory 4GB Disks SATA 128GB SSD NIC 1000Base-‐T (1Gbps) OS Ubuntu 14.04.1LTS Server Docker V1.2.0, build fa7b24f
Copyright(C) 2015 by Yoshitaka Kuwata 17
moodle
Docker Repository
Primary System
push
pull & run
pull & run
push Docker Engine
Secondary System
moodle
MURORAN INSTITUTE OF TECHNOLOGY
6. Seing of Moodle • Seing 15 Sessions for each classes
– Plase course material for each session • Allocate Numbers of classes and check the required
resources.
Copyright(C) 2015 by Yoshitaka Kuwata 18
MURORAN INSTITUTE OF TECHNOLOGY
6. Results : overview
• Image size increases with every entry of courses – Only the difference is stored in the images
• Time to push repositories depends on the image size
Copyright(C) 2015 by Yoshitaka Kuwata 19
Linux moodle Japanese locale Course 1 Course 2 Course N
≒1GB 32MB 32MB
MURORAN INSTITUTE OF TECHNOLOGY
6. Results: number of course VS image size
Copyright(C) 2015 by Yoshitaka Kuwata 20
Image Size
Time to put repo.
MURORAN INSTITUTE OF TECHNOLOGY
7. Discussions
Copyright(C) 2015 by Yoshitaka Kuwata 21
• Limita*on of snapshot on Docker – Docker can have only 127 level of snapshot
# docker build –t test . Sending build context to Docker daemon 4.096 kB Sending build context to Docker daemon Step 0 : FROM ubuntu -‐-‐-‐> 5ba9dab47459 Step 1 : RUN date -‐-‐-‐> Running in b4849b849f31 Tue Feb 17 09:59:37 UTC 2015 -‐-‐-‐> 36904b4e1c8d Removing intermediate container b4849b849f31 Step 2 : RUN date -‐-‐-‐> Running in ae5abb842961 Tue Feb 17 09:59:42 UTC 2015 -‐-‐-‐> 7c94c175a4b6 Removing intermediate container ae5abb842961 Step 3 : RUN date -‐-‐-‐> Running in f107b4d4ec6e Tue Feb 17 09:59:44 UTC 2015 -‐-‐-‐> 95dd7644aeca Removing intermediate container f107b4d4ec6e : : : Step 121 : RUN date INFO[0296] Cannot create container with more than 127 parents
This is not the limita*on of file system but the limita/on of Docker.
i.e. happens with AUFS, Overlay FS, and LVM
MURORAN INSTITUTE OF TECHNOLOGY
7. Discussions (cont.) • Work around for 127-‐parent limit (1) Merge layers with export and import
Copyright(C) 2015 by Yoshitaka Kuwata 22
# container_id=$(docker run -‐d test2 /bin/bash -‐c "") # docker export $container_id | docker import -‐ test2 34e82f0c70802fed06348fd805628ffc9d53374939b82fc2fb2cdcdec14f34de # docker images -‐-‐tree Warning: '-‐-‐tree' is deprecated, it will be removed soon. See usage. ├─34e82f0c7080 Virtual Size: 188.1 MB Tags: test2:latest └─511136ea3c5a Virtual Size: 0 B └─27d47432a69b Virtual Size: 192.5 MB └─5f92234dcf1e Virtual Size: 192.7 MB └─51a9c7c1f8bb Virtual Size: 192.7 MB └─5ba9dab47459 Virtual Size: 192.7 MB Tags: ubuntu:latest
MURORAN INSTITUTE OF TECHNOLOGY
7. Discussions (cont.) • Work around for 127-‐parent limit (2) Take a snapshot and con*nue the image
Copyright(C) 2015 by Yoshitaka Kuwata 23
Moodle base:V1
Image Instance
Moodle base:V1
Moodle base:V1.1
Moodle base:V1.1.1
run
Stop & commit
Moodle base:V1.1 run
Stop & commit
Moodle base:V1.1.1 run
Stop & commit Moodle base:V1.1.1.1
MURORAN INSTITUTE OF TECHNOLOGY
7. Discussions (cont.) • Work around for 127-‐parent limit (2) Take a snapshot and con*nue the image
Copyright(C) 2015 by Yoshitaka Kuwata 24
Moodle base:V1
Image Instance
Moodle base:V1
Moodle base:V1.1
Moodle base:V1.2
run
Stop & commit
Moodle base:V1.1
start
Stop & commit
Moodle base:V1.3
Moodle base:V1.2 Stop & commit
start
start
MURORAN INSTITUTE OF TECHNOLOGY
8. Conclusions
Copyright(C) 2015 by Yoshitaka Kuwata 25
• Light-‐weight DR architecture with Docker is proposed
• Prototype system is built and evaluated – Func*ons of the system are verified – Limita*on of Docker was found and work-‐around is proposed.
•