Hiroshi Miura - events.static.linuxfound.org · Merit of OpenStack Swift 6 merit ・Bundle Low cost server and HDD to single virtual storage ・Open Architecture ・Freedom to change

Copyright © 2015 NTT DATA Corporation

○○○○年○○月○○日株式会社ＮＴＴデータ ○○ ○○ ○○ ○○

Scalable Private Cloud Storage with Full OSS Stack

Hiroshi Miura System Platform Sector, NTT DATA Corporation

CloudOpen Japan

5, June 2015

2 Copyright © 2015 NTT DATA Corporation

Disclaimer

• Any product name, service name, software name and other marks are trade mark

or registered mark of corresponding companies.

• This presentation is in a purpose of providing the knowledge gained from our

first and biggest swift project.

• A presenter and NTT Data Corporation provide information in as-is basis and

have no responsiveness for results that you got according to information in this

presentation material.


Agenda

1. Back ground and issues

2.Categorize “cloud storage”

3. Building and utilizing private cloud storage

1. OpenStack Swift (, Ceph) and sheepdog

2. ownCloud, seafile


Top 5 Myth of Cloud Storage


Myth 1

Cloud Storage is less secure than local storage.

It has a risk of information leak.


Background： Cloud Storage is convenient

- free price for some volume

- many consumer utilize cloud storage service

- many student, professors utilize in college campus

- many office worker collaborate with outside of company

etc… Several type of

Cloud storage

smart phones and tablet

Free wi-fi, mobile connetions Broaden users not only home also

campus or office

http://www.freeversions.ru/dropbox-free-download


Background： but Cloud Storage is less secure

Leaking Risk

- Storing sensitive

data out of

campus.

- Out of control

for external

service

- Leakage of

information

Lost risk

Continuous risk

shift asset to cost

Data size limitations Performance limitations


Myth 2

Impossible to manage growing data such as IoT, smart city and smart factory in private system.


2004 2007 2010 2013 2016

Amount of Unstructured Data

Growing Data

• Data Explosion on Enterprise – Amount of Unstructured Data has been growing.

• We need storage with Scalability, Durability, Availability.

Examples of Unstructured Data

• Media (Images, Videos, Audios)

• Web Contents

• Documents

• Backups/Archives

• IoT/M2M

Growing exponentially

EB or PB scale


Myth 3

Private storage can archive higher performance, but more expensive than public cloud service.


Myth 4

Tire-3 storage (SATA, NL-SAS) is cheaper, higher capacity than enterprise grade storage.

It is less quality and just a consumer grade.


Server for cloud providers

• There are IA server for storage system High capacity

Long term support

Fit for cloud storage architecture

• Main target is Cloud service providers Who can maintain by itself

Use a big volume

• Cloud storage software help us utilize such a hardware


Cloud Storage Architecture

• Virtualized infrastructure

• Made up of many distributed resources - federated storage clouds

• Fault tolerant through redundancy and distribution of data

• Durable through the creation of versioned copies

• Typically eventually consistent with regard to data replicas

https://en.wikipedia.org/wiki/Cloud_storage


You should know about Cloud Storage

• Cloud Storage Infrastructure OpenStack Swift

Ceph

Sheepdog

• Cloud Storage Service software ownCloud

SeaFile/SeaHub


OpenStack Swift


What is Swift?

• Swift is...

• A storage system with Scalability, Durability, Availability.

• The REST-ful Distributed Object Storage likely Amazon S3.

• One of OpenStack Core Components.

• Implemented by Python.

• A Open Source Software.

① Block Storage (Cinder)

② Object Storage (Swift)


3 Key features of Swift

1

2

3

Durability

Scalability

Openness


Durability

• Protect data from various defeats

Datacenter

Rack

Node

Disk


Scalability

• Flexibly adopt to the growth of data

1y 2y 5y

10TB

100TB

3PB?

150TB?


Openness

• Free from limited model and maintenance period of hardware

A B C

Vendor A B C


3 key features of Swift

Durability Scalability Openness

Datacenter

Software

Hardware

Disk

1y 2y 5y

10TB

100TB

3PB?

150TB?

A B C

Vendor A B C

Protect data from various defeats

Flexibly adopt to the grow of data

No vendor lock-in


Merit of OpenStack Swift

6

merit

・Bundle Low cost server and HDD to single virtual storage

・Open Architecture

・Freedom to change

・developed by 370 organization in 139 countries.

・Small start, extend on demand

・automated sync data bet/ Multiple data centers

・Long term storage

Cost effective

durability

Disaster respose

Scalability

Stability

Freedom

※2014年6月現在


Swift as a storage for a variety of applications

Swift

System Backup

REST API

CMS

Cyber

Duck

FTP-like use Digital

Distribution Web Apps …

♫


Architecture: Nodes

• 2-type of Nodes: Proxy Node and Storage Node.

Storage Node Storage Node Storage Node Storage Node

…

Proxy Node Proxy Node …

HTTP Load balancer

Forward Data to node

Store data

Application

Proxy Node


HTTP Load balancer

Architecture: The Ring

• The Ring (static table for data allocation on storage node) decide the optimal Storage Node by Name.

Storage Node Storage Node Storage Node Storage Node

…

Proxy Node Proxy Node Proxy Node … Ring Ring Ring

Application

Ring Ring Ring Ring


Scalability

Proxy

Storage Storage Storage

Proxy (expand)

Proxy


Proxy

Storage (Expand)

（１） Expand proxy server

“Throughput” （２）Expand Storage servers or

disks “volume”

More Throughput

More Volume


Replicator

Node 1 Node 2 Node 3 Node 4

Node 2 Node 3 Node 4

(1) Each nodes checks data in others


(5) Recover disk

(6) recover data to original node

(4) Copy data to another node

Normal

Defeat

Recovery

Node 1 Node 2 Node 3 Node 4 (2) Disk defeat

(3) Detect disk trouble

Node 1


Replicator Disk

(7)Delete temporal data


Our use case


Use case: project overview

• Migration from existing high-end storage to distributed storage

• Backend Storage for application to store end user’s data

End Users High-end storage

【Swift】

Application server

・・・

Distributed Storage

Data migration


3 Challenges in the project

1 Durability assurance

2 Geographically distributed cluster

3 Delivery and quality


Challenge 1: Durability assurance

• Quality requirement in Japan

• Our customer is SUPER quality crazy.

• Everything should be under control

• System design for normal situation

• System design for defeat situation

– Even on distributed system

• Analyze every behavior before building system


Recovery test in variety of defeat pattern

• Variety of failure pattern

• (1) The point of failure • Disk, NIC, Process, Node, …

• (2) The number of failures • 1, 2, 3, 4, …

• (3) The range of failures • 1 node, multiple nodes/zones/regions, …

100s of test cases!!

Case #201

Proxy

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Zone1 Zone2

…

Region 1

Case #201

Proxy

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Zone1 Zone2

…

Region 1

Case #001

Proxy


Case #001

Proxy


Case #001

Proxy


Case #101

Proxy


Case #301

Proxy


Case #501

Proxy

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Sto

rage

Zone1 Zone2

…

Region 1


Site 1

Site 3 Site 2

Challenge 2: Geographically distributed cluster

• Geographically distributed swift cluster to realize disaster recovery

• Important points to evaluate global distribution

1. Client request

2. Durability

Proxy

Storage Storage Storage Proxy (Secondary)

300km~ 300km~

300km~

private network

×3


Pseudo-global cluster benchmark

• Pseudo-global cluster with simulated network latency

• Proxy and 3 Storage regions placed in different locations

• 10~200msec latency between locations simulated by tc

Proxy

Storage region 1

Storage region 2

Storage region 3

10~200msec latency

10~200msec latency

10~200msec latency

10~200msec latency

10~200msec latency

10~200msec latency

Client Proxy

Storage region1

TLmsec

TLmsec


Automated building

configuration server

swift proxy servers

swift storage servers

…

… OS

OS

Swift

Swift

puppet Kickstart

OS

Swift

OS

Swift

install &

configure

• Install and configure all software

• About 10 software and 50 configurations

• Kickstart and puppet


Result of automated building

Speed up of building

• 200 nodes/day, about 100 times as fast as manual building

Elimination of human error in building process

• 100% automated


Automated testing

storage servers …

…

Tempest

proxy servers

checking tool

Test all nodes

• Testing tools for

1. APIs : All swift APIs, including error case

2. Nodes : All swift nodes

• Extended Tempest and checking tool

Test all APIs


Result of automated testing

Quality assurance

• 100% API coverage, 100% node coverage

Speed up of testing

• 1 swift cluster(70 swift nodes) / 1 hour


Detailed system constitution

• 3 primary swift clusters over 3 geographical sites

• 3 backup swift cluster gathered in site 2

• 12PB total capacity: 1PB*3 for primary clusters, 1PB*1 for backup clusters

Site 2 Site 3

…

Primary Cluster1 r1 Storage(1PB)

…

Primary Cluster1 Secondary Proxy

…

Backup Cluster1 Storage(1PB)

…

Backup Cluster1 Proxy

…


…


…


…


…


…


…


…


…


…


…


…


…


…


Site 1

Monitoring (Act/Sby)

Visualizing (Act/Sby)

Configuration (Act/Sby)

…

Primary Cluster1 Primary Proxy

…


…









Sheepdog


• Replicated shared storage for VM

• snapshot, clone, and rollback

• Massive scale-out : 1000+ nodes

• Qemu/libvirt and iSCSI

Sheepdog – distributed object storage

Cut out

Consolidate

Storage pool

Sheepdog

provide

・IaaS back end storage

・Virtual integrated platform

IA servers

Internal disks

Virtual disk image


•Heavily developed by

NTT Lab and Alibaba/Taobao team

•Version 1.0 will soon be released

Sheepdog – developed heavily in east Asia

Yesterday session: “Sheepdog is ready!: Distributed Block Storage is

turning from experiment to commercial use” By Teruaki Ishizaki, NTT


• Manange data with building block of 4 MB chunk

• Distributed replica data to multiple servers as ordinal file – Realize redundancy

– Zero data use no actual space ⇒ Thin provisioning

Feature 1 High reliability

Fault tolerant through redundancy and distribution of data

Sheepdog cluster

VDI(Virtual Disk Image) - 1 GB 0 MB 4 MB 8 MB 16 MB 1024 MB

0 1 2

Create chunk of 4MByte data blocks from virtual disk

Distribution of files

3

0 1 0 1 0 3 3 1 3


• Node addition

– Accept a new node to a cluster automatically, when Sheepdog node connected

– Automated rebalance of distributed data

• Node removal

– Kick out a broken node from a cluster automatically, and continue working by rest of servers in cluster

– Made replica to another node of the data which has stored in broken node.

Feature2: Easy management

Sheepdog cluster

Automated Rebalance ex)node addition

No manual operation for cluster management


• FIO benchmark comparison among Sheepdog/Ceph/GlusterFS(LCJ 2014) – Test condition

– 12 node clusters

– Avr throughput from 4node 12VM

• configurations – Sheepdog：Erasure coding and 3 replica

– Ceph：3 replica

– GlusterFS：3 replica ＋4 stripe

Feature ３．High performance

Archive a good performance in some report

0.0

100.0

200.0

300.0

400.0

4MB 1MB 512kB 4kB

MB/s BandWidth (rand-read)

0.0

10.0

20.0

30.0

4MB 1MB 512kB 4kB

MB/s BandWidth (rand-write)

Sheepdog(EC)

Sheepdog(rep)

Ceph

GlusterFS


Sheepdog architecture

zookeeper/corosync

Linux

sheep

ストレージプール

VDI

IA Servers

Internal HDD

Softwares

Interfaces (qemu/iSCSI/swift)

Client or VM

• Sheepdog clusters • Manage raw HDD and connection interfaces

• Cluster Managers • ZooKeeper or CoroSync

• Client interfaces • Qemu

• NBD

• Swift

• iSCSI


pattern ①Block device for KVM/libvirt ②iSCSI Storage

Use case •OpenStack Iaas service •OpenStack Cinder and Horizon integration

•iSCSi storage for VMwareESX •Connection from Windows/Linux OS

•Backend storage for VirtualBox

architecture

4-type of use case

Linux (KVM Hypervisor)

sheep

qemu-dm

VM(Linux OS)

corosync/zookeeper

VDI

VM recognize it as block device

Linux

sheep

tgt

corosync/zookeeper

仮想ディスク

backing store of iSCSI target

target

Sheepdog driver


構成例 ③ObjectStorage ④NFSストレージとして利用

User case •Unified Storage with Object storage interface and block interface

•Shared file server

システム概略

ObjectStorage head

4-type of use case

Linux

sheep

corosync/zookeeper

VDI

OB Container mapped to VDI

REST IF

Sheepdog driver

One object mapped to sheepdog object

NFS head

Linux

sheep

corosync/zookeeper

VDI

NFS IF

Sheepdog driver

VFS communicate with Sheep


Distribution algorism

Consistent hash

0

25

50

75

100

125

150

175

NAME: B ID : 81

NAME: C ID : 133

NAME: A ID : 30

Managed with serverA

A

B

C


• Less rebalance when changing node count – Reduce performance damage when node brokage.l – Use Consistent Hashing

Strong point of Consistent Hashing


snapshot/clone/rollback

fe00

VID: fe

fe01 fe02 fe03

fe02

仮想ディスク

オブジェクトのファイルを示す

inodeオブジェクト

• Low disk consume when snapshot and clone creation

• Low overhead when snapshot, clone and rollback

fe01

fe00

fe03

snapshot取得後

fe00

VID: ff

fe01 fe02 fe03

仮想ディスク

fe00

VID: fe

fe01 fe02 fe03

snapshot

VID新規払出し inode情報はコピー

inodeオブジェクトを引き継ぎ


OwnCloud


ownCloud: Your own Google Drive/Docs

ownCloud owncloud.org


ownCloud: Your own Google Drive/Docs

Building your own Cloud Productivity Application

What is ‘ownCloud’? • File hosting application

• file sync among office, home and mobile devices

• Web Productivity application • edit note, manage calendar, share notes

• Support Standards: WebDAV, html5 – Good for modern browsers • Support mobile devices: iPhone, iPad and Android

・ Open Source: AGPLv3+GPL3


ownCloud: Many Features

Category features Note

File management

File Sync Sync files among many type of clients

File share and notification User can send and share file securely.

Versioning Versioning files

ODF/PDF Viewer View ODF/PDF on browser

Task Manage ToDo List and sync with mobile devices

Calendar Share calendar with users and groups

Address book Address book on browser

Gallery Mange your photos

Music player Manage your music files and enjoy everywhere

Video streaming Enjoy your video on browser

Security

LDAP/Active Directory Authenticate with LDAP、ActiveDirectory

Encryption Encrypt files stored on public cloud

Antivirus Check files

Extension Application Store Application store for 3rd party extensions

External storage Utilize AmazonS3, DropBox, Swift, GoogleDocs etc.


Seafile CC-BY

Dan Century


OpenSource Cloud Storage for teams and organizations

Easy sharing files in group


Seahub – Web-UI for seafile

Django based Web UI for Seafile


Seafile architecture

• GPLv2 Free Software

• https://github.com/haiwen/seafile


•Start from 2009

•Open Sourced in 2012

•Young Chinese entrepreneur and

hacker start the project and service

•Internationalized and support many

languages.

–Japanese translation added in 2015.

History


Combination of cloud storage software


ownCloud + Sheepdog

zookeeper/corosync Linux

sheep

ストレージプール

VDI

IA Servers

Internal HDD

Interfaces Qemu

Run ownCloud on VM Add user data disk on demand

• Combination of ownCloud and Sheepdog

– Scalable storage

– Productivity software

Scalability issue on RDBMS

backend such as MySQL,

PostgreSQL


Seafile and OpenStack Swift

Swift storage servers …

… Swift proxy servers

Browsers PC Clients Smartphones

Tablets

…

• File Syncing between PCs，Tablets and smartphones with Seafile.

• File manipulations, versioning management on Webpage.

• Scalable data architecture using Swift.

• Scalable head architecture by Seafile processing model.

Scale out Against for users and accesses

in every layers.

Seahub Sync

Sync


•Production ready SDS are ready.

•Cloud service software also is

emerged.

•Combination SDS with private

Cloud App provide good usability.

Conslution

