© 2016 NTT Software Innovation Center

Masakari と HA コミュニティの状況

2016/03/24室井 雅仁

2Copyright©2016 NTT corp. All Rights Reserved.

自己紹介

• 室井 雅仁 ( むろい まさひと )• 所属 : NTT• OpenStack を利用した OSS クラウドのアーキテクトを担当• 社内向け OpenStack 環境の運用、コミュニティへのアップス

トリーム• OpenStack Congress Core Reviewer

• https://wiki.openstack.org/wiki/Congress• https://thinkit.co.jp/article/8381

• IRC: masahito• エディタ : Emacs 派

3Copyright©2016 NTT corp. All Rights Reserved.

http://www.slideshare.net/masahito12/openstack-masakari

4Copyright©2016 NTT corp. All Rights Reserved.

本日のお品書き

• Masakari• OpenStack HA コミュニティの現状• Masakari デモ ( 時間があれば )

5Copyright©2016 NTT corp. All Rights Reserved.

Masakari

6Copyright©2016 NTT corp. All Rights Reserved.

What’s Masakari

Copyright © いらすとや. All Rights Reserved.

まさかり

7Copyright©2016 NTT corp. All Rights Reserved.

What’s Masakari in OpenStack

• Virtual Machine High Availability (VM-HA) service• Rescue Virtual Machine when any errors occur• Published as OSS at github https://github.com/ntt-sic/masakari

8Copyright©2016 NTT corp. All Rights Reserved.

Motivations

• Pets vs Cattle

• Unable to change all Apps to Cloud Native at once

• Open Source

9Copyright©2016 NTT corp. All Rights Reserved.

Requirements for Pets Model

• Detect 3 types of VM down• VM instance down• VM manager down• Host down

• Recover VM within 5 mins

• Work automatically

10Copyright©2016 NTT corp. All Rights Reserved.

Architecture OverviewC

ompu

te N

odes

Con

trolle

r Nod

es

& B

acke

nd N

odes

11Copyright©2016 NTT corp. All Rights Reserved.

How to detect the 3 down

• VM down• Monitoring libvert’s events

• Manager Process down • Monitoring manager process

• Host down• Monitoring Pacemaker status

12Copyright©2016 NTT corp. All Rights Reserved.

Detect VM Down

Libvirt

Masakari

1. Notify down VM’s Info(VM-ID, Host Name, etc.)

Libvirt Monitor

Detect VM downVM1 VM2 VM3

Libvirt

Libvirt Monitor

VM5 VM6

HostHost

Nova

2. Call Rebuild API for the down VM

3. Rebuild the VM

Down

0. Set callbacks for libvert’s events hook

13Copyright©2016 NTT corp. All Rights Reserved.

Manager Process Down

1. Restart manager process when it’s down

Process Monitor

Masakari

2. Notify manager process down if fail to restart few times

Libvirt Nova-compute

Host A

Libvirt Nova-compute

Host B

Nova

3. Notify Nova to disable schedule for Host A

Process Monitor

Down

14Copyright©2016 NTT corp. All Rights Reserved.

Host Down

RA

CIB

RA

RA

Node’s Status

pacemaker

Heartbeat communications or pacemaker-remote

Masakari

Check its Host’s status

1. Notify another host down

StartStopMonitor

WatchD

og & S

hutdowner

Host Fail Monitor

Polling RA

CIB

RA

RA

Node’s Status

pacemaker

StartStopMonitor

WatchD

og & S

hutdowner

Host Fail Monitor

Polling

Down

Host A Host B

Nova

2. Call Evacuate API for all VM on Host B

15Copyright©2016 NTT corp. All Rights Reserved.

How to use Masakari

1. Prerequisites• Set up Nova and Compute Nodes with KVM• Set up a shared storage per cluster for ephemeral disks (e.g. NFS)

2. Install and Configure Masakari• Download source from github

• https://github.com/ntt-sic/masakari• Install Masakari’s package• Initialize Masakari’s DB• Configure 4 Masakari’s config files

3. Start Masakari• Start all process• Add a reserved host prepared for host down

4. Wait any error• Masakari only works when any error occurs

16Copyright©2016 NTT corp. All Rights Reserved.

Run Masakari in local

• Run masakari-deploy!!http://github.com/ntt-sic/masakari-deploy • For quick QA or demo

17Copyright©2016 NTT corp. All Rights Reserved.

Challenges

• No branch from OpenStack master

18Copyright©2016 NTT corp. All Rights Reserved.

Masakari Next Release

• 1.1.0 is supposed be released until Austin Summit• Supports Pacemaker-Remote for hostmonitor• Supports PostgreSQL and other DB

• By using SQLAlchemy• Supports CentOS

• With python installer and rpm packaging• Uses python-client to call OpenStack API

19Copyright©2016 NTT corp. All Rights Reserved.

OpenStack HA コミュニティの現状

20Copyright©2016 NTT corp. All Rights Reserved.

OpenStack HA チーム

• openstack-ha チーム• VM HA のユースケース議論や今後の実装方針考えるチーム• IRC チャンネル : #openstack-ha• IRC ミーティング : 月曜 18:00 #openstack-meeting• Etherpad:

https://etherpad.openstack.org/p/automatic-evacuation • 参加メンバ ( 企業 )

• NTT, RedHat, SUSE, Intel, ...

ググっても多分みつかりません。チーム名が一般的過ぎて別ページがヒットします。

21Copyright©2016 NTT corp. All Rights Reserved.

チームで議論中の VM HA 方式

1. Masakari• https://github.com/ntt-sic/masakari

2. Resource-agent• https://

github.com/openstack/openstack-resource-agents

3. Mistral workflow• https://github.com/gryf/mistral-evacuate• Mistral プロジェクトを利用した VM-HA フロー

• Mistral: ワークフローエンジン as a Service• https://wiki.openstack.org/wiki/Mistral

22Copyright©2016 NTT corp. All Rights Reserved.

コミュニティ内関係図

VM HA 機能

Masakari Resource-agent

Mistral workflow

HA VMuser story 大

Product WG

執筆

……

VM HA 機能の既存実装

大HA チーム

HA チームとProduct WGで連携しよう

今後の方向性を議論中

あるべき姿の議論中

23Copyright©2016 NTT corp. All Rights Reserved.

コミュニティの今後の展望

• コミュニティとしての VM HA の仕様・実装• Product WG と協力

• 各種 VM HA の実装の方向性議論• Masakari と Mistral のインテグレーションなど

• Austin Summit でチームのサマリーに関するセッションも有ります• High Availability for pets and hypervisors - State of the Nation (

https://www.openstack.org/summit/austin-2016/summit-schedule/events/7327)

• コミュニティでの議論状況の共有が主な話題

• OpenStack HA チームの次回作に乞うご期待 !!

24Copyright©2016 NTT corp. All Rights Reserved.

Masakari デモ

25Copyright©2016 NTT corp. All Rights Reserved.

デモシナリオ

1. VM Instance down

Host1 Host2

VMdown 1.vm down

2.restartController

Nova Masakari

Host1 Host2

VM

down

1.host down

2.evacuateController

Nova Masakari

2. Host down

26Copyright©2016 NTT corp. All Rights Reserved.

Github: https://github.com/ntt-sic/masakariMail: muroi.masahito@lab.ntt.co.jp

Pull Request 気軽にどうぞ 日本語でも OK です。