Танки в облаках

Copyright © 2016 Mirantis, Inc. All rights reserved

www.mirantis.com

Tanks in The Clouds

Ivan Krovyakov, Mirantis


What we are going to discuss

Contents

● Wargaming.net: overview

● Cloud use case: what and why?

● How we did it

● Learnings and achievements

● What’s next

Speakers

Ivan KrovyakovCloud Solutions Architect, Mirantis


Wargaming and G-Core Labs in a nutshell

Wargaming.net

● International game developer● 18 years on the market● 2 Guinness World Records● 15 offices worldwide (incl. Austin, TX)● 4000+ employees● 150M+ users / 1,114,000 peak online

G-Core Labs

● Luxembourg-based IT provider● Europe/CIS/Asia/US DC coverage● Direct peering to major ISPs● HTTP/P2P CDN● 500+ GbpS bandwidth


Wargaming: initial statePeople, processes, technologies

Release Managers

Dev/QA DevOps

Administrators

G-Core team L1 support


Project Business Goals and Use Cases

● Business Goals

● Reduce costs by embracing open source platform and increasing utilization

● Reduce time-to-production by using API and automation for manual operations

● Use Cases: four-step workload lifecycle

● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code

● Staging Stable: pre-release testing of the new software versions

● Prod Test: pre-production small scale public A/B testing

● Production: publically available production workloads


Mirantis approach: Assess – Build – Sustain

SustainBuildAssess

Assess ● Analyze the business and technical goals● Map the goals to OpenStack capabilities● Lock down the MVP

Build ● Develop all the plugins needed● Build staging environment● Roll out the production environment(s)

Sustain ● Product support● Plugins and customizations maintenance● Roadmap including scaling and upgrades


Mirantis approach: AssessArchitecture Design Assessment (ADA)

● Identify cloud users and their use cases (technical and business)● Collect and prioritize all the requirements (any level)● Map MOS capabilities to the customer’s needs, lock MVP and get inputs for SoW

Goals

● Executive Summary and Overview● A review of business objectives, critical success factors, core use cases, users and stakeholders.

● Technical Response (MVP)● Focus on the technologies and products required for delivering the expected business value

● Functional & Operational Response● Recommendations on successful operation of proposed technical solution for MVP

Format ● 3-5 days onsite workshop with Mirantis architects and customer stakeholders● Offsite work to design and document the future cloud solution ● 10-15 business days project in total including customer review and readout

Artifacts

SustainBuildAssess


SustainBuildAssess

Mirantis approach: AssessWargaming ADA deliverables


Mirantis approach: BuildPhase 1: Development

SustainBuildAssess

Low-level design● Component-level decision making● Project scope refinement

Mirantis OpenStack plugin development● Dedicated lab environment● Separate QA process

Quality Assurance preparation activities● Test planning● Test automation using common tools: Rally, Tempest, Mirantis Health Check (OSTF)


Mirantis approach: BuildWargaming development phase deliverables

● Fuel plugins● Active Directory as authentication backend for Keystone● NFS as a storage backend (ephemeral, image, persistent)● Disabling anti-spoofing rules for VMs (multi-IP containerized apps)● Integration with external Puppet master

● Test plans and test report templates (Rally, Tempest)● Acceptance process● Demo scenarios

SustainBuildAssess


Mirantis approach: BuildPhase 2: Deployment

SustainBuildAssess

Hardware preparation Network infrastructure configuration Handover to Mirantis support

Pilot cloud deploymentQA acceptance according to test plan

Wargaming Ops start using cloudFeedback provided

Pilot cloud deployment(MVP 0.1)

Remaining clouds rollout

10 days each

CRs, scope refinement

Plugins / configuration changed according to feedback

Hardware preparation Network infrastructure configuration

Handover to Mirantis supportDocumentation

Pilot cloud deploymentQA acceptance according to test plan Handover to Wargaming Ops

Extra changes based on customer feedback

End of project

Retrospective


Mirantis approach: BuildWargaming deployment phase deliverables

● 4x clouds built on Mirantis OpenStack 7 (Kilo/Ubuntu 14.04/KVM)● 1x development MVP (WG internal sandbox)● 1x staging MVP (12x computes, 288x CPUs, 1.5TB RAM)● 2x production MVPs (19x computes, 372x CPUs, 2.6TB RAM)

● Basic set of OpenStack services (Glance, Keystone, Nova, Cinder, Ceilometer, Heat, Murano) + set of developed plugins

● Control plane HA: Mirantis reference architecture● Neutron: OpenVSwitch/VLANs, provider networks● Basic CMDB integration via Murano application (made by Wargaming)

SustainBuildAssess


Mirantis approach: TrainExtra step to make it work

Custom-tailored training based on the variety of existing offerings has been delivered

● OS100 – vanilla OpenStack 3-day course which covers various aspects of running OpenStack cloud core components● Architecture● Best practices● Provisioning workflow● Component interaction● Real-world challenges faced by OpenStack experts

● FUEL100 - 2-day course focused on Mirantis OpenStack extra components● Fuel● Murano● Sahara● Ceph

SustainBuildAssess

Train


Mirantis approach: TrainWargaming training deliverables

SustainBuildAssess

Train


Mirantis approach: Sustain

● Commercial enterprise-grade support subscription● Mirantis OpenStack as-is

● MOS packages● Reference architecture● Deployment and orchestration engine

● Publically available plugins and extensions

● Customer case-centric maintenance● Custom developed content maintenance● Assistance to roll out maintenance updates

● Upgrades● One-time upgrade● Upgrade subscription

SustainBuildAssess

L1 Support

L2 Support

Technical Escalations Team


Wargaming: business goalsInitial results

● Reduce time-to-production● DevOps and Admin teams were OpenStack enabled● Self-service IaaS is already in use

● DevOps team provisions virtual environments on their own

● Reduce costs● Workloads onboarding to OpenStack started

● > 100 VMs are already onboarded● Data collection for capacity management started

● Cost● Utilization

Source: http://cloudtweaks.com


Wargaming: lessons learned so far

● Don’t treat OpenStack as vCenter● Paradigm shift, not just a technology change● No feature parity now and a few years ahead

● Be prepared to revise your business processes● Think SLAs, cross-team interaction, working closely with product teams

● Pluggable cloud architecture: the key to long-term success● Modules: extra Dev $ spent -> Ops $$$ saved● Localized technical debt -> easier to deploy, sustain and upgrade

● Plan operational readiness beforehand● Again, people and processes >> technology ● Train and enable Ops on OpenStack before going into production


Wargaming and Mirantis: next steps

● Bare-metal -> OpenStack workloads onboarding● Kilo -> Mitaka (MOS 7 -> MOS 9) upgrade ● Scale out production clouds● Advanced IaaS features

● Load Balancing as a Service● DNS as a Service

● PaaS features● DB as a Service● Hadoop as a Service


Tank you for your attention!

http://static-ptl-us.gcdn.co/dcont/fb/image/military_month_684x280_3YmN6Fy.jpg


Wargaming workloads

What is a realm?

YAML which includes:○ region to deploy to (like “EU”)○ global services to use (“auth”, “dbms”, ...)○ games to deploy (“wot”, “wowp”, “wotb”, ...)

■ applications to deploy for each game ● names/IP-addresses of nodes to deploy to● Extra config info

dbms:- host: *wowp-aap-host0 version: '5.6' port: 3306 type: percona databases: - &wowp_aap_default name: aapwowpwgt21 host: *wowp-aap-host0 password: *** user: ***mqvhosts: - &wowp_aap_amqp_default host: *wowp-aap-host0 password: *** user: *** vhost: /aapwowpwgt21wowp: aap: version: 3 bonus_check_events: False pay_events: False amqp: default: *wowp_aap_amqp_default http: host: *wowp-aap-host0 ip: *wowp-aap-ip0 mysql: db: aapwowpwgt21 host: *wowp-aap-host0 password: *** user: *** db: default: *wowp_aap_default=== CUT ====


Wargaming: core projects

Currently, Wargaming is focused on its team-based MMO war series dedicated to mid-20th century warfare. This lineup includes the company’s flagship armored MMO World of Tanks, launched in April 2011; the fast-paced flight combat MMO World of Warplanes; and the epic naval warfare shooter World of Warships.150+ million users in Wargaming.net Universe and value each and every player.

■ Best European Publisher at EGA 2012■ 2 Guinness World Records (project: WoT, years: 2011, 2013);■ 2 Golden Joystick Awards (project: WoT, title: MMO game of the year, year: 2012 and 2013);


What is Mirantis

Mirantis is the pure play OpenStack company, delivering all the software, services, training, and support needed for running OpenStack. More customers rely on Mirantis than on any other company to get to production deployment of OpenStack at scale.

Mirantis is among the top three companies worldwide in contributing open source software to OpenStack, and has helped build and deploy some of the largest OpenStack clouds in the world, at companies such as Cisco, Comcast, Ericsson, NASA, Samsung and Symantec.


G-Core in a nutshell

G-Core is a Luxembourg-based global IT solutions provider offering a wide range of services, including hosting, CDN, peering network, and different levels of support, from smart hands to DDoS protection to advice on local regulations, and more. Initially developed to support online gaming, G-Core has three main KPIs: low latency, high availability and cost optimization. These parameters run through many online industries, such as banking or streaming services, expanding client base far beyond gaming.

1 114 000 peak players online


Tank you for your attention!

Any questions?


Wargaming before OpenStack - the Scale

● TBD


Wargaming: initial stack

● 3 level platform stack○ Hardware○ Virtualization layer, VMWare (optional) ○ Containerized applications, OpenVZ.

● Several engineering independent teams, responsible for their own part○ Release Managers○ Dev/QA engineers○ DevOps engineers○ Administrators○ G-Core team○ L1 support team

● JIRA-based provisioning workflow


Wargaming: cloud use cases

4 steps for a workload to get from developer to end-user

● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code● Staging Stable: pre-release testing of the new versions of software● Production Test: public beta-testing of limited deployments of production-ready versions of

software● Production: publically available production workloads


What was wrong? The business drivers

Wargaming has asked Mirantis to implement a cloud solution based on Mirantis OpenStack to ease their current pain points and offer a flexible solution to their Online Presence team cost effectively and without compromising on performance and scale.

● Reduce costs of existing and new infrastructure● Improve infrastructure utilization● Capability for automation of infrastructure provisioning● Shorten release cycle and increase business agility


Mirantis approach: BuildPhase 2: Deployment

Once content is developed and tested OpenStack rollout starts

● Pilot cloud deployed○ Hardware preparation, network infrastructure configuration○ QA acceptance according to the plan○ Handover to Mirantis support

○ Wargaming operations team starts playing with the cloud and provides feedback.

● Customer feedback addressed by Mirantis● Rest of the clouds deployed

○ Preparation○ Deployment○ QA acceptance○ Handover to support and operations○ Documentation

SustainBuildAssess


What has been done so far: numbers

4 cloud were built with Mirantis OpenStack 7 (Kilo/Ubuntu 14.04/KVM)

● 1 development cloud for WG team to play with● 1 staging cloud, 12x computes, 288x CPUs, 1.5TB RAM● 2 production clouds, 19x computes, 372x CPUs, 2.6TB RAM


What has been done so far: features

● Control plane High Availability - Mirantis reference architecture● MOS plugins

○ Active Directory as authentication backend for Keystone○ NFS as a storage backend (ephemeral, image, persistent)○ Disabling anti-spoofing rules for VMs (multi-IP containerized apps)○ Integration with external Puppet master

● Basic set of OpenStack services: Glance, Keystone, Nova, Cinder, Ceilometer, Heat, Murano

● Neutron - OpenVSwitch/VLANs, provider networks● Basic integration with external CMDB as a Murano application


Wargaming: business goalsInitial results

● DevOps and Admin teams were trained to operate OpenStack● Infrastructure as a Service has been provided to the cloud users

○ DevOps team provisions virtual environments on their own ■ “Business agility” driver is being addressed

● Migration of workloads has been started○ more than 100 VMs have been migrated so far

■ “Cost reduction” driver is being addressed

● Wargaming has started collecting data for long-term number-driven management of cloud capacity○ Cost○ Utilization


Project Business Goals and Use Cases

Wargaming has engaged Mirantis to implement a cloud solution based on Mirantis OpenStack to ease their current pain points and offer a flexible solution to their Online Presence team cost effectively and without compromising on performance and scale.

● Business Goals● Reduce CAPEX by replacing VMware with MOS and increasing infra utilization (~50% now)● Reduce time-to-production by using API and automation for manual operations

● Use Cases: four-steps workload lifecycle● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code● Staging Stable: pre-release testing of the new software versions ● Prod Test: pre-production small scale public A/B testing● Production: publically available production workloads

Technology

Танки в облаках