Copyright © 2016 Mirantis, Inc. All rights reserved
www.mirantis.com
Tanks in The Clouds
Ivan Krovyakov, Mirantis
Copyright © 2016 Mirantis, Inc. All rights reserved
What we are going to discuss
Contents
● Wargaming.net: overview
● Cloud use case: what and why?
● How we did it
● Learnings and achievements
● What’s next
Speakers
Ivan KrovyakovCloud Solutions Architect, Mirantis
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming and G-Core Labs in a nutshell
Wargaming.net
● International game developer● 18 years on the market● 2 Guinness World Records● 15 offices worldwide (incl. Austin, TX)● 4000+ employees● 150M+ users / 1,114,000 peak online
G-Core Labs
● Luxembourg-based IT provider● Europe/CIS/Asia/US DC coverage● Direct peering to major ISPs● HTTP/P2P CDN● 500+ GbpS bandwidth
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: initial statePeople, processes, technologies
Release Managers
Dev/QA DevOps
Administrators
G-Core team L1 support
Copyright © 2015 Mirantis, Inc. All rights reserved
Project Business Goals and Use Cases
● Business Goals
● Reduce costs by embracing open source platform and increasing utilization
● Reduce time-to-production by using API and automation for manual operations
● Use Cases: four-step workload lifecycle
● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code
● Staging Stable: pre-release testing of the new software versions
● Prod Test: pre-production small scale public A/B testing
● Production: publically available production workloads
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: Assess – Build – Sustain
SustainBuildAssess
Assess ● Analyze the business and technical goals● Map the goals to OpenStack capabilities● Lock down the MVP
Build ● Develop all the plugins needed● Build staging environment● Roll out the production environment(s)
Sustain ● Product support● Plugins and customizations maintenance● Roadmap including scaling and upgrades
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: AssessArchitecture Design Assessment (ADA)
● Identify cloud users and their use cases (technical and business)● Collect and prioritize all the requirements (any level)● Map MOS capabilities to the customer’s needs, lock MVP and get inputs for SoW
Goals
● Executive Summary and Overview● A review of business objectives, critical success factors, core use cases, users and stakeholders.
● Technical Response (MVP)● Focus on the technologies and products required for delivering the expected business value
● Functional & Operational Response● Recommendations on successful operation of proposed technical solution for MVP
Format ● 3-5 days onsite workshop with Mirantis architects and customer stakeholders● Offsite work to design and document the future cloud solution ● 10-15 business days project in total including customer review and readout
Artifacts
SustainBuildAssess
Copyright © 2016 Mirantis, Inc. All rights reserved
SustainBuildAssess
Mirantis approach: AssessWargaming ADA deliverables
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: BuildPhase 1: Development
SustainBuildAssess
Low-level design● Component-level decision making● Project scope refinement
Mirantis OpenStack plugin development● Dedicated lab environment● Separate QA process
Quality Assurance preparation activities● Test planning● Test automation using common tools: Rally, Tempest, Mirantis Health Check (OSTF)
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: BuildWargaming development phase deliverables
● Fuel plugins● Active Directory as authentication backend for Keystone● NFS as a storage backend (ephemeral, image, persistent)● Disabling anti-spoofing rules for VMs (multi-IP containerized apps)● Integration with external Puppet master
● Test plans and test report templates (Rally, Tempest)● Acceptance process● Demo scenarios
SustainBuildAssess
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: BuildPhase 2: Deployment
SustainBuildAssess
Hardware preparation Network infrastructure configuration Handover to Mirantis support
Pilot cloud deploymentQA acceptance according to test plan
Wargaming Ops start using cloudFeedback provided
Pilot cloud deployment(MVP 0.1)
Remaining clouds rollout
10 days each
CRs, scope refinement
Plugins / configuration changed according to feedback
Hardware preparation Network infrastructure configuration
Handover to Mirantis supportDocumentation
Pilot cloud deploymentQA acceptance according to test plan Handover to Wargaming Ops
Extra changes based on customer feedback
End of project
Retrospective
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: BuildWargaming deployment phase deliverables
● 4x clouds built on Mirantis OpenStack 7 (Kilo/Ubuntu 14.04/KVM)● 1x development MVP (WG internal sandbox)● 1x staging MVP (12x computes, 288x CPUs, 1.5TB RAM)● 2x production MVPs (19x computes, 372x CPUs, 2.6TB RAM)
● Basic set of OpenStack services (Glance, Keystone, Nova, Cinder, Ceilometer, Heat, Murano) + set of developed plugins
● Control plane HA: Mirantis reference architecture● Neutron: OpenVSwitch/VLANs, provider networks● Basic CMDB integration via Murano application (made by Wargaming)
SustainBuildAssess
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: TrainExtra step to make it work
Custom-tailored training based on the variety of existing offerings has been delivered
● OS100 – vanilla OpenStack 3-day course which covers various aspects of running OpenStack cloud core components● Architecture● Best practices● Provisioning workflow● Component interaction● Real-world challenges faced by OpenStack experts
● FUEL100 - 2-day course focused on Mirantis OpenStack extra components● Fuel● Murano● Sahara● Ceph
SustainBuildAssess
Train
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: TrainWargaming training deliverables
SustainBuildAssess
Train
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: Sustain
● Commercial enterprise-grade support subscription● Mirantis OpenStack as-is
● MOS packages● Reference architecture● Deployment and orchestration engine
● Publically available plugins and extensions
● Customer case-centric maintenance● Custom developed content maintenance● Assistance to roll out maintenance updates
● Upgrades● One-time upgrade● Upgrade subscription
SustainBuildAssess
L1 Support
L2 Support
Technical Escalations Team
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: business goalsInitial results
● Reduce time-to-production● DevOps and Admin teams were OpenStack enabled● Self-service IaaS is already in use
● DevOps team provisions virtual environments on their own
● Reduce costs● Workloads onboarding to OpenStack started
● > 100 VMs are already onboarded● Data collection for capacity management started
● Cost● Utilization
Source: http://cloudtweaks.com
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: lessons learned so far
● Don’t treat OpenStack as vCenter● Paradigm shift, not just a technology change● No feature parity now and a few years ahead
● Be prepared to revise your business processes● Think SLAs, cross-team interaction, working closely with product teams
● Pluggable cloud architecture: the key to long-term success● Modules: extra Dev $ spent -> Ops $$$ saved● Localized technical debt -> easier to deploy, sustain and upgrade
● Plan operational readiness beforehand● Again, people and processes >> technology ● Train and enable Ops on OpenStack before going into production
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming and Mirantis: next steps
● Bare-metal -> OpenStack workloads onboarding● Kilo -> Mitaka (MOS 7 -> MOS 9) upgrade ● Scale out production clouds● Advanced IaaS features
● Load Balancing as a Service● DNS as a Service
● PaaS features● DB as a Service● Hadoop as a Service
Copyright © 2015 Mirantis, Inc. All rights reserved
Tank you for your attention!
http://static-ptl-us.gcdn.co/dcont/fb/image/military_month_684x280_3YmN6Fy.jpg
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming workloads
What is a realm?
YAML which includes:○ region to deploy to (like “EU”)○ global services to use (“auth”, “dbms”, ...)○ games to deploy (“wot”, “wowp”, “wotb”, ...)
■ applications to deploy for each game ● names/IP-addresses of nodes to deploy to● Extra config info
dbms:- host: *wowp-aap-host0 version: '5.6' port: 3306 type: percona databases: - &wowp_aap_default name: aapwowpwgt21 host: *wowp-aap-host0 password: *** user: ***mqvhosts: - &wowp_aap_amqp_default host: *wowp-aap-host0 password: *** user: *** vhost: /aapwowpwgt21wowp: aap: version: 3 bonus_check_events: False pay_events: False amqp: default: *wowp_aap_amqp_default http: host: *wowp-aap-host0 ip: *wowp-aap-ip0 mysql: db: aapwowpwgt21 host: *wowp-aap-host0 password: *** user: *** db: default: *wowp_aap_default=== CUT ====
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: core projects
Currently, Wargaming is focused on its team-based MMO war series dedicated to mid-20th century warfare. This lineup includes the company’s flagship armored MMO World of Tanks, launched in April 2011; the fast-paced flight combat MMO World of Warplanes; and the epic naval warfare shooter World of Warships.150+ million users in Wargaming.net Universe and value each and every player.
■ Best European Publisher at EGA 2012■ 2 Guinness World Records (project: WoT, years: 2011, 2013);■ 2 Golden Joystick Awards (project: WoT, title: MMO game of the year, year: 2012 and 2013);
Copyright © 2016 Mirantis, Inc. All rights reserved
What is Mirantis
Mirantis is the pure play OpenStack company, delivering all the software, services, training, and support needed for running OpenStack. More customers rely on Mirantis than on any other company to get to production deployment of OpenStack at scale.
Mirantis is among the top three companies worldwide in contributing open source software to OpenStack, and has helped build and deploy some of the largest OpenStack clouds in the world, at companies such as Cisco, Comcast, Ericsson, NASA, Samsung and Symantec.
Copyright © 2016 Mirantis, Inc. All rights reserved
G-Core in a nutshell
G-Core is a Luxembourg-based global IT solutions provider offering a wide range of services, including hosting, CDN, peering network, and different levels of support, from smart hands to DDoS protection to advice on local regulations, and more. Initially developed to support online gaming, G-Core has three main KPIs: low latency, high availability and cost optimization. These parameters run through many online industries, such as banking or streaming services, expanding client base far beyond gaming.
1 114 000 peak players online
Copyright © 2016 Mirantis, Inc. All rights reserved
Tank you for your attention!
Any questions?
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming before OpenStack - the Scale
● TBD
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: initial stack
● 3 level platform stack○ Hardware○ Virtualization layer, VMWare (optional) ○ Containerized applications, OpenVZ.
● Several engineering independent teams, responsible for their own part○ Release Managers○ Dev/QA engineers○ DevOps engineers○ Administrators○ G-Core team○ L1 support team
● JIRA-based provisioning workflow
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: cloud use cases
4 steps for a workload to get from developer to end-user
● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code● Staging Stable: pre-release testing of the new versions of software● Production Test: public beta-testing of limited deployments of production-ready versions of
software● Production: publically available production workloads
Copyright © 2016 Mirantis, Inc. All rights reserved
What was wrong? The business drivers
Wargaming has asked Mirantis to implement a cloud solution based on Mirantis OpenStack to ease their current pain points and offer a flexible solution to their Online Presence team cost effectively and without compromising on performance and scale.
● Reduce costs of existing and new infrastructure● Improve infrastructure utilization● Capability for automation of infrastructure provisioning● Shorten release cycle and increase business agility
Copyright © 2016 Mirantis, Inc. All rights reserved
Mirantis approach: BuildPhase 2: Deployment
Once content is developed and tested OpenStack rollout starts
● Pilot cloud deployed○ Hardware preparation, network infrastructure configuration○ QA acceptance according to the plan○ Handover to Mirantis support
○ Wargaming operations team starts playing with the cloud and provides feedback.
● Customer feedback addressed by Mirantis● Rest of the clouds deployed
○ Preparation○ Deployment○ QA acceptance○ Handover to support and operations○ Documentation
SustainBuildAssess
Copyright © 2016 Mirantis, Inc. All rights reserved
What has been done so far: numbers
4 cloud were built with Mirantis OpenStack 7 (Kilo/Ubuntu 14.04/KVM)
● 1 development cloud for WG team to play with● 1 staging cloud, 12x computes, 288x CPUs, 1.5TB RAM● 2 production clouds, 19x computes, 372x CPUs, 2.6TB RAM
Copyright © 2016 Mirantis, Inc. All rights reserved
What has been done so far: features
● Control plane High Availability - Mirantis reference architecture● MOS plugins
○ Active Directory as authentication backend for Keystone○ NFS as a storage backend (ephemeral, image, persistent)○ Disabling anti-spoofing rules for VMs (multi-IP containerized apps)○ Integration with external Puppet master
● Basic set of OpenStack services: Glance, Keystone, Nova, Cinder, Ceilometer, Heat, Murano
● Neutron - OpenVSwitch/VLANs, provider networks● Basic integration with external CMDB as a Murano application
Copyright © 2016 Mirantis, Inc. All rights reserved
Wargaming: business goalsInitial results
● DevOps and Admin teams were trained to operate OpenStack● Infrastructure as a Service has been provided to the cloud users
○ DevOps team provisions virtual environments on their own ■ “Business agility” driver is being addressed
● Migration of workloads has been started○ more than 100 VMs have been migrated so far
■ “Cost reduction” driver is being addressed
● Wargaming has started collecting data for long-term number-driven management of cloud capacity○ Cost○ Utilization
Copyright © 2015 Mirantis, Inc. All rights reserved
Project Business Goals and Use Cases
Wargaming has engaged Mirantis to implement a cloud solution based on Mirantis OpenStack to ease their current pain points and offer a flexible solution to their Online Presence team cost effectively and without compromising on performance and scale.
● Business Goals● Reduce CAPEX by replacing VMware with MOS and increasing infra utilization (~50% now)● Reduce time-to-production by using API and automation for manual operations
● Use Cases: four-steps workload lifecycle● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code● Staging Stable: pre-release testing of the new software versions ● Prod Test: pre-production small scale public A/B testing● Production: publically available production workloads