DevOps!! 도데체 왜, 어떻게 할까??

PowerPoint

GS Shop 42016 6 3()

([email protected])FB, Flickr, Etsy, Netflix DevOps!

DevOps!! , ??

.

1

Speaker2. .

, ) ) CTO) UofN, Kona Sr. Software Engineer) Yahoo! APAC Listing/Map Eng. Leader) Yahoo! KR Local/Map Eng. Leader) Portaltone

https://www.facebook.com/kim.joseph.75

https://www.linkedin.com/in/josephkim75

Index , DevOps ?

2 FB, Flickr, Etsy, Netflix, Google

3 DevOps ?

1

4

DevOps ?

11. DevOps ?

1. DevOps ?5 .

1. DevOps ?62015 5 KPCB INTERNET TRENDS 2015

: http://www.kpcb.com/internet-trends

1. DevOps ?7: http://www.kpcb.com/internet-trends 20 People Connected 24/7 with Mobile Devices

1. DevOps ?8(http://www.reactivemanifesto.org/)Reactive Manifesto These changes are happening because application requirements have changed dramaticallyin recent years.

Only a few years ago a large application had tens of servers, seconds of response time, hours of offline maintenance and gigabytes of data.

Today applications are deployed on everything from mobile devices to cloud-based clusters running thousands of multi-core processors. Users expect millisecond response times and 100% uptime. Data is measured in Petabytes. Few years agoTodayLarge application?Tens of serversThousands of multi-core processorsSeconds of response timeMillisecond response timeHours of offline maintenance100% uptimeGigabytes of dataPetabytes of data

Today's demands are simply not met by yesterdays software architectures.

1. DevOps ?9, . . , Rush . ,

1. DevOps ?10, , .

1. DevOps ?11, Google, Facebook, Amazon ?

1. DevOps ?12The Hacker Way

Facebook IPO

Focus on ImpactMove FastBe BoldBe OpenBuild Social Value

. (Done is better than perfect.) . (Code wins arguments.) .

: http://www.looah.com/article/view/738/

1. DevOps ?13: http://www.google.com/intl/ko_KR/about/company/philosophy/

Ten Things Google Knows To Be True

Focus on the user and all else will followIts best to do one thing really, really well.Fast is better than slow.Democracy on the web works.You dont need to be on your desk to need an answer.You can make money without doing evil.There is always more information out there.The need for information crosses all border.You can be serious without a suit.Great just isnt good enough.

1. DevOps ?1414

Amazon Leadership Principles

Customer ObsessionOwnershipInvent and Simplify

Moving Fast

2013 280

1. DevOps ?15: http://www.slideshare.net/diannemarsh/from-code-to-the-monkeys-continuous-delivery-at-netflix/

Dianne Marsh (Director of Engineering, Netflix) said

1. DevOps ?16

Moving Fast!!Focus on Customer!!

1. DevOps ?17

, .

1. DevOps ?18CompanyDeploy FrequencyDeploy Lead TimeReliabilityCustomer ResponsivenessAmazon23,000 / dayMinutesHighHighGoogle5,500 / dayMinutesHighHighNetflix500 / dayMinutesHighHighFacebook1 / dayHoursHighHighTwitter3 / weekHoursHighHighTypical enterpriseOnce every 9 monthsMonths or quartersLow / MediumLow / Medium

: The Phoenix Project (2013), Amazon, Google, Netflix, Facebook, Twitter ? Measuring DevOps and IT Performances

Deploy frequency (Note: NOT delivery) Mean Time to Recover (MTTR) Lead Time for Changes

1. DevOps ?19 !!!

(2013) => (2014)

30+ (2013) 50 (2014 3) Qcon London 80 ~ 90 (2014 4) Chef Conf: Releng 2014 Keynote (https://www.youtube.com/watch?v=Nffzkkdq7GM)

1. DevOps ?20GOTO Berlin 2015 Dr. Nicole Forsgren (Chef the State of DevOps study ) DevOps: Next : GOTO Berlin 2015 DevOps: Next (https://www.youtube.com/watch?v=dMwGfRINpz0)

1. DevOps ?21GOTO Berlin 2015 Dr. Nicole Forsgren (Chef the State of DevOps study ) DevOps: Next : GOTO Berlin 2015 DevOps: Next (https://www.youtube.com/watch?v=dMwGfRINpz0)

1. DevOps ?22 DevOps ? , Damon Edwards (DTO solutions inc.)

DevOps is not about a technology, DevOps is about a business problem.

, DevOps ?

22. , DevOps ?

2. , DevOps ?24 .DevOps is NOT - A Job title.- An Organizational Unit. - A automation.- A tool.- Simply combining Dev & Ops...

2. , DevOps ?25But, there is not a clear definition yet.

WHAT???

2. , DevOps ?26DevOps (Wikipedia)DevOps , .

.

2. , DevOps ?27DevOps 1) DevOps (DevOps )2) CAMS or CALMS3) Devops.com 4) DevOps foundations5) The Three Ways Principles (Gene Kim)6) DevOps Area Practices (Patrick Debois)

2. , DevOps ?282007 Patrick Debois (godfather of DevOps) IDC . Patrick Debois . .

1) DevOps ( ): IT Revolution Press (http://itrevolution.com/the-history-of-devops/)[Patrick Debois (godfather of DevOps)]

2. , DevOps ?292008 8, Andrew Shafer Agile 2008 Conference Agile Infrastructure . . Patrick Debois!!Andrew Shafer Patrick Debois Patrick . Patrick Debois The Agile Systems Administration .1) DevOps ( ): IT Revolution Press (http://itrevolution.com/the-history-of-devops/)

[Andrew Shafer]

2. , DevOps ?302009 OReilly Velocity 2009 conference Flickr John Allspaw Paul Hammond 10+ Deploys a Day: Dev and Ops Cooperation at Flickr .

Patrick , Velocity . .

3 Dev, Ops, Days DevOpsDays Ghent 2009 10 .1) DevOps ( ): IT Revolution Press (http://itrevolution.com/the-history-of-devops/)

[2009 Velocity 2009 Conference John Allspaw Paul Hammond(https://www.youtube.com/watch?v=LdOe18KhtT4)]

2. , DevOps ?311) DevOps ( ): IT Revolution Press (http://itrevolution.com/the-history-of-devops/)[ http://devopsdays.org/ ]

2009 10 DevOpsDay .

DevOpsDay #DevOps .

2. , DevOps ?321) DevOps ( )By Damon Edwards (DTO Solutions inc.) - https://www.youtube.com/watch?v=o7-IuYS0iSE&feature=youtu.beFrom practitioners, by practitionersAn experience-based movementDecentralized and open to allDevOps

2. , DevOps ?332) CAMS or CALMShttps://www.chef.io/blog/2010/07/16/what-devops-means-to-me/http://devops.com/2015/05/13/surprise-broad-agreement-on-the-definition-of-devops/http://itrevolution.com/devops-culture-part-1/Devops is about CAMS. (John Willis) - C: Culture- A: Automation- M:Measurement- S: Sharing- L: Lean ( )CAMS: 2010 Devopsdays in Mountainview Damon Adwards John Willis CAMS CALMS: Continuous Delivery Jez Humble L (Lean)

2. , DevOps ?343) Devops.com DevOps exists to help the business winThe scope is broad, but centered on ITThe foundations are found in Agile and LeanCulture is very importantFeedback is fuel for innovationAutomation helpshttp://devops.com/2015/05/13/surprise-broad-agreement-on-the-definition-of-devops/

2. , DevOps ?354) DevOps foundationsLean (Lean Thinking)

Lean

Lean

2. , DevOps ?36Lean Software Development PrinciplesEliminate waste ( )Amplify learning ( )Decide as late as possible ( )Deliver as fast as possible ( delivery )Empower the team ( )Build quality in (quality )See the whole ( )4) DevOps foundations

2. , DevOps ?37OODA loop (OODA )

F-86 MiG-15 10:1 . .

Competed Observation: Orientation: Decision: Action: 4) DevOps foundations

2. , DevOps ?38http://itrevolution.com/the-three-ways-principles-underpinning-devops/

5) DevOps the Three ways principles (Gene Kim)Gene Kim DevOps Cookbook the Phoenix Project

2. , DevOps ?39http://itrevolution.com/the-three-ways-principles-underpinning-devops/5) DevOps the Three ways principles (Gene Kim)



2. , DevOps ?42Robert Johnson(Engineering Director at FB) InfoQ (2010 5) ... . .

http://www.infoq.com/presentations/Facebook-Moving-Fast-at-ScaleDevOps

2. , DevOps ?43http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/

6) DevOps Area Practices (Patrick Debois) DevOps IT (HR, Finance) collaboration optimization . DevQAOps, DevOpsSec, DevSecOps, BizDevOps DevOps (IT Performance => Org. Performance)

2. , DevOps ?44http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/6) DevOps Area Practices (Patrick Debois) DevOps (Devops lite) Dev Ops

2. , DevOps ?45http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/6) DevOps Area Practices (Patrick Debois)Patrick Debois 4 Areas, Dev Project, Ops Product .

2. , DevOps ?46http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/6) DevOps Area Practices (Patrick Debois)Practical examplesPractices => Patterns => PrinciplesAreasPractical examplesArea 1Extend delivery to productionCI/CD. chef/puppet configuration management Area 2Extend operations feedback to projectProduction(Ops) Logfiles, metrics, information Project(Dev) Area 3Embed Project knowledge into OperationsProject(Dev) Project(Dev) pagers Project(Dev) knowledge Production(Ops) Area 4Embed Operations knowledge into ProjectProject(Dev) Production(Ops) Production(Ops) Project(Dev) backlog

2. , DevOps ?47http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/6) DevOps Area Practices (Patrick Debois)You cant directly change culture. But you can change behavior, and behavior becomes culture

Lloyd Taylor VP Infrastructure, Ngmoco

2. , DevOps ?48- http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/Patrick Debois 4 Area CAMS

6) DevOps Area Practices (Patrick Debois)

FB, Flickr, Etsy, Netflix, Google

33. FB, Flickr, Netflix, Etsy, Google

3. FB, Flickr, Netflix, Etsy, Google 50

?

1)

Ars technica : http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/ : http://swprocess.egloos.com/3009704The Hacker Way: http://www.looah.com/article/view/738Robert Johnson InfoQ : http://www.infoq.com/presentations/Facebook-Moving-Fast-at-ScaleReleng 2014 Keynote 1: https://www.youtube.com/watch?v=Nffzkkdq7GM#t=275The Facebook Release Process InfoQ : http://www.infoq.com/presentations/Facebook-Release-Process

51

2) 2 ( )

3) Deployment Pipeline: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6449236


52

4) HipHop PHP -> C++ . 1.5 GB.1.5GB push BitTorrent BitTorrent (rack) 30 ( : 15, : 15, 2012 ) .

[Facebook . : http://www.looah.com/article/view/983]3. FB, Flickr, Netflix, Etsy, Google

53

5) FB Single stable branch , long-lived branche

6) Tools : Phabricator (http://phabricator.org/) : Watir (http://watir.com/) : Selenium (https://github.com/seleniumhq/selenium) : Perflab

7) IRC IRC ( 700)

8) , , Facebook


54

9) . . . , . (/ ) .

Facebook Hotfix Bar . . .: http://www.looah.com/article/view/9833. FB, Flickr, Netflix, Etsy, Google

551) 10+ deploys per day Flickr John Allspaw(Ops head) Paul Hammond(Dev head) Velocity 2009

: : https://www.youtube.com/watch?v=LdOe18KhtT43. FB, Flickr, Netflix, Etsy, Google

562)

[ ] Dev , Ops

John Allspaw Paul Hammond No!! !! Ops Dev . . , ?

? ? . ?

: http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr3. FB, Flickr, Netflix, Etsy, Google

572)

Tools Culture Risk

: http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr


583) Tools

Automated infrastructureFlickr Tools: Chef, Cfengine, Puppet, FAI, Cobbler

Shared version controlDev/Ops version control

One step build and deploy , ,


59

: http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

3. FB, Flickr, Netflix, Etsy, Google 3) Tools

603) Tools

Feature flags: on/off Trunk : Paul Hammond Always ship trunk . http://www.paulhammond.org/2010/06/trunk/alwaysshiptrunk.pdfBucket testingDark launchesShared metricsIRC and IM robotsDev, Ops IRC IM robots build, deploy, alerts monitors


614) Culture

Respect: , . Trust: Healthy attitude about failureAvoiding Blame

, !!


62

Etsy: , , ,

2008 (40+ engineers)Painful mergeHand off to deployersDeploy, site downRoll back deployFix bugs, go to step 2

6? 50 .

2014 (400+ engineers)Small changesets, deployed frequentlyEngineers deploy (not just engineers, but also Designers, Dogs)Deploys are fastChanges behind flagsCopious graphs/metricsFix fast & roll forward

(http://www.slideshare.net/beamrider9/continuous-deployment-at-etsy-a-tale-of-two-approaches)3. FB, Flickr, Netflix, Etsy, Google

631)

Netflix API (Tech Blog): http://techblog.netflix.com/2013/08/deploying-netflix-api.html

Self service build and deployment at Netflix: http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013?related=2

Dianne Marsh (Director of Engineering for Cloud Tools): http://www.slideshare.net/diannemarsh/from-code-to-the-monkeys-continuous-delivery-at-netflix?related=2

Continuous Delivery at Netflix: http://www.slideshare.net/robspieldenner/continuous-delivery-at-netflix?related=1

: http://techblog.netflix.com/2013/08/deploying-netflix-api.html3. FB, Flickr, Netflix, Etsy, Google

642) Deployment Flow

Unit TestRegression TestCanary AnalysisRed/BlackGlobal Release (Red)

: http://techblog.netflix.com/2013/08/deploying-netflix-api.html


653) Branches3 long-lived branches (Test/Release/Prod branch)

Test branch: test branch deploy production release branch merge

Release branch: weekly release. Release branch commit production staging deploy . Staging production deploy delivery team .

Prod branch: Release release branch prod branch merge. Prod branch Patch/Daily push . production deploy weekly release deploy prod branch commit. Prod branch commit release branch merge live portion canary cluster deploy. Canary analysis go global deploy


664) Canary

Canary? production small subset

Canary (HTTP status code, response time, , load avg ) 1000 metric 100 .

: http://techblog.netflix.com/2013/08/deploying-netflix-api.html


675) Region

Netflix 3 AWS region (2013 )API deploy Asgard (https://github.com/Netflix/asgard) Red/Black push production pushAWS Auto Scale Group( ASG) Red/Black push . ASG 10% ASG . . (red/red ) ASG . (red/black ) ASG . ASG

6) production push XMPP bot


681) Google DevOps? SRE (Site Reliability Engineering)

SRE operation 50:50 . SRE hire ops skill ., SRE , .3. FB, Flickr, Netflix, Etsy, Google

: https://landing.google.com/sre/interview/ben-treynor.html

69Developers run their own service. (self-support)- SRE creates tools and services to make this possible.Some services get allocated an SRE-team: (high priority services only)- Developers have run it for 6+ months.- Must pass a Hand-off Readiness Review before Hand-off. - In the future goes back to self-support if things go wrong a lot.3. FB, Flickr, Netflix, Etsy, Google

: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)

2) SRE Methodology

70Type and frequency of pages / alertsMaturity of the monitoring infrastructure: pages, dashboard, etcSystem architecture reviewRelease processBug counts / severitiesProduction hygiene3. FB, Flickr, Netflix, Etsy, Google

3) Hand-off Readiness Review: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)


4) Canary: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)


4) Canary: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)


5) The Reliability Budget: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)


6) Joint Responsibility: SRE@Google: Thousands of DevOps Since 2004 (https://www.youtube.com/watch?v=iIuTnhdTzK0)

75[] DevOps Team Topologies: http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/2013 10 Matthew Skelton What Team Structure Is Right for DevOps to Flourish? http://web.devopstopologies.com/7 Anti-Types 9 DevOps Team Topologies DevOps 3. FB, Flickr, Netflix, Etsy, Google

76[] DevOps Team Topologies3. FB, Flickr, Netflix, Etsy, Google Type 1: Dev and Ops Collaboration (Flickr, Etsy)

: http://web.devopstopologies.com/

77[] DevOps Team Topologies3. FB, Flickr, Netflix, Etsy, Google : http://web.devopstopologies.com/Type 2: Fully Shared Ops Responsibilities (Netflix, Amazon)

78[] DevOps Team Topologies3. FB, Flickr, Netflix, Etsy, Google : http://web.devopstopologies.com/Type 7: SRE Team (Google)

44. DevOps

4. DevOps 80 ? ? ?

The Phoenix Project Lean Enterprise .

4. DevOps 81

, ? (): Lean Enterprise

4. DevOps 82DevOps Branch , , , , , 2014 , Flickr John Allspaw Etsy SVP DevOps 6 . , .. !!

831) Lean (Lean Thinking)

: http://zzino.co.kr/blog/?p=1734. DevOps

842) DevOps Practice framework : http://zzino.co.kr/blog/?p=1734. DevOps , Feedback loop (// Daily Stand-up Meeting, All Hands Meeting), Small changesets frequently deploy. , (), (Dark launches)

Risk

853)

Best Practices 4. DevOps

86Best PracticesBranch ModelGit flowGithub flow (pull requests)ReleaseDark launching, Feature toggle, Infrastructure Configuration managementChef, Docker, Moses OrganizationFunctional Cross-functional,

3) 4. DevOps

4. DevOps 87, DevOps/Lean/CD

, , ( ) 6 (, 7) All Hands Meeting ( ) , All Hands Meeting Q&A 4)

4. DevOps 88, // Daily Stand-up Meeting

/ / . . , MSA

Source Repository dependency Deploy frequency 4)

4. DevOps 89, Feature Flags, Dark Launch, Canary Analysis

Feature off, Feature Flags Trunk dependency , A/B Testing feature toggles (http://martinfowler.com/articles/feature-toggles.html)Dark Launch Canary Analysis APM(New Relic) .4)

4. DevOps 90, , SE, DBA, Patrick Debois Area 4 SE, DBA, Sprint , TV Patrick Debois Area 2 Ops , MTTR 4)

4. DevOps 91, Cross-functional Team Functional 6 , , QA cross-functional team //QA/ , QA, ( on-call)/ Docker + Apache Mesos , API SCM SVN Git Git flow . Git flow Github flow Water-Scrum-fall, Kanban 4)

4. DevOps 92 : 2 2015 : 99%2015 : 2 2015 1 : 30% 2015 : 50% Mean Time to Recover(MTTR) 5) 1

4. DevOps 93DevOps . . . , . DevOps . , innovation Automation . . , .6)

94

References95KPCB : http://www.kpcb.com/internet-trendsReactive Manifesto: http://www.reactivemanifesto.org/Facebook ( ): http://www.looah.com/article/view/983/Development and Deployment at Facebook: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6449236[ ] ?: http://swprocess.egloos.com/3009704/InfoQ Facebook Moving Fast at Scale : http://www.infoq.com/presentations/Facebook-Moving-Fast-at-Scale10+ Deploys Per Day at Flickr : https://www.youtube.com/watch?v=LdOe18KhtT410+ Deploys Per Day at Flickr : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr/Continuous Deployment at Etsy : http://www.slideshare.net/beamrider9/continuous-deployment-at-etsy-a-tale-of-two-approachesCooperation Collaboration Awareness at Etsy & Flickr: http://www.slideshare.net/jallspaw/dev-and-ops-collaboration-and-awareness-at-etsy-and-flickrNetflix Culture: Freedom & Responsibility : http://www.slideshare.net/reed2001/culture-1798664?related=1

References96Netflix API deploy ( ): https://josephkim75.wordpress.com/2013/10/23/netflix%EC%97%90%EC%84%9C-api-deploy%ED%95%98%EA%B8%B0/Continuous Delivery at Netflix : http://www.slideshare.net/robspieldenner/continuous-delivery-at-netflix?related=1Principles and Practices in Continuous Deployment at Etsy: http://www.slideshare.net/mikebrittain/principles-and-practices-in-continuous-deployment-at-etsy?related=3Continuously Deploying Culture (Scaling Culture at Etsy) : http://www.slideshare.net/mcdonnps/continuously-deploying-culture-scaling-culture-at-etsy-14588485?related=4DevOps Patterns: http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/

Further Reading97