DevOps: The Future is Already Here — It’s Just Unevenly Distributed

DevOps: The Future is Already Here — It’s Just Unevenly Distributed

Damon Edwards @damonedwards

@damonedwards

Damon Edwards

DevOps Consulting

Automation Design

Support & Services

devopscafe.org

dev2ops.org

The history of DevOps is important…

A B

A B

OpsDev

“Agile Infrastructure”




June 23, 2009

"10+ Deploys Per Day: Dev and Ops Cooperation at Flickr”

June 23, 2009


June 23, 2009


June 23, 2009


June 23, 2009

October 30 - 31, 2009: Ghent, Belgium

DevOctober 30 - 31, 2009: Ghent, Belgium


Ops


OpsDays


OpsDays

*and many more!


# OpsDays

*and many more!

DevOps DaysGhent 2009

DevOps DaysGhent 2009Sydney 2010


Mountain View 2010


Mountain View 2010

+49 more as of Jan 2015devopsdays.org

The Rise of a New IT Operations Support Model

By 2015, DevOps will evolve from a niche strategy employed by large cloud providers into a mainstream strategy employed

by 20% of Global 2000 organizations

Why DevOps will emerge:

!DevOps is not usually driven from

Why DevOps will not emerge:

!Cultural changes are the hardest to

by 20% of Global 2000 organizations.

!DevOps is not usually driven from the top down and, thus, may be more easily accepted by IT operations teams.

!Cultural changes are the hardest to implement, and DevOps requires a significant rethinking of IT operations conventional wisdom.

!ITIL and other best practices frameworks are acknowledged to have not delivered on their goals, enabling IT organizations to look for

!There is a large body of work with respect to ITIL and other best practices frameworks that is already accepted within the industry enabling IT organizations to look for

new models.

!The growing interest in tools such as Chef, Puppet, etc., will help

accepted within the industry.

!Open source (OSS) management tools, which are more aligned with this approach, have not seen pp p

stimulate demand for OSS-based management

ppsignificant enterprise market share traction.

March 18, 2011

The Rise of a New IT Operations Support Model

By 2015, DevOps will evolve from a niche strategy employed by large cloud providers into a mainstream strategy employed

by 20% of Global 2000 organizations

Why DevOps will emerge:

!DevOps is not usually driven from

Why DevOps will not emerge:

!Cultural changes are the hardest to

by 20% of Global 2000 organizations.

!DevOps is not usually driven from the top down and, thus, may be more easily accepted by IT operations teams.

!Cultural changes are the hardest to implement, and DevOps requires a significant rethinking of IT operations conventional wisdom.

!ITIL and other best practices frameworks are acknowledged to have not delivered on their goals, enabling IT organizations to look for

!There is a large body of work with respect to ITIL and other best practices frameworks that is already accepted within the industry enabling IT organizations to look for

new models.

!The growing interest in tools such as Chef, Puppet, etc., will help

accepted within the industry.

!Open source (OSS) management tools, which are more aligned with this approach, have not seen pp p

stimulate demand for OSS-based management

ppsignificant enterprise market share traction.

March 18, 2011

Next ➔ October 19-21, 2015 in San Franciscodevopsenterprise.io

2014 State of DevOps Survey

9,200+ Respondents from 110 countries, across all industries

2014 State of DevOps Survey

9,200+ Respondents from 110 countries, across all industries

Why is the history important?

Why is the history important?It reminds us that DevOps is...


• from practitioners, by practitioners

It reminds us that DevOps is...



• not a “thing” (product, methodology, specification, job title)





• an experience-based movement






• decentralized and open to all






• decentralized and open to all


Biggest source of tensions in community is not understanding this!

What have we learned?

DevW

all of

Con

fusi

on

Ops

At first we forgot the context!

OpsDev

We are in business to give the customer... • What they want • When they want it • At the lowest cost possible

OpsDev

DevOps Problems!

Ah-ha!

OpsDev Remove

Shorten

Lead Time

Feedback

Ah-ha!

OpsDev Remove

Shorten

Lead Time

Feedback

• “My organization is stuck!… and I don’t know why”

• “Business pressuring to route around me”

• “I have no real visibility” • “Org has low confidence in

ability to deliver” • “Usual coping mechanisms

aren’t working” • “Don’t bring me more tools!”

• “My job is painful!.. and I know why”

• “I’m always waiting for someone or being undermined by someone else”

• “If you just let me do it myself…” • “We should automate everything

first” • “Using new tools like company x

will fix our problems!”

Executives Teams

“I think DevOps can help me” “I think DevOps can help me”











Executives Teams












Executives Teams












Executives Teams


Organizational AnalysisProcess RedesignManagement Coaching











Executives Teams



Software ToolsDesign Patterns and MethodologiesTraining











Executives Teams














Executives Teams




Execs

Teams

"I want to transform my organization"

"I want to transform how I work"

DevOps?

Organizational Transformation

Execs

"I want to transform my organization"

Predictability Short Time-to-Market High Quality Operations

My silo works just fine

It’s out of my control

Not the way we’ve always

done it

That just won’t work

here

Too busy running the business

What is important?

Didn’t work last time, why

now?

Where do I start?

Improved Time to Market & Quality

Quicker Feedbackfrom Users

BusinessIdea

RunningServices

Dev Ops

DevOps and Agile Ops

Security

Auditing

Compliance

Availability

Ops is in a particularly tough spot

Success: an organization that learns to fix itself

Focus is on: ● Seeing end-to-end ● Fast feedback

● Eliminating waiting, rework ● Catching problems early

Predictability Short Time-to-Market High Quality Operations

If you stand back far enough…

Silos are the #1 enemy of throughput and quality

Dev Release Ops Business





Handoff!

Handoff!

Handoff!



Application Knowledge

Handoff!

Handoff!

Handoff!




Operational Knowledge

Handoff!

Handoff!

Handoff!





Business Intent

Handoff!

Handoff!

Handoff!





Business Intent

Handoff!

Handoff!

Handoff!

Ownership but limited Accountability





Business Intent

Handoff!

Handoff!

Handoff!

Ownership but limited Accountability

Accountability but no Ownership

Organizational alignment is the key to overcoming silos


• 1000’s of small decisions made daily • How do we harness that?



• Top down is brittle • Conditions, people, tools are always

changing... how do we handle that?




changing... how do we handle that?• Alignment:

When you know that different individuals would independently see a set of conditions and arrive at the same decision that is directionally correct to achieve the company’s goals




changing... how do we handle that?• Alignment:

When you know that different individuals would independently see a set of conditions and arrive at the same decision that is directionally correct to achieve the company’s goals

• Startups have it; Enterprises lose it

Value Stream Mapping


Process1


Process1

?PD

TS

W

EP M

MM

TS

PDM

W

DWastes2


Process1

?PD

TS

W

EP M

MM

TS

PDM

W

DWastes2

?PD

TS

W

EP M

MM

TS

PDM

W

D

Ops involved

earlyUnify

ticket/reqtools

Write service tests in

dev

Automate from spec

SystemVerification

Tests Move into Dev

Self-Service

ACLsVerify from definition of done

Counter-measures

3

Core business service delivery process Current state value stream map

DevelopmentEnvironments

ReleaseBuild and deploy

QAEnvironments

PerformanceEnvironment

StagingEnvironment

ProductionEnvironment

Deploy to Performance

Web Assets, flows and

wireframes

Delta features and

fixesSet of service

branches

Documented test plans and cases

Request QA build

Daily Scrum meeting

Daily Scrum meeting

Deployment Meeting Business Systems

Support

Stan Walters

Change Control

Project Manager

Erica Switzer

Change request

Business Analyst

Erica Switzer

Steering Committee

Erica Switzer

Full Projects / Expedites

Estimates and

requirements

Technical Specification

Deployment Document

Change Control Forms

RT tickets

Jira issues

Jira issues

Tested packages

Development

Joe Harbaugh

L/T 8 weeksP/TH/CS/R

4 weeks525%

DevelopmentBuild and deploy

Joe Harbaugh

L/T 2 daysP/TH/CS/R

4 hours110%

Development Integration testing

Joe Harbaugh

L/T 1 dayP/TH/CS/R

1 day595%

Performance testing

Deploy toProduction

Build test plan

Raj Lee

L/T 5 daysP/TH/CS/R

4 days110%

Build test cases

Raj Lee


8 days220%

Fernando Gomez

L/T 8 hoursP/TH/CS/R

7 hours12%

Acceptance testing

Raj Lee

L/T 5 daysP/TH/CS/R

5 hours210%

Feature testing

Raj Lee


2.5 weeks420%

Allen Cannata

L/T 1 weekP/TH/CS/R

3 days166%

RT tickets

Deploy to Staging

Frances Middleton

L/T 6 daysP/TH/CS/R

90 minutes366%

Staging testing

Frances Middleton

L/T 1 weekP/TH/CS/R

3.5 days466%

Allen Cannata

L/T 9 hoursP/TH/CS/R

8 hours666%

Frances Middleton

L/T 6 daysP/TH/CS/R

90 minutes366%

DDx2

WD

PD

TSx5 Dx2 Wx2 PDx4

EP

DPDx3 Mx8

DWx2

PDx2

What should we be looking for?• Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding) • “Snowflake” servers and environments • Manual dependency management • Ad-hoc or unstructured artifact management

People

Process

Tools

Align work to customer value

Dev & QA Activity

Rel. / SRE Activity

Ops Activity

Business Activity

Dev & QA Activity

Rel. / SRE Activity

Ops Activity

Business Activity

Customer / Product Aligned Teams




Dev & QA Activity

Rel. / SRE Activity

Ops Activity

Business Activity




Environments as a

Service

QA as a

Service

Insights as a

Service

Security as a

Service

Environments Team

QA Team

Monitoring & Metrics

TeamSecurity

Team

What about cross-cutting concerns?

Process and Tooling Transformation

Teams

"I want to transform how I work"

Dev & QA Activity

Rel. / SRE Activity

Ops Activity

Business Activity




Environments as a

Service

QA as a

Service

Insights as a

Service

Security as a

Service

Environments Team

QA Team

Monitoring & Metrics

TeamSecurity

Team


Code

Source Repo

Config

CI Server

Package Repo

Operations Console

Shell Powershell

Pre-Production Environments

Shell Powershell

Production Environment

Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Drive all changes through a SDLC

Tests

CodeTests

Source Repo

Config Env SpecRun- book

Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE


Code

Source Repo


Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration

Tests


Code

Versioned Release

Tests

Source Repo


Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration


Code

Versioned Release

Tests

Source Repo


Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration

Dev Ops *

Execute Operations Procedures


Code

Versioned Release

Tests

Source Repo


Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration

Dev Ops *


Drive all changes through a SDLCSame People!!

Code

Versioned Release

Tests

Source Repo


Auto-mation

CI Server

Package Repo

Operations Console

Shell Powershell


Shell Powershell


Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration

Dev Ops *


SERVICEDrive all changes through a SDLC


(PO • Dev • Test • SRE)Cross Functional Delivery Team

CodeTests

SourceRepo

Config EnvSpecRun-book

Auto-mation

CIServer

PackageRepo

OperationsConsole

ShellPowershell

Pre-ProductionEnvironmentsPackages

SOURCE

QA Security EnvironmentsMonitoring

---

Metrics



CodeTests

SourceRepo


Auto-mation

CIServer

PackageRepo

OperationsConsole

ShellPowershell


SOURCE


---

Metrics

QA as aService

Security as aService

Metrics as aService

Env. as aService



CodeTests

SourceRepo


Auto-mation

CIServer

PackageRepo

OperationsConsole

ShellPowershell


SOURCE

pull pull pull pull


---

Metrics

QA as aService

Security as aService

Metrics as aService

Env. as aService

Be a service provider

pull

Cross-Cutting Concern X

✓ Standardized offerings

✓ Pulled by users (not pushed)

✓ On-demand and self-service

✓ Implementation knowledge not necessary for normal use

✓ Provider spends their time building service and coaching users

X as a Service

Be a service provider

pull


✓ Standardized offerings

✓ Pulled by users (not pushed)

✓ On-demand and self-service

✓ Implementation knowledge not necessary for normal use

✓ Provider spends their time building service and coaching users

X as a Service

Google: “Cloud Operations at Netflix”“Actionable Metrics Netflix”

Roy Rapoport Mark Maun

In this room at 1:30pm!

How to start being a service providerpull


X as a Service



X as a Service

Define your offerings1



X as a Service




X as a Service


Tame the tool sprawl2



X as a Service





X as a Service



Setup self-service interfaces3



X as a Service






X as a Service




Setup secure access4



X as a Service







X as a Service





Plug: Give Rundeck a try --> rundeck.org

What’s Next?

Microservices

Any organization that designs a system (defined broadly) will produce a design whose

structure is a copy of the organization's communication structure.

Melvyn Conway, 1967

Conway’s Law:

Martin Fowler

Martin Fowler

Martin Fowler

Go read: http://martinfowler.com/articles/microservices.html

Martin Fowler

http://martinfowler.com/articles/microservices.html

Adrian Cockcroft

Adrian Cockcroft

Adrian Cockcroft

Google: “Adrian Cockcroft Microservices”

Adrian Cockcroft

The Network

The pain is legendary“Our cloud is awesome… VMs in minutes… but then it takes two days to get the VM connected to anything”

“But the ACL policy worked yesterday”

“Firewall rules only get changed on Thursdays”

“I didn’t know the load would spike that way”

“That’ll take 4 tickets and an act of god”

“If we don’t predict traffic patterns right our app is hosed and there is no quick recovery”

“Only 3 people here know how to do that”

“After all that it turns out it was a DNS issue”

The network is changing

1990 2015

95% traffic left the DC New architectures: 5% traffic leaves the DC (Facebook: only 2%!)

John Willis

The demands are changing

• Disaggregation of Hardware/Software

• Web Scale

• Private Cloud Computing

• Containerization • Software Eating the Network• DevOps

Faster pace of change! Decentralized control!

John Willis

Expect acceleration

The Network Iceberg

Data Gravity Micro ServicesWho?How?

John Willis

Expect acceleration

The Network Iceberg

Data Gravity Micro ServicesWho?How?

John WillisIn this room at

3:00pm! @botchagalupe

Let’s close with… you.

What can you do starting now?

Does your solution….

DevOps Litmus Test

Reduce cycle time AND improve quality?


DevOps Litmus Test


Eliminate handoffs or reduce the friction of those handoffs that can't be eliminated?


DevOps Litmus Test



Improve tool-to-tool artifact flow and eliminate manual information flow?


DevOps Litmus Test




Eliminate manually-fulfilled request queues and other sources of waiting?


DevOps Litmus Test





Improve awareness and understanding of the current state and desired state of the end-to-end system?


DevOps Litmus Test





Improve awareness and understanding of the current state and desired state of the end-to-end system?


DevOps Litmus Test

✓✓

✓

✓

✓

@damonedwards

Damon Edwards

Technology

DevOps: The Future is Already Here — It’s Just Unevenly Distributed