Embed Size (px)
DESCRIPTION
In this analysis, we establish an economic and mathematical (queueing theory) framework to calculate how much it would cost to build the dev/test lab of your dreams. One in which developers never wait for testing resources, and every commit is tested on replicas of the production environment. Then, we contrast several different options to get there and evaluate all of them from an economic perspective. After the presentation, in order to illustrate that this is not only theory, I also gave a quick demonstration of how we at Ravello use our own technology to develop our own application. Each engineer can spin up as many instances of the production replica app as needed on demand for dev/test. We showed how we have integrated Ravello with Jenkins so that on every commit, we spin up the production replica application and run integration tests in parallel. If you have any questions, feel free to reach out. We are more than happy to discuss how this may be relevant to your development process. www.ravellosystems.com
Citation preview
The economic (in)feasibility of on-‐premise dev/test
The demand and supply mismatch of development and tes4ng resources
Navin R. Thadani, @navinthadani Ravello Systems, Inc.
Ravello Systems, Inc. | GlueCon 2013
On-‐premise development & test sucks
Sprint1
Sprint 2
Release Candidate
Product late to market and poor quality
Data center/private cloud capacity
Project Demand
Resources
Resources
Sprint1
Sprint 2
Release Candidate
Ravello Systems, Inc. | GlueCon 2013
#1 Lack of internal capacity
#2 Difficult to replicate/provision environments quickly
Ravello Systems, Inc. | GlueCon 2013
A simple applicaSon
router firewall load balancer
be1
OS
db2
OS
fe2
OS
fe1
OS
be2
OS
db1
OS
Front end Back end db 4er
q
OS
Ravello Systems, Inc. | GlueCon 2013
A simple development process
Unit test environments
Integra4on test environments
System test environments
for every commit
for every commit
batched
Ravello Systems, Inc. | GlueCon 2013
Fundamental quesSon How much capacity do you need to opSmally develop and test
your applica4on?
Ravello Systems, Inc. | GlueCon 2013
AssumpSon As far as possible, you want to
test on replicas of the produc4on environment
Ravello Systems, Inc. | GlueCon 2013
A queuing theory perspecSve
C2
C3
Test environments
C6 C7 C8
C1
Arrival rate = λ commits/hour
C5
Service rate = μ tests/hour
M-‐D-‐s queue Average Sme spent in the queue
Probability of waiSng more than 5 minutes
Overall resource uSlizaSon
Ravello Systems, Inc. | GlueCon 2013
Dev/test efficiency fronSer
Number of tesSng environments
Cost
Cost of tes4ng environments
Cost of “wai4ng” 4me
Total cost
X
TE1
Ravello Systems, Inc. | GlueCon 2013
8 person development team
5,000 commits (in 12 months)
300,000 lines of Java
Ravello Systems, Inc. | GlueCon 2013
Some more numbers
Unit tests Integra4on tests System test environments
λ
4/hr
μ
8/hr μ
5/hr Ravello Systems, Inc. | GlueCon 2013
Average wait Sme
1 2 3 4 5 6 7 8 9 10
1 2 0 0 0 0 0 0 0 0 0
2 4 0 0 0 0 0 0 0 0 0
3 9 1 0 0 0 0 0 0 0 0
4 24 1 0 0 0 0 0 0 0 0
5 oo 2 0 0 0 0 0 0 0 0
Number of integraSon tesSng environments
Commit rate
P0 =1/λµ( )
n
n!+λµ( )
S
S!1
1−λ / Sµ"
#$
%
&'
n=0
S−1
∑)
*
++++
,
-
.
.
.
.
Lq =λ /µ( )S λ / Sµ( )S! 1-λ /Sµ( )2
P0 Wq =Lqλ
Or simple simulaSon h_ps://github.com/geertj
Dust off your text books ;)
Ravello Systems, Inc. | GlueCon 2013
Dev/test efficiency fronSer (λ=4/hr)
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1 2 3 4 5 6 7 8 9 10 11 12
OpSmal point • 2 x central UT • 2 x IT • 1 x ST • 2 x manual QA
Number of tes4ng environments
Cost
Cost of “wai4ng” Capex + opex
Total cost fron4er
Ravello Systems, Inc. | GlueCon 2013
Reality bites (badly) Co
mmits per hou
r
0
5
10
15
20
25
30
10/2/12 11/2/12 12/2/12 1/2/13 2/2/13 3/2/13 4/2/13
Average 4 commits/ hour
Peak 27 commits/ hour
Ravello Systems, Inc. | GlueCon 2013
Dev/test efficiency fronSer
Number of tesSng environments
Cost
Cost of tes4ng environments Cost of
“wai4ng” 4me
For commit rate L1
For commit rate L2
TE1 TE2
X1
X2
Ravello Systems, Inc. | GlueCon 2013
Average wait Sme
Number of integraSon tesSng environments 1 2 3 4 5 6 7 8 9 10 1 2 0 0 0 0 0 0 0 0 0 2 4 0 0 0 0 0 0 0 0 0 3 9 1 0 0 0 0 0 0 0 0 4 24 1 0 0 0 0 0 0 0 0 5 100 2 0 0 0 0 0 0 0 0 6 100 4 1 0 0 0 0 0 0 0 7 100 6 1 0 0 0 0 0 0 0 8 100 11 1 0 0 0 0 0 0 0 9 100 26 2 0 0 0 0 0 0 0 10 100 100 3 1 0 0 0 0 0 0 11 100 100 4 1 0 0 0 0 0 0 12 100 100 7 1 0 0 0 0 0 0 13 100 100 12 2 0 0 0 0 0 0 14 100 100 26 2 1 0 0 0 0 0 15 100 100 100 3 1 0 0 0 0 0 16 100 100 100 5 1 0 0 0 0 0 17 100 100 100 7 1 0 0 0 0 0 18 100 100 100 12 2 1 0 0 0 0 19 100 100 100 27 3 1 0 0 0 0 20 100 100 100 100 3 1 0 0 0 0 21 100 100 100 100 5 1 0 0 0 0 22 100 100 100 100 7 2 1 0 0 0 23 100 100 100 100 13 2 1 0 0 0 24 100 100 100 100 26 3 1 0 0 0 25 100 100 100 100 100 4 1 0 0 0 26 100 100 100 100 100 5 1 0 0 0 27 100 100 100 100 100 8 2 1 0 0 28 100 100 100 100 100 13 2 1 0 0 29 100 100 100 100 100 28 3 1 0 0 30 100 100 100 100 100 100 4 1 0 0
Commit rate
Ravello Systems, Inc. | GlueCon 2013
Dev/test efficiency fronSer (λ=27/hr)
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1 2 3 4 5 6 7 8 9 10 11 12
OpSmal point • 5 x central UT • 8 x IT • 1 x ST • 3 x manual QA
Number of tes4ng environments
Cost Cost of “wai4ng”
Capex + Opex
Total cost fron4er
Ravello Systems, Inc. | GlueCon 2013
$100K – substandard lab WaiSng, inefficiency, resource
contenSon …
$250K – good lab service Efficient development
Hidden cost >>$1m (inefficiency)
Average uSlizaSon (<5%)
Ravello Systems, Inc. | GlueCon 2013
A new fronSer?
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1 2 3 4 5 6 7 8 9 10 11 12
Public Cloud
25x lower cost
Internal data center
Number of tes4ng environments
Ravello Systems, Inc. | GlueCon 2013
Reality check
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1 2 3 4 5 6 7 8 9 10 11 12
Change applicaSon to run on-‐premise (producSon) and cloud (development and test)
? Public Cloud
Internal data center
Number of tes4ng environments
Ravello Systems, Inc. | GlueCon 2013
Your opSons
1x
>>
2.5x
0.1x <0.1x
Wai4ng 4me
Capex + Opex
Modify your app to run in the cloud and internal data
center + automa4on
Internal sub opSmal lab
Internal opSmal lab
“Migrate” to public cloud
How we have solved it
Ravello Systems, Inc. | GlueCon 2013
DemonstraSon
Ravello Systems, Inc. | GlueCon 2013
Ravello Systems, Inc. | GlueCon 2013
