View
1.552
Download
0
Category
Preview:
DESCRIPTION
F-Secure
Citation preview
Protecting the irreplaceable | f-secure.com
Stop the Line + Stop Feature Development Lean practices for Software Product Development
Juan Gutiérrez Plaza | Gabor Gunyho | Régis Déau Senior Manager, Agile Practices Improvement Coach Manager, Testing Practices
21-Oct-2011
F-Secure – the company
• Founded in 1988,
listed on NASDAQ OMX
Helsinki
• Market cap ca 350 m€,
annual revenue ca 130 m€
(2010)
• Headquartered in Helsinki, 18
country offices, presence in
more than 100 countries
• 812 people, 300+ in R&D,
5 R&D offices in 4 countries
(2010)
© F-Secure Public 21-Oct-2011 2
Products and Services
© F-Secure Public 21-Oct-2011 3
Win, Mac, Linux, Android, iOS, RIM, Symbian, 20+ language versions
Customers: 200+ operator partners globally
© F-Secure Public 21-Oct-2011 4
0
2
4
6
8
10
12
14
16
18
20
22
Operator revenue (mEur/quarter)
About the Authors
© F-Secure Public 5
Gabor Gunyho Improvement Coach with the “R&D
Global Methods” team at F-Secure,
experienced Agile and Lean product
development expert, contributor and
reviewer of books on scaling Agile
and Lean SW development
Juan Gutiérrez Plaza Currently „Agile Practices Manager‟
at F-Secure‟s SDC unit, focusing on
the R&D transformation of the site.
Experienced coach who has helped
different teams to improve in eng.
and process practices
Régis Déau Testing practices Manager at F-Secure
SDC unit, focusing on developing an
agile testing culture and improve
quality engineering practices for
continuously improving the R&D
standards
21-Oct-2011
What is this presentation all about?
• No “recipe”
• Just to share
how we did it
© F-Secure Public 21-Oct-2011 6
Text source: http://easteuropeanfood.about.com/od/hungariansoups/r/gulyasleves.htm Image source: http://www.clker.com/clipart-9889.html
The Project
© F-Secure Public 21-Oct-2011 7
Project setup
• About 12 teams (around 100 people)
• Mostly in Helsinki, some in Kuala Lumpur, later also one in Poland
• Mostly feature teams
• Fairly mature in basic Scrum[1] and Agile engineering practices
• Some experience in multi-team projects[2][3] but not on this scale
• Major new product, significant changes in
• Business model
• Architecture
• Longer-Term Planning[4][5], including new backlog tooling
• Goal
• Upgrading the demo platform every sprint
© F-Secure Public 21-Oct-2011 8
Project timeline
• Started: Dec 2009
• This presentation counts data from March 2010
• Project Split and Spin-off: March 2010
• Intermediate Public Release: Sept 2010
• Limited scope
• Stop the Line practice: since Sept 2010 (1st draft in June)
• Simplification of the practice: Oct 2010
• STL enforcer added on: March 2011
• Stop Feature Development practice: since Sept 2010
• Two-week sprints: 46 so far
• Most resulted in a public Technology Preview release
• Public Release Oct 2011
© F-Secure Public 21-Oct-2011 9
Stop the Line
© F-Secure Public 21-Oct-2011 10
What is it?
A practice coming from Lean that is originated from the
Toyota Production System (TPS) [6]
Stop-the-Line
Work is stopped if an abnormality is found.
Work continues only when problem is fixed.
© F-Secure Public 21-Oct-2011 11
What is it? – The Line
“Line” refers to production/assembly lines in automobile industry where
one station takes the output of the previous station as input
© F-Secure Public 21-Oct-2011 12
Image source: http://www.fourwheeler.com/techarticles/body/129_0703_toyota_assembly_factory/photo_02.html
What is it? - Stopping
• If a problem is found, anyone can “pull the cord” that:
• Stops the line from moving ahead
• Signals the problem to everyone on the line pointing
to the station in trouble
© F-Secure Public 21-Oct-2011 13
Image source: http://www.resourcesystemsconsulting.com/blog/archives/78
Image source: http://www.flickr.com/photos/9516941@N08/3334795306/
Why it happened? How to avoid it?
• To get all the benefits of the Stop the Line practice, a root
cause analysis is done to find what caused the problem
• To avoid the same problem to happen again in the future,
fix the root cause too
© F-Secure Public 21-Oct-2011 14
Why to use it?
• Focus on quality at all times
• Identify recurrent (systemic) problems so they are solved
once and for all
• Avoid burying problems deep in the product where it‟s
more difficult to fix it, potentially adding more problems on
top of identified ones
• Everybody is aware of the problem so anyone who can
help, can contribute to fixing it
© F-Secure Public 21-Oct-2011 15
… and for us in SW. Development? (1/3)
• Detection
• A Stop-the-Line is raised when
• A build is failing (e.g. it doesn't compile or pass unit testing)
• Automated smoke tests fail for 2 or more consecutive times
• A problem prevents manual testing to be performed
© F-Secure Public 21-Oct-2011 16
… and for us in SW. Development? (2/3)
• Notification
• Stop-the-line radiator raises Stop-the-line flag for the “line” i.e.,
product area
© F-Secure Public 21-Oct-2011 17
… and for us in SW. Development? (3/3)
• Fixing
• A team or person claims the issue using the claim functionality in radiator (since March „10) and then starts investigating it
• Same or other team or person starts fixing the problem
• Issues not claimed before next day are handled in the daily Scrum of Scrums and picked by teams
• Team works on stop-the-line as high priority item until it is handled
• When radiator no longer declares stop-the-line, team is freed from this responsibility
• Prevention
• Team worked on the stop-the-line case conducts a root cause analysis for selected cases and records findings then sends note to project mailing list
© F-Secure Public 21-Oct-2011 18
In short…
© F-Secure Public 21-Oct-2011 19
In short…
© F-Secure Public 21-Oct-2011 20
Problem is Found
Stop the Line
Fix the problem
immediately
Root cause Analysis
Fix the root cause
The First Implementation Fix
• Rule #1 when the STL is on then do not commit new feature
development code to the module that has the STL, only commit bug
fixes
• Unfortunately not everyone was careful enough to follow this rule
systematically so some non-related commits were done whilst STL
events
• To prevent the human mistakes an automated tool was introduced to
enforce the rule #1, the “STL enforcer”
• Hook was added in the repository that checks if the commit is done
during a STL event, and if so, commits non-related with the fix, are
rejected
• Introduced in the middle of the project (March 2010)
© F-Secure Public 21-Oct-2011 21
Bug Handling & Stop Feature Development
© F-Secure Public 21-Oct-2011 22
Old Model
• High level concept:
• A bugs‟ warehouse
• Decision making order:
• Chief QE/Project bug review
• Team bug review
• Team member
© F-Secure Public 21-Oct-2011 23
• Bug count usage:
• Measuring quality
• Chief QE follows, reports and escalates (no real process to react)
• Bug life cycle:
• Store all, prioritize continuously
• Only high priority bugs get fixed
• Rest in warehouse (>95% of all)
• Maintenance gets all bugs that project did not have time to fix
• Maintenance never fixes these
QA (bug handling) process: Our Goal
• Very fast track in closing new cases
• Get all new cases closed in less than 4 weeks
• Make decision quickly, closest the actual place of work
• Avoid building a big batch (warehouse) of bugs by all
means
• To reduce recurring effort of prioritizing a long list
© F-Secure Public 21-Oct-2011 24
New Model
• Reversing the old decision-making order
• Decision making order:
• Team members
• Team bug review
• Team bug review with Product Owner (+other stakeholders if needed)
• Project bug review
© F-Secure Public 21-Oct-2011 25
New Model
• Using bug count limits – Stop Feature Development
• Work guidance:
• X bugs / team STOP new development in team
• Y bugs / project STOP the new development in whole project
• Bug life cycle:
• Extremely fast handling cycle:
• Fix in this sprint
• Fix in next sprint
• Trash (w/ reason category)
• For maintenance
• Yes we fix
• Trash (w/ reason category)
© F-Secure Public 21-Oct-2011 26
Stop Feature Development (SFD)
• What is it, then?
• An enhancement for STL
• Line is stopped not only when tests are not passing but when the number
of non-critical bugs go over a threshold:
• Per team
• Per project
• Why?
• Care for quality even more
• Prioritize bug fixing over new features
• When?
• >10 cases / team. Stop Feature Development for the team
• >100 cases / project. Stop Feature Development for the whole project
© F-Secure Public 21-Oct-2011 27
The QA process in a nutshell
© F-Secure Public 21-Oct-2011 28
Some valid bugs will
get trashed, but that
is OK in this process!
The QA process in a nutshell
• Bug is created: Choose the correct product area and prioritize the bug with
your best guess.
• Decision: A team decides whether the bug is fixed in this sprint, next sprint
or trashed.
• Fix and test: A team fixes and tests their fix.
• Closing: All new bugs are closed in less than 4 weeks.
• Inside the teams:
• Bugs not tracked if…
• The bug is fixed and tested by the team within the sprint
• The bug does not cross sprint or team boundaries
© F-Secure Public 21-Oct-2011 29
© F-Secure Public 21-Oct-2011 30
Statistics
© F-Secure Public 21-Oct-2011 31
STL+SFD Events vs. Releases
© F-Secure Public 21-Oct-2011 32
Mar‘10 Jan‘11 Now Aug‘11 Jun‘11
STL Enforcer STL + SFD
Sprint 29 – STL Root Cause Analysis Findings
• 11 STL cases was tracked for Sprint 29
• Frequent root cause categories:
1. Blind commits (or insufficiently tested commits)
2. Code/environment changes broke Test Automations
3. Large commits
• Actions that can prevent similar case in future:
1. Ensure sufficient testing before commit, commit to branch if needed
2. Monitor the radiator for smoke test results after commit (delay the commit to
next day if you plan to leave office soon)
3. Developers should test final builds manually more often
4. Make smaller and incremental commits
© F-Secure Public 21-Oct-2011 33
Detailed case breakdown
© F-Secure Public 21-Oct-2011 34
Root Cau
se C
atego
ry
Code/env
ironm
ent c
hange
s
broke
TA
Larg
e co
mm
its
Half im
plem
ented fe
ature
s
Blind co
mm
its
Test
Envir
onment C
hange
s
(IT)
How ca
n this
be pre
vente
d in
futu
re?
Hard /
Not worth
the
preve
ntion e
ffort
Monito
r Rad
iator
Smal
ler &
incr
emen
tal
com
mits
Fast
er TA
Har
dwar
e to
shorte
n TA cy
cle
Comple
te fe
ature
imple
menta
tion
Develo
pers sh
ould te
st R
ed
builds m
ore o
ften
Ensu
re su
fficie
nt te
stin
g
before
com
mit
Educa
te d
evelo
per on w
hat
to m
onitor
Case 1 1 1
Case 2 1 1 1 1 1
Case 3 1 1
Case 4 1 1 1 1 1 1
Case 5 1 1
Case 6 1 1
Case 7 1 1 1
Case 8 1 1 1 1
Case 9 1 1
Case 10 1 1
Total 3 2 1 4 1 3 3 2 1 1 3 5 1
Conclusions
© F-Secure Public 21-Oct-2011 35
Conclusions
• Overall quality of the product improved
• Number of STL events decreased by time
• STL enforcer helped to avoid some mistakes
• Not releasing every two weeks BECAME AN EXCEPTION and not a rule
• New bug handling process helped on focusing on important bugs
• SFD keeps the level of open bugs in a manageable number
• After a settle down period, these practices changed the mindset of the people
to be more quality focus
• Next step:
• STL and SFD are “brakes” to avoid accidents, now we are learning how to
drive at high speed safely (avoid making so many bugs)
© F-Secure Public 21-Oct-2011 36
Questions?
© F-Secure Public 21-Oct-2011 37
Acknowledgements
The authors would like to thank the whole project team and the whole R&D
organization of F-Secure and its management for making this presentation
possible and support the data collection and publishing
We‟d like to thank especially
• Petri Kuikka
• Risto Kumpulainen
• Pekka Kiviniemi
• Ferrix Hovi
for their contribution in the bug handling process, Continuous Integration and
Test Automation system and radiator design and implementation and data
visualization
© F-Secure Public 21-Oct-2011 38
Working at F-Secure - Keeping it fun!
• Coding Dojos
• Team building events
• Friday evening b**rs
• ‟Hamburger‟ nights
• Sports club: Football, squash, badminton…
• Trainings and book club
• Silent and… Musical hours ;-)
• Low hierarchy, small teams
• Innovation day and demo once per month
• Large and small company benefits
© F-Secure Public 21-Oct-2011 39
How does it look?
© F-Secure Public 21-Oct-2011 40
Wanna work for F-Secure?
Would you like to work as…
• C++, JavaScript and Python developer?
• Quality Engineer?
• Scrum Master?
• Product Owner?
We are looking for GOOD PEOPLE!
If you want to move to Bordeaux, LET ME KNOW!
If you are more interested in Helsinki, Oulu, San José or Kuala Lumpur,
check the position opens at:
http://www.f-secure.com/en_EMEA-Corp/careers/open-positions/
© F-Secure Public 21-Oct-2011 41
References
[1] Schwaber, K., Beedle, M.: “Agile Software Development with Scrum”, Prentice Hall (2001)
[2] Larman, C., Vodde, B.: “Scaling Lean & Agile Development: Thinking and Organizational Tools for
Large-Scale Scrum”, Addison-Wesley Professional (2008)
[3] Larman, C., Vodde, B.: “Practices for Scaling Lean & Agile Development: Large, Multisite, and
Offshore Product Development with Large-Scale Scrum”, Addison-Wesley Professional (2010)
[4] Leffingwell, D.: “Scaling Software Agility: Best Practices for Large Enterprises”, Addison-Wesley
Professional (2007)
[5] Leffingwell, D.: “Agile Software Requirements: Lean Requirements Practices for Teams, Programs,
and the Enterprise”, Addison-Wesley Professional (2011)
[6] Womack, J.P., Jones, D.T., Roos, D.: The machine that changed the world (1990, 2007)
[7] Poppendieck, M., Poppendieck, T.: “Implementing Lean Software Development: from Concept to
Cash”, Addison-Wesley (2007)
© F-Secure Public 21-Oct-2011 42
Contact Information juan.gutierrez@f-secure.com gabor.gunyho@f-secure.com regis.deau@f-secure.com
The authors did their best to attribute the authors of texts and images, and to recognize any copyrights, see more
details of copyrights, license terms and conditions for each source under the reference link provided. If you think
that anything in this material should be changed, added or removed, please contact the authors at the addresses
above
© F-Secure Public 21-Oct-2011 43
http://creativecommons.org/licenses/by-nd/3.0/
Recommended