Stop the line & Stop Feature Development practices

Embed Size (px)



Text of Stop the line & Stop Feature Development practices

  • 1. Stop the Line + Stop Feature DevelopmentLean practices for Software Product DevelopmentJuan Gutirrez Plaza| Gabor Gunyho| Rgis DauSenior Manager, Agile Practices Improvement Coach Manager, Testing Practices21-Oct-2011Protecting the irreplaceable |

2. F-Secure the company Founded in 1988,listed on NASDAQ OMXHelsinki Market cap ca 350 m,annual revenue ca 130 m(2010) Headquartered in Helsinki, 18country offices, presence inmore than 100 countries 812 people, 300+ in R&D,5 R&D offices in 4 countries(2010)2 21-Oct-2011 F-Secure Public 3. Products and ServicesWin, Mac, Linux, Android, iOS, RIM, Symbian, 20+ language versions3 21-Oct-2011 F-Secure Public 4. Customers: 200+ operator partners globally22201816141210 8 6 4 2 0Operator revenue (mEur/quarter)4 21-Oct-2011 F-Secure Public 5. About the AuthorsGabor GunyhoRgis DauJuan Gutirrez PlazaImprovement Coach with the R&D Testing practices Manager at F-Secure Currently Agile Practices ManagerGlobal Methods team at F-Secure, SDC unit, focusing on developing an at F-Secures SDC unit, focusing onexperienced Agile and Lean productagile testing culture and improve the R&D transformation of the site.development expert, contributor and quality engineering practices for Experienced coach who has helpedreviewer of books on scaling Agilecontinuously improving the R&Ddifferent teams to improve in eng.and Lean SW development standards and process practices5 21-Oct-2011 F-Secure Public 6. What is this presentation all about? No recipe Just to sharehow we did itImage source: Text source: 21-Oct-2011 F-Secure Public 7. The Project7 21-Oct-2011 F-Secure Public 8. Project setup About 12 teams (around 100 people)Mostly in Helsinki, some in Kuala Lumpur, later also one in PolandMostly feature teamsFairly mature in basic Scrum[1] and Agile engineering practicesSome experience in multi-team projects[2][3] but not on this scale Major new product, significant changes in Business model Architecture Longer-Term Planning[4][5], including new backlog tooling Goal Upgrading the demo platform every sprint8 21-Oct-2011 F-Secure Public 9. Project timeline Started: Dec 2009 This presentation counts data from March 2010 Project Split and Spin-off: March 2010 Intermediate Public Release: Sept 2010 Limited scope Stop the Line practice: since Sept 2010 (1st draft in June) Simplification of the practice: Oct 2010 STL enforcer added on: March 2011 Stop Feature Development practice: since Sept 2010 Two-week sprints: 46 so far Most resulted in a public Technology Preview release Public Release Oct 20119 21-Oct-2011 F-Secure Public 10. Stop the Line10 21-Oct-2011 F-Secure Public 11. What is it?A practice coming from Lean that is originated from theToyota Production System (TPS) [6] Stop-the-Line Work is stopped if an abnormality is found. Work continues only when problem is fixed.11 21-Oct-2011 F-Secure Public 12. What is it? The Line Line refers to production/assembly lines in automobile industry where one station takes the output of the previous station as inputImage source: 21-Oct-2011 F-Secure Public 13. What is it? - Stopping If a problem is found, anyone can pull the cord that: Stops the line from moving ahead Signals the problem to everyone on the line pointing to the station in troubleImage source: Image source: 21-Oct-2011 F-Secure Public 14. Why it happened? How to avoid it? To get all the benefits of the Stop the Line practice, a rootcause analysis is done to find what caused the problem To avoid the same problem to happen again in the future,fix the root cause too14 21-Oct-2011 F-Secure Public 15. Why to use it? Focus on quality at all times Identify recurrent (systemic) problems so they are solvedonce and for all Avoid burying problems deep in the product where itsmore difficult to fix it, potentially adding more problems ontop of identified ones Everybody is aware of the problem so anyone who canhelp, can contribute to fixing it15 21-Oct-2011 F-Secure Public 16. and for us in SW. Development? (1/3) Detection A Stop-the-Line is raised when A build is failing (e.g. it doesnt compile or pass unit testing) Automated smoke tests fail for 2 or more consecutive times A problem prevents manual testing to be performed16 21-Oct-2011 F-Secure Public 17. and for us in SW. Development? (2/3) Notification E-mail Stop-the-line radiator raises Stop-the-line flag for the line i.e., product area17 21-Oct-2011 F-Secure Public 18. and for us in SW. Development? (3/3) Fixing A team or person claims the issue using the claim functionality in radiator (since March 10) and then starts investigating it Same or other team or person starts fixing the problem Issues not claimed before next day are handled in the daily Scrum of Scrums and picked by teams Team works on stop-the-line as high priority item until it is handled When radiator no longer declares stop-the-line, team is freed from this responsibility Prevention Team worked on the stop-the-line case conducts a root cause analysis for selected cases and records findings then sends note to project mailing list18 21-Oct-2011 F-Secure Public 19. In short19 21-Oct-2011 F-Secure Public 20. In short Problem is FoundStop theLineFix the problem immediately Root causeAnalysisFix the root cause20 21-Oct-2011 F-Secure Public 21. The First Implementation Fix Rule #1 when the STL is on then do not commit new featuredevelopment code to the module that has the STL, only commit bugfixes Unfortunately not everyone was careful enough to follow this rulesystematically so some non-related commits were done whilst STLevents To prevent the human mistakes an automated tool was introduced toenforce the rule #1, the STL enforcer Hook was added in the repository that checks if the commit is done during a STL event, and if so, commits non-related with the fix, are rejected Introduced in the middle of the project (March 2010)21 21-Oct-2011 F-Secure Public 22. Bug Handling &Stop Feature Development22 21-Oct-2011 F-Secure Public 23. Old Model High level concept: Bug count usage: A bugs warehouse Measuring quality Chief QE follows, reports andescalates (no real process to react) Decision making order: Bug life cycle: Chief QE/Project bug review Store all, prioritize continuously Team bug review Only high priority bugs get fixed Team member Rest in warehouse (>95% of all) Maintenance gets all bugs thatproject did not have time to fix Maintenance never fixes these23 21-Oct-2011 F-Secure Public 24. QA (bug handling) process: Our Goal Very fast track in closing new cases Get all new cases closed in less than 4 weeks Make decision quickly, closest the actual place of work Avoid building a big batch (warehouse) of bugs by allmeans To reduce recurring effort of prioritizing a long list24 21-Oct-2011 F-Secure Public 25. New Model Reversing the old decision-making order Decision making order: Team members Team bug review Team bug review with Product Owner (+other stakeholders if needed) Project bug review25 21-Oct-2011 F-Secure Public 26. New Model Using bug count limits Stop Feature Development Work guidance: X bugs / team STOP new development in team Y bugs / project STOP the new development in whole project Bug life cycle: Extremely fast handling cycle: Fix in this sprint Fix in next sprint Trash (w/ reason category) For maintenance Yes we fix Trash (w/ reason category)26 21-Oct-2011 F-Secure Public 27. Stop Feature Development (SFD) What is it, then? An enhancement for STL Line is stopped not only when tests are not passing but when the number of non-critical bugs go over a threshold: Per team Per project Why? Care for quality even more Prioritize bug fixing over new features When? >10 cases / team. Stop Feature Development for the team >100 cases / project. Stop Feature Development for the whole project27 21-Oct-2011 F-Secure Public 28. The QA process in a nutshell Some valid bugs will get trashed, but that is OK in this process!28 21-Oct-2011 F-Secure Public 29. The QA process in a nutshell Bug is created: Choose the correct product area and prioritize the bug withyour best guess. Decision: A team decides whether the bug is fixed in this sprint, next sprintor trashed. Fix and test: A team fixes and tests their fix. Closing: All new bugs are closed in less than 4 weeks. Inside the teams: Bugs not tracked if The bug is fixed and tested by the team within the sprint The bug does not cross sprint or team boundaries29 21-Oct-2011 F-Secure Public 30. 30 21-Oct-2011 F-Secure Public 31. Statistics31 21-Oct-2011 F-Secure Public 32. STL+SFD Events vs. Releases STL + SFDSTL EnforcerMar10Aug11 Jan11Jun11 Now32 21-Oct-2011 F-Secure Public 33. Sprint 29 STL Root Cause Analysis Findings 11 STL cases was tracked for Sprint 29 Frequent root cause categories:1. Blind commits (or insufficiently tested commits)2. Code/environment changes broke Test Automations3. Large commits Actions that can prevent similar case in future:1. Ensure sufficient testing before commit, commit to branch if needed2. Monitor the radiator for smoke test results after commit (delay the commit to next day if you plan to leave office soon)3. Developers should test final builds manually more often4. Make smaller and incremental commits33 21-Oct-2011 F-Secure Public 34. TotalRoCase 9Case 8Case 7Case 6Case 5Case 4Case 3Case 2Case 1 otCase 1034C au seC at eg orCo yde311121-Oct-2011br / e n ok v i re oLa TA nmrgene tc211 cohamm ngesHaits lfim11pl emBlen inte ddfe F-Secure Public41111 co m atm ur es itsTest