Software quality - no more bugs!

1 Copyright © 2012 Retalix ||

NO MORE BUGS!By: Arnon Axelrod

SOFTWARE QUALITY

2 Copyright © 2012 Retalix |

About me


About you• Name• Company• Role• Number of people in the project• Average release cycle• % of man-months dedicated to bugs finding and

fixing• From 1 to 10 – how are you satisfied with the

quality of the software you deliver?


Definition of Done


What is quality?


The quality pyramid• Intuitive, sleek, “sexy”,

enjoyable…

• User friendly, HelpUseful

• Stable• Supportable• Secure

Reliable

• Useful features• Solves a user’s

problemUsable

Appealing


The quality pyramid – “reality”• Intuitive, sleek, “sexy”,

enjoyable…

• User friendly, HelpUseful• Stable• Supportable• Secure

Reliable


problemUsable

Appealing


The quality pyramid• Intuitive, sleek, “sexy”,

enjoyable…

• User friendly, HelpUseful

• Stable• Supportable• Secure

Reliable


problemUsable

Appealing

SbE / BDD / ATDD


Bugs


What is a bug?

Can a computer err?


What is a bug?Wikipedia:“A software bug is an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways”


Why there are bugs?• Expectations

gaps• Miscommunicatio

n• Coding errors• Miscommunication

between teams or team members

• Misunderstanding the business needs

• Incomplete or incorrect

definition of the expected results

• Complexity• Misuse of tools• Bugs in tools


Bugs - conclusion

By improving the communication and expectation coordination – we can prevent

many bugs


Problems with our current practices


Waterfall


Is Agile the answer?


Automated Testing


Unit Testing


Introduction to Specification by Example


The Agile ManifestoWe are uncovering better ways of developingsoftware by doing it and helping others do it.Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentation

Customer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items onthe right, we value the items on the left more.


Different names• Specification by Example• ATDD (Acceptance Test Driven Development)• Agile Acceptance Testing• Example Driven Development• Executable Specifications• BDD (Behavior Driven Development)


How does it work?

Specifications

Tests

Examples of user

scenarios


User Stories“are a promise for a future conversation”

– Ron Jeffries As a <role/persona>I want <goal/desire> so that <benefit>.

In order to <receive benefit>as a <role/persona>I want <goal/desire>

As a user closing the appI want to be prompted to save my workso that I won’t loose all I’ve done so far

In order not to loose all I’ve done so faras a User closing the appI want to be prompted to save my work


Deriving the Solution from the Goal


Specifying Collaboratively


Specifying Collaboratively• Attendees: BA, Developer, Tester• Looking for simplest solution

– break the task to small pieces that add value – work incrementally

• Focus on Simple Examples of User Scenarios


Specifying Collaboratively – DoD

How will the user use the solution? (examples!)

How can we demonstrate it?

How will we test it?

Definition of Done!


Automating the examples• Test should be as close as possible to the DoD

– Certain tools (e.g. FitNesse) makes it easy!

• Test must fail before the code is developed…


Developing the User Story Develop the fixtures first (follow the errors) Write the minimal code to pass the tests

Don’t add functionality that wasn’t requested! Apply TDD/Unit tests During and After development run all tests to make

sure nothing is broken Refactor as needed (remove duplication, give

meaningful names, etc.) Emergent design Make sure again that nothing has broken


Running the tests Locally, before check-in

Automatically (CI, Nightly)

Provide transparency!

Should always be green!


Living documentationAutomatic Specifications = Living Documentation!

Benefits:• A single, central source of truth• Provides visibility about the state of the project• Easy to identify contradicting requirements


Living documentationAutomatic Specifications = Living Documentation!

How?• Maintain the scenarios (+some explanation text)

in a shared location• Organize in a logical manner

– Reorganize as needed


Benefits of Specification by Example• Full automated test coverage

– Makes refactoring easy and safe!• Helps keeping the code clean all the time!

– Regressions are caught by the developer before check-in!• Virtually no bugs!

• Tests are easy to understand and maintain– Provide a comprehensive documentation which is always up to

date!– When conflicting requirements are caught (during

development), it’s easy to communicate it to the business people


Benefits of Specification by Example• Trust is built among stakeholders• Forming an ubiquitous language• Less rework – eliminate waste• Facilitate change

– Short delivery cycles• Receive feedback earlier!• Respond to new requirements quickly

• Delivering the RIGHT Software!


Real Example: Promotion Engine (FitNesse)


Game – Simulating SbE


Existing application – specifications:• Purpose: Point-of-Sale application• Can sell products from catalog• 2 Screens:

– Selling screen– Products catalog


Existing application – specifications:• Products catalog screen:

• Allows to create, delete and update products in the catalog• Each product has:

• Name• Is it sold it units or by weight?• Price per unit or Kg

• Validations:• Name is unique and non-empty• Price > 0


Existing application – specifications:• Selling screen:

• Only one selling at a time (no history)• Can add, edit and remove lines• Each line contains:

• Product name• Quantity

• For unit products – must be an integer• For weighable products – can be decimal

• Price – calculated by Product’s price * quantity


Existing application – specifications:• Selling screen (cont.)

• A product can appear in more than 1 line• The total is displayed at the bottom• “Next Customer” button:

• Enabled only there’s at least 1 line• Displays “Thank you for buying from us!”• Cleans the screen to prepare for next customer

• “Edit Products” button:• Enabled only when the selling is empty• Opens the Products Catalog screen


New Requirement:• Support Promotions from the following type:

– Given Bamba costs 4 NIS– Promotion: “Buy 3 Bamba in 10 NIS”


The Game Flow (phase 1)• We’ll split into 3 groups:

– Business Analysts– Developers– Testers

• Instructor represents the customer• Each team will have 5 minutes to complete their

role• The other groups are waiting outside the

classroom


The Game Flow (phase 1)• Business Analysts role:

– Write a specification document for the requirement• according to their understanding of the customer’s

requirement• Can ask the Customer clarification questions

– Provide the specification document to the developers and testers


The Game Flow (phase 1)• Developers role

– Read the Specifications written by the B.A.– Write a detailed flowchart or pseudo-code as needed– Specify all the changes that have to be done to the app


The Game Flow (phase 1)• Testers role

– Write a test plan according to the Spec the B.A wrote– Each test should contain clear steps and expected

results– The goal of the test plan is to find as many bugs as

possible in the Developer’s “code”


The Game Flow (phase 1)• Final stage:

– The instructor will “execute” the tests on the “Code”• Will write which tests passed and which failed

– The instructor (as the “Customer”) will perform additional tests and will indicate number of bugs and severity

– Note: In case we won’t find any serious bugs, we’ll repeat the game with more complex requirements

• Phase 2 of the game will be explained later…


SbE Game – Phase 2


The Game Flow (phase 2)Simulation of Specification by Example:

• All groups work together• Specify using examples• Can ask the Customer for clarifications


Understanding Specification by Example


Specification by Example and Automatic Testing


Manual Acceptance Tests• Tests that are conducted by the Customer or

Business Analyst to verify that a new requested feature or User Story is implemented as expected

• Usually done thought the UI


Manual Regression Tests• Manual tests, (usually conducted by a tester) to

verify that old functionality still works

• Usually done though the UI


Test Case (Scenario)• A test case represents a single scenario• A scenario is composed of steps that interact with

the system and verifies its behavior


Test Case Structure

Arrange Act Assert

Given

When Then


Test Case - Example• Create an Item called “Bamba” with price $3• Begin a selling transaction• Sell 2 “Bamba”• Ensure that the total is $6• Pay by cash• Ensure that the transaction is closed and its

balance is 0


Automatic Acceptance Test• An automatic test that verifies that a new

requested feature or User Story is implemented as expected

• After the test passes once, it becomes an automatic Regression test

• Automatic UI Testing has high maintenance costs…


Test Scope• Defintion1: The parts of the system and their

configurations that are tested by a particular test or group of tests

• Definition2: Everything that is not under the control of the test


Test Scope

End-to-End

UIView ModelClient LogicServer Proxy

Service LayerBusiness

LogicDALORM

DB


Test ScopeUI

View ModelClient LogicServer Proxy


LogicDALORM

DB

Usability, UX


Types of TestsUI



LogicDALORM

DB

Integration


Test ScopeUI



LogicDALORM

DBFunctional

Mock

Mock

Mock


Test ScopeUI


Service Layer

DALORM

DB

Unit tests

Business Logic


Test Scope• Smaller Scopes ensures that the tested

functionality is handled only inside this scope and nowhere else (loosely-coupled, lower maintenance costs). – It aids diagnosing the root cause of problems– Faster

• Larger scope verifies that the system behaves correctly as a whole


Writing Good Automatic Acceptance Tests

• Create an abstraction between the test case and the Scope.– The Test Layer reflects the scenario in the most

readable way. • The Test Layer is agnostic to the Scope of the Test

– The Automation Layer is an adaptor between the Test Layer and the SUT• The Automation Layer is bound to the Scope of the

Test


Automatic Tests Architecture

Automatic Test Case (Test Layer)Abstraction Layer

Automation Layer

UIView ModelClient LogicServer Proxy


LogicDALORM

DB


Choosing the right scope• Some scopes are easier to automate than others• Some scopes are more likely to change than

others– Causing higher maintenance costs of test code

Choose the scope that covers most of the risk, while still easy to

maintain


Complementary Testing Strategies• Manual Acceptance Tests• Exploratory tests• TDD / Unit tests• Usability tests• Performance tests• Load and scalability tests


Deriving the Solution from the Goal


• “If I had asked my customers what they wanted they would have said a faster horse.”

– Henry Ford


The F-16 Story


Deriving the Solution from the Goals• Collaboration empowers creativity and simplicity• Ask “Why?” and “Who?”

– Dig deep until you reach the “money” answer• Focus on the outputs first, only then on the inputs• If you’re given a solution, ask:

– How it would be useful?

– How will this solution help you achieve the goal better than what you’re doing now?


Specifying Collaboratively


Specifying Collaboratively• Benefits of Collaboration:

– Encourages Creativity– Prevents blind spots– Creates shared understanding– Promotes ubiquitous language– Produces specifications that are easy to understand – Produced tests that are easy to maintain


Tips for effective collaboration• Consider having introductory meetings• Make sure that everyone have the necessary

background• Make sure that participants have the business

answers– Involve the relevant stakeholders– Product owner to get as much information as possible

before hand– Don’t over-prepare!


The Power of Examples


Game: The Power of Examples


Illustrating using examples• Examples is probably the best way to ensure

understanding• Examples should be precise – but not over-

detailed– Focus on the relevant domain concepts and Business

Rules– Avoid technical details like specific UI details

• Expected result should be specific and measurable as possible


Illustrating using examples• Examples should be realistic

– But can be simplified to focus on the point at hand and be easy to understand

• Examples and expected results should be described using the user’s perspective– Avoid relying on database or logs (unless the logs are aimed for the

user)• Add a meaningful title to an example or a set of example

and a short description• Tip: start with basic examples and explore what’s missing• Given-When-Then!


Illustrating Non-Functional Requirements

• Many “nonfunctional” requirements can be expressed in deterministic examples and measurements– E.g. performance…

• Qualitative non measurable requirements (e.g. usability) can be expressed manually using examples of “good” and “bad”

• Use UI sketches• Use check-lists for cross-cutting concerns


Writing Good Executable Examples• Use Business Domain language• Describe from User’s perspective

– Think about different roles/personas• Focus on the relevant details!

– Hide irrelevant details in the automation layer– Focus on the Business Logic, not on the UI


Writing Good Executable Examples• Start from the Happy-Path and explore• Avoid having dependencies on preconfigured,

predefined, shared data• Have automatic tests run in an isolated

environment


Game: Improving Executable Examples


Improving Executable Examples – SolutionName: Promotion Can Add Points To CustomerDescription: A promotion can add points to a customer (which he can use later in other promotions). The user can examine his points balance in the web-site.Steps:• Define item: bamba (price 3NIS)• Define promotion: For every 2 bamba – add 200 points to the customer account• Create new customer “Arnon”, with stating balance of 100• Begin selling for customer “Arnon”• Add: 5 bamba• Pay total• Check receipt shows added points: 400• Check balance for customer “Arnon” is 500 (though web-site)


Automating the Examples to match the Specifications


Automating the Examples to match the Specifications

• Choose the Tool appropriate for you– Features and ease-of-use– Price– Programming Language

• Choose the automation scope


FitNesse


Concordion


Cucumber, JBehave, SpecFlow


Twist


MS-Test, NUnit


Maximizing the Effectiveness of Executable Specifications


Validating Frequently• Run the tests automatically• Run Frequently

– Most Preferred: Continuous Integration– Nightly– Weekly– Iteration– Least Preferred: Release


Validating Frequently• Separate short and long tests• Allow developers to run the tests locally before

check-in• Optimize tests• Run tests in parallel (on different

machines) to reduce time


Reliable Automation• Same Code Quality as Production Code!• Fix test issues quickly!• Use isolated environment• Ban dependencies between tests• Avoid shared data• Avoid relying on time

– Rely on events rather than timeouts– Mock system time


Provide transparency• Provide High-Level Report to management• Provide detailed report for failing tests for

Developers– Make it easy to find the root cause

• Mark failing tests for incomplete user stories differently from regression failures


Provide Transparency• Manage failing tests

– Avoid marking tests as “Ignore”– Make tests that fail due to known bugs so that they’ll

appear in reports• Link to other ALM artifacts


Evolving a Documentation System


Living Documentation should be Easy to Understand• Executable Examples should be:

– Readable and clear for Business People– Focused

• Provide some textual/graphical explanations as needed

• Evolve the Language and “Refactor” to make it consistent


Organizing the Executable Specifications• Specifications should be easy to find• Create links between tests/concepts (Wiki)• Have a primary hierarchical structured according

to functional areas– Evolve it as needed

• Use Tags for easy searching• Have secondary indexes for other categorizations

100

Copyright © 2012 Retalix |

Changing the Process

101


General Advice…• Know the benefits of the new process• Know the benefits for each stakeholder

– But also their concerns• Listen! Listen! Listen!

– Get to know your organization and business• Have a clear vision!• Be enthusiastic and catch others!• Once you started – be determined!

102


Benefits for Business Analysts• Developers will actually read and understand the

specifications– Developers won’t skip parts of the specifications

• You can track development progress easily• You can easily identify conflicts in business rules

103


Challenges for Business Analysts• I don’t do tests – that’s not my job• It’s extra work• They will only look at the tests and won’t read the

requirements• What if I leave something out?• I can’t cover all the specifications with Examples

104


Benefits for Testers• You’ll be involved from the start• You’ll build quality in from the start• You’ll have more influence of the developed

product• You will have much better understanding of the

domain• You’ll delegate a lot of dull work to developers• You’ll be more free for exploratory testing• Better relationships with the Developers

105


Challenges for Testers• Will I loose my job because of automation?• Automated Tests won’t catch all problems• I don’t want to loose control over testing• It’s cheating – developers will know in advance

what I’m going to test…

106


Benefits for Developers• Functional gaps will be flushed out before

development• Business Analysts will really understand special

cases• Automated tests will be your target for

development• You’ll have a clear Definition of Done• You’ll be able to give better estimations

107


Challenges for Developers• I don’t have time to write fixtures or maintain

tests• Acceptance Tests turn out to be incomplete• Can’t we just write the tests ourselves?• We already write unit tests

108


Tips for implementing SbE• Ride on a wider process change (if applicable)• Start small and show results• Focus on improving quality• Focus on reducing TTM• Engage management• Start from test automation and improve• Create a small “Sanity” automation suite to show

benefits quickly

109


Tips for changing the process• Avoid testing terminology while talking to non-testers• Don’t focus on the tool

– Introduce a free tool instead of an expensive one…– Avoid fighting on the right tool

• Avoid buzzwords and “Agile” terminology• Leverage transparency in order to enforce • Conduct effective retrospectives to learn how to

improve!• Listen carefully to what people tell you!

110


Conclusion

111


Conclusion

• What did you learn today?• What will you take with you from this workshop?

• Make a commitment!

• Verifying the Definition of Done

112


THANK YOU!

113


Resources• Gojko Adzic, Bridging the Communications Gap:

Specification by Example and Agile Acceptance Testing (Neuri, 2009)

• My blog: • Gojko Adzic, Specification By Example: How

successful teams deliver the right software (Manning, 2011)

• My blog: http://blogs.microsoft.co.il/blogs/arnona

http://blogs.microsoft.co.il/blogs/arnona