Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
TestabilityReid Holmes
http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf https://www.youtube.com/watch?v=1keyEiJHqPw https://www.youtube.com/watch?v=XcT4yYu_TTs
REID HOLMES - CPSC 410: Advanced Software Engineering
TestabilityThe degree to which a system or component facilitates the establishment of test objectives and the execution of tests to determine whether those objectives have been achieved.
REID HOLMES - CPSC 410: Advanced Software Engineering
Given a finite amount of time and resources, how can we validate that the system has an acceptable risk of costly or dangerous defects.
—Bob Binder
REID HOLMES - CPSC 410: Advanced Software Engineering
Develop
Test
Check-In
REID HOLMES - CPSC 410: Advanced Software Engineering
Why not test?‣ Good reasons:‣ I don’t know how!‣ Legacy code
‣ Bad reasons:‣ Bad design‣ Doesn’t catch bugs (now)‣ Slow‣ Boring‣ Hard to change‣ That’s QA’s job
REID HOLMES - CPSC 410: Advanced Software Engineering
Common assumption‣ “The cost of fixing faults rises exponentially with
how late [e.g., requirements, design, implementation, deployment] they are detected.”‣ This is commonly stated but is based on evidence
from 20+ years ago.‣ This assumption does not seem to hold for
modern processes, tools, and languages.‣ That said, it is still necessary to validate that the
system works.‣ Also important that it continues to work as the
system evolves.
REID HOLMES - CPSC 410: Advanced Software Engineering
Terminology‣ Efficiency: number of tests per unit of effort.‣ Effectiveness: the probability of detecting a bug per
unit of effort.‣ Higher testability: more effective tests, same cost.‣ Lower testability: fewer weaker tests, same cost.
‣ Repeatability: the likelihood that running the same test twice will yield the same result.
‣ SUT/CUT: System/Code Under Test‣ White-box: tests consider internals of CUT.‣ Black-box: tests are oblivious of internals of CUT.
http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf
REID HOLMES - CPSC 410: Advanced Software Engineering
Anatomy of a test‣ To reveal a fault, a test must:‣ Reach some code‣ Trigger a defect‣ Propagate an incorrect result‣ The result must be observed‣ The result must be interpreted as incorrect
‣ Test threats:‣ Non-deterministic dependencies‣ Threading/Race Conditions/Deadlock‣ Shared data
REID HOLMES - CPSC 410: Advanced Software Engineering
Properties of Testability‣ Controllability‣ The extent to which the SUT can be made to
perform specific actions of interest.‣ Observability‣ The extent to which the response of the SUT to a
test can be verified.‣ Isolateability‣ The degree to which the element under test can
be validated on its own.‣ Automatability‣ The ability to execute the test programmatically.
REID HOLMES - CPSC 410: Advanced Software Engineering
Controllability‣ What do we have to do to run a test?‣ How expensive is it?‣ Does the SUT make running a test impractical?‣ Give a test goal, do we have enough information to
create an adequate suite?‣ How much tooling can we afford?
REID HOLMES - CPSC 410: Advanced Software Engineering
Observability‣ What do we have to do to identify pass/fail?‣ How expensive is it to do this?‣ Can we extract the result from the SUT?‣ Do we know enough to identify pass/fail?
REID HOLMES - CPSC 410: Advanced Software Engineering
Isolateability‣ Can the element being tested be isolated?‣ What is the cost to do this?‣ If an element cannot be naturally isolated, can we
simulate it (e.g., with mocks / stubs)?‣ Why bother?‣ Isolated components are:‣ Simpler to reason about (e.g., root cause analysis)‣ Less prone to non-determinism‣ Faster
‣ Simulated dependencies can also more enable validating unusual states.
REID HOLMES - CPSC 410: Advanced Software Engineering
Automatability‣ Can tests be executed without human intervention?‣ Huge economic advantages:‣ Setting up automation: 5 hours‣ Running manual test: 30 minutes.‣ Automation pays for itself after just 10 iterations.
‣ What is the cost of automated infrastructure?‣ What is the benefits of using a test infrastructure?‣ Executions can be batched.‣ Run on same configuration / hardware.‣ Global visibility of results.
‣ Enables regression testing.
REID HOLMES - CPSC 410: Advanced Software Engineering
Challenges‣ Tests are code too.‣ Also subject to their own faults.
‣ Not all test failures uncover faults:‣ Defect in test itself‣ Flaky test (due to some form of non-determinism)‣ Requirements shortcoming (undefined behaviour)‣ Implicit assumptions often surfaced by tests
‣ How to retrospectively recognize a ‘true’ failure?‣ Developer changed the source (not test), test
passed on the next iteration.
REID HOLMES - CPSC 410: Advanced Software Engineering
Develop
Test
Check-In
Must: Be Fast Be Reliable Isolate Failures
REID HOLMES - CPSC 410: Advanced Software Engineering
Kinds of tests
REID HOLMES - CPSC 410: Advanced Software Engineering
Test value
Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you onlyneed to search that small unit under test to find the bug.
Writing effective unit tests requires skills in areas such as dependency management, mocking, and hermetic testing. Iwon't cover these skills here, but as a start, the typical example offered to new Googlers (or Nooglers) is how Googlebuilds and tests a stopwatch.
With endtoend tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally forall endtoend tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, thatbug could be anywhere in the product.
Although endtoend tests do a better job of simulating real user scenarios, this advantage quickly becomesoutweighed by all the disadvantages of the endtoend feedback loop:
Unit EndtoEnd
Fast
Reliable
Isolates Failures
Simulates a Real User
Unit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work welltogether. But even then, you do not necessarily need endtoend tests. For that, you can use an integration test. Anintegration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that theycoherently work together.
If two units do not integrate properly, why write an endtoend test when you can write a much smaller, more focusedintegration test that will detect the same bug? While you do need to think larger, you only need to think a little larger toverify that units work together.
Even with both unit tests and integration tests, you probably still will want a small number of endtoend tests to verifythe system as a whole. To find the right balance between all three test types, the best visual aid to use is the testingpyramid. Here is a simplified version of the testing pyramid from the opening keynote of the 2014 Google TestAutomation Conference:
Unit Tests vs. EndtoEnd Tests
Integration Tests
Testing Pyramid
Kirkland (1)
Kurt Alfred Kluever (1)
Mark Ivey (1)
Mark Striebeck (1)
Marko Ivanković (1)
Michael Bachman (1)
Mike Wacker (1)
Mona El Mahdy (1)
Noel Yap (1)
Patricia Legaspi (1)
Peter Arrenbrecht (1)
Phil Rollet (1)
Pooja Gupta (1)
Radoslav Vasilev (1)
Rajat Dewan (1)
Rajat Jain (1)
Rich Martin (1)
Richard Bustamante (1)
Roshan Sembacuttiaratchy (1)
Ruslan Khamitov (1)
Sean Jordan (1)
Sharon Zhou (1)
Stephen Ng (1)
Tejas Shah (1)
Test Analytics (1)
Vojta Jína (1)
October 2015 (1)
August 2015 (1)
June 2015 (1)
May 2015 (2)
April 2015 (2)
March 2015 (1)
February 2015 (1)
January 2015 (2)
December 2014 (2)
November 2014 (1)
October 2014 (2)
September 2014 (2)
August 2014 (2)
July 2014 (3)
June 2014 (3)
May 2014 (2)
April 2014 (2)
Archives
This site uses cookies from Google to deliver its services, to personaliseads and to analyse traffic. Information about your use of this site isshared with Google. By using this site, you agree to its use of cookies.
LEARN MORE GOT IT
REID HOLMES - CPSC 410: Advanced Software Engineering
Test pyramid
http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
REID HOLMES - CPSC 410: Advanced Software Engineering
Continuous Integration‣ Every project will build and execute automated tests.‣ TravisCI (https://travis-ci.com/ (NOT ORG)) has
provided the class with large numbers of commits‣ You can request access online, we will grant it
REID HOLMES - CPSC 410: Advanced Software Engineering
Activity‣ In your groups:‣ Choose one of your use cases‣ Describe one end to end test for this use case‣ Describe 2 integration tests for this use case‣ Describe 3 unit tests for this use case