System Testing Only Strategy

康美国（Lance Kind）

Pros/Cons of System Test ONLY

Background –康美国, Lance Kind

• Working in a XP environment since 1999

• Agile Coaching since 2006

• Development Lead

• Certified Scrum Master

• Certified Professional Snowboard/Ski Instructor

• TDD fanatic

• 15 years in software development and Internet technologies

• Science fiction author (LancerKind.com)

LancerKind@gmail.com

Why Automate Testing?

• Developers using debuggers as manual unit testing tools takes a long time and it’s not repeated before each check in.

• If testing isn’t significantly automated, we can’t easily check whether the system is running.

• The two major types of automated testing are testing the whole system as a black box (System Testing) and testing each individual piece directly as small black boxes (Unit Tests).

What Automated Testing doesn’t do well!

• Find bugs for which there ARE NOT automated tests.

• There’ll always be a need for some amount of manual testing.

The difference between System Tests and Unit Tests

• System tests tell us if the system does what the customer expects

• Unit tests tell us if a *part* of the system does what the developers expect

System tests • Tell us whether the system does what the

customer expects

• Enable developers to know they’ve satisfied requirements

• Can be run automatically at any time by anyone

Traditional balance of tests

Manual GUI Acceptance

Tests

Unit Tests

Automated GUI Tests

• Easy to Create • Very familiar – what we always do • Typically tedious • How do we know coverage? • There are thousands of these.

• Need Automation specialists • Automation good for performance • Seems like we always rewrite • Sometimes fragile

• What is Development testing? • How do we know what these are? • How do we know when they fail?

(Shamelessly borrowed from Mike Cohn)

Testing Pyramid

GUI Acceptance

Tests

FitNesse

Unit Tests

•Fewest in number •Automate as much as possible •Less than 100 and take minutes to hours

•Drive development and acceptance

•Hundreds and take hours to days

•Do the most •Create Test Driven Design •Thousands and run in minutes

Quality Decision Makers

Traditional:

• QA Management

• Architects

• Testing Teams

Agile:

• Same as the left but add:

• Development engineers

Real World Data Collected from the following:

• As a developer and consultant:

– AT&T Wireless internal app

– KeyBank College Loan web app

• As a consultant:

– Large Chinese Co. that makes router products.

AT&T Wireless

• Project profile:

– Internal web application

– JEE, Java script, AJAX, SQL server

– View heavy and DB heavy application

– Six years of history (team of 8):

• 600+ system tests (from view layer, down), 4 days for full system test pass, suite of intermittent failures

• 300 unit tests, a minute for full unit test pass (JSUnit, JUnit)

AT&T Wireless Results

• Good: – Shipped to customer every 4 months – Functional Quality very high

• Impediments: – Code/design quality middle to low – View layer was thick as there was no technical/practical

reason to make small – Developers were reluctant to improve design (refactoring)

as the code coverage was low and the feedback loop slow (days to discover regressions).

– When regression occurred, it took hours to a day to discover why the system tests failed.

China Router Co.’s

• Project profile: – Embedded controller application to build routers used

by telecoms

– C++/VX Works

– Separate system test team using Ruby system test scripts

– 3 month project (team of 7 but system testers on external team): • High hundreds of system tests, days to a week for full system

test pass

• No unit tests.

China Router Co.’s biggest impediment

• If a system test failed, it took days to weeks to resolve what caused the failure.

• No unit tests were being developed.

• Code/design quality was low with lots of duplicate code and commented out blocks of code.

• No one ever refactored.

• The team feared System Test failure reports!

KeyBank

• Project profile: – External web application used by college students

– Java Server Faces, TSQL

– View heavy and DB heavy application

– Four month project (team of 5): • 80 system tests (from view layer down, and at Web

Service Layer), 2 hours days for full system test pass, suite of intermittent failures

• 30unit tests, seconds for full unit test pass (JSUnit, JUnit)

KeyBank month #3 of 4 Results

• Good: – The developers were very productive in creating system tests and

functionality. – Code checkins were happening every 30 minutes to an hour.

• Bad: – Few unit tests were being developed. – Code/design quality was only medium to low. Month #3 code review report

was poor. – View layer was thick because unit testing was being skipped, so we didn’t have

small, decoupled objects. – System Tests seemed unstable because upon each execution, different tests

would fail. The fixes sometimes maybe were already checked in, but no one knew for sure until the next morning when they could view the results before work started again.

– Everyone agreed that the situation was bad. The code review was a wakeup call.

We took action!

• Even though we were in a hurry, (who isn’t?) we refactored like madmen!

• Results after one week: – The big view layer classes were thinned and many small unit

tested objects were created. – 80 unit tests were added – Duplicate code was reduced – We were proud of our source code again! – System tests still acted unstable as they couldn’t keep up with

the pace of our changes. Only see green bar for system tests two times a week.

– We stopped/slowed our changes 3 days before release and finally got GREEN BAR on the system test results

– Our release was successful and no bugs reported by client.

Summary

• System Test Only is easier to “roll out” in an organization” than System Test & TDD because:

– Without TDD, developers don’t need to change their habits and be responsible for their own testing.

– A separate System test team can create system tests against interfaces.

– An organization doesn’t need to change itself very much.

– Poor code design doesn’t have to be redesigned so it’s testable for unit tests.

Summary part 2

• System Test Only has problems: – Slow feedback means we can’t quickly find out our project

status. – When a system test fails, the scope is so large it takes half a day

to weeks to find the problem where unit tests rarely take more than an hour and often 15-20 minutes.

– Design quality is low as there’s nothing (other than code review) forcing developers to write decoupled code with good interface design.

– Relying on code review to improve code is slow: (code->manual test (and maybe wait for system test report)->debug->have a code-review meeting->back to code step to fix code) Because it’s slow, people in a hurry will not improve code much.

– Refactoring is infrequent because it’s not possible to discover in seconds to minutes if there’s a regression.

To make a System-Test-Only strategy work well

• Even if it’s possible to get the same code coverage as TDD (90%+) … then there are still these problems:

• Slow feedback--System Tests are too slow. Somehow, we need our full System Test suit to execute in seconds to minutes like Unit Tests.

• Hard to FIND the bug: System Tests take hours to weeks to find the bug where Unit Tests take minutes to an hour.

• Until then these problems are solved, I’ll keep using Test Driven Development to make Unit Tests. Acceptance Driven Development doesn’t help these problems at all.

Contact me: 康美国, Lance Kind

Mobile: 18606926591

LancerKind@gmail.com

Twitter: lancerkind

Skype: lancerkind

And contact me through Bill Li （李国彪）

of UPerform: 18602103637