כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?

IBM Haifa Research Lab

© 2010 IBM Corporation

Combinatorial Test Design and the FoCuS Tool

ItaiItai SegallSegall

Software Testing Verification and Reviews Software Testing Verification and Reviews

GroupGroup



About us

� Software Testing, Verification and Review Methodologies group at IBM Haifa Research Labs

– Concurrent testing

– Code review methodology and tool support

– Code coverage analysis

– Test selection

– Functional modeling and test planning


© 2010 IBM Corporation33

Outline

� Combinatorial Test Design (CTD) Concepts and Overview

� The FoCuS Tool and the Modeling and Reduction Process



Observation - Omission is a Major Risk in Testing

� Choosing what to test out of an enormous test space

– What is a test space? The set of parameters/values that define the

scenarios, inputs, configurations or conditions to be tested, such as

• Customer type (business, residential, government, …)

• Customer geography (provinces)

• Transaction type (add service, delete service, request credit …)

� When you have a list of 3000 tests, can it be reviewed?

� How do you know what’s there and what is missing?

– Coverage (in retrospect)

– Preventive techniques using CTD



An Example of a Test Design

� The configuration space for a cellphone application

– Bluetooth – Yes / No

– GPS – Yes / No

– GPS software

– Support for different GSM frequencies

– Support for different 3G frequencies

– Support for EMS and MMS

� Total number of combinations

– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120 Too many!



Do we really need to test all 5120 configurations?

66

� Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html used in slides 6,8,9

So, to balance cost and risk, we look to select a subset of tests that covers the interactions of configuration variables at some level of interaction (pairs, three-

way, etc.)



How do we reduce the 5120 potential test cases?

� Interactions !

– Interaction level of every 2 means that for any two variables all the combinations of their values have been tested together.

– Every three is the same for every three variables

� IBM’s FoCuS tool models the variables, and allows us to adjust the level of interaction/coverage to achieve acceptable cost/risk tradeoff

� Explicit identification of what is tested and what is not

tested




� The configuration space for a cellphone application

– Bluetooth – Yes / No

– GPS – Yes / No

– GPS software

– Support for 3 different GSM frequencies

– Support for 2 different 3G frequencies

– Support for EMS and MMS

� Total number of combinations

– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120

� Question: How many tests are required for covering every combination of every 2 parameters?

� Answer: only 16 ! (99.7% reduction from 5120)




TRUETRUEFALSETRUEFALSETRUEFALSEFALSENoneFALSEFALSE

TRUETRUETRUEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE

FALSEFALSEFALSETRUETRUETRUETRUETRUEuDriveTRUETRUE

TRUETRUEFALSEFALSEFALSETRUEFALSEFALSEiGoTRUEFALSE

FALSEFALSEFALSEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE

FALSEFALSETRUEFALSEFALSETRUEFALSETRUENoneFALSETRUE

TRUEFALSEFALSETRUETRUETRUETRUETRUEmioMapTRUEFALSE

TRUETRUEFALSETRUEFALSEFALSEFALSEFALSEDestinatorTRUETRUE

TRUETRUETRUEFALSETRUETRUEFALSEFALSEDestinatorTRUETRUE

FALSETRUEFALSEFALSEFALSEFALSEFALSEFALSEmioMapTRUETRUE

FALSEFALSETRUETRUETRUETRUEFALSETRUEiGoTRUETRUE

TRUETRUEFALSETRUETRUEFALSETRUETRUEiGoTRUEFALSE

TRUEFALSETRUEFALSEFALSEFALSETRUETRUEmioMapTRUEFALSE

TRUEFALSEFALSETRUETRUEFALSETRUETRUENoneFALSETRUE

FALSEFALSETRUETRUEFALSEFALSETRUEFALSEuDriveTRUEFALSE

FALSEFALSETRUEFALSETRUEFALSETRUETRUEDestinatorTRUEFALSE

MMSEMSUMTS2100UMTS850GSM1800GSM900GSM850Wi_Fi

GPS_Soft

wareGPSBluetooth

Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120





















GPS_Soft

wareGPSBluetooth






















GPS_Soft

wareGPSBluetooth






















GPS_Soft

wareGPSBluetooth




Outline

� Combinatorial Test Design (CTD) Concepts and Overview

� The FoCuS Tool and The Modeling and Reduction Process



A Comment on the Use of CTD in the Industry

� Many CTD tools were created

– The mathematics are very nice to play with

– The problem is solved with advanced combinatorics

� However, most are almost toy-tools

– Real world problems are not toy examples

– Real world problems may be hard to model

– Not all value combinations are possible – there are restrictions

– Existing tests need to be evaluated or taken into account

– Advanced requirements such as distribution requirements on the values in the resulting tests

� Most tools miss many things testers need



CTD within our FoCuS tool

� Tackles complexity of modeling by Cartesian product methodology

� Fully supports restrictions on the valid combinations

� Highly scalable

� Supports advanced features

– Distribution requirements

– Good path and bad path test planning

� Considers existing tests

– Report uncovered areas

– Suggest complementary test plans

15



The cell phone example in FoCuS



Example – cont.

� Cartesian product consists of 5120 possible configurations:



Example – Cont.

� Some combinations do not make sense, and should be removed:



Example – Cont.

� Now the model contains 2560 legal tasks:



Example – Cont.

� A test plan can now be created:



Example – Cont.

� 16 tests suffice:



Example – Cont.

� Suppose I have only a few Bluetooth components



Example – Cont.

� The new test plan:



Example – Cont.

� More complex coverage requirements may be stated:



Example – cont.

� And the resulting test plan:



Example – cont.

� “.. But I already have some tests implemented, I want to use them”



Example – cont.

� FoCuS can analyze what is not covered:



Example – cont.



Example – cont.

� .. and suggest a complementary test plan:



Comparing Manually Created Test Plans to Usage of Modeling and CTD

�Manual test plans will usually have many large holes

– And it is not known where they are

�Manual test plans are more expensive

– Take a lot of time to design

– Manual test plans contain more tests, so longer time to implement the tests and longer time to execute them

�Known quality situation

– With CTD you can get the desired quality based on risk

– With CTD you can compare alternatives, as once the model is created, creating alternatives takes minutes



Additional Advantages of Systematic Test Plan Design

� Once the methodology is learned, it is much less work – less tests to execute

� The tool tells you how many test configurations are needed and specific configurations that fulfill your requirements

– You can discuss alternatives for how the budget will be used

� Having a model instead of a spreadsheet test list is much more useful

– Less effort to maintain the model

– From the model you can generate lists after changes are made

• A new operating system was added

• Need to add a new variable that interacts with the current ones

• Need to remove variables or values and make sure that the test plan maintains quality

– The model is much easier to review than traditional test plans as the

generation scheme explicitly describes what is tested



Thank You

� Trial download: http://www.alphaworks.ibm.com/tech/focus

� Contact me at: [email protected]

Documents

כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?