Path selection criteria

Tor Stålhane

Why path selection criteria

Doing white box testing using the test-all-paths strategy can be a tedious and expensive affaire. The strategies discussed here are alternative ways to reduce the number of paths to be tested.

As with all white box tests, it should only be used for small chunks of code – less than say 200 lines.

define, use, kill (duk) – 1

We define three usages of a variable:

• d – define the variable

• u – use the variable

• k – kill the variable.

A large part of those who use this approach will only use define and use – du.

Based on the usages we can define a set of patterns potential problems.

duk – 2 We have the following nine patterns:• dd: define and then define again – error• dk: define and then kill – error• ku: kill and then used – error • kk: kill and then kill again – error • du: define and then use – OK • kd: kill and then redefine – OK • ud: use and then redefine – OK• uk: use and then kill – OK • uu: use and then use – OK

duk examples - 1

Define x

Define xUse x

Define x

Define xUse x

Use x Use x

ddudu

duk examples - 2

Define y

Use y

Use y

Define y

Use y

Kill y

Use y

Kill y

udukudk

duk examples - 3

Kill z

Use zUse z Kill z

Define z Kill zDefine z

Define zDefine z

Kill z

Use zUse z

kuuudkkduud

Use zUse z

Test strategy – 1 Based on the three usages we can define a

total of seven testing strategies. We will have a quick look at each

• All definitions (AD): test cases cover each definition of each variable for at least one use of the variable.

• All predicate-uses (APU): there is at least one path of each definition to p-use of the variable

Test strategy – 2 • All computational uses (ACU): there is at

least one path of each variable to each c-use of the variable

• All p-use/some c-use (APU+C): there is at least one path of each variable to each c-use of the variable. If there are any variable definitions that are not covered then cover a c-use

Test strategy – 3

• All c-uses/some p-uses (ACU+P): there is at least one path of each variable to each c-use of the variable. If there are any variable definitions that are not covered then cover a p-use.

• All uses (AU): there is at least one path of each variable to each c-use and each p-use of the variable.

Test strategy – 4

• All du paths (ADUP): test cases cover every simple sub-path from each variable definition to every p-use and c-use of that variable.

Note that the “kill” usage is not included in any of the test strategies.

Application of test strategies – 1

Define x

Define xc-use xc-use z

Kill zc-use xDefine z

p-use yKill z

Define yp-use z

c-use c-use z

Kill yDefine z

All definitionsAll c-use

Define x

Define xc-use xc-use z

Kill zc-use xDefine z

p-use yKill z

Define yp-use z

c-use c-use z

Kill yDefine z

All p-use

Application of test strategies – 2

Define x

Define xc-use xc-use z

Kill zc-use xDefine z

p-use yKill z

Define yp-use z

c-use c-use z

Kill yDefine z

ACUAPU+C

Relationship between strategiesAll paths

All du-paths

All uses

All c/some p All p/some c

All c-uses

All p-uses

All defs

Branch

Statement The higher up in the hierarchy, the better is thetest strategy

Acknowledgement

The material on the duk patterns and testing strategies are taken from a presentation made by L. Williams at the North Carolina State University.

Use of coverage measures

Tor Stålhane

Model – 1

We will use the following notation:• c: a coverage measure• r(c): reliability• 1 – r(c): failure rate• r(c) = 1 – k*exp(-b*c)

Thus, we also have that ln[1 – r(c)] = ln(k) – b*c

Model – 2The equation ln[1 – r(c)] = ln(k) – b*c is of

the same type as Y = α*X + β.

We can thus use linear regression to estimate the parameters k and b by doing as follows:

1.Use linear regression to estimate α and β

2.We then have– k = exp(α)– b = - β

Coverage measures considered

We have studied the following coverage measures:

• Statement coverage: percentage of statements executed.

• Branch coverage:percentage of branches executed

• LCSAJLinear Code Sequence And Jump

Statement coverage

Statment

-ln(F

5)

1,00,90,80,70,60,50,40,3

13

12

11

10

9

8

7

Scatterplot of -ln(F5) vs Statment

Graph summary-ln(F

5)

0,80,60,40,20,0

12

10

8

0,80,60,40,20,0

12

10

8

Statment Branch

LCSAJ

Scatterplot of -ln(F5) vs Statment; Branch; LCSAJ

Equation summary

Statements:

-ln(F) = 6.5 + 6.4 Cstatement, R2(adj) = 85.3

Branches:

-ln(F) = 7.5 + 6.2 Cbranches, R2(adj) = 82.6

LCSAJ

-ln(F) = 6.5 + 6.4 CLCSAJ, R2(adj) = 77.8

Usage patterns – 1

Not all parts of the code are used equally often. When it comes to reliability, we will get the greatest effect if we have a high coverage for the code that is used most often.

This also explains why companies or user groups disagrees so much when discussing the reliability of a software product.

Usage patterns – 2

input domain

X

X

X

X

X

X

X

X

X

XInput space A

Corrected

Usage patterns – 3

As long as we do not change our input space – usage pattern – we will experience no further errors.

New user groups with new ways to use the system will experience new errors.

Usage patterns – 4

input domain

X

X

XX

X

X

Input space A

Input space B

Extended model – 1 We will use the following notation:• c: coverage measure• r(c): reliability• 1 – r(c): failure rate• r(c) = 1 – k*exp(-a*p*c)• p: the strength of the relationship between

c and r. p will depend the coupling between coverage and faults.

• a: scaling constant

Extended model – 2

C

1 - k

R(C)

1

Large p

Small p

1.00.0

Residual unreliability

Extended model - commentsThe following relation holds:ln[1 – r(c)] = ln(k) – a*p*c

• Strong coupling between coverage and faults will increase the effect of test coverage on the reliability.

• Weak coupling will create a residual gap for reliability that cannot be fixed by more testing, only by increasing the coupling factor p – thus changing the usage pattern.

Bishop’s coverage model – 1

Bishop’s model for predicting remaining errors is different from the models we have looked at earlier. It has a

• Simpler relationship between number of remaining errors and coverage

• More complex relationship between number of tests and achieved coverage

Bishop’s coverage model – 2 We will use f = P(executed code fails). Thus,

the number of observed errors will depend on three factors

• Whether the code– Is executed – C– Fails during execution – f

• Coupling between coverage and faults - p

N0 – N(n) = F(f, C(n, p))

C(n) = 1 – 1/(1 + knp)

Bishop’s coverage model – 3

Based on the assumptions and expression previously presented , we find that

If we use the expression on the previous slide to eliminate C(n) we get

)1)]((1[

)(11

)(

0

0pp fnCf

nC

N

nNN

1)(

11

)(

0

0

pnfkN

nNN