MUTATION TESTING

Woot?EXAMPLE 1

public void testMethod() {

Something something = new Something();

something.executeSomeMethod();

}

EXAMPLE 2

public void testConstructor() {

Something something = new Something();

}

LITTLE HISTORYFIRST PROPOSED in 1971 by Richard Lipton

A. Jefferson Offutt and R. H. Untch “Mutation 2000: Uniting the Orthogonal”

FIRST DEFINED in 1978

R. A. De Millo, R. J. Lipton, F. G. Sayward “Hints on test data selection”

FIRST IMPLEMENTED in 1980 by Timothy Budd

T. A. Budd, “Mutation Analysis of Program Test Data”

PROBLEM

SOLUTION

EXPECTATIONS

TESTS - TDD?

TESTS - NO TDD?

TEST

MUTATION

TESTING MUTATED CODE...

EXPECTED RESULT

SOMETHING WRONG?

FLOW1. Run tests2. Mutate production code3. Run tests again4. Generate report

KILL THE MUTANT!

TESTS RESULTS?PASSED (SURVIVED) FAILED (KILLED)

other results applicable (e. g. timeout, memory error)

EXAMPLEORIGINAL

if (i >= 0) {

return "foo";

} else {

return "bar";

}

MUTATED

if (i > 0) {

return "foo";

} else {

return "bar";

}

conditionals boundary mutation

COMMON MUTATORS #1CONDITIONALS MUTATORS

< -> <=

<= -> <

> -> >=

>= -> >

== -> !=

!= -> ==

MATH MUTATORS

+ -> -

- -> +

>> -> <<

<< -> >>

* -> /

/ -> *

COMMON MUTATORS #2RETURN VALUES MUTATOR

returns true if original returns false

returns false if original returns true

returns 0 if original returns int != 0

returns 1 if original returns 0

METHOD CALLS MUTATOR

removes void method call

removes non-void method call, returns default value for the return type

replaces members with null in constructors

SURVIVORS STORIES

HOW TO GET STARTED?PITEST for JAVA http://pitest.org

Parasoft INSURE++ for C/C++ https://www.parasoft.com/product/insure/

MutPy for Python https://pypi.python.org/pypi/MutPy/0.4.0

Maven: mvn org.pitest:pitest-maven:mutationCoverage

PERFORMANCE100 classes

10 tests / class

1 ms / test

tTOTAL = 100 x 10 x 1ms = 1s

10 mutations / class

tALL = 10 x 100 x 1s ~ 17min

tSMART = 10 x 100 x (10 x 1ms) = 10s

1000 classes

10 tests / class

1 ms / test

tTOTAL = 1000 x 10 x 1ms = 10s

10 mutations / class

tALL = 10 x 1000 x 10s ~ 28h

tSMART = 10 x 1000 x (10 x 1ms) ~ 2min

real life example: 481 test cases fully mutated on 16GB i7 -> 03:24 min

ATTRIBUTIONSHands Up - Avel Chuklanov - https://unsplash.com/photos/9cx4-QowgLc (CC0 1.0)

Bridge - Franc - https://thenounproject.com/search/?q=bridge&i=122542 (CC BY 3.0 US)

Cargo Truck - No way - https://thenounproject.com/search/?q=truck&i=154864 (CC0 1.0)

Philosoraptor - OnlyOneKenobi - http://wallpaperswide.com/philosoraptor-wallpapers.html

Chainsaw - James Keuning - https://thenounproject.com/search/?q=chainsaw&i=11328 (CC0 1.0)

Refugees - Gerald Wildmoser - https://thenounproject.com/search/?q=drown&i=208787 (CC BY 3.0 US)

Resurrection - Isabel Martinez Isabel - https://thenounproject.com/search/?q=survived&i=202842 (CC BY 3.0 US)

Pac Man - iconoci - https://thenounproject.com/search/?q=pac+man&i=9215 (CC BY 3.0 US)

Stopwatch - Nick Holroyd - https://thenounproject.com/search/?q=stopwatch&i=12666 (CC BY 3.0 US)

Zombie analysis - clement127 - https://www.flickr.com/photos/clement127/16089737804/ (CC BY-NC-ND 2.0)

in order of appearance

THANK YOU!Mateusz Bryła