Upload
dahlia-hendrix
View
44
Download
0
Embed Size (px)
DESCRIPTION
Funkcinis testavimas. Programų testavimas remiantis įėjimo, išėjimo ir vidinių būsenų kintamaisiais. Testavimo atvejai. Testavimo duomenys. Testavimo rezultatai. Testavimo ataskaitos. Suprojektuoti testavimo atvejus. Pa ruošti testavimo duomenis. Paleisti programą su test. duom. - PowerPoint PPT Presentation
Citation preview
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 1
Funkcinis testavimas
Programų testavimas remiantis įėjimo, išėjimo ir vidinių būsenų kintamaisiais.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 2
Defektų testavimo procesas
Testavimo atvejai
Testavimo atvejai
Testavimo duomenys
Testavimo duomenys
Testavimo rezultatai
Testavimo rezultatai
Testavimo ataskaitos
Testavimo ataskaitos
Suprojektuoti testavimo atvejus
Suprojektuoti testavimo atvejus
Paruošti testavimo duomenis
Paruošti testavimo duomenis
Paleisti programą su test. duom.
Paleisti programą su test. duom.
Palyginti rezultatus su testavimo atvejais
Palyginti rezultatus su testavimo atvejais
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 3
“Juodos dėžės “ testavimas
Testavimo metodas, kai programa yra įsivaizduojama kaip “juoda dėžė”.
Programos testavimo atvejai yra paremti sistemos specifikacija.
Testo planavimas gali prasidėti labai anksti programinės įrangos kūrimo procese.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 4
Funkcinė specifikacija Sąsajos ekranas ir vykdoma užduotis Paprastai pagal funkcinę specifikaciją testuoja ne
programos realizuotojas Geriausiai tinka nepriklausomiems fragmentams Susietiems fragmentams turi įtaką vykdymo
tvarka
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 5
Vidinės būsenos Kiekvienas automatas remiasi perėjimais tarp
būsenų Testavimo metu turi būti nagrinėjami visi
perėjimai tarp būsenų Būsenų gali būti labai daug
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 6
B01 architecture BEHAV of b01 is constant a:integer:=0; constant b:integer:=1; constant c:integer:=2; constant e:integer:=3; constant f:integer:=4; constant g:integer:=5; constant wf0:integer:=6; constant wf1:integer:=7;
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 7
B01 when a => if line1='1' and
line2='1' then stato:=f; else stato:=b ; end if; outp <= line1 xor
line2; overflw <= '0' ;
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 8
Perėjimų tarp būsenų grafas
A
F
B
G
C
1
0
E
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 9
“Juodos dėžės “ testavimas
e
e
I
O
SistemaSistema
Įvesti testavimo duomenys
Įšvesti testavimo rezultatai
Įvedimas, sukeliantis nenormalų elgesį.
Išvedimai, kurie parodo defektų
buvimą.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 10
Ekvivalentinis dalinimas Įvedami duomenys ir išvedami rezultatai dažnai
paskirstomi į atskiras klases, kur visi klasių nariai yra susieti.
Kiekviena iš šių klasių yra lygiaverčio suskirstymo rezultatas, kur programos elgesys su kiekvienu klasės nariu yra toks pats (ekvivalentiškas).
Testiniai atvejai turi būti pasirinkti iš kiekvienos dalies.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 11
Ekvivalentinis dalinimas
SistemaSistema
Teisingos įvestys Neteisingos įvestys
Išvestys
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 12
Padalinti sistemos įėjimų ir išėjimų reikšmes į “ekvivalentiškas aibes”.
Jei įvedamas -5 skaitmenų sveikas skaičius tarp 10000 ir 99999, ekvivalentinis padalinimas yra <10000, 10000-99999 ir > 99999.
Išrinkti testinius atvejus ribose šių aibių 0000, 09999, 10000, 99999, 100000.
Ekvivalentinis dalinimas
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 13
Ekvivalentinis padalinimas
999910000 50000
10000099999
34 7
1110
Mažiau nei 4 Tarp 4 ir 10 Daugiau nei 10Mažiau nei 4 Tarp 4 ir 10 Daugiau nei 10
Įvedimų reikšmių skaičius
Mažiau nei 10000 Tarp 10000 ir 99999 Daugiau nei 99999Mažiau nei 10000 Tarp 10000 ir 99999 Daugiau nei 99999
Įvedamos reikšmės
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 14
Paieškos procedūros specifikacija
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 15
Išskiriami įėjimo poveikiai (įvestis), kurie atitinka išankstines sąlygas.
Išskiriami įėjimo poveikiai, kur išankstinės sąlygos nėra palaikomos.
Įėjimo poveikis, kur nurodytas elementas yra masyvo narys.
Įėjimo poveikis, kur pagrindinis elementas nėra masyvo narys.
Įėjimo poveikių padalinimas paieškos procedūrai
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 16
Testavimo nuorodos masyvams Testuoti programinę įrangą kai masyvas turi tik
vieną elementą. Skirtingiems testams naudoti skirtingo dydžio
masyvus. Parinkti testus taip, kad būtų nagrinėjamas
pirmas, vidurinis ir paskutinis masyvo elementas. Testuoti nulinio ilgio masyvus.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 17
Testai paieškos procedūraiArray ElementSingle value In sequenceSingle value Not in sequenceMore than 1 value First element in sequenceMore than 1 value Last element in sequenceMore than 1 value Middle element in sequenceMore than 1 value Not in sequence
Input sequence (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 29, 21, 23 17 true, 141, 18, 9, 31, 30, 16, 45 45 true, 717, 18, 21, 23, 29, 41, 38 23 true, 421, 23, 29, 33, 38 25 false, ??
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 18
Binary search (Java)
class BinSearch {
// This is an encapsulation of a binary search function that takes an array of// ordered objects and a key and returns an object with 2 attributes namely// index - the value of the array index// found - a boolean indicating whether or not the key is in the array// An object is returned because it is not possible in Java to pass basic types by// reference to a function and so return two values// the key is -1 if the element is not found
public static void search ( int key, int [] elemArray, Result r ){
int bottom = 0 ;int top = elemArray.length - 1 ;int mid ;r.found = false ; r.index = -1 ;while ( bottom <= top ){
mid = (top + bottom) / 2 ;if (elemArray [mid] == key){
r.index = mid ;r.found = true ;return ;
} // if partelse{
if (elemArray [mid] < key)bottom = mid + 1 ;
elsetop = mid - 1 ;
}} //while loop
} // search} //BinSearch
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 19
Patenkintos išankstinės sąlygos, pagrindinis elementas masyve.
Patenkintos išankstinės sąlygos, pagrindinio elemento nėra masyve.
Nepatenkintos išankstinės sąlygos, pagrindinis elementas masyve.
Nepatenkintos išankstinės sąlygos, pagrindinio elemento nėra masyve.
Įėjimo masyvas turi vienintelę reikšmę. Įėjimo masyvas turi lyginį reikšmių skaičių. Įėjimo masyvas turi nelyginį reikšmių skaičių.
Ekvivalentinis sudalinimas dvejetainei paieškai
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 20
Ekvivalentinių klasių ribiniai atvejai
Mid-point
Elements < Mid Elements > Mid
Equivalence class boundaries
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 21
Dvejetainės paieškos testiniai atvejai
Functional TestingSpecification Based Testing
(Black Box testing)
22
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 23
Why do it? Increase our confidence that the software works
and works correctly
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 24
Devise a test plan
A program reads 3 integer values. The 3 values areinterpreted as representing the lengths of the sidesof a triangle. The program prints a message thatstates whether the triangle is scalene ( nelygiašonis), isosceles (lygiašonis), or equilateral (lygiakraštis).
Write test cases that would adequately test this program.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 25
Myers Test Cases 1. Valid scalene (5, 3, 4) => scalene 2. Valid isoscleles (3, 3, 4) => isosceles 3. Valid equilateral (3, 3, 3,) => equilateral 4. First permutation of 2 sides (50, 50, 25) =>
isosceles 5. Second perm of 2 sides (25, 50, 50) => isosceles 6. Third perm of 2 sides (50, 25, 50) => isosceles 7. One side zero (1000, 1000, 0) => invalid
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 26
More test cases 8. One side has negative length (3, 3, -4) => invalid 9. first perm of two equal sides (5, 5, 10) => invalid 10. Second perm of 2 equal sides (10, 5, 5) => invalid 11. Third perm of 2 equal sides (5, 10, 5) => invalid 12. Three sides >0, sum of 2 smallest < largest (8,2,5) => invalid 13. Perm 2 of line lengths in test 12 (2, 5, 8) => invalid 14. Perm 3 of line lengths in test 12 (2, 8, 5) => invalid
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 27
More test cases 15. Perm 4 of line lengths in test 12 (8, 5, 2) => inv 16. Perm 5 of line lengths in test 12 (5, 8, 2) => inv 17. Perm 6 of line lengths in test 12 (5, 2, 8) => inv 18. All sides zero (0, 0, 0) => inv 19. Non-integer input, side a (@, 4, 5) => inv 20. Non-integer input, side b (3, $, 5) => inv 21. Non-integer input, side c (3, 4, %) => inv
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 28
More test cases 22. Missing input a (, 4, 5) => invalid 23. Missing input b (3, , 5) => invalid 24. Missing input c (3, 4, ) => invalid 25. Three sides > 0, one side equals the sum of the other
two (12, 5, 7) => inv 26. Perm 2 of line lengths in test 25 (12, 7, 5) => inv 27. Perm 3 of line lengths in test 25 (7, 5, 12) => inv 28. Perm 4 of line lengths in test 25 (7, 12, 5) => inv 29. Perm 5 of line lengths in test 25 (5, 12, 7) => inv 30. Perm 6 of line lengths in test 25 (5, 7, 12) => inv
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 29
More test cases 31. Three sides at max values (32767, 32767, 323767) => inv 32. Two sides at max values (32767, 32767, 1) => inv 33. One side at max values (32767, 1, 1) => inv
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 30
Testing principles
1. Complete testing is not possible
2. Testing work is creative & difficult• need to understand the system
• most systems are complex
• need business knowledge, testing experience, creativity, insight and testing methodology
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 31
Testing principles
4. Testing is risk based• safety-critical systems need high level
• use risk as basis for allocating the test time available and selecting what to test
5. Testing must be planned • plan overall approach
» what to test and when to stop
• design tests
• establish expected results for each selected test case
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 32
Testing principles
6. Testing requires independence• unbiased measurement e.g. test team
• goal is to measure software quality accurately
• often conflict with the requirement to understand the system being tested (principle 2)
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 33
Black-box methods equivalence partitioning boundary-value analysis error guessing cause-effect graphing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 34
Equivalence partitioning based on the specification; no need to see the code divide up the input domain into equivalence
partitions or classes of data each class is treated identically any datum chosen from a class is as valid as any
other advantage: reduces the input domain to a
manageable size
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 35
Equivalence partitioning (cont) to assist in identifying partitions, look in specs
for terms such as “range”, “set” and other similar words
equivalence classes should not overlap in addition to choosing one datum from each
class, invalid data may also be chosen
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 36
Equivalence partitioning (cont)example #1
consider the exam processing program where we categorise the grade as A, B, C, D, F
assume A>=90, B>=80, C>=70, D>=60 we can partition the grades, g, into the following
partitions: 0<=g<60, 60<=g<70, 70<=g<80, 80<=g<90, g>=90 and g>100, g<0
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 37
Equivalence partitioning (cont)example #1
input Output Expected output50 F
65 D75 C85 B95 A-5 Invalid input
105 Invalid input
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 38
Equivalence partitioning (cont)example #2
string searching program: the program prompts the user for a positive integer in the range 1 to 20 and then for a string of characters of that length. The program then prompts for a character and returns the position in the string at which the character was first found or a message indicating that the character was not present in the string. The user has the option to search for more characters.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 39
Equivalence partitioning (cont)example #2
There are 3 equivalence classes to consider• one class of integers in the stated range
• two classes of integers above and below the range
output domain consists of two classes• the position at which the character is found in the string
• a message stating that it was not found
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 40
Equivalence partitioning (cont)example #2
x input string search char response expectedoutput
34 wrong input0 wrong input3 abc c pos = 3
yk not in string
n
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 41
strengths and weakness of equivalence partitioning
strength: it does reduce size of input domain well suited to data processing applications where
input variables may be easily identified and take on distinct values
not easily applied to applications where input domain is simple yet the processing is complex
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 42
strengths and weakness of equivalence partitioning
problem: although the specification may suggest that a class of data is processed identically, this may not be the case
example: y2k problem also, the technique does not provide an algorithm
for finding the partitions
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 43
Boundary value analysis used in conjunction with equivalence partitioning this technique focuses on likely sources of faults:
boundaries of equivalence classes the technique relies on having created the
equivalence classes
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 44
Boundary value analysis (cont)input Output Expected output
50 F
65 D75 C85 B95 A-5 Invalid input
105 Invalid input60 D70 C80 B90 A
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 45
Boundary value analysis (cont)string processing program
integer values of 0, 1, 20 and 21 are obvious choices,
as well as finding the character in the first and last position
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 46
Boundary value analysis (cont)
x input string search char response expectedoutput
21 out of range0 out of range1 a a pos = 1
yX not in string
n20 abcdefghijkl
mnopqrsta pos = 1
yt pos = 20
n
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 47
Error guessing an ad hoc approach, based on intuition and
experience identify tests that are likely to expose errors make a list of possible errors or error prone
situations and then develop tests based on the list
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 48
Error guessing (cont)some items to try:
empty or null lists/strings zero instances/occurrences blanks or null characters in strings negative numbers
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 49
Error guessing (cont)strengths/weaknesses
intuition frequently accurate the technique is efficient technique relies on experience; not always
available the ad hoc nature leaves doubt about quality of
the test: “did I forget any typical error situations?”
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 50
Cause-effect graphing functional or spec-based technique systematic approach to selecting a set of high-
yield test cases that explore combinations of input conditions
rigorous method for transforming a natural-language spec into a formal-language spec
exposes incompleteness and ambiguities in the spec
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 51
Cause-effect graphing (cont) the spec is analyzed, and all possible causes are identified: inputs, stimuli,
anything that will elicit (išaiškinti) a response from the system
all possible effects are identified: outputs, changes in the system state
causes and effects must be stated so that they can be evaluated as either true or false
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 52
Cause-effect graphing (cont) causes and effects are combined into a boolean
graph that describes their relationship each cause and effect is allocated a unique
number for reference create a boolean graph that shows the links
between cause and effect construct test cases that cover all possible
combinations of cause/effect
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 53
Cause-effect graphing (cont) graphs are combined using operators:
• not
• and
• or
• nor
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 54
Cause-effect graphing (cont)exam processing program
causes:• (1) integer in range 1-20
• (2) search char is in string
• (3) search for another char
effects• (20) integer out of range
• (21) report position of char in string
• (22) char not found in string
• (23) program terminates
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 55
Cause-effect graphing (cont)
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 56
Cause-effect graphing (cont)strengths/weaknesses
exercises combinations of test data expected results are part of test creation process major drawback is boolean graph: large number
of causes and effects produce highly complex graph
soln to weakness: identify sub-problems
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 57
Black-Box Test Reduction Using Input-Output Analysis
ISSTA ‘00
Patrick J. Schroeder, Bogdan Korel
Department of Computer Science
Illinois Institute of Technology
Chicago, IL USA
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 58
1) Motivation
2) Black-Box Testing
3) Input-Output Analysis
4) Determining Input-Output Relationships
5) Conclusion
Presentation Overview
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 59
Experience in Industry as a black-box tester of large, complex, telecommunications systems
Root cause analysis of customer-found faults often discovered ineffective or redundant black-box tests
Additional information from the software implementation often improves black-box testing (after the initial release!)
Motivation for Research
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 60
Black-Box Testing
Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification
Testing usually conducted without knowledge of software implementation - system treated as a “black box”
Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 61
Observations on Black-Box Testing
Testing from the black-box perspective generates a large number of tests (e.g. AT&T Definity PBX > 3 million tests)
All testers are faced with resource limitations; the number of tests must be reduced
Test reduction done randomly or based on “engineering judgement” may lead to ineffective or redundant tests
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 62
Combinatorial Testing
Combinatorial testing: For programs with multiple inputs, one must decide how to test different combinations of selected test data values
The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests
However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 63
Combinatorial Explosion
If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is 1010
X1 X2 X3 . . . Xn
Program P
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 64
Current Approaches
Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgement
Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise or n-way combinations of inputs
Difficult to determine impact on fault-detection capability of the test set using these techniques
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 65
Definition of the Problem
In combinatorial testing, black-box testers face the situation where the number of possible test cases far exceeds the time and resources available to execute them
How can we reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing?
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 66
Proposed Technique
Observations:Current techniques do not consider program output variables
For a class of programs, not all program inputs affect all program outputs
We propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs
This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 67
Simple Example using IO Analysis
Input data selected using equivalence partitioning:
A={1, 3} B={N,S,E,W} C={TDC, BDM}
W ZTotal Number of Combinatorial Tests: 2 * 4 * 2 = 16
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 68
IO Analysis Applied
A={1, 3} B={N,S,E,W} C={TDC, BDM}
W Z
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 69
IO Analysis Applied
T(W)(1, N, TDC)(1, N, BDM)(3, N, TDC)(3, N, BDM)
T(Z)(1, N, TDC)(1, S, TDC)(1, E, TDC)(1, W, TDC)
A={1, 3} B={N,S,E,W} C={TDC, BDM}
W Z T(W) T(Z) (1, N, TDC)(1, N, BDM)(3, N, TDC)(3, N, BDM)(1, S, TDC)(1, E, TDC)(1, W, TDC)
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 70
Optimization Problem
• Create the smallest possible test set which covers all data combinations in the individual output test sets
• Test set optimized using “Don’t Care” (DC) conditions
Toptimized(1, N, TDC)(1, S, BDM)(3, E, TDC)(3, W, BDM)
T(W)(1, DC, TDC)(1, DC, BDM)(3, DC, TDC)(3, DC, BDM)
T(Z)(DC, N, DC)(DC, S, DC)(DC, E, DC)(DC, W, DC)
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 71
Determining Input-Output Relationships
Information may be gathered in a variety of ways (e.g. from specifications, code)
Manual collection of information difficult and time consuming
Automated collection possible through:
Static Analysis
Dynamic Analysis
Execution-oriented Analysis
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 72
Automated Analysis Techniques
Static techniques analyze a program’s source code - static program dependencies used to determine IO relationships
Dynamic techniques analyze a program’s execution trace across a set of test data - dynamic dependencies used to determine IO relationships
Execution-oriented techniques manipulate individual inputs across multiple program executions in a test harness to determine IO relationships
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 73
Technique Advantages/Disadvantages
Static techniques reliably determine the absence of relationships between inputs and outputs; however, they may identify relationships that are not actually present
Dynamic analysis and execution-oriented analysis reliably determine the presence of input-output relationships; however, they cannot guarantee that all relationships will be detected
Dynamic analysis and execution-oriented analysis may be used when static analysis, perhaps in over-estimation, indicates that all program inputs affect all program outputs
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 74
Experimental Study Results
1) Warehouse Management System
Combinatorial Tests: 3,125
Using IO Analysis 40
2) DACS Configuration System
Combinatorial Tests 22,500
Using IO Analysis 158
3) Liquidity Ratio Spreadsheet
Combinatorial Tests 1.95 Million
Using IO Analysis 625
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 75
Conclusion
For the class of program and type of problem defined, input-output analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set
Research continues on identifying program types and testing situations where our approach is applicable
Additional experimentation with execution-oriented analysis to provide program dependencies, and other information useful in black-box testing, is underway
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 76
Example 1
Identify subdomains using cases in the specification Procedure is a black box; intervals are hidden
Int max(int a, int b)// effects: a > b => returns a// a < b => returns b// a = b => returns a
3 obvious tests:(4,3) => 4 (NOTE: any input in the
subdomain of a > b)(3,4) => 4 (3,3) => 3
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 77
Example 2
Int find(int[] a, int value)// effects: returns the smallest i// such that a[I]==value,// unless value not in a // Missing
2 obvious tests:({4,5,6},5) => 1({4,5,6},7) => missing
1 more subtle test:({4,5,5},5) => 1
Approach: look for any statement in the effects (or requires) clause that suggests multiple cases
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 78
Example 3
Test a function that uses the quadratic formula to determine the two roots of a second-degree polynomial ax2+bx+c.
For simplicity, assume that we are going to work only with real numbers, and print an error message if it turns out that the two roots are complex numbers (numbers involving the square root of a negative number).
• What will be the test data for each of the four cases, based on values of the polynomial's discriminant (b2-4ac)?
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 79
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 80
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 81
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 82
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 83
Exceptions to be Considered:
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 84
Test Design Tips
A command to a program may have immediate effects but also enable, disable, or modify later commands. It's important to both check the immediate effect and also try later commands.
Question: Which commands? And how? Answer: The ones that a user would likely
perform, in the way a user would perform them.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 85
Example 4: Consider a product with password protection. You could change the password:
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 86
Example 5: Here's a bug in prerelease
version 0.7.6 of the AbiWord word processor:
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 87
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 88
Equivalence Partitioning
useruserqueriesqueries mousemouse
pickspicks
outputoutputformatsformats
promptspromptsdatadata
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 89
Boundary Value Analysis
userqueries mouse
picks
outputformats
prompts data
outputdomaininput domain
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 90
As with white-box testing, we should test our code with each set of test data.
If the answers match, then our code passes the black-box test.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 91
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 92
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 93
Unit Testing • Informal:
• • Incremental coding
• Static Analysis:• • Hand execution: Reading the source code• • Walk-Through (informal presentation to others)• • Code Inspection (formal presentation to others)• • Automated Tools checking for• • syntactic and semantic errors• • departure from coding standards
• Dynamic Analysis:• • Black-box testing (Test the input/output behavior)• • White-box testing (Test the internal logic of the subsystem or• object)• • Data-structure based testing (Data types determine test cases)
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 94
Unit Testing
modulemoduleto beto betestedtested
test casestest cases
resultsresults
softwaresoftwareengineerengineer
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 95
Unit Testing
interface
local data structures
boundary conditions
independent pathserror handling paths
modulemoduleto beto betestedtested
test casestest cases
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 96
What Should be in a Test Case?
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 97
How to Design Unit Test Spec & Cases ?
A 7-step general process for developing a unit test specification as a set of individual unit test cases.
For each step of the process, suitable test-case design techniques are suggested.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 98
Step 1 - Make it run: Execute the unit under test in the simplest way possible
–Specification derived tests –Equivalence partitioning
Step 2 - Positive testing: Walk through the relevant specifications; each test case should test one or more statements of design specification for the unit
–Specification derived tests –Equivalence partitioning
Step 3 - Negative testing: Show that the software does not do anything that it is not specified to do -- error guessing
–Error guessing –Boundary value analysis –Internal boundary value testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 99
Step 4 - Special considerations: Check performance, safety requirements and security requirements.
Specification derived tests Step 5 - Coverage tests
Branch testing Condition testing Data definition-use testing
Step 6 - Test execution: Yield a measure of test coverage, indicating whether coverage objectives have been achieved.
Step 7 - Coverage completion• There may be complex decision conditions, loops and branches within the code for which coverage targets may
not have been met when tests were executed• Infeasible paths or conditions • Unreachable or redundant code • Ideally, the coverage completion step should be conducted without looking at the actual code
Branch testing Condition testing Data definition-use testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 100
Test Case Design Techniques
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 101
Example: Specification derived tests
Consider the specification for a function to calculate the square root of a real number: • Input - real number • Output - real number
1. When given an input of zero or greater, the positive square root of the input shall be returned.
2. When given an input of less than 0, the error message "Square root error - negative input" shall be displayed and a value of 0 returned.
3. The library routine Print_Line shall be used to display the error message.
Consider the specification for a function to calculate the square root of a real number: • Input - real number • Output - real number
1. When given an input of zero or greater, the positive square root of the input shall be returned.
2. When given an input of less than 0, the error message "Square root error - negative input" shall be displayed and a value of 0 returned.
3. The library routine Print_Line shall be used to display the error message.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 102
There are three statements in this specification, which can be addressed by two test cases
Print_Line conveys structural information in the specification • Test Case 1: Input 4, Return 2
» Exercises the first statement in the specification ("When given an input of 0 or greater, the positive square root of the input shall be returned.").
• Test Case 2: Input -10, Return 0, Output "Square root error - illegal negative input" using Print_Line
» Exercises the second and third statements in the specification ("When given an input of less than 0, the error message "Square root error - illegal negative input" shall be displayed and a value of 0 returned. The library routine Print_Line shall be used to display the error message.").
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 103
Example: Equivalence partitioning
Equivalence partitioning assumes that all values within any individual partition are equivalent for test purposes
Test cases should therefore be designed to test one value in each partition
• The square root function used in the previous example. The square root function has two input partitions and two output partitions
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 104
Four partitions can be tested with two test cases: • Test Case 1: Input 4, Return 2
» Exercises the >=0 input partition (ii) » Exercises the >=0 output partition (a)
• Test Case 2: Input -10, Return 0, Output "Square root error - illegal negative input" using Print_Line» Exercises the <0 input partition (i) » Exercises the "error" output partition (b)
ii
i
Partitions for Square
Root
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 105
Example: Boundary value analysis
Test cases are designed to exercise the software on and at either side of boundary values
Input partition boundaries in
Square Root: • The zero or greater partition has a
boundary at 0 and a boundary at the most positive real number.
• The less than zero partition shares the boundary at 0 and has another boundary at the most negative real number.
• The output has a boundary at 0, below, which it cannot go.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 106
Test Case 1: Input {the most negative real number}, Return 0, Output "Square root error - illegal negative input" using Print_Line
» Exercises the lower boundary of partition (i).
Test Case 2: Input {just less than 0}, Return 0, Output "Square root error - illegal negative input" using Print_Line
» Exercises the upper boundary of partition (i).
Test Case 3: Input 0, Return 0 » Exercises just outside the upper boundary of partition (i), the lower boundary
of partition (ii) and the lower boundary of partition (a).
Test Case 4: Input {just greater than 0}, Return {the positive square root of the input}
» Exercises just inside the lower boundary of partition (ii). Test Case 5: Input {the most positive real number}, Return {the
positive square root of the input} » Exercises the upper boundary of partition (ii) and the upper boundary of
partition (a).
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 107
Example: Branch testing
Given a structural specification for a unit, specifying the control flow within the unit, test cases can be designed to exercise branches.
Such a structural unit specification will typically include a flowchart or flow graph. • For the square root example, a test designer could assume that
there would be a branch between the processing of valid and invalid inputs, leading to the following test cases:
• Test Case 1: Input 4, Return 2 Exercises the valid input processing branch
• Test Case 2: Input -10, Return 0, Output "Square root error - illegal negative input" using Print_Line.
Exercises the invalid input processing branch
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 108
However, there could be many different structural implementations of the square root function, which may not be all covered by the two test cases
However, there could be many different structural implementations of the square root function, which may not be all covered by the two test cases
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 109
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 110
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 111
Example: Condition testing
To test the individual elements of logical expressions, both within branch conditions and within other expressions in a unit -- "white box" test design from a structural specification for a unit.
• Consider the specification for the square root function, which uses successive approximation.
• We decide to limit the algorithm to a maximum of 10 iterations, because after 10 iterations the answer would be as close as it would ever get. The PDL specification for the unit could specify an exit condition:
:EXIT_LOOP WHEN(error<desired
accuracy)or(iterations=10) :
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 112
Test cases have to prove that both error<desired accuracy and
iterations=10 can independently affect the outcome of the decision. Test Case 1: 10 iterations, error>desired accuracy for all
iterations. • Both parts of the condition are false for the first nine iterations. On the tenth
iteration, the first part of the condition is false and the second part becomes true, showing that the iterations=10 part of the condition can independently affect its outcome.
Test Case 2: 2 iterations, error>=desired accuracy for the first iteration, and error<desired accuracy for the second iteration. • Both parts of the condition are false for the first iteration. On the second
iteration, the first part of the condition becomes true and the second part remains false, showing that the error<desired accuracy part of the condition can independently affect its outcome.
Other Test Cases: Both conditions are true or false.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 113
Step 1 - Make it run: Execute the unit under test in the simplest way possible
–Specification derived tests –Equivalence partitioning
Step 2 - Positive testing: Walk through the relevant specifications; each test case should test one or more statements of design specification for the unit
–Specification derived tests –Equivalence partitioning
Step 3 - Negative testing: Show that the software does not do anything that it is not specified to do -- error guessing
–Error guessing –Boundary value analysis –Internal boundary value testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 114
Step 4 - Special considerations: Check performance, safety requirements and security requirements.
Specification derived tests Step 5 - Coverage tests
Branch testing Condition testing Data definition-use testing
Step 6 - Test execution: Yield a measure of test coverage, indicating whether coverage objectives have been achieved.
Step 7 - Coverage completion• There may be complex decision conditions, loops and branches within the code for which coverage targets may
not have been met when tests were executed• Infeasible paths or conditions • Unreachable or redundant code • Ideally, the coverage completion step should be conducted without looking at the actual code
Branch testing Condition testing Data definition-use testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 115
Example: Data definition-use testing
To test pairs of data definitions and uses. A data definition is anywhere that the value of a
data item is set, and a data use is anywhere that a data item is read or used. • Consider one PDL specification for the square
root function, which sends every input to the maths co-processor and uses the co-processor status to determine the validity of the result.
• The first step is to list the pairs of definitions and uses. In this specification, there are a number of definition-use pairs.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 116
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 117
These pairs of definitions and uses can then be used to design test cases. Two test cases are required to test all six of these definition-use pairs:
• Test Case 1: Input 4, Return 2 Tests definition-use pairs 1, 2, 5, 6
• Test Case 2: Input -10, Return 0, Output "Square root error - illegal negative input" using Print_Line.
Tests definition-use pairs 1, 2, 3, 4
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 118
Example: Internal boundary value testing
Consider a fragment of the successive approximation version of the square root unit specification:
:
Calculate first approximation
LOOP
Calculate error
EXIT_LOOP WHEN error<desired accuracy
Adjust approximationEND_LOOP
RETURN the answer :
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 119
The calculated error can be in one of two partitions about the desired accuracy, which is not apparent from a purely functional specification.
An analysis of internal boundary values yields three conditions for which test cases need to be designed:
• Test Case 1: Error just greater than the desired accuracy
• Test Case 2: Error equal to the desired accuracy • Test Case 3: Error just less than the desired
accuracy