Testing CS446 GW 1

Testing -- CS446

Gio Wiederhold

Fall 1999

Testing only proves the presence of errors, not their absence [Edsger Dijkstra]

Testing CS446 GW 2

Objective ?

• Assure that program executes?

• Assure that it provides results?

• Assure that the results look good ?

• Assure that the results are correct wrt– expectations ?– specifications ?– customer expectations ?– interfaces to other programs ?

Testing CS446 GW 3

IMPORTANCE ?

• Fraction of development cost - Significant ?

• Objectives -- relevant ?

• Setting to do testing -- simple/complex?

Testing CS446 GW 4

IMPORTANCE ?• Fraction of development cost = 40%

– more for real-time programs -- why?– prior to delivery, % in Maintenance?

• Objectives -- conflicting relevance+ show program correctness - confidence– find errors to debug - actual approach

• Setting to do testing -- complex requirements – Fixed environment -- reproducible / observable– Inputs -- all values / extreme+branch values / timings– Outputs -- values / performance metrics

• customer expectations / analytical predictions / • past values

Testing CS446 GW 5

Extent of testing• Exhaustive

– every combination of all paths– all specified / possible input values– all times for real time

• Complete– all paths at least once, each branchpoint– all extreme values, all branch values– typical, high rates, low rates

• Affordable– assess value -- find errors vs. correct paths

P r o

b l e m

s ?

Testing CS446 GW 6

Frequency ?

• At delivery of integrated system– likely to be a disaster

• At completion of module– definition of module,– requires environment or scaffolding

• When programmers have time

• When prior regression cycle completed

• Daily build– regression longer than cycle

Testing CS446 GW 7

Types of Testing

• Unit test / component test– requires scaffolding

• Subsystem / integration test – requires service specifications

• System test / delivery test– requires copy of customer environment and data

• also ( orthogonal ) :– feature testing– performance testing– load (to maximum specification) testing– stress (to failure) testing– stability testing versus environments and input sets– reliability testing customer reports?

Testing CS446 GW 8

Regression test

After any / significant changes / updates• verify that all old stuff still works• use collection of all / most prior tests and

their results• automate comparison of results

differences found are intentional gooddifferences found are insignificant acceptabledifferences found are annoying post on error reportdifferences found are errors to be fixed error report

• act on error reports detail

Testing CS446 GW 9

Decision process

classify

cost of fix

prioritize

reallocatecurrent work

schedule into current work

ignore(but still log!)

schedule for next release

error log__ __ __ ___ __ __ ___ __ __ ___ __ __ _

cost outcome

experienceError reports

next regression test

logging

1.

5a,b,c,d.

4.

3.

6.

7.

2.

product testingregression testcustomer

product insertion

Testing CS446 GW 10

Decision process: 1,2

Error report

1. check against log: is it old or new ?

2. classify as to seriousness – critical other stuff fails– important in specification / contract– unacceptable seriously impacts usability– limited affects non-essential function– minor easily bypassed or ignored

Testing CS446 GW 11

Decision process: 3, 4

Error category

3. estimate cost – architectural change try to workaround or defer

– module interface need multiple assessments, as below

– released module cost of fix and dissemination

– completed, non-released module re-assignment cost

– module in-process cost of partial redo

4. prioritize – by combining seriousness & cost

phase

cost[Boehm]

phase

a

i

rcp 10

n*

30

100?

1

Testing CS446 GW 12

Decision process: 5, 6, 7

Repair action

5. schedule appropriately – get specific schedule– buy-in, estimate from staff

6. monitor progress– actual start-date, staff allocation– log actual costs & time up to insertion

7. perform regression tests

Testing CS446 GW 13

When is testing done?• When no bugs remain?

Asymptote:– measure rate of <new> bug reports

– correct for state: , number of customers

• When the customer has no complaints?– specifications versus expectations– complaints are good ?!

Indicates that product is useful !

How to be reimbursed?

time

time

What does it mean if no errors are found?

Testing CS446 GW 14

Value of testing over time?• Rate of error discovery

• Tests per error found

• Difficulty of errortype

• Combination: – cost/bug found=The Pesticide Paradox [Beizer:83]

time

time

time

time

Testing CS446 GW 15

White box / Black box

Internals are Visible• tester understands code• amount of code is

managable• tests are based on internal

conditions

early, unit, academic?

Internals are Hidden• tester understands function,

customer needs

• codes can be large, composed

• tests based on specifications

integration, final, business

Internals are Abstracted• test plan is formalized• code can be substantial

Pre Integration, reuse metrics

Grey box?

Testing CS446 GW 16

Specifications

• contractual– negotiated by customer / supplier– size of database in terms of employees

• derived– created during software design – size of database in terms of records

• implicit– left to insight of tester– interface / input performance at full scale

Testing CS446 GW 17

Implicit specifications

Example: Dialog design advice [Shneiderman] dont1. Strive for consistency FTP login - user2. Enable shortcuts for experts deep menu trees3. Offer informative feedback illegal operation4. Design dialogs to yield closure FTP bin mode5. Offer simple error handling editable form6. Permit easy reversal of actions log, ctrl Z7. Support internal locus of control no ptr to home page8. Reduce short-term memory load > 72 items

Assignment? List violations from your experience

Testing CS446 GW 18

Bug types• assumptions

– implit expectations • conflicts

– features vs usability vs speed -- balance

• algorithm– publications often don’t state limits– only report good aspect -- concurrency control

• overflow– arrays, files, record types -- language dependent

• leaks in memory allocation– language/executable dependent, tools available

Testing CS446 GW 19

Testability

What makes programs hard to test?• Boundary values hard to enter• Coverage hard to assess• Design for testability

– internal breakpoints for inputs at branches – reports -- voluminous -- analysis tools– switches to bypass healthy areas (WB / BB)

– temporary/permanent breakpoints/switches

• Path to report mapping

Testing CS446 GW 20

Alternatives• Verification

– objective: versus formal specification– only performed after specifications are fixed

• Validation– subjective: versus customer intent– implicit: expectations from the state-of-the-art– can be performed thoughout

• Replication of coding and comparison– used where no realistic tests can be made – costly separation of staff, – still specification dependence, questionable

• Code review ‘Static testing’

Testing CS446 GW 21

Conclusion• Planning for Testing is worthwhile

– explicit cost item, relate to quality objective– distinct, but related to debugging

• Logging is crucial– input to next test, further development

• Requires knowledge of code, function– experience with programs, customer needs– can qualified testers be obtained, retained

• Testing receives insufficient attention