Istqb Fl Studyguide

8/13/2019 Istqb Fl Studyguide

1/75

Chapter 1

Principles of Testing

1.1 OVERVIEW

In this module you will get an overview of the most fundamental principles of testing.

Firstly, we point out that although you will come across different terminology throughoutyour career, professional testers are all pretty much agreed on these basic ideas that wedescribe in this module.

Secondly, we take a look at the need for proper testing look at the cost of getting it wrongand we show why exhaustive testing is neither possible not practical. What are errors andhow do they get into the software?

hirdly, we describe a fundamental test process, base on industry standards, and underlineimportance of planning! tests and determining expected results in advance of test execution.

We conclude with a look at re"testing and regression testing# why it is so important to go thatextra mile and run a suite of tests again, $ust when you thought it was safe to hit the releasedate.

1.2 OBJECTIVES

%fter completing this module you will&

' (nderstand basic testing terminology.' (nderstand why testing is necessary.

' )e able to define error, fault and failure.' %ppreciate why errors occur and how costly they can be.' (nderstand that you cannot test everything and that testing is therefore a risk managementprocess.' (nderstand the fundamental test process.' (nderstand that developers and testers have different mindsets.' *earn how to communicate effectively with both developers and testers.' Find out why you cannot test your own work.' (nderstand the need for regression testing.' (nderstand the importance of specifying your expected results in advance.' (nderstand how and why tests should be prioriti+ed.


2/75

1.3 TESTI!TER"IO#O!$

here is no generally accepted set of testing definitions used by the worldwide testingcommunity. he )ritish Standard for Software -omponent esting published in %ugust /provides a new source of testing definitions. When involved in a testing pro$ect it isimportant to ensure everyone understands terminology adopted# you may find differentterminology is used in your organi+ation.

Exercise

he )ritish Standard for Software -omponent esting is known as )S___________% useful glossary of terms used in software testing is called )S00000000000

%lthough it is a )ritish Standard published by the )SI 1)ritish Standards Institute!, theSpecialist Interest 2roup in Software esting 1SI2IS! developed it over a period of severalyears.

ANSWERS: 7925-PART2, 7925-PART1

1.% W&$ISTESTI!ECESS'R$(

his section explains why testing is necessary and closely at the cost and conse3uences oferrors in computer software.

1.%.1 )efinitions Errors* +a,lts an- +ail,res

%n error is a human action that produces an incorrect result. % fault is a manifestation of anerror in software. Faults are also known collo3uially as defaults or bugs. % fault, ifencountered, may cause a fault, which is a deviation of the software from its existing delivery

or service.

We can illustrate these points with the true story 4ercury spacecraft. he computer programaboa spacecraft contained the following statement wri the F566%7 programminglanguage.

85 99 i : .9

he programmer;s intention was to execute a succeeding statements up to line 99 ten timesthen creating a loop where the integer variable I was using the loop counter, starting andending at 9.

(nfortunately, what this code actually does is writing variable ido to decimal value . andit does that once only. herefore remaining code is executed once and not 9 times within theloop. %s a result spacecraft went off course and mission was abort considerable costowever, this is compounded by the fact that we all operate underreal world pressures such as tight deadlines, budget restrictions, conflicting priorities and soon.

1.%.% Cost of errors

he cost of an error can vary from nothing at all to large amounts of money and even loss oflife. he aborted 4ercury mission was obviously very costly but surely this is $ust an isolated

example. 5r is it? here are hundreds of stories about failures of computer systems that havebeen attributed to errors in the software. % few examples are shown below&

% nuclear reactor 0as sh,t -o0nbecause a single line of code was coded as : @ insteadof :%)S 1@! i.e. the absolute value of @ irrespective of whether @ was positive ornegative.

)lue -ross of Wisconsin installed a new A=99m claims processing system " it sent out AB9million in unwarranted and -,plicate pa/ents. %lso, when a data entry clerk typed ;none;in the town field the system sent hundreds of checks to the non"existent town of ;757C; inWisconsin.

In 4ay =, Depsi fan a promotion in the Dhilippines. It told customers they could win amillion pesos 1approx. AE9,999! if they bought a bottle of Depsi and found number Estamped on the underside of the bottle cap. (nfortunately, due to a software error, /99,999bottle caps were produced with number E instead of one, which was an e3uivalent of AE=billion in pri+e money. It cost the company dearly as some people pursued their claimsthrough the courts and Pepsi pai- o,t illions of -ollars in copensation.

%nother story was printed in the 7ew @ork imes on /th February E. -hemical )ank


4/75

managed to allow AG million to be withdrawn incorrectly from 99,999 accounts " a singleline error in the program caused every %4 on their network to process the transactiontwice.

1.%. 0hat happene- on Octoer 2th 4 25th1662(

he #on-on ',lance Ser7ice 8#'S9 covers an area of $ust over B99 s3uare miles and isthe largest ambulance service in the world. It covers a resident population of some B./

million, but its daytime population is larger, especially in central *ondon. Since HE SouthWest hames 6egional >ealth %uthority has managed it.

he *%S carries over G999 patients every day and receives between =999 and =G99 callsdaily 1including between 99 and B99 emergency calls i.e. calls!. In terms of resourcesthe *%S has some =H99 staff, HG9 ambulances and a small number of various other vehiclesincluding helicopter. *%S make almost = million patient $ourneys each year. In =its budgeted income was JB.H million.

5n the =Bth and =7th5ctober = the new system failed, ambulances failed to turn up andpeople lost their lives. %lthough no -oroners -ourt ever apportioned blame for any deaths

directly to the computer systems failure, it was by any standards a ma$or disaster and madethe main evening news bulletins on several occasions.

1.%. #on-on ',lance Ser7ice

In summary the problems were&-omputer %ided 8ispatch "

he system relied on near perfect information to propose optimum resource to be allocated toan emergency. >owever, there were imperfections in the information and changes inoperational procedures made it difficult for staff to correct the system.

his was not a problem when it went live at H am on =Bth 5ctober = as the system loadwas light# however as the number of emergency calls increased throughout the"day it becameincreasingly difficult for staff to correct errors# this led to&

Door, duplicated and delayed allocations.

)uild"up of exception messages and awaiting attention list.

Slow"down of system as messages and lists built up.

Increased number of callbacks and hence delays in telephone answering.

he cost of these errors were ultimately that ambulances didn;t turn up and people lost theirlives although the official en3uiry report did not attribute any fatalities to the systemproblems. he costs in terms of loss of confidence in the computer system, industrialrelations and so on were probably also high.

E


5/75

1.%.5 E:ha,sti7e testing 0h/ not test e7er/thing(

It is now widely accepted that you cannot test everything. Cxhausted testers you will find, butexhaustive testing you will not. -omplete testing is neither theoretically, nor practicallypossible. -onsider a 9"character string that has =/9 possible input streams andcorresponding outputs. If you executed one test per microsecond it would take approx. Etimes the age of the (niverse to test this completely. For a survey of the methodology andlimitations of formal proofs of program correctness see K4anna H/L

1.%.; Testing an- risow much testing would you be willing to perform if the risk of failure were negligible?%lternatively, how much testing would you be willing to perform if a single defect could costyou your life;s savings, or, even more significantly, your life? K>et+el //L.

he amount of testing performed depends on the risks involved. 6isk must be used as thebasis for allocating the test time that is available and for selecting what to test and where toplace emphasis. % priority must be assigned to each test.

est 4anagers and Dro$ect 4anagers come up with different prioriti+ation schemes but htbasic principle is that you must focus the testing effort on those areas of the system that arelikely to have the most defects. %nother key principle is that you must execute the mostimportant test first. 5therwise, if you run out of time, which is likely, you will not haveexercised the tests that give you the best payback in terms of faults found.

1.%.6 Testing an- =,alit/

esting identifies faults whose removal increases the software 3uality by increasing thesoftware;s potential reliability. esting is the measurement of software 3uality. We measurehow closely we have achieved 3uality by testing the relevant factors such as correctness,

reliability, usability, maintainability, reusability, testability etc.

1.%.1> Testing an- legal* contract,al* reg,lator/ or an-ator/ re=,ireents

5ther factors that may determine the testing performed may be legal, contractualre3uirements, normally defined in industry specific standards or based on agreed bestpractice 1or more realistically non"negligent practice!.

1.%.11 &o0 ,ch testing is eno,gh(

It is difficult to determine how much testing is enough. esting is always a matter of $udgingrisks against cost of extra testing effort. Dlanning test effort thoroughly before you begin, andsetting completion criteria will go some way towards ensuring the right amount of testing isattempted. %ssigning priorities to tests will ensure that the most important tests have beendone should you run out of time.

G


6/75

1. +?)'"ET'#TESTPROCESS

1..1 Intro-,ction

esting must be planned. his is one of )ill >et+el;s B testing principles K>et+el // p=GL andhe says we are all agreed on this one. >owever, he points out that the problem is that most ofus do not discipline ourselves to act upon it. 2ood testing re3uires thinking out an overallapproach, designing tests and establishing expected results; for each of the test cases we

choose.

@ou have seen already that we cannot test everything, we must make a selection, and theplanning and care we expend on that selection accounts for much of the difference betweengood and poor testers.

he 3uality and effectiveness of software testing are primarily determined by the 3uality ofthe test processes used KMit GL. his is one of Cd Mit;s B essentials of software testing. estgroups that operate within organi+ations that have an immature development process will feelmore pain than those that do not. >owever, the test group should strive to improve its owninternal testing processes. his section of the course shows a fundamental test process, based

on the )SH=G"= standard for software component testing.

he fundamental test process comprises planning, specification, execution, recording andchecking for completion. @ou will find organi+ations that have slightly different names foreach stage of the process and you may find some processes that have $ust E stages, where E NG are combined, for example. >owever, you will find that all good test processes adhere tothis fundamental structure.

1..2 Test process stages

See )SH=G"= for diagram of test process. est planning involves producing a document that

describes an overall approach and test ob$ectives noting any assumptions you have made andstating any exceptions to your overall test strategy for your pro$ect. est planning can beapplied at all levels. -ompletion or exit criteria must be specified so that you know whentesting 1at any stage! is complete. 5ften a coverage target is set and used as test completioncriteria.

Test specification 1sometimes referred to as test design! involves designing test conditionsand test cases using recogni+ed test techni=,es identified at the planning stage. >ere it isusual to produce a separate document or documents that fully describe the tests that you willcarry out. It is important to determine the e:pecte- res,ltsprior to test execution.

Test e:ec,tion involves actually running the specified test on a computer system eithermanually or by using an automated test tool.

Test recor-ing involves keeping good records of the test activities that you have carried out.Oersions of the software you have tested and the test specifications are software you havetested and the test specifications are recorded along with the actual outcomes of each test.

Chec


7/75

instances it may be appropriate to design some new test cases to meet a particular coveragetarget.

7ote that )SH=G"= does not specify the format of any test documentation. >owever, heICCC standard, known as /=, specifies in detail a standard for software test documentation.

)SH=G"= shows a diagram of a suggested hierarchy of test documentation.

&O"E WOR@

Exercise

Dutting aside management problems. 6ead through test documentation examples in )SH=G"= and answer following 3uestions&

What test techni=,es -oes coponent test strateg/ stip,late(

What percentage of -ecision co7erage is re=,ire-(

What sho,l- e -one if errors are fo,n-(

The proAect coponent test plan is ,sef,l eca,se the approach o,tline- allo0s

a9 Strict a-herence to the coponent test strateg/

9 "ore fa,lts to e i-entifie- in the #O! coponents

c9 ' asic 0or>D )C 0hereas strateg/ re=,ire- 6>D(

Which test case -eals 0ith nonn,eric inp,t(

#ist the e:pecte- o,tcoe an- the test con-ition

Wh/ -oes the CTP ha7e a--itionalaltere- test cases(

What action has een ta


8/75

1..3 S,ccessf,l tests -etect fa,lts

%s the ob$ective of a test should be to detect faults, a successful test is one that does detect afault. his is counter"intuitive, because faults delay progress# a successful test is one thatmay cause delay. he successful test reveals a fault which, if found later, may be many moretimes costly to correct so in the long run, is a good thing.

1..% "eaning of copletion or e:it criteria

-ompletion or exit criteria are used to determine when testing 1at any stage! is complete.hese criteria may be defined in terms of cost, time, faults found or coverage criteria.

1.. Co7erage criteria

-overage criteria are defined in terms of items that are exercised by test suites, such asbranches, user re3uirements, and most fre3uently used transactions etc.

1. T&EPS$C&O#O!$O+TESTI!

1..1 P,rpose

he purpose of this section is to explore differences in perspective between tester anddeveloper 1buyer N builder! and explain some of the difficulties management and staff facewhen working together developing and testing computer software.

1..2 )ifferent in-sets

We have already discussed that none of the primary purposes of testing is to find faults in

software i.e., it can be perceived as a destructive process. he development process on theother hand is a naturally creative one and experience shows that staff that work indevelopment have different mindsets to that of testers.

We would never argue that one group is intellectually superior to another, merely that theyview systems development from another perspective. % developer is looking to build newand exciting software based on user;s re3uirements and really wants it to work 1first time ifpossible!. >e or she will work long hours and is usually highly motivated and verydetermined to do a good $ob.

% tester, however, is concerned that user really does get a system that does what they want, isreliable and doesn;t do thing it shouldn;t. >e or she will also work long hours looking forfaults in software but will often find the $ob frustrating as their destructive talents take theirtool on the poor developers. %t this point, there is often much friction between developer andtester. 8eveloper wants to finish system but tester wants all faults in software fixed beforetheir work is done.

/


9/75

In summary:

8evelopers&

%re perceived as very creative " they write code without which there would be nosystem< .

%re often highly valued within an organi+ation.

%re sent on relevant industry training courses to gain recogni+ed 3ualifications.

%re rarely good communicators 1sorry guys!ey Fred. >ere;s a fault report %6=. *ook at this code. Who wrote this? Was ityou? Why, you couldn;t program your way out of a paper bag. We really want this fixed by G

o;clock or else.R

We were unable to print Fred;s reply because of the language< 7eedless to say Fred did notfix the fault as re3uested.


10/75

Exercise

@our trainer will split you into small test teams. 5ne of you will be the test team leader. @ouhave found several faults in a program and the team leader must report these to the developer1your trainer!. he background is that your team has tested this program twice before andtheir are still 3uite a lot of serious faults in the code. here are also several spelling mistakesand wrong colors on the screen layout. he test team is getting a bit fed up. >owever, you

have to be as nice as possible to the developer.

1.. Wh/ canFt 0e test o,r o0n 0or


11/75

1. 5 RETESTI!')RE!RESSIOTESTI!

We find and report a fault in *52 , which is duly fixed by the developer and included in thelatest release which we now have available for testing. What should we do now?

Cxamples of regression tests not carried out include&

he day the phones stopped. . *%S failure on Eth 7ovember 1perhaps!

%riane G failure.

Whenever a fault is detected and fixed then the software should be re"tested to ensure that theoriginal fault has bee successfully removed. @ou should also consider testing for similar andrelated faults. his is made easier if your tests are designed to be repeatable, whether they aremanual or automated.

6egression testing attempts to verify that modifications have not caused unintended adverseside effects in the unchanged software 1regression faults! and that the modified system still

meets re3uirements. It is performed whenever the software, or its environment, is changed.

4ost companies will build up a regression test suite or regression test pack over time andwill add new tests, delete unwanted test and maintain tests as the system evolves. When ama$or software modification is made then the entire regression pack is likely to be run 1albeitwith some modification!. For minor planned changes or emergency fixes then during the testplanning phase the test manager must be selective and identify how many of the regressiontests should be attempted. In order to react 3uickly to an emergency fix the test manager maycreate a subset of regression test pack for immediate execution in such situations.

6egression tests are often good candidates for automation provided you have designed and

developed automated scripts properly 1see automation section!.

In order to have an effective regression test suite then good configuration management ofyour test assets is desirable if not essential. @ou must have version control of your testDocumentation (testplans, scripts etc.! as well as your test data and baseline databases. %ninventory of your test environment 1hardware configuration, operating system version etc.! isalso necessary.


12/75

1.; EGPECTE)RES?#TS

he specification of expected results in advance of test execution is perhaps one of the mostfundamental principles of testing computer software. If this step is omitted then humansubconscious desire for tests to pass will be overwhelming and tester may perhaps interpret aplausible, yet erroneous result, as correct outcome. .

%s you will see when designing test using black box and white box techni3ues there is ampleroom within the test specification to write down you expected results and therefore no realexcuse for not doing it. If you are unable to determine expected results for a particular testthat you had in mind then it its not a good test as you will not be able to 1a! determinewhether it has passed or not and 1b! you will never be able to repeat it.

Cven with a 3uick and dirty ad"hoc test it is advisable to write down beforehand what youexpect to happen. his may all sound pretty obvious but many test efforts have floundered byignoring this basic principle.

R he ma$or difference between a thing that might go wrong and a thing that cannot possibly

go wrong is that when a thing that cannot possibly go wrong does go wrong it usually turnsout to be impossible to get at or repair.R

--Douglas Adams

=


13/75

SO"E TESTI! TER"IO#O!$

+a,lts " a mistake in the code that causes the software to not behave as expected (causes)

+ail,res " the act of a product not behaving as expected " the manifestation of a fault(symptoms)

Vali-ation " establishing the correspondence between the software and its specification ""are e !uilding te rigt product#"

Test care " the collection of inputs, predicted results and execution conditions for a singletest

'-lia-hoc test care " a test executed without prior planning " especially if the expectedbehaviors is not known prior to running the test.

Passfail criteria " decision rules used to determine whether a product passes or fails a giventest

Coinci-ental correctness " when behavior appears to be what is expected, but it is $ust acoincidence

Test s,ite " a collection of test cases necessary to Rade3uatelyR test a product

Test plan " a document describing the scope, approach, resources and schedule of intendedtesting activity " identifies features to be tested, the testing tasks, who will do each task, andany risks re3uiring contingency planning.

Oracle " a procedure, process or magical phenomenon that can determine if the actual

behavior matches the expected behavior

Inci-ent " when a test produces an unexpected outcome, " further effort is necessary toclassify the incident as a software error, a design error, a specification error, a testing error,etc.

B,g report " a method of transmitting the occurrence of a discrepancy between actual andexpected output to someone who cares for Rfollow"upR " also known as discrepancy report,defect report, problem report, etc.

Wor


14/75

$y $rite %rograms#

-oncept& If you want your computer to do exactly what you want it to do, you must write aprogram.

% computer does nothing on its own. In fact, a computer is a dumb machine with nointelligence whatsoever. 8espite what you might read in science fiction stories, a computer

does nothing more than blindly follow instructions supplied by a programmer. -omputerscannot think.

8efinition& % program is a set of instructions that tells the computer exactly what to do.

When someone buys a computer today, the computer sits on the desk doing nothing until heloads a program into the computer;s internal memory and starts running the program. Pust asa O-6 does not record shows on its own without being programmed to do so, a computerre3uires detailed instructions found only in programs.

Suppose that you own rental properties and want to use your computer to track your tenantrecords. @our computer will not help you out in any way until you load and run a rentalproperty program. Where do you find such a program? here are two ways to obtainprograms for computers. @ou can

1! )uy one and hope that the program does exactly what you want it to do.

1=! Write your own program.

It;s much easier and faster to buy a program that you need. housands of programs are on themarket today. In fact, there are so many programs out there that you might not see the needfor writing your own programs.

If you can find a program that does exactly what you want, you are ahead of the game. If youfind a program that meets your exact re3uirements, you should buy that program becausepurchasing a program is often less expensive and much 3uicker than writing the sameprogram yourself or hiring programmers to write it for you.

hink about this for a moment, though& If there are so many programs sold today that dovirtually everything, why are programming languages such as Oisual )asic continuing tobreak previous sales records each year? he answer is simple& Deople buy a computer so thatthe computer will do $obs that they need done. Firms cannot adapt their business to acomputer program. hey must find programs, or write their own programs, so that the

computer processes information according to the business procedures already in place. heonly way to ensure that a program exactly fits the needs of a firm is for the firm to develop itsown programs.

E


15/75

)usiness people are not the only ones who need custom"designed programs. 7o two peoplemanage their finances exactly the same way# no two scientists need computers for exactly thesame kinds of computations# and no two graphic artists need the same kinds of computerdrawing tools. %lthough people buy spreadsheets and word processors for their general"purpose computing needs, many people re3uire speciali+ed programs for specific $obs.

he art of programming computers is rewarding not only from a re3uirements standpoint, but

also on a more personal level. Drogramming computers is fun< Pust as a sculptor looks on afinished work of clay, programmers are often proud of the programs that they write. )y thetime you finish this book, you will have written programs that were not available before youwrote them. When you want your computer to do something specific and you cannot find aprogram that does the $ob exactly the way you want, you will be able to design and write theprogram yourself.

Some Drograms are -hangeable& here is a third method for getting exactly the program thatyou need if you want to computeri+e your company;s accounting records. %ccountingsoftware firms often sell not only accounting programs but also thesource code for thoseprograms. he source code is a listing of the program;s instructions. )y having access to thesource code, you can take what the software company wrote and modify the behavior of theprogram to suit your own re3uirements.

)y starting with a functional program instead of starting from scratch, you save programmingtime and money. Sadly, most non"accounting software firms do not supply the source code.4ost programs sold today have been compiled. %fter compiling, the source code is translatedinto a locked"in executable program. he bottom line is that you cannot easily change thebehavior of compiled programs. For most programs, therefore, you have the choice of buyingthem or writing them yourself from scratch.

)efinition& -ode is another name for program.

&e'ie 7o single program pleases everyone. When a company sells a program, it must begeneral enough to please most purchasers. Some people need programs to behave in aspecific manner in order to fulfill a specific need. hey must resort to writing their ownprograms. *uckily, Oisual )asic takes a lot of the pain out of writing programs.

G


16/75

' Brief &istor/ of Te:t,al Prograing

Concept -omputers cannot understand $ust any language. @ou must learn a language thatyour computer knows before you can write programs for your computer.

)efinition %n application is yet another name for program.

4any people use computers all day long for word processing, database storage, and

spreadsheet analysis, without reali+ing what is going on behind the scenes. @ou must alwayskeep in mind that computers cannot think. @our computer does not know how to be a wordprocessor. If you want your computer to do word processing, you must supply detailedinstructions in the form of a program. 5nly by following the detailed instructions of a wordprocessor program that you load can your computer perform word processing.

It would be nice if writing a program is as easy as telling the computer what you want done.4any people can handle ambiguous instructions, but computers are not smart enough tounderstand vague re3uirements. -omputers can only follow orders given to them, and youmust supply those orders in the form of a program. herefore, you must supply the programsthat you write. Writing programs, especially complex programs, takes time and severalprocedural steps. Oisual )asic speeds the process of creating programs, but even with Oisual)asic some programs take time to write and perfect.

)efinition& % bug is a program error.

hese are the typical steps that most programmers go through when writing programs. First,you have an idea for a program. 7ext, you use a program"development system, such asOisual )asic, to write the program. Crrors, or bugs, often appear in programs because of thedetails needed for even the simplest of programs. herefore, you must test the programthoroughly and fix the errors. Fixing errors is called debugging. 5nce all the errors are out ofthe program, you have a finished application.

PRO!R'""I!

% computer is an information"processing machine. KCxecutes given commands.

(ses memory.

Information : data. here are various types of data e.g. numerical, text, graphics or

pictures, sound signals etc.

% computer program is a se3uence or set of instructions in a programming language.

%ll data and instructions for a program are stored in the computer;s memory in coded

form. his coded form is similar to the mores code 1"9"""9"99"!. he coding or

translation is now done automatically. I.e. by commercial or computer programming. Drograms will be written in the programming languages of which there are many.

*ike Dascal, -obalt, 1used for records!, Oisual )asic, etc.

B


17/75

For the purpose of this module, Oisual )asic 1O)! programming language will be

used. >ence the O) Integrated 8evelopment Cnvironment 1I8C! will do thetranslation into coded form, which is a software program.

*522I72 I75 OIS(%* )%SI-* 1adds topics!, enter 5pen, enterWhen in the form use "a!el . Caption : "ello Class of .///"

o delete an ob$ect from the form Select Cdit menu.

E&&0& 12 %&03&A44123

here are three common errors1Syntax errors are due to structure or grammar of the language 1rules! applied, e.g.more or less spacing, full stops, commas etc.2 Routine errors are due to non"existing situations like divided by 9, is impossibility.hese are errors where the program has instructed the computer to perform an impossibleoperation e.g. as shown above.3Loi!a" errors are those in the meaning of the program.

o save, always S'VE +OR" 'S, followed by S'VE PROJECT 'S.

The Cost of B,gs

Drogramming, and testing, to its use by the public, there;s the potential for bugs to be found.he Figure below shows how the cost of fixing these grows over time.

-ost

Specification 8esign -ode

H


18/75

ime When )ug Is Found5e cost to fix !ugs increased dramatically o'er time6

he cost are logarithmic " that is, they increase tenfold as time increases. % bug found andfixed during the early stages when the specification is being written might cost next tonothing or 9 pence in our example. he same bug, if not found until the software is codedand tested, might cost J to J9. If a customer finds it, the cost could easily top J99.

W&$ )OES SO+TW'RE &'VE B?!S(

4iscommunication or no communication " as to specifics of what an application

should or shouldn;t do 1the application;s re3uirements!.

Software complexity " the complexity of current software applications can be difficult

to comprehend for anyone without experience in modern"day software development.Windows"type interfaces, client"server and distributed applications, datacommunications, enormous relational databases, and sheer si+e of applications haveall contributed to the exponential growth in softwaresystem complexity. %nd the useof ob$ect"oriented techni3ues can complicate instead of simplify a pro$ect unless it is

well engineered.

Drogramming errors " programmers, like anyone else, can make mistakes.

-hanging re3uirements " the customer may not understand the effects of changes, or

may understand and re3uest them anyway " redesign, rescheduling of engineers,effects on other pro$ects, work already completed that may have to be redone orthrown out, hardware re3uirements that may be affected, etc. If there are many minorchanges or any ma$or changes, known and unknown dependencies among parts of thepro$ect are likely to interact and cause problems, and the complexity of keeping trackof changes may result in errors. Cnthusiasm of engineering staff may be affected. Insome fast"changing business environments, continuously modified re3uirements maybe a fact of life. In this case, management must understand the resulting risks, and Q%and test engineers must adapt and plan for continuous extensive testing to keep theinevitable bugs from running out of control

ime pressures " scheduling of software pro$ects is difficult at best, often re3uiring a

lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.

Cgos " people prefer to say things like& ;no problem;, ;piece of cake;, ;I can whip thatout in a few hours; ;it should be easy to update that old code;Instead of& ;that adds a lot of complexity and we could end up making a lot ofmistakes; or we have no idea if we can do that# we;ll wing it;, ;I can;t estimate howlong it will take, until I take a close look at it;, ;we can;t figure out what that oldspaghetti code did in the first place;

If there are too many unrealistic ;no problems;, the result is bugs.

/


19/75

Doorly documented code " it;s tough to maintain and modify code that is badly written

or poorly documented# the result is bugs. In many organi+ations managementprovides no incentive for programmers to document their code or write clear,understandable code. In fact, it;s usually the opposite& they get points mostly for3uickly turning out code, and there;s $ob security if nobody else can understand it 1;ifit was hard to write, it should be hard to read;!.

Software development tools " visual tools, class libraries, compilers, scripting tools,

etc. often introduce their own bugs or are poorly documented, resulting in addedbugs.

E:aple "icrosoft Wor-

CS 75 I7D(S CDC-C8 6CS(*S

I -lick on File hold down and 5pen dialog appears8rag to 5pen

= 6ight click on paragraph 4enu options appear " font, paragraph etc.-ut -opy Daste are grayed out

6ight click within a table = menu options appear related to table

%ctions e.g. insert row, delete row

E Spelling Spelling checking starts and displays firstCrror but check grammar off18epends on option!

G File Drint 8isplays printer dialog

B -lick on print icon Will attempt to print to default printer

)ri7ing #icense E:ercise

-onsider the following simple program&

IF ageT B and age U:BG >C7Issue drivers *icenseIF age U=B >C7

Set Insurance Dremium to >I2>

C*SC

Set Insurance Dremium to S%78%68 C78 IF

C*SC

Crror message 8*55l "R Sorry, unable to issue drivers licenseR


20/75

C78 IF

TEST O IP?TS EGPECTE) RES?#TS

9

= B

H

E =G

G =BB BG

H H9

=9


21/75

'T" E:ercise

Some simplified rules for an % 4&

a! -ard must be valid for this bankb! -orrect DI7 must be enteredc! Withdrawal must not take account overdrawn unless there is an overdraft agreedd! 7otes available are J 9 and J=9 only


Oalid -ard

Incorrect DI7JG9 re3uested ;R

= Oalid -ard

-orrect DI7

J=99 re3uested

)alance JG97o overdraft i-

Oalid -ard

-orrect DI7

JG9 re3uested

)alance JG97o overdraft

E Oalid -ard

-orrect DI7

JG9 re3uested

)alance JE99Withdrawn JB9 earlier "

Telephones E:ercise


Dick (8 receiver

= 8ial 99

8iaE= or =

E 8ial

G 8ial )usy 7umber

B Dress G %fter >earingCngaged tone

H 8iaEH

/ 8iaGH

=


22/75

1. 6 PrioritiHation of Tests

Since we have discussed already that there is never enough time to do all of the testing thatyou would like an obvious approach is to prioriti+e the tests his will then mean thatwhenever you stop testing you will have done the best testing possible in the time available.

he test manager should identify the criteria to hi used when prioriti+ing tests. hese willinclude but are not limited to&

Severity.

Drobability.

Oisibility of failure.

)usiness priority assigned to the re3uirement.

-ustomer wants.

>ow much change the function has undergone.

>ow error prone is this module.

>ow critical is it to the business.

>ow technically complex is it.

>ow difficult is it to test.

>ow costly is it to test.

Meep the test priority with the test at all times. his0 will prevent you from developing anddesigning lo0 priority tests that never get run. (se a scheme of high, medium, low or anumeric scheme or ,=, , E but try not to have more than about E categories.

O TEST C'SE )ESCRIPTIO PRI

% manual block on a bank is introduced 1using the >5*8 function!-ausing payments to be moved between priority classes by the

Scheduler

= Failure of the primary scheduler process

Internal authentication failures

E 7ormal close of day

G 7ormal system start up

B 5ne of each type of re3uest to the scheduler

H Dayments are released correctly from the (SC6 class when limits

Cxceeded but funds received from another bank causing position to

Improve

/ Dayments can be routed to the scheduler 3ueue from several sourceswith the correct class and user defined priority 1including payments

that have been referred or repaired!

Dayments cancelled manually be central control

==


23/75

9 6egression testing of existing telecommunications links

6egression testing of recompiled interface with mainframe system

= 6emote bank changes their limit on us causing payments to be6eleased from the (SC6 class 1if held due to exceeding limit!

6emote bank issues temporary raise

E 6oute all payments to the scheduler with parameters set forimmediate release

G Special processing when internal cut"off time reached

B Suspendactivate l imits

H Suspendactivate scheduler

1.1> S,ar/

@ou should now be familiar with some of the fundamental principles of testing and you willrecogni+e much of the terminology. he horror stories that you will have heard are not $ustsensationalist to motivate the course# they are true stories and there are 1unfortunately! manymore stories about the costs of not testing systems properly.

In particular you can now&

6ecogni+e and use basic testing terminology.

(nderstand why testing is necessary.

8efine error, fault and failure.

Cxplain why errors occur.

2ive examples of the cost of errors.

(nderstand why exhaustive testing is unachievable.

%ppreciate why testing is a risk management process.

(nderstand the fundamental test process.

(nderstand the difference between the mindsets of developers and testers. (nderstand why you cannot test your own work.

(nderstand the need for regression testing.

(nderstand the importance of specifying your expected result in advance.

*ist some criteria for prioriti+ing your tests.

=


24/75

Chapter2

Testing thro,gho,t #ifec/cle

RWhat is clear from the In3uiry eam;s investigations is that neither the -omputer %ided

8ispatch 1-%8! system itself, nor its users, were ready for full implementation on =Bth5ctober =. he -%8 software was not complete, not properly tuned, and not fully tested.he resilience of the hardware under a full load had not been tested. he fall back option tothe second file server had certainly not been tested.R

Cxtract from the main conclusions of the official report into the failure of the *ondon%mbulance Service;s -omputer Systems on 5ctober =Bth and =Hth =.

2.1 OVERVIEW

his module covers all of the different testing activities that take place throughout thepro$ect lifecycle. We introduce various models for testing, discuss the economics oftesting and then describe in detail component, integration, system and acceptancetesting. We conclude with a brief look at maintenance testing.


2.2 OBJECTIVES

(nderstand the difference between verification and validation testingactivities

(nderstand what benefits the O model offers over other models.

)e aware of other models in order to compare and contrast.

(nderstand the cost of fixing faults increases as you move the producttowards live use.

(nderstand what constitutes a master test plan.

(nderstand the meaning of each testing stage.

=E


25/75

2.3 "o-els for Testing

his section will discuss various models for testing. 8efinitions of these models willdiffer however the fundamental principles are agreed on by experts and practitionersalike. We cover verification and validation 1O N O!, the O"4odel and briefly discuss

a 6apid %pplication 8evelopment 16%8! approach to testing.

2.3.1 Verification an- 7ali-ation 8V4V9

Oerification is defined by )/H=G as the process of evaluating a system or componentto determine whether the products of the given development phase satisfy theconditions imposed at the start of that phase.

Oalidation is defined by )/H=G as determination of the correctness of the productsof software development with respect to the user needs and re3uirements.

=G


26/75

E:ercise

Oalidation and Oerification

-omplete the definition of testing& 0000000000000000000000000000000000000000

esting is defined by )SH=G as the process of exercising software to that it satisfiesspecified re3uirements and to00000000000000000000000000000000000000000000

In simpler terms, verification answers the 3uestion Rhave we built the product rightR,i.e. is it correct and free from errors, does it meet the specification whereasOalidation asks the 3uestionR is this the right product?R have the users got what theywanted. o help you remember this use the following&

Oerification Is it error free, does it do what was specified?

Oalidation Is it valid, is this what you really, really want?

2.3.2 Vo-el

here are many models used to describe the se3uence of activities that make aSystems 8evelopment *ife -ycle 1S8*-!. S*8- is used to describe activities ofboth development and maintenance work. hree models are worth mentioning.

Se=,ential 1the traditional waterfall model!.

Increental 1the function by function incremental model!.

Spiral 1the incremental, iterative, evolutionary, 6%8, prototypemodel!.

he three models would all benefit from earlier attention to the testing activity thathas to be done at some time during the S8*-.

%ny reasonable model for S8*- must allow for change and spiral approach allowsfor this with emphasis on slowly changing 1evolving! design. We have to assumechange is inevitable will have to design for change.

8act .)usiness is always changing

=.Finding a fault causes change

&esult Case of fixing a fault defines ease of responding to change

Corollary

If we want systems that can be modified and hence maintained,the earlier we start testing and try the change process, the earlierwe will find out how easy it is going to be to maintain the system

=B


27/75

2.3.3 Se=,ential "o-el

Initial %ppraisal

Feasibility Study

6e3uirements, specification

Functional Specification

echnical Specification

DrogramSpecification

-ode %cceptance est

%nalysis

he se3uential model often fails to bring satisfactory results because of the late attention tothe testing activity. When earlier phases in the development cycle slip, it is the testing phasethat gets s3uee+ed. his can lead to a limited amount of testing being carried out with theassociated production ;teething; problems.

2.3.% Plan for 0ante- 0aterfall o-el -e7elopent of s/ste

he overall pro$ect plan for development of a system might be as shown below&

%ctivities ime

)usiness study """"6e3uirements analysis """"(ser level design """"echnical design """"Drogram specification """"-reation of code """"(nit testing """"Integration testing """""System testing """""%cceptance testing """""Implementation """"""

Q-

=H


28/75

his is a typical Specify, 8esign and )uild pro$ect plan.

%ll testing and 3uality control points come late in pro$ect and only done if

there is time.

When testing is done so late in pro$ect it can reveal costly errors.

Dro$ect plan has testing done late because people think that only physicaldeliverables such as code can be tested. -learly there has to be a betterway.

he challenge it to devise a better way of developing systems. here is aneed to introduce 3uality control points earlier in the S8*-.

=/


29/75

2.3. Se=,ential o-el pl,s testing gi7es FVF -iagra

he O diagram is another way of looking at the se3uential development but this timefrom viewpoint of testing activities that need to be completed later in S8*-.

he ;O; diagram in this simple form has been around for a long time and is especiallyuseful as it easily demonstrates how testing work done early in S8*- is used as input

to assurance work later in development.

he O model of S8*- offers considerable benefits over others as it emphasi+esbuilding of test data and test scenarios during development and not as an afterthought. he O model also allows for establishment of versions, incrementaldevelopment and regression testing.

4anagement needs to rename activities, referred to variously as systems testing oracceptance testing. here has always been a phase of development traditionallythought of as the testing phase. his is an historical perception. esting is not a phasebut rather an activity that must be carried out all through development, giving rise to

the principle of otal Quality.

In the past, system testing was the only type of testing carried out. esting waschecking that programs, when linked together, met the systems specification.Whether design itself was correct was another matter. he concept of RtestingR designbefore programs were coded was given only the most perfunctory attention. his wasfor two reasons&

=


30/75

. )y the time physical design had been done the system was very difficult toalter# modifications caused by design reviews were therefore very unwelcome.

=. he design was documented in terms of physical file layouts and programspecifications, neither of which the user could comprehend. he 3uestion ofwhether physical design was correct could be reviewed, but the more important3uestion& R8id the system do what the user wanted?R Was largely neglected.

9


31/75

2.3. FVF 0ith test recogniHe- -eli7erale

he activity of )usiness %nalysis has, as deliverables, the Specification of6e3uirements from which %cceptance est Dlan is constructed. o have created, forexample, System %rchitecture without integration est Specification is to do onlyalf te 9o!:


32/75

2.3.5 Re7ise- plan for s/ste -e7elopent

he overall pro$ect plan for development of a system might be as shown below. 7ote the new early3uality control points.

his plan shows that the creation and running of actual tests are separated. he creation of test

material 1acceptance test plans, user system test scripts, technical system tests such as integration,link, recovery restart, etc., and unit test data! is done as the relevant design is done. he potential forautomation is very good and the use of tools to capture the test cases, scripts, etc. will play a big partin making the running tests efficient. he early creation of test material will make the process ofdeveloping a system effective. he emphasis must be on first being effecti7e then being efficient.

=


33/75

2.3.; Rapi- 'pplication )e7elopent

he spiral, 6apid %pplication 8evelopment 16%8! model has the benefit of theevolutionary approach. his is an incremental process of build a little then test a little,which has the benefit of attempting to produce a usable but limited version early.

he 6%8 approach relies upon the 3uality of the 6%8 team.

he management issues to address are&

>ave I got knowledgeable user input in my team?

>ave I got experienced designers and developersin my team?

%m I leaving a good audit trail to support future4aintenance?

-ode

%cceptance est

(ser6e3uirement


34/75

2.% ECOO"ICS O+ TESTI!

his section looks at some of the economic factors involved in test activities. %lthough someresearch has been done to put forward the ideas discussed, few organi+ations have yet toprovide accurate figures to confirm these theories.

4a$or retailers and car manufacturers often issue product recall notices when they reali+ethat there is a serious fault in one of their products. Derhaps you can think of other examples.he fixing of the so"called millennium but is probably one of the greatest product recallnotices in history.

)oehm;s research suggests that cost of fixing faults increases dramatically as we movesoftware product towards field use. If fault is detected at an early stage of design, it may beonly design documentation that has to change resulting in perhaps $ust a few hours work.>owever, as pro$ect progresses and other components are built based on faulty design, morework is obviously needed to correct fault once it has been found. his is because designwork, coding and testing will have to be repeated for components of the system that were

previously thought to have been completed.

If faults are found in documentation, then development based on that documentation mightgenerate many related faults, which multiply the effect of the original fault.

%nalysis of specifications during test preparation 1early test design! often brings faults inspecifications to light. his will also prevent faults from multiplying i.e. if removed earlierthey will not propagate into other design documents.

In summary we suggest that it is generally cost effective to use resources on testingthroughout the pro$ect lifecycle starting as soon as possible. he alternative is to potentially

incur much larger costs associated with the effort re3uired to correct and re"test ma$or faults.6emember that the amount of resources allocated to testing is a management decision basedon an assessment of the associated risks. >owever, few organi+ations are able to accuratelycompare the relative costs of testing and the costs associated with re"work.

E


35/75

2. RE#'TIVE COSTS TO +IG ERROR

he cost of fixing errors escalates as we move the pro$ect towards field use. From an analysis

of sixty"three pro$ects cited in )oehm& Software Cngineering Cconomics 1)oehm, /!.

G


36/75

2. &igh le7el test planning

@ou should be aware that many people use the term ;test plan; to describe a documentdetailing individual tests for a component of a system. We are introducing the concept ofhigh level test plans to show that there are a lot more activities involved in effective testingthan $ust writing test cases.

he ICCC standard for test documentation 1ICCC %7SI, / KStd /="/ 8,affectionately known as /=, defines a master validation test plan as follows&

Durpose of master test plan is to prescribe scope, approach, resources and schedule of testing

activities. % master test plan should include the following&

. est Dlan Identifier=. 6eferences. IntroductionE. est Items

G. Software 6isk IssuesB. Features to be testedH. Features not to be tested/. %pproach. Item DassFail -riteria9. Suspension -riteria. 6esumption 6e3uirements=. est 8eliverables. 6emaining esting asksE. Cnvironmental 7eedsG. Staffing and raining 7eeds

B. 6esponsibilitiesH. Schedule/. Dlanning 6isks and -ontingencies. %pprovals=9. 2lossary

B


37/75

=.B. SD%-C " the final frontier 1of testing?!

his may be useful acronym to help you remember what you should include in any high leveltest plan document. he actual format is up to you and will depend upon your own companystandards and other internal considerations. Space stands for Scope, Deople, %pproach,-riteria and Cnvironment and we have tried to map this onto the ICCC /= headings asfollows&

Scope Deople %pproach -riteria Cnvironment

I.est plan identifierG. Staffing andtraining

/. %pproach. Itempassfail

E.Cnvironmental

=.6eferences needs criteria needs.Introduction

E.est Items B. 6esponsibilities =. est deliverables 9.Suspensio

nest lab

B.Features to be tested. 6emaining testtasks

criteria est network

H.Features notto be tested

G.Software risk issues H. Schedule . %pprovals.%ssumption

est data

re3uirements

(se of live data

/. Dlanning risks and 5ffice re3uirements (se of testing tools 7umber of

contingencies cycles es

=9.2lossary 4otivationrewards -onfigurationmanagement

H


38/75

Cxercise

&igh #e7el Planning

We will use the case study for the *ondon %mbulance Service;s new computer aided8ispatch system to discuss the test planning issues. ake a few moments to thinkabout how you would test such a system and we will complete the outline plan as a

group exercise.

Scope

Deople

%pproach

-riteria

Cnvironment

/


39/75

2.5 Coponent testing

-omponent testing is described fully in )S"H=G and should be aware that component testingis also known as unit testing, module testing or Drogram esting. he definition from)SH=G is simply the testing of in-i7i-,al soft0are coponents.

raditionally, the programmer carries out component testing. his has proved to be lesseffective than if someone else designs and funs the tests for the component.

R)uddyR testing, where two developers test each other;s work is more independent and oftenmore effective. >owever, the coponent test strateg/ should describe what level ofindependence is applicable to a particular component.

(sually white box 1structural! testing techni3ues are used to design test cases for componenttests but some black box tests can be effective as well.

We have already covered a generic test process in this course. he component test process isshown in the following diagram&

2.; ITE!R'TIO TESTI!

Integration is the process of combining components into larger assemblies. From the standard)S"H=G integration testing is defined as Rtesting performed to expose faults in theinterfaces and in the interaction between integrated componentsR, >owever, in this sectionwe look at two interpretations of integration testing known as integration testing in the largeand integration testing in the small.

B/ Integration Testing in the #arge we mean testing the integration of the new system orsoftware package with other 1complete! systems. his would include the identification of,


40/75

and risk associated with, all interfaces to these other systems. %lso included is testing of anyinterfaces to external organi+ations 1e.g. C8I " electronic data interchange, Internet! but nottesting of the processing or operation of those external systems.

Cxercise

-ompare the definition of integration testing in the large with )/H=G 1G9G! definition of

interface testing. %ny comments?

We use Integration esting in the Small in the more traditional sense of integration testingwhere components are assembled into sub"systems and subsystems are linked together toform complete systems. Integration strategies may be incremental or non"incremental andinclude&

bin"ban

op"down

)ottom"up

Sandwich

esting approach is directly related to integration strategy chosen.

Stu#s $ %ri&ers

% stub is a skeletal or special purpose implementation of a software module, used to developor test a component that calls or is otherwise dependent on it.

% test driver is a program or test tool used to execute software against a test case suite.

5&A12E& 205E

$y not refer to te pro9ect E; example in *7 in te component test plan for te

component 03? it as a good diagram soing stu!s and dri'ers6

E9


41/75

2.6 S$STE" TESTI!

System testing is defined as the process of testing an integrated system to verify that it meetsspecified re3uirements.

@ou will come across two very different types of system testing, Functional system testingand non"functional system testing. In plain Cnglish functional system testing is focuses ontesting the system based on what it is supposed to do. 7on"functional system testing looks atthose aspects that are important yet not directly related to what functions the systemperforms. For example if the functional re3uirement is to issue an airline ticket, the non"functional re3uirement might be to issue it within 9 seconds.

% functional re3uirement is R a re3uirement that specifies a function that a system or systemcomponent must performR. 6e3uirements"based testing means that the user re3uirementsspecification and the system re3uirements specification 1as used for contracts! are used toderive test cases. )usiness process"based testing based on expected user profiles 1e.g.scenarios, use cases!.

7on"functional re3uirements cover the following areas&

. *oad=. Derformance. StressE. SecurityG. (sabilityB. StorageH. Oolume/. Install ability. 8ocumentation9. 6ecovery

7on"functional re3uirements are $ust as important as functional re3uirement

E:ercise

*ist some non"functional re3uirements for *%S -%8 system.

E


42/75

=.9 %--CD%7-C CSI72

he definition of acceptance testing in )SH=G states that "acceptance testing is formaltesting conducted to enable a user, customer, or other authori+ed entity to determine whetherto accept a system or componentR. %cceptance testing may be the only form of testingconducted by and visible to a customer when applied to a software package. he mostcommon usage of the term relates to the user acceptance testing 1O%! but you should beaware that there are several other uses of acceptance testing, which we briefly describe here.

?ser acceptance testing " the final stage of validation. -ustomer should perform or beclosely involved in this. -ustomers may choose to do any test they wish, normally based ontheir usual business processes. % common approach is to set up a model office where systemsare tested in an environment as close to field use as is achievable.

Contract acceptance testing" a demonstration of the acceptance criteria, which would havebeen defined in the contract, being met.

'lpha 4 eta testing" In alpha and beta tests, when the software seems stable, people who

represent your market use the product in the same way 1s! that they would if they bought thefinished version and give you their comments. %lpha tests are performed at the developer;ssite, while beta tests are performed at the user;s sites.

Cxercise

What problems would *%S have avoided had they set some acceptance criteria?

2.11 "'ITE'CE TESTI!

"aintenance testing is not specifically defined in )SH=G but it is all about testing changesto the software after it has gone into productions.

here are several problems with maintenance testing. If you are testing old code the originalspecifications and design documentation may be poor or in some cases non"existent. Whendefining the scope of the maintenance testing it has to be $udged in relation to the changedcode# how much has changed, what areas does it really affect etc. his kind of impactanalysis is difficult and so there is a higher risk when making changes " it is difficult todecide how much regression testing to do.

E=


43/75

Chapter 3

)/naic Testing Techni=,es

RVhe system was not fully tested to a satisfactory level of 3uality and resiliencebefore full implementation on =B 5ctober =.R

Cxtract from the main conclusions of the official report into the failure of the *ondon %mbulanceService;s -omputer Systems on 5ctober =Bth and =Hth and 7ovember Eth =.

3.1 OVERVIEW

his module introduces idea of a test case design techni3ue. )roadly, these arecategori+ed as either functional or structural testing techni3ues. he advantage ofusing these proven design methods is that they provide a more intelligent andeffective means of identifying tests than a purely intuitive approach. @ou will beexpected to know two functional techni3ues and two structural techni3ues in detail

and be aware there are many other techni3ues that can be applied to test case design.here are many excellent books on testing techni3ues some of which are listed in theappendix.

3.2 OBJECTIVES


(nderstand the difference between black box 1functional! and white box1structural! testing techni3ues.

)e able to name at least three black box techni3ues.

(nderstand how to use e3uivalence partitioning and boundary value analysisto design test cases.

%ppreciate the use of state transition testing.

)e able to name at least three white box techni3ues.

(nderstand the meaning of statement testing and branch testing.

)e aware that the )SH=G standard details some 1but not all! of the recogni+edtesting techni3ues.

Mnow that all of these testing techni3ues can be supplemented by errorguessing.

E


44/75

3.3 ITRO)?CTIO

Cach of the techni3ues we are about to describe has its strengths and weaknesses. %useful rule of thumb is that if you are having difficulty applying a particular techni3ueto a testing problem then perhaps you ought to try a different techni3ue. his section

introduces the different types of testing techni3ue and discusses the differencebetween them.

3.3.1 +,nctional test techni=,es

Functional test techni3ues are often referred to as ;black box; test techni3ues and the commonparlance is that we are ;doing black box testing;. % functional test techni3ue will help designtest cases based on functionality of component or system under test, without necessarilyhaving to understand underlying detail of software design. -onsider functionality of systemof determine test inputs and expected results.

Structural test techni3ues are sometime called white box text techni3ues hence term ;whitebox testing;. 2lass box testing is less widely used term for structural test case design.Structural test techni3ues help design test cases based on internal structure and design ofcomponent or system under test. *ook at code, database spec, data model, etc., to determinetest inputs and expected results.

E:ercise

What;s the difference between )lack and White box testing?

)%SC8 57 (SC8 )@ S(I%)*C F56

)lack )ox 6e3uirements Independent tester System test

Functionality (sers %cceptance test

White )ox Internal structure 8eveloper (nit test

-ode 8) design 8esigner *ink testSub"system test

EE


45/75

3.% B#'C@ BOG TEC&I?ES

he following list of black box techni3ues is from )S H=G"=. 5n this course we willdescribe and give an example of only those ones highlighted in bold&

C3uivalent Dartitioning. )oundary Oalue %nalysis

State ransition esting

-ause"Cffect 2raphing ; Syntax esting

C3uivalence partitioning 1CD! is a test case design techni3ue that is based on the premise thatthe inputs and outputs of a component can be partitioned into classes that, according to thecomponent;s specification, will be treated similarly by the component.. hus the result oftesting a single value from an e3uivalence partition is considered representative of thecomplete partition.

%s an example consider any program that accepts days of ht week and months of they year asinputs. Intuitively you would probably not expect to have to test every date of the year. @ouwould obviously try months with 9 days 1e.g. Pune! and months with days 1e.g. Panuary!and you may even remember to try out the special case of February for both non"leap year1=/ days! and leap years 1= days!. C3ually, looking at the days of the week you would not,depending on the application, test every day. @ou may test for weekdays 1e.g. uesday! andweekends 1e.g. Sunday!. What you are in effect doing is deciding on e3uivalence classes forthe set of data in 3uestion.

7ot everyone will necessarily pick the same e3uivalence classes# there is some sub$ectivity

involved. )ut the basic assumption you are making is that anyone value from thee3uivalence, class, is as good as any other when we come to design the test.

We hope that you can see how this techni3ue can dramatically reduce the number of tests thatyou may have for a particular software component.

EG


46/75

E:ercise

-onsider a software component called 2C7C6%C"26%8I72, which is used by the systemto calculate student;s grade based on an examination mark and a coursework mark. hecomponent is passed an exam mark 1out ofHG! and a coursework mark 1out of =G! from which

it generates a grade fro the course in the range; %; to ;8;. he grade is calculated from theoverall mark, which is calculated as the sum of the exam mark and coursework marks, asfollows.

5OC6%** 4%6M 26%8C

2reater than or e3ual to H9 %

2reater than or e3ual to G9 but less than H9 )

2reater than or e3ual to 9 but less than G9 -

*ess than 9 8

Cxercise

Identify valid partitions&000000000

Identify invalid partitions&00000000

Identify output partitions&000000000

7ow derive some test cases based on input exam mark partition&

CS -%SC =

Input 1exam mark!

Input 1coursework!

otal mark 1calculated!

Dartition tested

Cxpected output

EB


47/75

he standard goes on to derive test cases form the other partitions but in this course we willnot have time to do this completely.

Bo,n-ar/ Val,e 'nal/sis is base on the following premise. Firstly, the inputs and outputs

of a component can be partitioned into classes that, according to the component;sspecification, will be treated similarly by the component and, secondly, that developers areprone to marking errors in their treatment of the boundaries of these classes. hus test casesare generated to exercise these boundaries.

E:ercise

*ook at the boundaries for input exam mark. -hoose values on each boundary and one either

side of it in order to generate a set of test cases.

CS -%SC = E G B

Input 1exam mark!

Input 1coursework!

)oundary tested

Cxpected output

%gain the standard continues to describe boundary tests for all of the other valid boundaries.

EH


48/75

.G $ite *ox 5ecni@ues

he following list of white box techni3ues is from )SH=G=. 5n this course we will describeand give an example of only those ones highlighted in bold.

Statement testing

)ranch 8ecision esting.

8ata Flow esting.

)ranch -ondition esting. .

)ranch -ondition -ombination esting

4odified -ondition 8ecision esting. *-S%P esting.

6andom esting.

Statement testing is a structural techni3ue based on decomposition of a software componentinto constituent statements. Statements are identified as executable or non"executable. Foreach test case, you must specify inputs to component, identification of statement1s! to beexecuted and expected outcome of test case.

Cxample of program with E statements&

:I7D(#@:E#

IF T@ >C74essage: R*imited exceededR

C*SC

4essage: R7o problems reportedR C78 IF

+:o

)ranch testing re3uires model of source code, which identifies decisions and decisionoutcomes. % decision is an executable statement, which may transfer control to anotherstatement depending upon logic of decision statement. ypical decisions are found in loops

and selections. Cach possible transfer of control is a decision outcome. F or each test caseyou must specify inputs to component, identification of decision outcomes to be executed bytest case and expected outcome of test case.

E/


49/75

3..1 Coparison of 0hite o: techni=,es fro Soft0are Testing in the Real Worl-

Cach column in this figure represents a distinct method of white"box testing, and each row1"E! defines a different test characteristics. For a given method 1column!, R@R in a given rowmeans that test characteristic is re3uired for method. R7R signifies no re3uirement. RImplicitRmeans test characteristic is achieved implicitly by other re3uirements of method. 1 ,E Software 8evelopment echnologies! reproduced with permission.

3.5Summary

In module three you have learnt that applying a formal, recogni+ed testing techni3ue in asystematic way is the most effective approach to finding errors in software. In particular youcan now

Cxplain the difference between black box 1functional! and white box1structural! testing techni3ues.

7ame at least three black box techni3ues.

se e3uivalence partitioning and boundary value analysis to design testcases.

6ecogni+e a state transition testing techni3ue.

7ame at least three white box techni3ues.

(nderstand what the meaning of statement testing and branch testing.

se the standard )SH=G to find out more about testing techni3ues.

Mnow when to apply error"guessing techni3ues to supplement the formal

E


50/75

techni3ues.

Chapter %

Static testing

R5n 7ovember Eth the system did fail. his was caused by a minor programmingerror that caused the system to RcrashR, the automatic change over to the back upsystem had not been ade3uately tested, and thus the whole system was brought

downR.

Cxtract from the main conclusions of the official report into the failure of the *ondon %mbulance Service ;s

-omputer Systems on 5ctober =Bth and =Hth and 7ovember E th=.

E.9 S%I- CSI72

%.1 O7er7ie0

Static testing techni3ues is used to find errors before software is actually executedand contrasts therefore with dynamic testing techni3ues that are applied to a working

system. he earlier we catch an error, the cheaper it is, usually, to correct. hismodule looks at a variety of different static testing techni3ues. Some are applied todocumentation 1e.g. walkthroughs, reviews and Inspections! and some are used toanaly+e the physical code 1e.g. compilers, data flow analy+ers!. his is a huge sub$ectand we can only hope to give an introduction in this module. @ou will be expected toappreciate the difference between the various review techni3ues and you will need tobe aware of how and when static analysis tools are used.

%.2 OAecti7es


(nderstand the various review techni3ues that you can apply todocumentation and code.

%ppreciate the difference between walkthroughs, formal reviews andinspections.

(nderstand how static analysis techni3ues can detect errors in code.

(nderstand two complexity metrics 1lines ofcode and 4c-abe;s metric!.

G9


51/75

%.3 REVIEWS ') T&E TEST PROCESS

%.3.1 What is a re7ie0(

% review is a fundamental techni3ue that must be used throughout the development

lifecycle. )asically a review is any of a variety of activities involving evaluation of

technical matter by a group of people working together. he ob$ective of any review

is to obtain reliable information, usually as to status andor work 3uality.

%.3.2 Soe histor/ of re7ie0s

8uring any pro$ect, management re3uires a means of assessing a measuring progress. he so"called progress review evolved as means of achieving this. >owever, results of those earlyreviews proved to be bitter experiences for many pro$ect managers. Pust how long can apro$ect remain at 9X complete? hey found that they could not measure ;true; progress untilthey had a means of gauging the 3uality of the work performed. hus the concept of thetechnical review emerged to examine the 3uality of the work and provide input to theprogress reviews.

%.3.3 What can e re7ie0e-(

here are many different types of reviews throughout the development life cycle.

Oirtually any work produced during development can be 1and is! reviewed. his

includes, re3uirements documents, designs, database specifications, designs, data

models, code, test plans, test scripts, test documentation and so on.

%.3.% What has this got to -o 0ith testing(

he old fashioned view that reviews and testing are totally different things stemsfrom the fact that testing used to be tacked onto the end of the development lifecycle.>owever as we all now view testing as a continuous activity that must be started asearly as possible you can begin to appreciate the benefits of reviews. 6eviews are theonly testing techni3ue that is available to us in the early stages of testing. %t earlystages in the development lifecycle we obviously cannot use dynamic testingtechni3ues since the software is simply not ready for testing.

6eviews share similarities to test activities in that they must be planned 1what are wetesting!, what are the criteria for success 1expected results! and who will do the work

1responsibilities!. he next section examines the different types of review techni3uesin more detail.

G


52/75

%.% T$PES O+ REVIEW

Walk"through, informal reviews, technical reviews and Inspections are fundamentaltechni3ues that must be used throughout the development process. %ll have theirstrengths and weaknesses and their place in the pro$ect development cycle. %ll fourtechni3ues have some ground rules in common as follows&

% structured approach must be for the review process.

)e sure to know what is being reviewed " each component must have a uni3ueidentifier.

-hanges must be configuration controlled.

6eviewers must prepare.

6eviewers must concentrate on their own speciali+ation.

)e sure to review the product, not the person.

here must be&

otal group responsibility.

-orrect si+e of review group.

-orrect allocation of time.

-orrect application of standards.

-hecklists must be used.6eports must be produced.Quality must be specified in terms of&

%dherence to standards.

6eliability re3uired.

6obustness re3uired.

4odularity.

%ccuracy.

%bility to handle errors.

G=


53/75

%. REVIEW TE'" SIKE ') CO"POSITIO

Droblems with small teams must be avoided# bring in extra people, 1perhaps use theesting eam! to bring extra minds to bear on the issues.

5pinion is often divided as to whether or not the author should participate in a review. hereare advantages to both scenarios. Since specifications and designs must be capable of beingunderstood without the author present, an Inspection without them tests the document.%nother reason for excluding the author from the review is that there should be teamownership of the product and team responsibility for the 3uality of all the deliverables,maintaining ownership via the author runs against this.

%lternatively, including the author can be a valuable aid to team building. C3ually, an authormay well be able to clear up misunderstandings in a document more 3uickly than anotherteam member, thus saving the reviewer valuable time. From a practical perspective however,it is worth remembering that an author is the least likely person to identify faults in thedocument.

he one person who should not attend reviews is the manager. If, as in some cases, themanager is also a contributor to the deliverables, he should be included but treated the sameas other group members. It is important that reviewers are a peer group process.

%. T$PE 1* 2 ') 3 REVIEW PROCESSES

6eview process is both most effective and universal test method and managementneeds to make sure that review process is working as effectively as possible. (sefulmodel for manager is ,=, model.

, =, model is derived from work of first working party of )ritish -omputer Society

Specialist Interest 2roup in Software esting and book that came from this work# inSoftware 8evelopment.

ype testing is process of making sure that product 1document, program, screendesign, clerical procedure or Functional Specification! is built to standards andcontains those features that we would expect from the name of that product. It is a testto make sure that produce conforms to standards, is internally consistent, accurate andunambiguous.

ype = testing is process of testing to see if product conforms to re3uirements as specified byoutput of preceding pro$ect stage and ultimately Specifications of 6e3uirements for whole

pro$ect. ype = testing is backward looking and is checking that product is consistent withpreceding documentation 1including information on change!.

ype testing is forward looking and is about specification of certification processand test that are to be done on delivered product. It is asking 3uestion " -an we canbuild deliverables 1test material, training material, next stage analysisdocumentation!?

G


54/75

%..1 "a


55/75

8efective %ll agreed otal rework

GG


56/75

8efective but %ll agreed 6ework andsoluble solution review all

acceptable material

Dossible Some but not all Seek explanationdefect, needs agree possibly review

explanation some of work

Quality issue Drefer an -osts comparedalternative to standards

%cceptable %ll accept Droceed

5ver 4ost agree Droceed butdeveloped for review budgetsthe budget

)e sensitive to voice of concerned but not necessarily assertive tester in team# thisperson may well have observed a fault that all others have missed. It should not bethat person with load voice or strong personality is allowed to dominate.

%.5 ISPECTIOS

his section on Inspections is based on an edited version of selected extracts fromom 2ib and 8orothy 2raham;s book K2ib, 2raham L.

%.5.1 Intro-,ction

4ichael C. Fagan at I)4 Mingston 7@ *aboratories developed the Inspectiontechni3ue. Fagan, a certified 3uality control engineer, was a student of the methods ofthe 3uality gurus W. Cdwards 8eming and P. 4. Puran.

Fagan decided, on his own initiative, to use industrial hardware statistical 3ualitymethods on a software pro$ect he was managing in H="HE. he pro$ect consisted ofthe translation of I)4Ys software from %ssembler *anguage to D*S, a high"levelprogramming language assembler. Fagan;s achievement was to make statistical3uality and process control methods work on ;ideas on paper;. In HB he reported hisresults outside I)4 in a now famous paper KFagan, HBL.

he Inspection techni3ue was built further by -aroline *. Pones and 6obert 4ays atI)4 K$ones, /GL who created a number of useful enhancements&

he kickoff meeting, for training, goal setting, and setting a strategy for thecurrent inspection

Inspection cycle#

he causal analysis meeting#

he action database#

he action team.

GB


57/75

%.5.2 Re7ie0s an- 0al


58/75

effective at finding faults, and this causes problems at later stages, test execution, andoperational use.

We need to learn from both Inspection and test experiences. Inspection and testingshould both ideally 1but all too rarely in practice! produce product"fault metrics andprocess improvement metrics, which can be used to evaluate the softwaredevelopment process. 8ata should be kept on faults found in Inspection, faults found

in testing, and faults that escaped both Inspection and test, and were only discoveredin the field. his data would reflect fre3uency, document location, security, cost offinding, and cost of fixing.

here is a trade"off between fixing and preventing. he metrics should be used to fine"tunethe balance between the investment in the fault detection and fault prevention techni3uesused. he cost of Inspection, test design, and test running should be compared with the costof fixing. he faults at the time they were found, in order to arrive at the most cost"effectivesoftware development process.

%.5. )ifferences et0een Inspection an- testing

Inspection can be used long before executable code is available to run tests.Inspection can be applied mud earlier than dynamic testing, but can also be appliedearlier than test design activities. est can only be defined when a re3uirements ordesign specification has been written, since that specification is the source forknowing the expected result of a test execution.

he one key thing that testing does and Inspection does not, is to evaluate thesoftware while it is actually performing its function in its intended 1or simulated!environment. Inspection can only examine static documents and models# testing canevaluate the product working.

Inspection, particularly the process improvement aspect, is concerned with preventingsoftware engineers from inserting any form of fault into what they write. heinformation gained from faults found in running tests could be used in the same way,but this is rare in practice.

%.5. Benefits of Inspection

5pinion is divided over whether Inspection is a worthwhile element of any product;development; process. -ritics argue that it is costly# it demands too much ;upfront;time and is unnecessarily bureaucratic. Supporters claim that the eventual savings and

benefits outweigh the costs and the short"term investment is crucial for long"termsavings.

G/


59/75

In I"Start;s 8evelopers 2uide 1/!, it is estimated that ;he cost of non"3uality softwaretypically accounts for 9X of development costs and G9X to B9X of the lifecycle costs;.Faults then are costly, and this cost increases the later they are discovered. Inspection appliedto all software is arguably the prime techni3ue to reduce defect levels 1in some cases tovirtually +ero defects! and to provide an increased maturity level through the use ofInspection metrics. he savings can be substantial.

)irect sa7ings

)e7elopent pro-,cti7it/ is ipro7e-.

Fagan, in his original article, reported a =X increase in ;coding productivity alone;

using Inspection KFagan, HB, I)4 Systems Pournal, p /HL. >e later reported

further gains with the introduction of moderator training, design and code change

control, and test fault tracking.

)e7elopent tiescale is re-,ce-.

-onsidering only the development timescales, typical net savings for pro$ect

development are GX to G9X.

Cost an- tie ta


60/75

OrganiHational an- people enefits.

For software professionals Inspection means their work is of better 3uality and moremaintainable. Furthermore, they can expect to live under less intense deadlinepressure. heir work should be more appreciated by management, and theircompany;s products will gain a competitive edge.

%.5.5 Costs of Inspection

he cost of running an Inspection is approximately 9X " GX of the developmentbudget. his percentage is about the same as other walkthrough and review methods.>owever, Inspection finds far more faults for the time spent and the upstream costscan be $ustified by the benefits of early detection and the lower maintenance costs thatresult.

%s mentioned earlier, the costs of Inspection include additional ;up front; time in thedevelopment process and increased time spent by authors writing documents theyknow will be Inspected. Implementing and running Inspections will involve long"

term costs in new areas. %n organi+ation will find that time and money go on&

Inspection leading training.

4anagement training.

4anagement of the Inspection leaders.

4etric analysis.

Cxperimentation with new techni3ues totry to improve Inspection results.

Dlanning, checking and meeting activity& the entire Inspection process itself.

Quality improvement& the work of the process improvement teams.

he company may also find it effective to consider computeri+ed tools fordocumentation and consistency checking. %nother good investm

Documents

Istqb Fl Studyguide