106
Get Homework/Assignment Done Homeworkping.com Homework Help https://www.homeworkping.com/ Research Paper help https://www.homeworkping.com/ Online Tutoring https://www.homeworkping.com/ click here for freelancing tutoring sites Chapter 1 Principles of Testing 1.1 OVERVIEW In this module you will get an overview of the most fundamental principles of testing. Firstly, we point out that although you will come across different terminology throughout your career, professional testers are all pretty much agreed on these basic ideas that we describe in this module. Secondly, we take a look at the need for proper testing look at the cost of getting it wrong and we show why exhaustive testing is neither possible not practical. What are errors and how do they get into the software? Thirdly, we describe a fundamental test process, base on industry standards, and underline importance of planning) 1

202158425 istqb-fl-studyguide

Embed Size (px)

Citation preview

Page 1: 202158425 istqb-fl-studyguide

Get Homework/Assignment Done Homeworkping.comHomework Help https://www.homeworkping.com/

Research Paper helphttps://www.homeworkping.com/

Online Tutoringhttps://www.homeworkping.com/

click here for freelancing tutoring sites

Chapter 1Principles of Testing

1.1 OVERVIEW

In this module you will get an overview of the most fundamental principles of testing.

Firstly, we point out that although you will come across different terminology throughout your career, professional testers are all pretty much agreed on these basic ideas that we describe in this module.

Secondly, we take a look at the need for proper testing look at the cost of getting it wrong and we show why exhaustive testing is neither possible not practical. What are errors and how do they get into the software?

Thirdly, we describe a fundamental test process, base on industry standards, and underline importance of planning) tests and determining expected results in advance of test execution.

We conclude with a look at re-testing and regression testing; why it is so important to go that extra mile and run a suite of tests again, just when you thought it was safe to hit the release date.

1.2 OBJECTIVES

After completing this module you will:

» Understand basic testing terminology.» Understand why testing is necessary.

1

Page 2: 202158425 istqb-fl-studyguide

» Be able to define error, fault and failure.» Appreciate why errors occur and how costly they can be.» Understand that you cannot test everything and that testing is therefore a risk management process. » Understand the fundamental test process. » Understand that developers and testers have different mindsets.» Learn how to communicate effectively with both developers and testers.» Find out why you cannot test your own work.» Understand the need for regression testing.» Understand the importance of specifying your expected results in advance.» Understand how and why tests should be prioritized.

1.3 TESTING TERMINOLOGY

There is no generally accepted set of testing definitions used by the worldwide testing community. The British Standard for Software Component Testing published in August 1998 provides a new source of testing definitions. When involved in a testing project it is important to ensure everyone understands terminology adopted; you may find different terminology is used in your organization.

Exercise

The British Standard for Software Component Testing is known as BS___________A useful glossary of terms used in software testing is called BS___________

Although it is a British Standard published by the BSI (British Standards Institute), the Specialist Interest Group in Software Testing (SIGIST) developed it over a period of several years.

ANSWERS: 7925-PART2, 7925-PART1

1.4 WHY IS TESTING NECESSARY?

This section explains why testing is necessary and closely at the cost and consequences of errors in computer software.

1.4.1 Definitions Errors, Faults and Failures

An error is a human action that produces an incorrect result. A fault is a manifestation of an error in software. Faults are also known colloquially as defaults or bugs. A fault, if encountered, may cause a fault, which is a deviation of the software from its existing delivery

2

Page 3: 202158425 istqb-fl-studyguide

or service.

We can illustrate these points with the true story Mercury spacecraft. The computer program aboa spacecraft contained the following statement wri1 the FORTRAN programming language.

DO 100 i = 1.10

The programmer's intention was to execute a succeeding statements up to line 100 ten times then creating a loop where the integer variable I was using the loop counter, starting 1 and ending at 10.

Unfortunately, what this code actually does is writing variable i do to decimal value 1.1 and it does that once only. Therefore remaining code is executed once and not 10 times within the loop. As a result spacecraft went off course and mission was abort considerable cost!

The correct syntax for what the programmer intended is DO 100 i =1,10

Exercise

What do you think was the error, fault and failure in this example?

The error is __________

The fault is ___________

The failure is __________

1.4.2 Reliability

Reliability is the probability that software will not cause the failure of a system for a specified time under specified conditions. Measures of reliability include MTBF (mean time between failure), MTTF (mean time to failure) as well as service level agreements and other mechanisms.

1.4.3 Errors and how they occur

Why do we make errors that cause faults in computer software leading to potential failure of our systems? Well, firstly we are all prone to making simple human errors. This is an unavoidable fact of life. However, this is compounded by the fact that we all operate under real world pressures such as tight deadlines, budget restrictions, conflicting priorities and so on.

1.4.4 Cost of errors

The cost of an error can vary from nothing at all to large amounts of money and even loss of life. The aborted Mercury mission was obviously very costly but surely this is just an isolated example. Or is it? There are hundreds of stories about failures of computer systems that have

3

Page 4: 202158425 istqb-fl-studyguide

been attributed to errors in the software. A few examples are shown below:

A nuclear reactor was shut down because a single line of code was coded as X = Y instead of X=ABS (Y) i.e. the absolute value of Y irrespective of whether Y was positive or negative.

Blue Cross of Wisconsin installed a new $200m claims processing system - it sent out $60 million in unwarranted and duplicate payments. Also, when a data entry clerk typed 'none' in the town field the system sent hundreds of checks to the non-existent town of 'NONE' in Wisconsin.

In May 1992, Pepsi fan a promotion in the Philippines. It told customers they could win a million pesos (approx. $40,000) if they bought a bottle of Pepsi and found number 349 stamped on the underside of the bottle cap. Unfortunately, due to a software error, 800,000 bottle caps were produced with number 349 instead of one, which was an equivalent of $42 billion in prize money. It cost the company dearly as some people pursued their claims through the courts and Pepsi paid out millions of dollars in compensation.

Another story was printed in the New York Times on 18th February 1994. Chemical Bank managed to allow $15 million to be withdrawn incorrectly from 100,000 accounts - a single line error in the program caused every ATM on their network to process the transaction twice.

1.4.5 what happened on October 26th & 27th 1992?

The London Ambulance Service (LAS) covers an area of just over 600 square miles and is the largest ambulance service in the world. It covers a resident population of some 6.8 million, but its daytime population is larger, especially in central London. Since 1974 South West Thames Regional Health Authority has managed it.

The LAS carries over 5000 patients every day and receives between 2000 and 2500 calls daily (including between 1300 and 1600 emergency calls i.e. 999 calls). In terms of resources the LAS has some 2700 staff, 750 ambulances and a small number of various other vehicles including 1 helicopter. LAS make almost 2 million patient journeys each year. In 1992/1993 its budgeted income was £69.7 million.

On the 26th and 27th October 1992 the new system failed, ambulances failed to turn up and people lost their lives. Although no Coroners Court ever apportioned blame for any deaths directly to the computer systems failure, it was by any standards a major disaster and made the main evening news bulletins on several occasions.

1.4.6 London Ambulance Service

In summary the problems were:Computer Aided Dispatch - 1

The system relied on near perfect information to propose optimum resource to be allocated to an emergency. However, there were imperfections in the information and changes in operational procedures made it difficult for staff to correct the system.

4

Page 5: 202158425 istqb-fl-studyguide

This was not a problem when it went live at 7 am on 26th October 1992 as the system load was light; however as the number of emergency calls increased throughout the-day it became increasingly difficult for staff to correct errors; this led to:

Poor, duplicated and delayed allocations. Build-up of exception messages and awaiting attention list. Slow-down of system as messages and lists built up. Increased number of callbacks and hence delays in telephone answering.

The cost of these errors were ultimately that ambulances didn't turn up and people lost their lives although the official enquiry report did not attribute any fatalities to the system problems. The costs in terms of loss of confidence in the computer system, industrial relations and so on were probably also high.

1.4.7 Exhaustive testing why not test everything?

It is now widely accepted that you cannot test everything. Exhausted testers you will find, but exhaustive testing you will not. Complete testing is neither theoretically, nor practically possible. Consider a 10-character string that has 280 possible input streams and corresponding outputs. If you executed one test per microsecond it would take approx. 4 times the age of the Universe to test this completely. For a survey of the methodology and limitations of formal proofs of program correctness see [Manna 78]

1.4.8 Testing and risk

How much testing would you be willing to perform if the risk of failure were negligible? Alternatively, how much testing would you be willing to perform if a single defect could cost you your life's savings, or, even more significantly, your life? [Hetzel 88].

The amount of testing performed depends on the risks involved. Risk must be used as the basis for allocating the test time that is available and for selecting what to test and where to place emphasis. A priority must be assigned to each test.

Test Managers and Project Managers come up with different prioritization schemes but ht basic principle is that you must focus the testing effort on those areas of the system that are likely to have the most defects. Another key principle is that you must execute the most important test first. Otherwise, if you run out of time, which is likely, you will not have exercised the tests that give you the best payback in terms of faults found.

1.4.9 Testing and quality

Testing identifies faults whose removal increases the software quality by increasing the software's potential reliability. Testing is the measurement of software quality. We measure how closely we have achieved quality by testing the relevant factors such as correctness, reliability, usability, maintainability, reusability, testability etc.

5

Page 6: 202158425 istqb-fl-studyguide

1.4.10 Testing and legal, contractual, regulatory or mandatory requirements

Other factors that may determine the testing performed may be legal, contractual requirements, normally defined in industry specific standards or based on agreed best practice (or more realistically non-negligent practice).

1.4.11 How much testing is enough?

It is difficult to determine how much testing is enough. Testing is always a matter of judging risks against cost of extra testing effort. Planning test effort thoroughly before you begin, and setting completion criteria will go some way towards ensuring the right amount of testing is attempted. Assigning priorities to tests will ensure that the most important tests have been done should you run out of time.

1.5 FUNDAMENTAL TEST PROCESS

1.5.1 Introduction

Testing must be planned. This is one of Bill Hetzel's 6 testing principles [Hetzel 88 p25] and he says we are all agreed on this one. However, he points out that the problem is that most of us do not discipline ourselves to act upon it. Good testing requires thinking out an overall approach, designing tests and establishing expected results' for each of the test cases we choose.

You have seen already that we cannot test everything, we must make a selection, and the planning and care we expend on that selection accounts for much of the difference between good and poor testers.

The quality and effectiveness of software testing are primarily determined by the quality of the test processes used [Kit 95]. This is one of Ed Kit's 6 essentials of software testing. Test groups that operate within organizations that have an immature development process will feel more pain than those that do not. However, the test group should strive to improve its own internal testing processes. This section of the course shows a fundamental test process, based on the BS7925-2 standard for software component testing.

The fundamental test process comprises planning, specification, execution, recording and checking for completion. You will find organizations that have slightly different names for each stage of the process and you may find some processes that have just 4 stages, where 4 & 5 are combined, for example. However, you will find that all good test processes adhere to this fundamental structure.

1.5.2 Test process stages

See BS7925-2 for diagram of test process. Test planning involves producing a document that describes an overall approach and test objectives noting any assumptions you have made and

6

Page 7: 202158425 istqb-fl-studyguide

stating any exceptions to your overall test strategy for your project. Test planning can be applied at all levels. Completion or exit criteria must be specified so that you know when testing (at any stage) is complete. Often a coverage target is set and used as test completion criteria.

Test specification (sometimes referred to as test design) involves designing test conditions and test cases using recognized test techniques identified at the planning stage. Here it is usual to produce a separate document or documents that fully describe the tests that you will carry out. It is important to determine the expected results prior to test execution.

Test execution involves actually running the specified test on a computer system either manually or by using an automated test tool.

Test recording involves keeping good records of the test activities that you have carried out. Versions of the software you have tested and the test specifications are software you have tested and the test specifications are recorded along with the actual outcomes of each test.

Checking for test completion involves looking at the previously specified test completion criteria to see if they have been met. If not, some test may need to be rerun and in some instances it may be appropriate to design some new test cases to meet a particular coverage target.

Note that BS7925-2 does not specify the format of any test documentation. However, The IEEE standard, known as 829, specifies in detail a standard for software test documentation.

BS7925-2 shows a diagram of a suggested hierarchy of test documentation.

HOME WORK

Exercise

Putting aside management problems. Read through test documentation examples in BS7925-2 and answer following questions:

What test techniques does component test strategy stipulate?

What percentage of decision coverage is required?

What should be done if errors are found?

The project component test plan is useful because the approach outlined allows:

a) Strict adherence to the component test strategyb) More faults to be identified in the LOG componentsc) A basic working systems to be established as early as possible d) Isolation of the components within the test strategy

The component test plan must consist of a single document? TRUE/FALSE

7

Page 8: 202158425 istqb-fl-studyguide

The component test plan must specify test completion criteria? TRUE/FALSE

Why does component test plan specify 100% DC whereas strategy required 90%?

Which test case deals with non-numeric input?

List the expected outcome and the test condition

Why does the CTP have additional/altered test cases?

What action has been taken as a result of the test report?

8

Page 9: 202158425 istqb-fl-studyguide

1.5.3 Successful tests detect faults

As the objective of a test should be to detect faults, a successful test is one that does detect a fault. This is counter-intuitive, because faults delay progress; a successful test is one that may cause delay. The successful test reveals a fault which, if found later, may be many more times costly to correct so in the long run, is a good thing.

1.5.4 Meaning of completion or exit criteria

Completion or exit criteria are used to determine when testing (at any stage) is complete. These criteria may be defined in terms of cost, time, faults found or coverage criteria.

1.5.5 Coverage criteria

Coverage criteria are defined in terms of items that are exercised by test suites, such as branches, user requirements, and most frequently used transactions etc.

1.6 THE PSYCHOLOGY OF TESTING

1.6.1 Purpose

The purpose of this section is to explore differences in perspective between tester and developer (buyer & builder) and explain some of the difficulties management and staff face when working together developing and testing computer software.

1.6.2 Different mindsets

We have already discussed that none of the primary purposes of testing is to find faults in software i.e., it can be perceived as a destructive process. The development process on the other hand is a naturally creative one and experience shows that staff that work in development have different mindsets to that of testers.

We would never argue that one group is intellectually superior to another, merely that they view systems development from another perspective. A developer is looking to build new and exciting software based on user's requirements and really wants it to work (first time if possible). He or she will work long hours and is usually highly motivated and very determined to do a good job.

A tester, however, is concerned that user really does get a system that does what they want, is reliable and doesn't do thing it shouldn't. He or she will also work long hours looking for faults in software but will often find the job frustrating as their destructive talents take their tool on the poor developers. At this point, there is often much friction between developer and tester. Developer wants to finish system but tester wants all faults in software fixed before their work is done.

9

Page 10: 202158425 istqb-fl-studyguide

In summary:

Developers:

Are perceived as very creative - they write code without which there would be no system! .

Are often highly valued within an organization. Are sent on relevant industry training courses to gain recognized qualifications. Are rarely good communicators (sorry guys)! Can often specialize in just one or two skills (e.g. VB, C++, JAVA, SQL).

Testers:

Are perceived as destructive - only happy when they are finding faults! Are often not valued within the organization. Usually do not have any industry recognized qualifications, until now Usually require good communication skills, tack & diplomacy. Normally need to be multi-talented (technical, testing, team skills).

1.6.3 Communication b/w developer and tester

It is vitally important that tester can explain and report fault to developer in professional manner to ensure fault gets fixed. Tester must not antagonize developer. Tact and diplomacy are essential, even if you've been up all night trying to test the wretched software.

1.6.4 How not to approach

Tester: "Hey Fred. Here's a fault report AR123. Look at this code. Who wrote this? Was it you? Why, you couldn't program your way out of a paper bag. We really want this fixed by 5 o'clock or else."

We were unable to print Fred's reply because of the language! Needless to say Fred did not fix the fault as requested.

10

Page 11: 202158425 istqb-fl-studyguide

Exercise

Your trainer will split you into small test teams. One of you will be the test team leader. You have found several faults in a program and the team leader must report these to the developer (your trainer). The background is that your team has tested this program twice before and their are still quite a lot of serious faults in the code. There are also several spelling mistakes and wrong colors on the screen layout. The test team is getting a bit fed up. However, you have to be as nice as possible to the developer.

1.6.6 Why can't we test our own work?

This seems to be a human problem in general not specifically related to software development. We find it difficult to spot errors in our own work products. Some of the reasons for this are:

We make assumptions We are emotionally attached to the product (it's our baby and there's nothing

wrong with it). We are so familiar with the product we cannot easily see the obvious faults. We're humans. We see exactly what we want to see. We have a vested interest in passing the product as ok and not finding faults.

Generally it is thought that objective independent testing is more effective. There are several levels of independence as follows:

Test cases are designed by the person(s) writing the software. Test cases are designed by another person(s). Test cases are designed by a person(s) from a different section. Test cases are designed by a person(s) from a different organization. Test cases are not chosen by a person.

The discussion of independence test groups and outsourcing is left to another section.

11

Page 12: 202158425 istqb-fl-studyguide

1. 7 RE-TESTING AND REGRESSION TESTING

We find and report a fault in LOG 3, which is duly fixed by the developer and included in the latest release which we now have available for testing. What should we do now?

Examples of regression tests not carried out include:

The day the phones stopped. . LAS failure on 4th November (perhaps) Ariane 5 failure.

Whenever a fault is detected and fixed then the software should be re-tested to ensure that the original fault has bee successfully removed. You should also consider testing for similar and related faults. This is made easier if your tests are designed to be repeatable, whether they are manual or automated.

Regression testing attempts to verify that modifications have not caused unintended adverse side effects in the unchanged software (regression faults) and that the modified system still meets requirements. It is performed whenever the software, or its environment, is changed.

Most companies will build up a regression test suite or regression test pack over time and will add new tests, delete unwanted test and maintain tests as the system evolves. When a major software modification is made then the entire regression pack is likely to be run (albeit with some modification). For minor planned changes or emergency fixes then during the test planning phase the test manager must be selective and identify how many of the regression tests should be attempted. In order to react quickly to an emergency fix the test manager may create a subset of regression test pack for immediate execution in such situations.

Regression tests are often good candidates for automation provided you have designed and developed automated scripts properly (see automation section).

In order to have an effective regression test suite then good configuration management of your test assets is desirable if not essential. You must have version control of your testDocumentation (test plans, scripts etc.) as well as your test data and baseline databases. An inventory of your test environment (hardware configuration, operating system version etc.) is also necessary.

12

Page 13: 202158425 istqb-fl-studyguide

1.8 EXPECTED RESULTS

The specification of expected results in advance of test execution is perhaps one of the most fundamental principles of testing computer software. If this step is omitted then human subconscious desire for tests to pass will be overwhelming and tester may perhaps interpret a plausible, yet erroneous result, as correct outcome. .

As you will see when designing test using black box and white box techniques there is ample room within the test specification to write down you expected results and therefore no real excuse for not doing it. If you are unable to determine expected results for a particular test that you had in mind then it its not a good test as you will not be able to (a) determine whether it has passed or not and (b) you will never be able to repeat it.

Even with a quick and dirty ad-hoc test it is advisable to write down beforehand what you expect to happen. This may all sound pretty obvious but many test efforts have floundered by ignoring this basic principle.

" The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong does go wrong it usually turns out to be impossible to get at or repair."

--Douglas Adams

13

Page 14: 202158425 istqb-fl-studyguide

SOME TESTING TERMINOLOGY

Faults - a mistake in the code that causes the software to not behave as expected (causes)

Failures - the act of a product not behaving as expected - the manifestation of a fault (symptoms)

Validation - establishing the correspondence between the software and its specification - "are we building the right product?"

Test care - the collection of inputs, predicted results and execution conditions for a single test

Ad-lib/ad-hoc test care - a test executed without prior planning - especially if the expected behaviors is not known prior to running the test.

Pass/fail criteria - decision rules used to determine whether a product passes or fails a given test

Coincidental correctness - when behavior appears to be what is expected, but it is just a coincidence

Test suite - a collection of test cases necessary to "adequately" test a product

Test plan - a document describing the scope, approach, resources and schedule of intended testing activity - identifies features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.

Oracle - a procedure, process or magical phenomenon that can determine if the actual behavior matches the expected behavior

Incident - when a test produces an unexpected outcome, - further effort is necessary to classify the incident as a software error, a design error, a specification error, a testing error, etc.

Bug report - a method of transmitting the occurrence of a discrepancy between actual and expected output to someone who cares for "follow-up" - also known as discrepancy report, defect report, problem report, etc.

Work-around - a procedure by which an error in the product can be "by-passed" and the desired function achieved.

14

Page 15: 202158425 istqb-fl-studyguide

Why Write Programs?

Concept: If you want your computer to do exactly what you want it to do, you must write a program.

A computer does nothing on its own. In fact, a computer is a dumb machine with no intelligence whatsoever. Despite what you might read in science fiction stories, a computer does nothing more than blindly follow instructions supplied by a programmer. Computers cannot think.

Definition: A program is a set of instructions that tells the computer exactly what to do.

When someone buys a computer today, the computer sits on the desk doing nothing until he loads a program into the computer's internal memory and starts running the program. Just as a VCR does not record shows on its own without being programmed to do so, a computer requires detailed instructions found only in programs.

Suppose that you own rental properties and want to use your computer to track your tenant records. Your computer will not help you out in any way until you load and run a rental property program. Where do you find such a program? There are two ways to obtain programs for computers. You can

(1) Buy one and hope that the program does exactly what you want it to do.

(2) Write your own program.

It's much easier and faster to buy a program that you need. Thousands of programs are on the market today. In fact, there are so many programs out there that you might not see the need for writing your own programs.

If you can find a program that does exactly what you want, you are ahead of the game. If you find a program that meets your exact requirements, you should buy that program because purchasing a program is often less expensive and much quicker than writing the same program yourself or hiring programmers to write it for you.

Think about this for a moment, though: If there are so many programs sold today that do virtually everything, why are programming languages such as Visual Basic continuing to break previous sales records each year? The answer is simple: People buy a computer so that the computer will do jobs that they need done. Firms cannot adapt their business to a computer program. They must find programs, or write their own programs, so that the computer processes information according to the business procedures already in place. The only way to ensure that a program exactly fits the needs of a firm is for the firm to develop its own programs.

15

Page 16: 202158425 istqb-fl-studyguide

Business people are not the only ones who need custom-designed programs. No two people manage their finances exactly the same way; no two scientists need computers for exactly the same kinds of computations; and no two graphic artists need the same kinds of computer drawing tools. Although people buy spreadsheets and word processors for their general-purpose computing needs, many people require specialized programs for specific jobs.

The art of programming computers is rewarding not only from a requirements standpoint, but also on a more personal level. Programming computers is fun! Just as a sculptor looks on a finished work of clay, programmers are often proud of the programs that they write. By the time you finish this book, you will have written programs that were not available before you wrote them. When you want your computer to do something specific and you cannot find a program that does the job exactly the way you want, you will be able to design and write the program yourself.

Some Programs are Changeable: There is a third method for getting exactly the program that you need if you want to computerize your company's accounting records. Accounting software firms often sell not only accounting programs but also the source code for those programs. The source code is a listing of the program's instructions. By having access to the source code, you can take what the software company wrote and modify the behavior of the program to suit your own requirements.

By starting with a functional program instead of starting from scratch, you save programming time and money. Sadly, most non-accounting software firms do not supply the source code. Most programs sold today have been compiled. After compiling, the source code is translated into a locked-in executable program. The bottom line is that you cannot easily change the behavior of compiled programs. For most programs, therefore, you have the choice of buying them or writing them yourself from scratch.

Definition: Code is another name for program.

Review: No single program pleases everyone. When a company sells a program, it must be general enough to please most purchasers. Some people need programs to behave in a specific manner in order to fulfill a specific need. They must resort to writing their own programs. Luckily, Visual Basic takes a lot of the pain out of writing programs.

16

Page 17: 202158425 istqb-fl-studyguide

A Brief History of Textual Programming

Concept: Computers cannot understand just any language. You must learn a language that your computer knows before you can write programs for your computer.

Definition: An application is yet another name for program.

Many people use computers all day long for word processing, database storage, and spreadsheet analysis, without realizing what is going on behind the scenes. You must always keep in mind that computers cannot think. Your computer does not know how to be a word processor. If you want your computer to do word processing, you must supply detailed instructions in the form of a program. Only by following the detailed instructions of a word processor program that you load can your computer perform word processing.

It would be nice if writing a program is as easy as telling the computer what you want done. Many people can handle ambiguous instructions, but computers are not smart enough to understand vague requirements. Computers can only follow orders given to them, and you must supply those orders in the form of a program. Therefore, you must supply the programs that you write. Writing programs, especially complex programs, takes time and several procedural steps. Visual Basic speeds the process of creating programs, but even with Visual Basic some programs take time to write and perfect.

Definition: A bug is a program error.

These are the typical steps that most programmers go through when writing programs. First, you have an idea for a program. Next, you use a program-development system, such as Visual Basic, to write the program. Errors, or bugs, often appear in programs because of the details needed for even the simplest of programs. Therefore, you must test the program thoroughly and fix the errors. Fixing errors is called debugging. Once all the errors are out of the program, you have a finished application.

PROGRAMMING

A computer is an information-processing machine. [Executes given commands. Uses memory. Information = data. There are various types of data e.g. numerical, text, graphics or

pictures, sound signals etc. A computer program is a sequence or set of instructions in a programming language. All data and instructions for a program are stored in the computer's memory in coded

form. This coded form is similar to the mores code (-0---0-00-). The coding or translation is now done automatically. I.e. by commercial or computer programming.

Programs will be written in the programming languages of which there are many. Like Pascal, Cobalt, (used for records), Visual Basic, etc.

17

Page 18: 202158425 istqb-fl-studyguide

For the purpose of this module, Visual Basic (VB) programming language will be used. Hence the VB Integrated Development Environment (IDE) will do the translation into coded form, which is a software program.

LOGGING INTO VISUAL BASIC VBS (adds topics), enter Open, enterWhen in the form use - Label 1. Caption = "Hello Class of 2000" To delete an object from the form Select Edit menu.

ERRORS IN PROGRAMMING

There are three common errors1 Syntax errors: are due to structure or grammar of the language (rules) applied, e.g.more or less spacing, full stops, commas etc.2 Routine errors: are due to non-existing situations like 1 divided by 0, is impossibility. These are errors where the program has instructed the computer to perform an impossible operation e.g. as shown above.3 Logical errors: are those in the meaning of the program.

To save, always SAVE FORM AS, followed by SAVE PROJECT AS.

The Cost of Bugs

Programming, and testing, to its use by the public, there's the potential for bugs to be found. The Figure below shows how the cost of fixing these grows over time.

Cost

Specification Design Code

Time When Bug Is Found

18

Page 19: 202158425 istqb-fl-studyguide

The cost to fix bugs increased dramatically over time.

The cost are logarithmic - that is, they increase tenfold as time increases. A bug found and fixed during the early stages when the specification is being written might cost next to nothing or 10 pence in our example. The same bug, if not found until the software is coded and tested, might cost £1 to £10. If a customer finds it, the cost could easily top £100.

WHY DOES SOFTWARE HAVE BUGS?

Miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements).

Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well engineered.

Programming errors - programmers, like anyone else, can make mistakes.

Changing requirements - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of keeping track of changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control

Time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.

Egos - people prefer to say things like: 'no problem', 'piece of cake', 'I can whip that out in a few hours' 'it should be easy to update that old code'Instead of: 'that adds a lot of complexity and we could end up making a lot of mistakes' or ‘we have no idea if we can do that; we'll wing it', 'I can't estimate how long it will take, until I take a close look at it', 'we can't figure out what that old spaghetti code did in the first place'

If there are too many unrealistic 'no problems', the result is bugs.

Poorly documented code - it's tough to maintain and modify code that is badly written

19

Page 20: 202158425 istqb-fl-studyguide

or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').

Software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.

Example - Microsoft Word

TEST NO INPUTS EXPECTED RESULTS

I Click on File hold down and Open dialog appearsDrag to Open

2 Right click on paragraph Menu options appear - font, paragraph etc.Cut Copy Paste are grayed out

3 Right click within a table 12 menu options appear related to tableActions e.g. insert row, delete row

4 Spelling Spelling checking starts and displays firstError but check grammar off(Depends on option)

5 File Print Displays printer dialog6 Click on print icon Will attempt to print to default printer

Driving License Exercise

Consider the following simple program:

IF age> 16 and age <=65 THENIssue drivers LicenseIF age <26 THEN

Set Insurance Premium to HIGH

ELSE

Set Insurance Premium to STANDARD END IF

ELSE

Error message DLOOl -" Sorry, unable to issue drivers license" END IF

20

Page 21: 202158425 istqb-fl-studyguide

TEST NO INPUTS EXPECTED RESULTS

1 102 163 174 255 266 657 70

21

Page 22: 202158425 istqb-fl-studyguide

ATM Exercise

Some simplified rules for an A TM:

a) Card must be valid for this bankb) Correct PIN must be enteredc) Withdrawal must not take account overdrawn unless there is an overdraft agreed d) Notes available are £ 1 0 and £20 only

TEST NO INPUTS EXPECTED RESULTS

1 Valid CardIncorrect PIN£50 requested '"

2 Valid CardCorrect PIN£200 requestedBalance £50No overdraft 'i-

3 Valid CardCorrect PIN£50 requestedBalance £50No overdraft /

4 Valid CardCorrect PIN£50 requestedBalance £400Withdrawn £160 earlier 1-

Telephones Exercise

TEST NO INPUTS EXPECTED RESULTS

1 Pick UD receiver2 Dial 1003 Dia1142 or 1924 Dial 9995 Dial Busy Number6 Press 5 After Hearing

Engaged tone7 Dia114718 Dia11571

22

Page 23: 202158425 istqb-fl-studyguide

1. 9 Prioritization of Tests

Since we have discussed already that there is never enough time to do all of the testing that you would like an obvious approach is to prioritize the tests This will then mean that whenever you stop testing you will have done the best testing possible in the time available.

The test manager should identify the criteria to hi used when prioritizing tests. These will include but are not limited to:

Severity. Probability. Visibility of failure. Business priority assigned to the requirement. Customer wants. How much change the function has undergone. How error prone is this module. How critical is it to the business. How technically complex is it. How difficult is it to test. How costly is it to test.

Keep the test priority with the test at all times. This_ will prevent you from developing and designing lo_ priority tests that never get run. Use a scheme of high, medium, low or a numeric scheme or 1,2, 3, 4 but try not to have more than about 4 categories.

NO TEST CASE DESCRIPTION PRI

1 A manual block on a bank is introduced (using the HOLD function)Causing payments to be moved between priority classes by theScheduler

2 Failure of the primary scheduler process3 Internal authentication failures4 Normal close of day5 Normal system start up6 One of each type of request to the scheduler7 Payments are released correctly from the USER class when limits

Exceeded but funds received from another bank causing position toImprove

8 Payments can be routed to the scheduler queue from several sourceswith the correct class and user defined priority (including paymentsthat have been referred or repaired)

9 Payments cancelled manually be central control10 Regression testing of existing telecommunications links

23

Page 24: 202158425 istqb-fl-studyguide

11 Regression testing of recompiled interface with mainframe system12 Remote bank changes their limit on us causing payments to be

Released from the USER class (if held due to exceeding limit)13 Remote bank issues temporary raise14 Route all payments to the scheduler with parameters set for

immediate release15 Special processing when internal cut-off time reached16 Suspend/activate limits17 Suspend/activate scheduler

1.10 Summary

You should now be familiar with some of the fundamental principles of testing and you will recognize much of the terminology. The horror stories that you will have heard are not just sensationalist to motivate the course; they are true stories and there are (unfortunately) many more stories about the costs of not testing systems properly.

In particular you can now:

Recognize and use basic testing terminology. Understand why testing is necessary. Define error, fault and failure. Explain why errors occur. Give examples of the cost of errors. Understand why exhaustive testing is unachievable. Appreciate why testing is a risk management process. Understand the fundamental test process. Understand the difference between the mindsets of developers and testers. Understand why you cannot test your own work. Understand the need for regression testing. Understand the importance of specifying your expected result in advance. List some criteria for prioritizing your tests.

24

Page 25: 202158425 istqb-fl-studyguide

Chapter2: Testing throughout Lifecycle

"What is clear from the Inquiry Team's investigations is that neither the Computer Aided Dispatch (CAD) system itself, nor its users, were ready for full implementation on 26th October 1992. The CAD software was not complete, not properly tuned, and not fully tested. The resilience of the hardware under a full load had not been tested. The fall back option to the second file server had certainly not been tested."

Extract from the main conclusions of the official report into the failure of the London Ambulance Service's Computer Systems on October 26th and 27th 1992.

2.1 OVERVIEW

This module covers all of the different testing activities that take place throughout the project lifecycle. We introduce various models for testing, discuss the economics of testing and then describe in detail component, integration, system and acceptance testing. We conclude with a brief look at maintenance testing.

After completing this module you will:

2.2 OBJECTIVES

Understand the difference between verification and validation testing activities

Understand what benefits the V model offers over other models.

Be aware of other models in order to compare and contrast.

Understand the cost of fixing faults increases as you move the product towards live use.

Understand what constitutes a master test plan.

Understand the meaning of each testing stage.

25

Page 26: 202158425 istqb-fl-studyguide

2.3 Models for Testing

This section will discuss various models for testing. Definitions of these models will differ however the fundamental principles are agreed on by experts and practitioners alike. We cover verification and validation (V & V), the V-Model and briefly discuss a Rapid Application Development (RAD) approach to testing.

2.3.1 Verification and validation (V&V)

Verification is defined by B87925 as the process of evaluating a system or component to determine whether the products of the given development phase satisfy the conditions imposed at the start of that phase.

Validation is defined by B87925 as determination of the correctness of the products of software development with respect to the user needs and requirements.

26

Page 27: 202158425 istqb-fl-studyguide

Exercise

Validation and Verification

Complete the definition of testing: ________________________________________

Testing is defined by BS7925 as the process of exercising software to that it satisfies specified requirements and to____________________________________________

In simpler terms, verification answers the question "have we built the product right", i.e. is it correct and free from errors, does it meet the specification whereasValidation asks the question" is this the right product?" have the users got what they wanted. To help you remember this use the following:

Verification Is it error free, does it do what was specified?Validation Is it valid, is this what you really, really want?

2.3.2 V-model

There are many models used to describe the sequence of activities that make a Systems Development Life Cycle (SDLC). SLDC is used to describe activities of both development and maintenance work. Three models are worth mentioning.

Sequential (the traditional waterfall model). Incremental (the function by function incremental model). Spiral (the incremental, iterative, evolutionary, RAD, prototype

model).

The three models would all benefit from earlier attention to the testing activity that has to be done at some time during the SDLC.

Any reasonable model for SDLC must allow for change and spiral approach allows for this with emphasis on slowly changing (evolving) design. We have to assume change is inevitable will have to design for change.

Fact 1.Business is always changing

2.Finding a fault causes change

Result Ease of fixing a fault defines ease of responding to change

CorollaryIf we want systems that can be modified and hence maintained, the earlier we start testing and try the change process, the earlier we will find out how easy it is going to be to maintain the system

27

Page 28: 202158425 istqb-fl-studyguide

2.3.3 Sequential Model

Initial AppraisalFeasibility Study

Requirements, specification

Functional Specification

Technical Specification

Program Specification

Code Acceptance TestAnalysis

The sequential model often fails to bring satisfactory results because of the late attention to the testing activity. When earlier phases in the development cycle slip, it is the testing phase that gets squeezed. This can lead to a limited amount of testing being carried out with the associated production 'teething' problems.

2.3.4 Plan for wanted waterfall model development of system

The overall project plan for development of a system might be as shown below:

Activities Time

Business study ----Requirements analysis ---- User level design ----Technical design ---- Program specification ----Creation of code ----Unit testing ---- Integration testing -----System testing -----Acceptance testing -----Implementation ------

QC

28

Page 29: 202158425 istqb-fl-studyguide

This is a typical Specify, Design and Build project plan.

All testing and quality control points come late in project and only done if there is time.

When testing is done so late in project it can reveal costly errors.

Project plan has testing done late because people think that only physical deliverables such as code can be tested. Clearly there has to be a better way.

The challenge it to devise a better way of developing systems. There is a need to introduce quality control points earlier in the SDLC.

29

Page 30: 202158425 istqb-fl-studyguide

2.3.5 Sequential model plus testing gives 'V' diagram

The V diagram is another way of looking at the sequential development but this time from viewpoint of testing activities that need to be completed later in SDLC.

The 'V' diagram in this simple form has been around for a long time and is especially useful as it easily demonstrates how testing work done early in SDLC is used as input to assurance work later in development.

The V model of SDLC offers considerable benefits over others as it emphasizes building of test data and test scenarios during development and not as an after thought. The V model also allows for establishment of versions, incremental development and regression testing.

Management needs to rename activities, referred to variously as systems testing or acceptance testing. There has always been a phase of development traditionally thought of as the testing phase. This is an historical perception. Testing is not a phase but rather an activity that must be carried out all through development, giving rise to the principle of Total Quality.

In the past, system testing was the only type of testing carried out. Testing was checking that programs, when linked together, met the systems specification. Whether design itself was correct was another matter. The concept of "testing" design before programs were coded was given only the most perfunctory attention. This was for two reasons:

30

Page 31: 202158425 istqb-fl-studyguide

1. By the time physical design had been done the system was very difficult to alter; modifications caused by design reviews were therefore very unwelcome.

2. The design was documented in terms of physical file layouts and program specifications, neither of which the user could comprehend. The question of whether physical design was correct could be reviewed, but the more important question: "Did the system do what the user wanted?" Was largely neglected.

31

Page 32: 202158425 istqb-fl-studyguide

2.3.6 'V' with test recognized deliverable

The activity of Business Analysis has, as deliverables, the Specification of Requirements from which Acceptance Test Plan is constructed. To have created, for example, System Architecture without integration Test Specification is to do only half the job!

32

Page 33: 202158425 istqb-fl-studyguide

2.3.7 Revised plan for system development

The overall project plan for development of a system might be as shown below. Note the new early quality control points.

This plan shows that the creation and running of actual tests are separated. The creation of test material (acceptance test plans, user system test scripts, technical system tests such as integration, link, recovery restart, etc., and unit test data) is done as the relevant design is done. The potential for automation is very good and the use of tools to capture the test cases, scripts, etc. will play a big part in making the running tests efficient. The early creation of test material will make the process of developing a system effective. The emphasis must be on first being effective then being efficient.

33

Page 34: 202158425 istqb-fl-studyguide

2.3.8 Rapid Application Development

The spiral, Rapid Application Development (RAD) model has the benefit of the evolutionary approach. This is an incremental process of build a little then test a little, which has the benefit of attempting to produce a usable but limited version early.

The RAD approach relies upon the quality of the RAD team.

The management issues to address are:

Have I got knowledgeable user input in my team? Have I got experienced designers and developers

in my team? Am I leaving a good audit trail to support future

Maintenance?

Code

Acceptance Test

UserRequirement

34

Page 35: 202158425 istqb-fl-studyguide

2.4 ECONOMICS OF TESTING

This section looks at some of the economic factors involved in test activities. Although some research has been done to put forward the ideas discussed, few organizations have yet to provide accurate figures to confirm these theories.

Major retailers and car manufacturers often issue product recall notices when they realize that there is a serious fault in one of their products. Perhaps you can think of other examples. The fixing of the so-called millennium but is probably one of the greatest product recall notices in history.

Boehm's research suggests that cost of fixing faults increases dramatically as we move software product towards field use. If fault is detected at an early stage of design, it may be only design documentation that has to change resulting in perhaps just a few hours work. However, as project progresses and other components are built based on faulty design, more work is obviously needed to correct fault once it has been found. This is because design work, coding and testing will have to be repeated for components of the system that were previously thought to have been completed.

If faults are found in documentation, then development based on that documentation might generate many related faults, which multiply the effect of the original fault.

Analysis of specifications during test preparation (early test design) often brings faults in specifications to light. This will also prevent faults from multiplying i.e. if removed earlier they will not propagate into other design documents.

In summary we suggest that it is generally cost effective to use resources on testing throughout the project lifecycle starting as soon as possible. The alternative is to potentially incur much larger costs associated with the effort required to correct and re-test major faults. Remember that the amount of resources allocated to testing is a management decision based on an assessment of the associated risks. However, few organizations are able to accurately compare the relative costs of testing and the costs associated with re-work.

35

Page 36: 202158425 istqb-fl-studyguide

2.5 RELATIVE COSTS TO FIX ERROR

The cost of fixing errors escalates as we move the project towards field use. From an analysis of sixty-three projects cited in Boehm: Software Engineering Economics (Boehm, 1981).

36

Page 37: 202158425 istqb-fl-studyguide

2.6 High level test planning

You should be aware that many people use the term 'test plan' to describe a document detailing individual tests for a component of a system. We are introducing the concept of high level test plans to show that there are a lot more activities involved in effective testing than just writing test cases.

The IEEE standard for test documentation (IEEE/ ANSI, 1983 [Std 829-1983 D, affectionately known as 829, defines a master validation test plan as follows:

Purpose of master test plan is to prescribe scope, approach, resources and schedule of testing activities. A master test plan should include the following:

1. Test Plan Identifier2. References3. Introduction4. Test Items5. Software Risk Issues6. Features to be tested7. Features not to be tested8. Approach9. Item Pass/Fail Criteria10. Suspension Criteria11. Resumption Requirements12. Test Deliverables13. Remaining Testing Tasks14. Environmental Needs15. Staffing and Training Needs16. Responsibilities17. Schedule18. Planning Risks and Contingencies 19. Approvals20. Glossary

37

Page 38: 202158425 istqb-fl-studyguide

2.6.1 SPACE - the final frontier (of testing?)

This may be useful acronym to help you remember what you should include in any high level test plan document. The actual format is up to you and will depend upon your own company standards and other internal considerations. Space stands for Scope, People, Approach, Criteria and Environment and we have tried to map this onto the IEEE 829 headings as follows:

Scope People Approach Criteria Environment

I.Test plan identifier 15. Staffing and training 8. Approach 9. Item

pass/fail14. Environmental

2.References needs criteria needs3.Introduction

4.Test Items 16. Responsibilities 12. Test deliverables 10. Suspension Test lab

6.Features to be tested 13. Remaining test tasks criteria Test network

7.Features notto be tested

5.Software risk issues 17. Schedule 19. Approvals 11. Assumption Test data

requirements Use of live data

18. Planning risks and Office requirements Use of testing tools Number ofcontingencies cycles es

20.Glossary Motivation/rewards Configurationmanagement

38

Page 39: 202158425 istqb-fl-studyguide

Exercise

High Level Planning

We will use the case study for the London Ambulance Service's new computer aided Dispatch system to discuss the test planning issues. Take a few moments to think about how you would test such a system and we will complete the outline plan as a group exercise.

Scope

People

Approach

Criteria

Environment

39

Page 40: 202158425 istqb-fl-studyguide

2.7 Component testing

Component testing is described fully in BS-7925 and should be aware that component testing is also known as unit testing, module testing or Program Testing. The definition from BS7925 is simply the testing of individual software components.

Traditionally, the programmer carries out component testing. This has proved to be less effective than if someone else designs and funs the tests for the component.

"Buddy" testing, where two developers test each other's work is more independent and often more effective. However, the component test strategy should describe what level of independence is applicable to a particular component.

Usually white box (structural) testing techniques are used to design test cases for component tests but some black box tests can be effective as well.

We have already covered a generic test process in this course. The component test process is shown in the following diagram:

2.8 INTEGRATION TESTING

Integration is the process of combining components into larger assemblies. From the standard BS-7925 integration testing is defined as "testing performed to expose faults in the interfaces and in the interaction between integrated components", However, in this section we look at two interpretations of integration testing known as integration testing in the large and integration testing in the small.

By Integration Testing in the Large we mean testing the integration of the new system or software package with other (complete) systems. This would include the identification of,

40

Page 41: 202158425 istqb-fl-studyguide

and risk associated with, all interfaces to these other systems. Also included is testing of any interfaces to external organizations (e.g. EDI - electronic data interchange, Internet) but not testing of the processing or operation of those external systems.

Exercise

Compare the definition of integration testing in the large with B87925 (50115) definition of interface testing. Any comments?

We use Integration Testing in the Small in the more traditional sense of integration testing where components are assembled into sub-systems and subsystems are linked together to form complete systems. Integration strategies may be incremental or non-incremental and include:

bin-ban Top-down Bottom-up Sandwich

Testing approach is directly related to integration strategy chosen.

Stubs & Drivers

A stub is a skeletal or special purpose implementation of a software module, used to develop or test a component that calls or is otherwise dependent on it.

A test driver is a program or test tool used to execute software against a test case suite.

TRAINER'S NOTES:

Why not refer to the project EX example in BS7925; in the component test plan for the component LOG3 it has a good diagram showing stubs and drivers.

41

Page 42: 202158425 istqb-fl-studyguide

2.9 SYSTEM TESTING

System testing is defined as the process of testing an integrated system to verify that it meets specified requirements.

You will come across two very different types of system testing, Functional system testing and non-functional system testing. In plain English functional system testing is focuses on testing the system based on what it is supposed to do. Non-functional system testing looks at those aspects that are important yet not directly related to what functions the system performs. For example if the functional requirement is to issue an airline ticket, the non-functional requirement might be to issue it within 30 seconds.

A functional requirement is " a requirement that specifies a function that a system or system component must perform". Requirements-based testing means that the user requirements specification and the system requirements specification (as used for contracts) are used to derive test cases. Business process-based testing based on expected user profiles (e.g. scenarios, use cases).

Non-functional requirements cover the following areas:

1. Load 2. Performance3. Stress 4. Security5. Usability6. Storage 7. Volume 8. Install ability 9. Documentation10. Recovery

Non-functional requirements are just as important as functional requirement

Exercise

List some non-functional requirements for LAS CAD system.

42

Page 43: 202158425 istqb-fl-studyguide

2.10 ACCEPTANCE TESTING

The definition of acceptance testing in BS7925 states that "acceptance testing is formal testing conducted to enable a user, customer, or other authorized entity to determine whether to accept a system or component". Acceptance testing may be the only form of testing conducted by and visible to a customer when applied to a software package. The most common usage of the term relates to the user acceptance testing (VAT) but you should be aware that there are several other uses of acceptance testing, which we briefly describe here.

User acceptance testing - the final stage of validation. Customer should perform or be closely involved in this. Customers may choose to do any test they wish, normally based on their usual business processes. A common approach is to set up a model office where systems are tested in an environment as close to field use as is achievable.

Contract acceptance testing - a demonstration of the acceptance criteria, which would have been defined in the contract, being met.

Alpha & beta testing - In alpha and beta tests, when the software seems stable, people who represent your market use the product in the same way (s) that they would if they bought the finished version and give you their comments. Alpha tests are performed at the developer's site, while beta tests are performed at the user's sites.

Exercise

What problems would LAS have avoided had they set some acceptance criteria?

2.11 MAINTENANCE TESTING

Maintenance testing is not specifically defined in BS7925 but it is all about testing changes to the software after it has gone into productions.

There are several problems with maintenance testing. If you are testing old code the original specifications and design documentation may be poor or in some cases non-existent. When defining the scope of the maintenance testing it has to be judged in relation to the changed code; how much has changed, what areas does it really affect etc. This kind of impact analysis is difficult and so there is a higher risk when making changes - it is difficult to decide how much regression testing to do.

43

Page 44: 202158425 istqb-fl-studyguide

Chapter 3:Dynamic Testing Techniques

"…The system was not fully tested to a satisfactory level of quality and resilience before full implementation on 26 October 1992."

Extract from the main conclusions of the official report into the failure of the London Ambulance Service's Computer Systems on October 26th and 27th and November 4th 1992.

3.1 OVERVIEW

This module introduces idea of a test case design technique. Broadly, these are categorized as either functional or structural testing techniques. The advantage of using these proven design methods is that they provide a more intelligent and effective means of identifying tests than a purely intuitive approach. You will be expected to know two functional techniques and two structural techniques in detail and be aware there are many other techniques that can be applied to test case design. There are many excellent books on testing techniques some of which are listed in the appendix.

3.2 OBJECTIVES

After completing this module you will:

Understand the difference between black box (functional) and white box (structural) testing techniques.

Be able to name at least three black box techniques.

Understand how to use equivalence partitioning and boundary value analysis to design test cases.

Appreciate the use of state transition testing.

Be able to name at least three white box techniques.

Understand the meaning of statement testing and branch testing.

Be aware that the BS7925 standard details some (but not all) of the recognized testing techniques.

Know that all of these testing techniques can be supplemented by error guessing.

44

Page 45: 202158425 istqb-fl-studyguide

3.3 INTRODUCTION

Each of the techniques we are about to describe has its strengths and weaknesses. A useful rule of thumb is that if you are having difficulty applying a particular technique to a testing problem then perhaps you ought to try a different technique. This section introduces the different types of testing technique and discusses the difference between them.

3.3.1 Functional test techniques

Functional test techniques are often referred to as 'black box' test techniques and the common parlance is that we are 'doing black box testing'. A functional test technique will help design test cases based on functionality of component or system under test, without necessarily having to understand underlying detail of software design. Consider functionality of system of determine test inputs and expected results.

Structural test techniques are sometime called white box text techniques hence term 'white box testing'. Glass box testing is less widely used term for structural test case design. Structural test techniques help design test cases based on internal structure and design of component or system under test. Look at code, database spec, data model, etc., to determine test inputs and expected results.

Exercise

What's the difference between Black and White box testing?

BASED ON USED BY SUITABLE FORBlack Box Requirements Independent tester System test

Functionality Users Acceptance testWhite Box Internal structure Developer Unit test

Code DB design Designer Link testSub-system test

45

Page 46: 202158425 istqb-fl-studyguide

3.4 BLACK BOX TECHNIQUES

The following list of black box techniques is from BS 7925-2. On this course we will describe and give an example of only those ones highlighted in bold:

/Equivalent Partitioning. Boundary Value Analysis State Transition Testing Cause-Effect Graphing /'

Syntax Testing /

Equivalence partitioning (EP) is a test case design technique that is based on the premise that the inputs and outputs of a component can be partitioned into classes that, according to the component's specification, will be treated similarly by the component.. Thus the result of testing a single value from an equivalence partition is considered representative of the complete partition.

As an example consider any program that accepts days of ht week and months of they year as inputs. Intuitively you would probably not expect to have to test every date of the year. You would obviously try months with 30 days (e.g. June) and months with 31 days (e.g. January) and you may even remember to try out the special case of February for both non-leap year (28 days) and leap years (29 days). Equally, looking at the days of the week you would not, depending on the application, test every day. You may test for weekdays (e.g. Tuesday) and weekends (e.g. Sunday). What you are in effect doing is deciding on equivalence classes for the set of data in question.

Not everyone will necessarily pick the same equivalence classes; there is some subjectivity involved. But the basic assumption you are making is that anyone value from the equivalence, class, is as good as any other when we come to design the test.

We hope that you can see how this technique can dramatically reduce the number of tests that you may have for a particular software component.

46

Page 47: 202158425 istqb-fl-studyguide

Exercise

Consider a software component called GENERATE-GRADING, which is used by the system to calculate student's grade based on an examination mark and a coursework mark. The component is passed an exam mark (out of75) and a coursework mark (out of 25) from which it generates a grade fro the course in the range' A' to 'D'. The grade is calculated from the overall mark, which is calculated as the sum of the exam mark and coursework marks, as follows.

OVERALL MARK GRADE

Greater than or equal to 70 AGreater than or equal to 50 but less than 70 BGreater than or equal to 30 but less than 50 CLess than 30 D

Exercise

Identify valid partitions:_________

Identify invalid partitions:________

Identify output partitions:_________

Now derive some test cases based on input exam mark partition:

TEST CASE 1 2 3

Input (exam mark)

Input (coursework)

Total mark (calculated)

Partition tested

Expected output

47

Page 48: 202158425 istqb-fl-studyguide

The standard goes on to derive test cases form the other partitions but in this course we will not have time to do this completely.

Boundary Value Analysis is base on the following premise. Firstly, the inputs and outputs of a component can be partitioned into classes that, according to the component's specification, will be treated similarly by the component and, secondly, that developers are prone to marking errors in their treatment of the boundaries of these classes. Thus test cases are generated to exercise these boundaries.

Exercise

Look at the boundaries for input exam mark. Choose values on each boundary and one either side of it in order to generate a set of test cases.

TEST CASE 1 2 3 4 5 6

Input (exam mark)

Input (coursework)

Boundary tested

Expected output

Again the standard continues to describe boundary tests for all of the other valid boundaries.

48

Page 49: 202158425 istqb-fl-studyguide

3.5 White Box Techniques

The following list of white box techniques is from BS79252. On this course we will describe and give an example of only those ones highlighted in bold.

Statement testing Branch Decision Testing. Data Flow Testing. Branch Condition Testing. .

Branch Condition Combination Testing Modified Condition Decision Testing. LCSAJ Testing. Random Testing.

Statement testing is a structural technique based on decomposition of a software component into constituent statements. Statements are identified as executable or non-executable. For each test case, you must specify inputs to component, identification of statement(s) to be executed and expected outcome of test case.

Example of program with 4 statements:

X=INPUT;Y=4;IF X>Y THENMessage= "Limited exceeded"

ELSE

Message= "No problems reported" END IFz=o

Branch testing requires model of source code, which identifies decisions and decision outcomes. A decision is an executable statement, which may transfer control to another statement depending upon logic of decision statement. Typical decisions are found in loops and selections. Each possible transfer of control is a decision outcome. F or each test case you must specify inputs to component, identification of decision outcomes to be executed by test case and expected outcome of test case.

49

Page 50: 202158425 istqb-fl-studyguide

3.5.1 Comparison of white box techniques from "Software Testing in the Real World"

Each column in this figure represents a distinct method of white-box testing, and each row (1-4) defines a different test characteristics. For a given method (column), "Y" in a given row means that test characteristic is required for method. "N" signifies no requirement. "Implicit" means test characteristic is achieved implicitly by other requirements of method. (@ 1993, 1994 Software Development Technologies) reproduced with permission.

3.7 Summary

In module three you have learnt that applying a formal, recognized testing technique in a systematic way is the most effective approach to finding errors in software. In particular you can now

Explain the difference between black box (functional) and white box (structural) testing techniques.

Name at least three black box techniques.

Use equivalence partitioning and boundary value analysis to design test cases.

Recognize a state transition testing technique.

Name at least three white box techniques.

Understand what the meaning of statement testing and branch testing.

Use the standard BS7925 to find out more about testing techniques.

Know when to apply error-guessing techniques to supplement the formal techniques.

50

Page 51: 202158425 istqb-fl-studyguide

Chapter 4:

Static testing

"On November 4th the system did fail. This was caused by a minor programming error that caused the system to "crash", the automatic change over to the back up system had not been adequately tested, and thus the whole system was brought down".

Extract from the main conclusions of the official report into the failure of the London Ambulance Service 's Computer Systems on October 26th and 27th and November 4th 1992.

4.0 STATIC TESTING

4.1 Overview

Static testing techniques is used to find errors before software is actually executed and contrasts therefore with dynamic testing techniques that are applied to a working system. The earlier we catch an error, the cheaper it is, usually, to correct. This module looks at a variety of different static testing techniques. Some are applied to documentation (e.g. walkthroughs, reviews and Inspections) and some are used to analyze the physical code (e.g. compilers, data flow analyzers). This is a huge subject and we can only hope to give an introduction in this module. You will be expected to appreciate the difference between the various review techniques and you will need to be aware of how and when static analysis tools are used.

4.2 Objectives

After completing this module you will:

Understand the various review techniques that you can apply to documentation and code.

Appreciate the difference between walkthroughs, formal reviews and inspections.

Understand how static analysis techniques can detect errors in code. Understand two complexity metrics (lines of code and McCabe's metric).

51

Page 52: 202158425 istqb-fl-studyguide

4.3 REVIEWS AND THE TEST PROCESS

4.3.1 What is a review?

A review is a fundamental technique that must be used throughout the development lifecycle. Basically a review is any of a variety of activities involving evaluation of technical matter by a group of people working together. The objective of any review is to obtain reliable information, usually as to status and/or work quality.

4.3.2 Some history of reviews

During any project, management requires a means of assessing a measuring progress. The so-called progress review evolved as means of achieving this. However, results of those early reviews proved to be bitter experiences for many project managers. Just how long can a project remain at 90% complete? They found that they could not measure 'true' progress until they had a means of gauging the quality of the work performed. Thus the concept of the technical review emerged to examine the quality of the work and provide input to the progress reviews.

4.3.3 What can be reviewed?

There are many different types of reviews throughout the development life cycle. Virtually any work produced during development can be (and is) reviewed. This includes, requirements documents, designs, database specifications, designs, data models, code, test plans, test scripts, test documentation and so on.

4.3.4 What has this got to do with testing?

The old fashioned view that reviews and testing are totally different things stems from the fact that testing used to be tacked onto the end of the development lifecycle. However as we all now view testing as a continuous activity that must be started as early as possible you can begin to appreciate the benefits of reviews. Reviews are the only testing technique that is available to us in the early stages of testing. At early stages in the development lifecycle we obviously cannot use dynamic testing techniques since the software is simply not ready for testing.

Reviews share similarities to test activities in that they must be planned (what are we testing), what are the criteria for success (expected results) and who will do the work (responsibilities). The next section examines the different types of review techniques in more detail.

52

Page 53: 202158425 istqb-fl-studyguide

4.4 TYPES OF REVIEW

Walk-through, informal reviews, technical reviews and Inspections are fundamental techniques that must be used throughout the development process. All have their strengths and weaknesses and their place in the project development cycle. All four techniques have some ground rules in common as follows:

A structured approach must be for the review process.

Be sure to know what is being reviewed - each component must have a unique identifier.

Changes must be configuration controlled. Reviewers must prepare. Reviewers must concentrate on their own specialization. Be sure to review the product, not the person.

There must be:

Total group responsibility. Correct size of review group. Correct allocation of time. Correct application of standards.

Checklists must be used.Reports must be produced.Quality must be specified in terms of:

Adherence to standards. Reliability required. Robustness required. Modularity. Accuracy. Ability to handle errors.

53

Page 54: 202158425 istqb-fl-studyguide

4.5 REVIEW TEAM SIZE AND COMPOSITION

Problems with small teams must be avoided; bring in extra people, (perhaps use the Testing Team) to bring extra minds to bear on the issues.

Opinion is often divided as to whether or not the author should participate in a review. There are advantages to both scenarios. Since specifications and designs must be capable of being understood without the author present, an Inspection without them tests the document. Another reason for excluding the author from the review is that there should be team ownership of the product and team responsibility for the quality of all the deliverables, maintaining ownership via the author runs against this.

Alternatively, including the author can be a valuable aid to team building. Equally, an author may well be able to clear up misunderstandings in a document more quickly than another team member, thus saving the reviewer valuable time. From a practical perspective however, it is worth remembering that an author is the least likely person to identify faults in the document.

The one person who should not attend reviews is the manager. If, as in some cases, the manager is also a contributor to the deliverables, he should be included but treated the same as other group members. It is important that reviewers are a peer group process.

4.6 TYPE 1, 2 AND 3 REVIEW PROCESSES

Review process is both most effective and universal test method and management needs to make sure that review process is working as effectively as possible. Useful model for manager is 1,2,3 model.

1, 2, 3 model is derived from work of first working party of British Computer Society Specialist Interest Group in Software Testing and book that came from this work; in Software Development.

Type 1 testing is process of making sure that product (document, program, screen design, clerical procedure or Functional Specification) is built to standards and contains those features that we would expect from the name of that product. It is a test to make sure that produce conforms to standards, is internally consistent, accurate and unambiguous.

Type 2 testing is process of testing to see if product conforms to requirements as specified by output of preceding project stage and ultimately Specifications of Requirements for whole project. Type 2 testing is backward looking and is checking that product is consistent with preceding documentation (including information on change).

Type 3 testing is forward looking and is about specification of certification process and test that are to be done on delivered product. It is asking question - Can we can build deliverables (test material, training material, next stage analysis documentation)?

54

Page 55: 202158425 istqb-fl-studyguide

4.6.1 Make reviews incremental

Whilst use of 1, 2, 3 model will improve review technique, reviewing task can be made easier by having incremental reviews throughout product construction.

This will enable reviewers to have more in-depth understanding of product that they are reviewing and to start construction type 3 material.

4.6.2 General review procedure for documents

Test team will need general procedure for reviewing documents, as this will probably form large part of team's work.

1. Establishing standards and format for document. 2. Check contents list.3. Check appendix list.4. Follow up references outside documents.5. Cross check references inside document.6. Check for correct and required outputs.7. Review each screen layout against appropriate standard, data dictionary, processing rules and files/data base access.

8. Review each report layout against appropriate standard, data dictionary, processing rules and files/data base access.9. Review comments and reports on work andreviews done prior to this review.

10. Documents reviewed will range from whole reports such as Specification of Requirements to pages from output that system is to produce. All documents will need careful scrutiny.

4.6.3 Report on reviewReport should categorize products:

Type1 Type 2

Type 3

55

Page 56: 202158425 istqb-fl-studyguide

Defective All agreed Total reworkDefective but All agreed Rework andsoluble solution review all

acceptable materialPossible Some but not all Seek explanationdefect, needs agree possibly reviewexplanation some of workQuality issue Prefer an Costs compared

alternative to standardsAcceptable All accept ProceedOver Most agree Proceed butdeveloped for review budgetsthe budget

Be sensitive to voice of concerned but not necessarily assertive tester in team; this person may well have observed a fault that all others have missed. It should not be that person with load voice or strong personality is allowed to dominate.

4.7 INSPECTIONS

This section on Inspections is based on an edited version of selected extracts from Tom Gib and Dorothy Graham's book [Gib, Graham 93].

4.7.1 Introduction

Michael E. Fagan at IBM Kingston NY Laboratories developed the Inspection technique. Fagan, a certified quality control engineer, was a student of the methods of the quality gurus W. Edwards Deming and J. M. Juran.

Fagan decided, on his own initiative, to use industrial hardware statistical quality methods on a software project he was managing in 1972-74. The project consisted of the translation of IBM’s software from Assembler Language to PLS, a high-level programming language assembler. Fagan's achievement was to make statistical quality and process control methods work on 'ideas on paper'. In 1976 he reported his results outside IBM in a now famous paper [Fagan, 1976].

The Inspection technique was built further by Caroline L. Jones and Robert Mays at IBM [jones, 1985] who created a number of useful enhancements:

The kickoff meeting, for training, goal setting, and setting a strategy for the current inspection

Inspection cycle; The causal analysis meeting; The action database; The action team.

56

Page 57: 202158425 istqb-fl-studyguide

4.7.2 Reviews and walk-through

Reviews and walkthroughs are typically peer group discussion activities - without much focus on fault identification and correction, as we have seen. They are usually without the statistical quality improvement, which is an essential part of Inspection. Walkthroughs are generally a training process, and focus on learning about a single document. Reviews focus more on consensus and buy-in to a particular document.

It may be wasteful to do walkthroughs or consensus reviews unless a document has successfully exited from Inspection. Otherwise you may be wasting people's time by giving them documents of unknown quality, which probably contain far too many opportunities for misunderstanding, learning the wrong thing and agreement about the wrong things.

Inspection is not an alternative to wal1cthroughs for training, or to reviews for consensus. In some cases it is a pre-requisite. The difference processes have different purposes. You cannot expect to remove faults effectively with walkthroughs, reviews or distribution of documents for comment. However, in other cases it may be wasteful to Inspect documents, which have not yet 'settled down' technically. Spending time searching for and removing faults in large chucks, which are later discarded, is not a good idea. In this case it may be better to aim for approximate consensus documents. The educational walkthrough could occur either before or after Inspection.

4.7.3 Statistical quality improvement

The fundamental differences between Inspection and other review methods is that Inspection provides a tool to help improve the entire development process, through a well-known quality engineering method, statistical process control, [Godfrey, 1986].

This means that the data which is gathered and analyzed as part of the Inspection process - data on faults, and the hours spent correcting them - is used to analyze the entire software engineering process. Widespread weakness in a work process can be found and corrected. Experimental improvement to work processes can be confirmed by the Inspection metrics - and confidently spread to other software engineers.

4.7.4 Comparison of Inspection and testing

Inspection and testing both aim at evaluating and improving the quality of the software engineering product before it reaches the customers. The purpose of both is to find and then fix errors, faults and other potential problems.

Inspection and testing can be applied early in software development, although Inspection can be applied earlier than test. Both Inspection and test, applied early, can identify faults, which can then be fixed when it is still much cheaper to do so.

Inspection and testing can be done well or badly. If they are done badly, they will not be

57

Page 58: 202158425 istqb-fl-studyguide

effective at finding faults, and this causes problems at later stages, test execution, and operational use.

We need to learn from both Inspection and test experiences. Inspection and testing should both ideally (but all too rarely in practice) produce product-fault metrics and process improvement metrics, which can be used to evaluate the software development process. Data should be kept on faults found in Inspection, faults found in testing, and faults that escaped both Inspection and test, and were only discovered in the field. This data would reflect frequency, document location, security, cost of finding, and cost of fixing.

There is a trade-off between fixing and preventing. The metrics should be used to fine-tune the balance between the investment in the fault detection and fault prevention techniques used. The cost of Inspection, test design, and test running should be compared with the cost of fixing. The faults at the time they were found, in order to arrive at the most cost-effective software development process.

4.7.5 Differences between Inspection and testing

Inspection can be used long before executable code is available to run tests. Inspection can be applied mud earlier than dynamic testing, but can also be applied earlier than test design activities. Test can only be defined when a requirements or design specification has been written, since that specification is the source for knowing the expected result of a test execution.

The one key thing that testing does and Inspection does not, is to evaluate the software while it is actually performing its function in its intended (or simulated) environment. Inspection can only examine static documents and models; testing can evaluate the product working.

Inspection, particularly the process improvement aspect, is concerned with preventing software engineers from inserting any form of fault into what they write. The information gained from faults found in running tests could be used in the same way, but this is rare in practice.

4.7.6 Benefits of Inspection

Opinion is divided over whether Inspection is a worthwhile element of any product 'development' process. Critics argue that it is costly; it demands too much 'upfront' time and is unnecessarily bureaucratic. Supporters claim that the eventual savings and benefits outweigh the costs and the short-term investment is crucial for long-term savings.

58

Page 59: 202158425 istqb-fl-studyguide

In IT-Start's Developers Guide (1989), it is estimated that 'The cost of non-quality software typically accounts for 30% of development costs and 50% to 60% of the lifecycle costs'. Faults then are costly, and this cost increases the later they are discovered. Inspection applied to all software is arguably the prime technique to reduce defect levels (in some cases to virtually zero defects) and to provide an increased maturity level through the use of Inspection metrics. The savings can be substantial.

Direct savings

Development productivity is improved.Fagan, in his original article, reported a 23% increase in 'coding productivity alone' using Inspection [Fagan, 1976, IBM Systems Journal, p 187]. He later reported further gains with the introduction of moderator training, design and code change control, and test fault tracking.

Development timescale is reduced.Considering only the development timescales, typical net savings for project development are 35% to 50%.

Cost and time taken for testing is reduced.Inspection reduces the number of faults still in place when testing starts because they have been removed at an earlier stage. Testing therefore runs more smoothly, there is less debugging and rework and the testing phase is shorter. At most sites Inspection eliminates 50% to 90% of the faults in the development process before test execution starts.

Lifetime costs are reduced and software reliability increased.Inspection can be expected to reduce total system maintenance costs due to failure reduction and improvement in document intelligibility, therefore providing a more competitive product.

Indirect savings

Management benefits.Through Inspection, managers can expect access to relevant facts and figures about their software engineering environment, meaning they will be able to identify problems earlier and understand the payoff for dealing with these problems.

Deadline benefits.Although it cannot guarantee that an unreasonable deadline will be met, through quality and cost metrics Inspection can give early warning of impending problems, helping avoid the temperature of inadequate correction nearer the deadline.

59

Page 60: 202158425 istqb-fl-studyguide

Organizational and people benefits.For software professionals Inspection means their work is of better quality and more maintainable. Furthermore, they can expect to live under less intense deadline pressure. Their work should be more appreciated by management, and their company's products will gain a competitive edge.

4.7.7 Costs of Inspection

The cost of running an Inspection is approximately 10% - 15% of the development budget. This percentage is about the same as other walkthrough and review methods. However, Inspection finds far more faults for the time spent and the upstream costs can be justified by the benefits of early detection and the lower maintenance costs that result.

As mentioned earlier, the costs of Inspection include additional 'up front' time in the development process and increased time spent by authors writing documents they know will be Inspected. Implementing and running Inspections will involve long-term costs in new areas. An organization will find that time and money go on:

Inspection leading training. Management training. Management of the Inspection leaders. Metric analysis. Experimentation with new techniques to

try to improve Inspection results. Planning, checking and meeting activity: the entire Inspection process itself. Quality improvement: the work of the process improvement teams.

The company may also find it effective to consider computerized tools for documentation and consistency checking. Another good investment might be improved meeting rooms or sound insulation so members of the Inspection team can concentrate during checking.

4.7.8 Product Inspection steps The Inspection process is initiated with a request for Inspection by the author

or owner of a task product document. The Inspection leader checks the document against entry criteria, reducing the

probability of wasting resources on a product destined to fail. The Inspection objectives and tactics are planned. Practical details are decided

upon and the leader develops a master plan for the team. A kickoff meeting is held to ensure that the checkers are aware of their

individual roles and the ultimate targets of the Inspection process. Checkers work independently on the product, document using source

documents, rules, procedures and checklists. Potential faults are identified and recorded.

A logging meeting is convened during which potential faults and issues requiring explanations, identified by individual checker, are logged. The

60

Page 61: 202158425 istqb-fl-studyguide

checkers now work as a team aiming to discover further faults. And finally suggestions for methods of improving the process itself are logged.

An editor (usually the author) is given the log of issues to resolve. Faults are now classified as such and a request for permission to make the correction and improvements to the product is made to the document's owner. Footnotes might be added to avoid misinterpretation. The editor may also make further process improvement suggestions.

The leader ensures that the editor has taken action to correct all known faults, although the leader need not check the actual corrections.

The exit process is performed by the Inspection leader who uses application generic and specific exit criteria.

The Inspection process is closed and the product made available with an estimate of the remaining faults in a 'warning label'.

Exercise

Comparison between Various Techniques

Take a few moments to complete the following table.

TECHNIQUE Primarily Used for INVOLVES LED BY FORMALITY

Walkthroughs Education; Dry runs Peer group Author Fairly informal

Informal reviews Fault detection Anyone Individual Undocumented; Cheap. Useful

Technical reviews Fault detection

Peer group; Technical experts; No managers

IndividualFormal; Documented; No metrics kept

InspectionsFault detection; Process improvement

Defined roles Trained moderator (not the author)

Very formal Rules, checklists Metrics kept

4.9.2 McCabe’s complexity metric

McCabe's complexity metric is a measure of the complexity of a module's decision structure. It is the number of linearly independent paths and therefore, the minimum number of paths that should be tested. The metric can be calculated in three different ways. The number of decisions plus one, the number of 'holes' or connected regions (bearing in mind that there is a fictitious link between the entry and exit of the program), or thirdly the equation:

M=L-N+2P

Where: L = the no. Of links in graph

N = the no. Of nodes in the graph

61

Page 62: 202158425 istqb-fl-studyguide

P = the no. Of disconnected parts of the graph

Despite its simplicity the McCabe metric is based on deep properties of program structure. The greatest advantage is that it is almost as easy to calculate as the 'lines of code' metric, and results in a considerably better correlation of complexity to faults and the difficulty of testing.

McCabe advises partitioning programs where complexity is greater than la and this has been supported by studies such as Walsh who found that 23% of the routines with an M value of greater than 10 contained 53% of the faults. There does appear to be a discontinuous jump in the fault rate around M = 10. As an alternative to partitioning, others have suggested that the resources for development and testing should be allocated in relation to the McCabe measure of complexity, giving greater attention to modules ht exceed this value.

The weakness of the McCabe metric is found in the assumption that faults are proportional to decision complexity, in other words that processing complexity and database structure, amongst other things, are irrelevant. Equally it does not distinguish between different kinds of decisions. A simple "IF-THEN-ELSE" statement is treated the same as a relatively complicated loop yet we intuitively know that the loop is likely to have more faults. Also CASE statements are treated the same as nested IF statements which is again counter intuitive

62

Page 63: 202158425 istqb-fl-studyguide

Chapter 5:Test Management

"... Project management throughout the development and implementation process was inadequate and at times ambiguous. A major systems integration project such as CAD

Requires full time, professional, experienced project management. This was lacking...”

"... The early decision to achieve full CAD implementation in one phase was misguided. In an implementation as far reaching as CAD it would have been preferable to implement in a step wise approach, proving each phase totally before moving on to the next...”

Extract from the main conclusions of the official report into the failure of the London Ambulance Service'sComputer Systems on October 26th and 27th 1992.

5.1 Overview

This module covers the overall management of the test effort for a particular project and attempts to answer several key questions such as:

How many testers do we need?

How shall the testers be organized?

What's the reporting structure and who is in charge?

How will we estimate the amount of testing effort required for this project?

How do we keep versions of our test material in line with the development deliverables?

How do we ensure the test effort remains on track?

How do we know that we have finished testing?

What is the process for logging and tracking incidents?

63

Page 64: 202158425 istqb-fl-studyguide

5.2 Objectives

After completing this module you will:

Understand how testing might be organized.

Understand the different roles in a test team.

Understand the importance of test estimation, monitoring and control.

Appreciate the need for configuration management of the test assets.

Understand how and why incidents must be logged and tracked.

5.3 Organization

"We trained hard ... but it seemed that every time we were beginning to form up into teams we would be reorganized... I was to learn later in life that we meet any new situation by reorganizing and a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency, -and demoralization."

A fundamental feature of our lives today is that nothing stays the same. Over time both internal and external pressures on the organizational structures (that we carefully put in place) must change and adapt if our business is to remain competitive. As development and testing organizations grow and evolve, a different structure is required to cope with the changing demands placed upon them. The approach adopted over time may look something like this:

Testing may be each individual developer's responsibility.Testing is the development team's collective responsibility (either through buddy testing or assigning one person on the team to be the tester).There is a dedicated independent test team (who do no development).Internal test consultants 'centers of excellence' provide advice to projects.A separate company does the testing - this is known as outsourcing.

An excellent description of how the test function can be organized within a company can be found in Ed Kit's book, Software Testing in The Real World [KIT95l

64

Page 65: 202158425 istqb-fl-studyguide

5.5 Configuration management (CM)

We all appreciate the need for testing and assuring quality in our development systems. But how many of us appreciate that Configuration Management is a precursor to these goals?

Configuration Management provides us with the balance to ensure that:

Systems are complete.Systems are predictable in content.Testing required is identified.Change can be ring-fenced as complete.An audit trail exits.

We've always practiced CM. Such activities an aggregating and releasing software to the production environment may, in the past, have been uncontrolled - but we did it.

Many famous organizations have found the need to develop a standard for CM that they have then since taken into the market place.

Configuration management encompasses much more that simply keeping a version control of your software and test assets, although that is a very good start. Configuration management is crucial to successful testing, especially regression testing because, in order to make repeatable, you must be able to recreate exactly the software and hardware environment that was used in the first instance.

Typical symptoms of poor CM might include:

Unable to match source and object code. Unable to identify which version of a compiler generated the object code.Unable to identify the source code changes made in a particular version of the software simultaneous changes are mad e to the same source code by multiple developers (and changes lost).

65

Page 66: 202158425 istqb-fl-studyguide

5.6 Definitions

ISO (International Standards Organization) definition of CM:

.

Configuration management (CM) provides a method to identify, build, move, control and recover any baseline in any part of the life cycle, and ensures that is secure and free from external corruption.

Configuration identification requires that all configuration items (CI) and their versions in test system are known.

Configuration control is maintenance of CI’s in a library and maintenance of records on how CI’s change over time.

Status accounting is the function of recording and tracking problem reports, change requests, etc.

Configuration auditing is the function to check on the contents of libraries, etc. for standards compliance, for instance.

CM can be very complicated in environments where mixed hardware and software platforms are being used, but sophisticated cross platform CM tools are increasingly available.

5. 7 Simple CM life cycle process

CM contains a large number of components. Each component has its own process and contributes to the overall process.

Let's take a look at a simple process that raises a change, manages it through the life cycle and finally executes an implementation to the production environment. Here we can see how to CM life cycle operates by equating actions with aspects of CM.

As a precursor activity we 'Evaluate Change'. All changes are evaluated before they enter the CM Life Cycle:

1.Raise Change Packet, Uses Change Management functions to identify and register a change. Uses Change Control functions to determine that action is valid and authorized.

2.Add Configurable Item to Change Packet.Select and assign configurable items to the Change packet. Execute Impact Analysis to determinate the items that also require some action as a result of the change and the order in which the actions take place.

66

Page 67: 202158425 istqb-fl-studyguide

3.Check-In.Apply version CI back under CM control.

4.Create Executable.Build an executable for every contained CI in the order indicated.

5.Sign-Off. .Uses Change Control to verify that the auctioneer signaling that the auctioneer signaling that testing is complete for the environment in which the change is contained is authorized to do so, and that the action is valid.

6.Check-OK to Propagate.Uses Change Control, Co Requisite to verify request to move a Change Packet through the life cycle is valid.

a) All precursor tacks completed successfully.b) Next life cycle environment fit to receive.c) Subsequent change in current environment has not invalidated Change Packet.

7. PropagationAffect next environment population by releasing Change Packet and then distributing over a wider-area.

Note: You might notice that it is composed of a number of singular functions and series executed as a cycle. Of particular note is that 'Create Executable' is a singular function. This is because we should only ever build once if at all possible. This, primarily, saves time and computer resources. However, re-building an element in a new environment may negate testing carried out in proceeding one and can lead to a lengthy problem investigation phase.

5.8 What does CM control?

CM should control everything element that is a part of a system application.

Nominally, CM:

1. Configurable Items:Maintains registration and position of all of our CI's. These may be grouped into logically complete change packets as a part of a development of maintenance exercise.

2. Defines Development Life Cycle:It is composed of a series of transition points, each having its own Entry/Exit criteria and which maps to a specific test execution stage.

3. Movement:Controls and Change Packet move and progresses it through the Life Cycle.

4. Environment:Testing takes place in a physical environment configured specifically for the stage testing that the life cycle transition point and stage reflects.

67

Page 68: 202158425 istqb-fl-studyguide

5.9 How is it controlled?

CM is like every other project in that it requires a plan. It is particularly important that CM has a plan of what it is to provide as it forms a framework for life cycle management in which to work consistently and securely.

The CM plan cover what needs to be done, not by when, and defines three major areas:

The Processes will define or include:Raising a changeAdding elementsBooking elements in and out Exit/Entry criteriaLife cycle definitionLife cycle progressionImpact analysis,Ring-fencing change, release aggregationChange controls.Naming standardsGenetic processes for build and other activities

The Roles & responsibilities covering who and what can be done: Configuration Manager & Librarian Project Manager, Operations Personnel Users, Developers and others as necessary

Records, providing necessary audit trail will include:What is managed, the status of the life cycle position arid change status, Who did what, where, when and under what authority. Also the success factor for activities.Only once the CM plan and the processes that support it are defined can we consider automation.

5.10 What does CM look like?

CM has several hubs and functions that will make or break it. Hubs of system are defined as areas where information and source code are stored. Typically major hubs are central inventory and central repository. Surrounding those are four major tool sets that allow us to work on the data:

Version ManagementAllows us access to any version or revision of a stored element.

Configuration Control Allows us to group elements into manageable sets.

Change Control & ManagementThis is global name given to processes that govern change through application development life cycle and stages it passes through from an idea through to implementation. It may include:

68

Page 69: 202158425 istqb-fl-studyguide

Change Control Panel or Board to assess and evaluate change;Controls: Governing who can do what, when and under what circumstances.Management: Carrying out an action or movement through the life cycle once

the controls have been satisfied

Build & ReleaseControl is how our elements are built and manner in which our change is propagated through life cycle.

The view is about as close to genetic global view of CM as you can get. It won't match all tools 100% as it covers all aspects of CM - and very few of the tools (although they might claim to) can do this.

69

Page 70: 202158425 istqb-fl-studyguide

Exercise

Configuration management -1

Make list of items that you think Test Manager should insist placed under configuration management control.

Exercise

Configuration management - 2

There are very many points to consider when implementing CM. We have summarized them into the following three categories:

CM ProcessesThe framework that specifies how our CM system is to operate and what it is to encompass.

Roles & ResponsibilitiesWho does what and at what time?

CM RecordsThe type of records we keep and the manner in which we keep and maintain them.

Quite a short list you might say. Using the information we have learned so far, try and construct a minimal Configuration Management Plan. Do not try and expand the processes required, but give hem suitable titles in an appropriate sequence.Additionally, for every process you identify, try and match it to one or more segments of the CM Bubble diagram.

70

Page 71: 202158425 istqb-fl-studyguide

5.11 Test estimation, monitoring and control

Test estimationEffort required to perform activities specified in high-level test plan must be calculated in advance. You must remember to allocate time for designing and writing the test scripts as well as estimating the test execution time. If you are going to use the test automation, there will be a steep learning curve for new people and you must allow for this as well. If your tests are going to run on multiple test environments add in extra time here too. Finally, you will never expect to complete all of the testing in one cycle, as there will be faults to fix and test will have to be re-run. Decide on how many test cycles you will require and try and estimate the amount of rework (fault fixing and re-testing time).

Test monitoringMany test efforts fail despite wonderful plans. One of the reasons might be that the test team was so engrossed in detailed testing effort (working long hours, finding many faults) that they did not have time to monitor progress. This however is vitally important if the project is to remain on track. (e.g. use a weekly status report).

71

Page 72: 202158425 istqb-fl-studyguide

Exercise

Try and list what you might think are useful measures for tracking test progress.

Test Manager will have specified some exit (or completion) criteria in the master test plan and will use the monitoring mechanism to help judge when the test effort should be concluded. The test manager may have to report on deviations from the project/test plans such as running out of time before completion criteria have been achieved.

Test control - in order to achieve the necessary test completion criteria it may be necessary to re-allocate resources, change the test schedule, increase or reduce test environments, employ more testers, etc.

5.12 Incident Management

An incident is any significant, unplanned event that occurs during testing that requires subsequent investigation and/or correction. Incidents are raised when expected and actual test results differ.

5.12.1 what is an incident?

You may now be thinking that incidents are simply another name for faults but his is not the case. We cannot determine at the time an incident has occurred whether there is really a fault in the software, whether environment was perhaps set up incorrectly or whether in fact test script was incorrect. Therefore we log incident and move on to the next test activity.

5.12.2 Incidents and the test process

An incident occurs whenever an error, query or problem arises during the test process. There must be procedures in place to ensure accurate capture of all incidents. Incident recording begins as soon as testing is introduced into system's development life cycle. First incidents that will be raised therefore are against documentation as project proceeds; incidents will be raised against database designs, and eventually program code of system under test.

72

Page 73: 202158425 istqb-fl-studyguide

5.12.3 Incident logging

Incidents should be logged when someone other than author of product under test performs testing. When describing incident, diplomacy is required to avoid unnecessary conflicts between different teams involved in testing process (e.g. developers and testers). Typically, information logged on an incident will include:

. Name of tester(s), data/time of incident, Software under test ID

. Expected and actual results

. Any error messages

. Test environment

. Summary description

. Detailed description including anything deemed relevant to reproducing/fixing potential fault (and continuing with work)

. Scope

. Test case reference

. Severity (e.g. showstopper, unacceptable, survivable, trivial)

. Priority (e.g. fix immediately, fix by release date, fix in next release)

. Classification. Status (e.g. opened, fixed, inspected, retested, closed)

. Resolution code (what was done to fix fault)

Incidents must be graded to identify severity of incidents and improve quality of reporting information. Many companies use simple approach such as numeric scale of I to 4 or high, medium and low. Beizer has devised a list and weighting for faults as follows:

1Mild Poor alignment, spelling etc.2Moderate Misleading information, redundant information3Annoying Bills for 0.00, truncation of name fields etc.4Disturbing Legitimate actions refused, sometimes it works, sometimes not

5Serious Loss of important material, system loses track of data, records etc.

6Very serious The mis-posting of transactions7Extreme Frequent and widespread mis-postings

8Intolerable Long term errors from which it is difficult or impossible to recover

9Catastrophic Total system failure or out of control actionslaInfectious Other systems are being brought down

In practice, in the commercial world at least, this list is over the top and many

73

Page 74: 202158425 istqb-fl-studyguide

companies use a simple approach such as numeric scale of 1 to 4 as outlined below:

1Showstopper Very serious fault and includes GPF, assertion failure orcomplete system hang

2Unacceptable Serious fault where software does not meet businessrequirements and there is no workaround

3Survivable Fault that has an easy workaround - may involve partial manualoperation

4Cosmetic Covers trivial faults like screen layouts, colors, alignments, etc

Note that incident priority is not the same as severity. Priority relates to how soon the fault will be fixed and is often classified as follows:

1. Fix immediately.2.Fix before the software is released. 3.Fix in time for the following release.4.No plan to fix.

It is quite possible to have a severity 1 priority 4 incident and vice versa although the majority of severity 1 and 2 faults are likely to be assigned a priority of 1 or 2 using the above scheme.

5.12.4 Tracking and analysis

Incidents should be tracked from inception through various stages to eventual close-out and resolution. There should be a central repository holding the details of all incidents.

For management information purposes it is important to record the history of each incident. There must be incident history logs raised at each stage whilst the incident is tracked through to resolution for trace ability and audit purposes. This will also allow ht formal documentation of the incidents (and the departments who own them) at a particular point in time.

Typically, entry and exit criteria take the form of the number of incidents outstanding by severity. For this reason it is imperative to have a corporate standard for the severity levels of incidents.

Incidents are often analyzed to monitor test process and to aid in test process improvement. It is often useful to look at sample of incidents and try to determine the root cause.

5.13 Standards for testing

74

Page 75: 202158425 istqb-fl-studyguide

There are now many standards for testing, classified as QA standards, industry-specific standards and testing standards. These are briefly explained in this section. QA standards simple specify that testing should be performed, while industry-specific standards specify what level of testing to perform. Testing standards specify how to perform testing.

Ideally testing standards should be referenced from the other two.

The following table gives some illustrative examples of what we mean:

Type StandardQA Standards ISO 9000Industry specific Railway signalingstandard standardTesting Standards BS 7925-1, BS 7925-2

5.14 Summary .

In module five you have learnt that the Test Manager faces an extremely difficult challenge in managing the test team and estimating and controlling a particular test effort for a project. In particular you can now:

Suggest five different ways in which a test team might be organized.

Describe at least five different roles that a test team might have.

Explain why the number of test cycles and re-work costs are important factors in estimating.

Describe at least three ways that a test effort can be monitored.

List three methods of controlling the test effort to achieve the necessary completion criteria.

Prioritize incidents.

Understand the importance of logging all incidents.

Understand the need for tracking and analysis of incidents.

75