The Testing Practioner - Je aanbieder voor Televisie, …members.chello.nl/~j.martijn/The Testing Practitioner v0... · Web viewTest policy ( in quality management system. The quality

The Testing Practitioner

Erik van VeenendaalUTN Publishers, Den Bosch, 2002

ISBN 90-72194-65-9

Summary: Marc Mooij (May/June 2005)

VVersion Date Author Content0.1 29-05-2005 M. Mooij Initial version summary0.2 3-06-2005 M. Mooij en V. Staal Chapters 17 – 19 added

TPI is vooral in het Nederlands, met de kern begrippen vertaald in het Engels. Dit stukje had ik nog liggen....

03 4-6-2005 M. Mooij Schema’s Risk based testing en Structuur test Strategy / Plan toegevoegd (appendix). Tot koptekst 5 doorgevoerd

DisclaimerThis document is only a summary. In order to get the full knowledge and understanding of the course material, you should pay attention in class and read the readers yourself.Standards are not fully included. Study these yourself, especially the terminology.

In this version a English / British spelling checker was used. Some articles are written by American authors. So the spelling of some words might be changed. Spelling and writing mistakes do occur in this version. There was no time for a thorough check-up of all the text.

The use of this summary as a part of your study is therefore on your own risk. If any information is missing or incorrect this is unfortunate, but feel free to correct, change or improve the document yourself.

Good luck with your exam!

Let op: nog niet geheel nagekeken

/tt/file_convert/5ab7bb057f8b9ac10d8c174c/document.docPage 2 of 82

Contents

1 Testing Fundamentals.................................................................................................................................................51.1 Test principles and definitions...............................................................................................................................51.2 Test process.........................................................................................................................................................61.3 Testing and the software life cycle........................................................................................................................71.4 Test levels.............................................................................................................................................................81.5 Traditional versus new..........................................................................................................................................9

2 TMAP test process......................................................................................................................................................92.1 Testing as a process.............................................................................................................................................92.2 Test process model...............................................................................................................................................9

3 Testing and standards...............................................................................................................................................103.1 Standards............................................................................................................................................................103.2 Software testing in context..................................................................................................................................103.3 A software testing model.....................................................................................................................................113.4 Software testing framework................................................................................................................................123.5 Final remarks......................................................................................................................................................12

4 Risk based testing.....................................................................................................................................................134.1 The bad game.....................................................................................................................................................134.2 Understanding necessary quality levels..............................................................................................................134.3 Setting priorities in testing...................................................................................................................................134.4 More effective testing..........................................................................................................................................154.5 Making testing cheaper.......................................................................................................................................154.6 Cutting test work.................................................................................................................................................154.7 Strategies for prevention.....................................................................................................................................164.8 Priority rules........................................................................................................................................................16

5 Good enough testing................................................................................................................................................. 165.1 The language of risk...........................................................................................................................................165.2 How much testing is enough?.............................................................................................................................175.3 When we should stop testing?............................................................................................................................175.4 When is a product ‘good enough’?.....................................................................................................................185.5 How good is your testing?...................................................................................................................................19

6 Measuring Software Quality......................................................................................................................................196.1 Project context....................................................................................................................................................196.2 Quality model: ISO 9126.....................................................................................................................................196.3 Selection of quality characteristics......................................................................................................................206.4 Identification of quality metrics............................................................................................................................206.5 Defining completing criteria.................................................................................................................................206.6 Real –life measurements....................................................................................................................................206.7 Release decision.................................................................................................................................................20

7 Test Point Analysis: a method for test estimation.....................................................................................................217.1 Philosophy..........................................................................................................................................................217.2 Basic procedure..................................................................................................................................................217.3 Principles............................................................................................................................................................227.4 The technique in detail........................................................................................................................................227.5 Test Point Analysis at an early stage..................................................................................................................26

8 Formal review types..................................................................................................................................................278.1 Document reviews..............................................................................................................................................278.2 Participants, roles and responsibilities................................................................................................................278.3 Inspections..........................................................................................................................................................288.4 Technical reviews...............................................................................................................................................298.5 Walkthroughs......................................................................................................................................................308.6 Combing review types.........................................................................................................................................318.7 Success factors of implementation.....................................................................................................................32

9 Reading techniques................................................................................................................................................... 329.1 Current practices.................................................................................................................................................339.2 A reading procedure template.............................................................................................................................339.3 The requirements customer perspective.............................................................................................................339.4 The requirements test perspective......................................................................................................................349.5 Customizing reading techniques.........................................................................................................................359.6 Experiences........................................................................................................................................................36

10 Making inspections work...........................................................................................................................................3610.1 Basic definitions..................................................................................................................................................3610.2 Benefits and current best practice......................................................................................................................3610.3 Improving Inspections.........................................................................................................................................37

11 Static Analysis...........................................................................................................................................................3911.1 Static fault versus dynamic failure......................................................................................................................3911.2 When faults become failure................................................................................................................................3911.3 Early versus late detection..................................................................................................................................39

Version 0.3 June 4th, 2005


11.4 Measurement for static analysis.........................................................................................................................3911.5 Coverage: how much is enough?.......................................................................................................................4011.6 Approaches to static analysis.............................................................................................................................40

12 Testing Techniques; why bother?.............................................................................................................................4112.1 What are techniques?.........................................................................................................................................4112.2 An overview........................................................................................................................................................4112.3 Are techniques for testers only?.........................................................................................................................4112.4 Why use techniques at all...................................................................................................................................4212.5 When should techniques be used?.....................................................................................................................4212.6 Software and system modelling..........................................................................................................................4212.7 New technology and techniques.........................................................................................................................4212.8 Advantage / disadvantages.................................................................................................................................43

13 Black Box Techniques...............................................................................................................................................4313.1 Characteristics....................................................................................................................................................4313.2 Equivalence partitioning......................................................................................................................................4413.3 Boundary value analysis.....................................................................................................................................4413.4 Cause/effect graphing (cause/effect analysis of decision table testing).............................................................4413.5 Classification tree method...................................................................................................................................4513.6 Data cycle test....................................................................................................................................................4513.7 Elementary comparison test...............................................................................................................................4613.8 Process cycle test...............................................................................................................................................4613.9 Semantic test......................................................................................................................................................4613.10 State transition test..........................................................................................................................................4713.11 Syntax test.......................................................................................................................................................4813.12 Test uses cases...............................................................................................................................................48

14 Exploratory testing.....................................................................................................................................................4914.1 Introduction.........................................................................................................................................................4914.2 Practising Exploratory testing.............................................................................................................................5014.3 Where exploratory testing fits.............................................................................................................................5114.4 Exploratory testing in action................................................................................................................................5114.5 Productivity.........................................................................................................................................................51

15 Usability testing.........................................................................................................................................................5115.1 What is usability..................................................................................................................................................5115.2 Contexts of use...................................................................................................................................................5215.3 Designing for usability.........................................................................................................................................5215.4 Planning and managing usability testing.............................................................................................................5215.5 Why is usability testing different?........................................................................................................................5315.6 Designing usability tests.....................................................................................................................................5415.7 Early life cycle techniques...................................................................................................................................5415.8 Late life cycle techniques....................................................................................................................................5415.9 Additional sources of information........................................................................................................................55

16 Performance testing..................................................................................................................................................5516.1 Performance testing............................................................................................................................................5516.2 Motivation............................................................................................................................................................5516.3 Conducting a performance test...........................................................................................................................5616.4 Risk management...............................................................................................................................................5716.5 Experiences from real life...................................................................................................................................57

17 The bug reporting Processes....................................................................................................................................5817.1 Definitions...........................................................................................................................................................5817.2 IEEE 1044 bug process......................................................................................................................................5817.3 Bug reporting process.........................................................................................................................................5817.4 A hypothetical case study...................................................................................................................................5917.5 Classification: beyond the failure description......................................................................................................5917.6 Process quality indicators...................................................................................................................................6017.7 Handling challenges............................................................................................................................................6117.8 Implementing changes........................................................................................................................................62

18 Testing Maturity Model..............................................................................................................................................6318.1 History and Background......................................................................................................................................6318.2 TMM Maturity levels............................................................................................................................................6318.3 TMM Structure....................................................................................................................................................6318.4 TMM Level 2: definition.......................................................................................................................................6318.5 From Detection to prevention..............................................................................................................................63

19 Test Process improvement........................................................................................................................................6519.1 How good is your test process?..........................................................................................................................6519.2 The model...........................................................................................................................................................6519.3 The Improvement process..................................................................................................................................6519.4 Conclusions and remarks...................................................................................................................................65

20 Test tool overview.....................................................................................................................................................6920.1 A closer look.......................................................................................................................................................6920.2 Advantages.........................................................................................................................................................69



20.3 Considerations....................................................................................................................................................6920.4 Test tool overview...............................................................................................................................................69

21 Tool evaluation and selection....................................................................................................................................7121.1 Do you really need a tool?..................................................................................................................................7121.2 The need for a formal evaluation........................................................................................................................7121.3 Identify and document requirements...................................................................................................................7121.4 Conduct market research....................................................................................................................................7121.5 Organise supplier presentations.........................................................................................................................7121.6 Formally evaluating the test tool.........................................................................................................................7121.7 Post evaluation activities.....................................................................................................................................72

22 Test execution automation........................................................................................................................................7222.1 Advantages.........................................................................................................................................................7222.2 Record and playback..........................................................................................................................................7222.3 Test programming...............................................................................................................................................7222.4 Data driven tests.................................................................................................................................................7222.5 Testing with action words....................................................................................................................................7322.6 Consequences....................................................................................................................................................73

23 Team Building...........................................................................................................................................................7423.1 Steaming along or just plodding..........................................................................................................................7423.2 What makes a successful team?........................................................................................................................7423.3 People in a team.................................................................................................................................................7523.4 How to build a more effective team.....................................................................................................................7523.5 Management.......................................................................................................................................................7523.6 Retaining test expertise......................................................................................................................................7623.7 Your career belongs to you!................................................................................................................................7623.8 How do we spot a good tester?..........................................................................................................................7623.9 Communications.................................................................................................................................................7723.10 The end game..................................................................................................................................................77

24 Test career paths......................................................................................................................................................7824.1 Three dimensions...............................................................................................................................................7824.2 The test career cube...........................................................................................................................................78



Part 1 Test principles and Process1 Testing FundamentalsIsabel EvansAim: refreshing the reader’s understanding of the fundamentals of testing:- the reasons for- the principles behind- place of testing in the life cycleOverview of test process, levels and types, testing techniques and principals task of the management Standard reference for test vocabulary and definitions

1.1 Test principles and definitionsTesting is necessary To identify errors or faults in order tot reduce faultsHumans make mistakes / errors, which cause faults in products. They may be unnoticed or they may cause failures.A fault is the result of an error – it is something wrong which might occur in an interim product such as a specification or code or user documentation . A failure is the deviation of the product from its expected delivery of service. (BS7925-1)

Definition of software qualityTesting is one of the verification and validation methods applied. Other methods: Proof reading, reviewing and auditing of processes.Software quality: is delivered on time, to budget and to specification ( must have correct specification, a correct time scale and a correct budget. The delivered system must subsequently meet the specification Validation = is this the right specification and verification: is the system correct to specification.Validation = Determination of the correctness of the products of software development with respect to the users needs and requirements Is this the right specification (BS0725-1)Verification = The process of evaluating a system or component to determine whether the products of the given development phase satisfy the conditions imposed at the start of that phase Is the system correct to the specification.

Other definitions of qualityGarvin: five distinct definitions of quality could be recognized. Veenendaal: The definition of software quality will affect how a project defines testing.Product based definition: Quality is based on a well-defined set of software quality attributes. These attributes must be measured in an objective and quantitative way. Differences in the quality of products of the same type can be traced back to the way the specific attributes have been implemented.User based definition: Quality fitness for use. This definition says that software quality should be determined by the user(s) of a product in a specific business solution. Different business characteristics require different qualities of a software product. Quality can have many subjective aspects and cannot determined on basis of only quantitative and mathematical metricsManufacturing based Definition: This definition points to the manufacturing, i.e. the specification, design and construction, process of software products. Quality depends on the extent to which requirements have been implemented in a software product in conformance with the original requirements. Quality is based on inspection, registration and (statistical) analysis of faults and failures in (intermediate) products.Value Based Definition: This definition states that Software quality should always be determined by means of a decision process on trade-off’s between time, effort and cost-effort aspects. The value based definitions emphasizes the need to make trade-off’s, this is often done by means of communication with all parties involved, e.g. sponsors, customers, developers and producers.Transcendent definition: This ‘esoteric’ definition states the quality can in principle be recognized easily depending on the perceptions and the affective feeling of an individual or group of individuals towards a type of software product. Although the least operational one, this definition should not be neglected in practice. Often a transcendent statement about quality can be a first step towards the explicit definition and measurement of quality.

Project pressures – risk, time, budget and scopeTesters need to consider the scope of testing against the risk for the customer and the business. This is critical in understanding the test strategy, especially: which interim and final products must be tested (all, some depending on risks and budget) What types of testing are required? non-functional as well functional testing depending on risk and

acceptance criteria (defined and implicit) How independent must testing be? depending on risk: designers or independent testing

Triangle: Specification – time pressure – cost pressure.

What needs testing?Intermediate and final products during a project All levels of specification and design Code Documentation for the user group / user guides, help messages Operational documentation for IT maintenance/support Any service level agreements



Definitions of testingInitial: we want to know that the software worksHetzel (1973): Testing is the process of establishing that the program or system does what it is supposed to. Positive / conformationMyers (1979): Testing is the process of executing a program or system with the intent of finding errors. But what is an error? define precise results expected from the testsIEEE (610.12, 1990): Testing is the process of exercising or evaluation a system by manual or automatic means to verify that it satisfies requirements or to identify differences between actual and expected results Consider wider aspects of software quality not just functionality but also capabilities of the software whether functionality (correctness, testability) or non-functional (reliability, usability, maintainability) measured against requirements and risks Hetzel (1984): Testing is any activity aimed at evaluating an attribute or capability of a program or system. Testing is the measurement of software quality meets requirementsEvans et al,(1996): testing is demonstrating that the system is fit for purpose --> user-based definition / more pragmatic definition Pol & Van Veenendaal, 1996: Testing is a process of planning, preparation and execution to establish the characteristic of a software product tot determine the difference between the actual and required status. covers both validation and verification & non-functional testingGelperin & Hetzel, 1988: Testing is a process of all life cycle activities concerned with checking software and software-related work products more inclusive activity and not just executing code.BS7925-1, 1998: Testing: the process of exercising software tot verify that it satisfies specified requirements and to detect errors bringing the other definitions together (missing: key questions of quality and testing) / manufacturing based definition.ISEB Foundation 1999: As the objective of a test should be to detect faults, a ‘successful’ test is one that does not detect a fault not value based or transcendent definition of quality.All inclusive definition: The planning, preparation and execution of tasks to establish the characteristics of a software product and to determine the difference between the actual and required status, in order to meet the quality requirements of the customer and to mitigate risk.

1.2 Test processIEEE 1008: Standard for Software Unit TestingTest process = circular a number of steps which may need to be repeated several times. See also BS79-2 Standard for Component Testing. Component test – prove the component works in isolationTest process 3 major activities: Planning the testing including strategy & detailed test planning Acquiring the test designing tests, building testcases & testdata, getting tools ready, getting management /

reporting ready. Measuring the product executing tests, recording results, dealing with any discrepancies and test incidents,

checking for test completion.

Planning the test strategyTest strategy at least rationale for amount, breath and depth of testing for the project. It should: Summarize the risks (vary from project to project) including non-functional test (incl. Risk rate) Summarise any constraints time, budget, personnel, equipment and the effect of the constraints on risk Document entry and exit criteria for each test level or phase purpose: agreement on minimum standard for a

product entering testing and for test completion Document what types of testing will be done and decide what is out of scope (especially if results are not as the

customer or the team expected) Document pass/fail criteria at a high level.Test plan good starting place for new writing test strategy or reviewing an existing strategy.IEEE 829 full strategy = more comprehensiveTest policy in quality management system. The quality plan or test procedures High Level test Plan (Test strategy = overall policy document)

Planning – detailed test plansEach type of testing is planned in detail. Overall strategy broken down in several detailed strategies for different areas of the project. Each area (Level, type [functional / non-functional] how the strategy is applied. Depending on risk there may be exceptions to the strategy.

Acquiring – the tests and surrounding environment Test designTest design / test specification. Test cases are designed for each of the functional and non-functional tests using formal techniques, error guessing or test cases from a library of previously designed tests. Acquisitions parallel: Acquiring test data building it / extracting it use , deciding which library to use Acquiring the test environment in some form (building, booking or checking) Checking and making the tools ready incl. Test management tools / techniques (e.g. reporting, metrics gathering,

defect logging and tracking). Note: too late in this stage to introduce new tools

Test readiness and entry criteriaSoftware ready test, test environment, data, tools, team, test management must be ready. Help; checklist or formal meeting. The questions: Have we got everything we need? Has everything met the entry criteria, which we set in the strategy?



Testing cannot start until the product which is to be tested is ready Entry criteria for test phase: include exits from previous test phases, product build stages an test preparation activities. Often: ill-prepared environment. Work around: deal with areas which are ready.

Measuring – the product and the testingProduct is measured against its acceptance criteria. Executes designed test cases – measure coverage – log results. Important: manage test configuration by recording the identity and version of products and tests for test results, completeness and ease of audit checking.

Test incidentsLog any difference between actual and expected results as a test incident or discrepancy. Investigate cause: Test problem environment was wrong, used data were wrong, version of test specification was wrong, input

was mis-keyed Product problem fault in code, omission in the requirements, fault in user documentation Some other problem: cause not yet clear, outside the test team’s control Depending on the cause of the problem repeat one or more test activities, make changes of products,

environments or test materials. Any changes can adversely affect the whole product.

Exit criteria for test completion and test managementUse previously defined exit criteria to check whether the tests are complete. Not complete continue (or circle back to more plan test). Exit criteria include: Test completion criteria based on time, budget, risk, number, or requirements covered, product acceptance

criteria met, BB test coverage measurements, WB test coverage measurements, number of outstanding faults and their priorities and any other coverage criteria.

Test management criteria including test filing in the test library, updating test packs and sign off for any test incidents raised.

Suspension and resumption criteriaSuspension criteria: when testing has to stop mid-execution because of severity of failureResumption: when a test execution can be restartedMost Important Test first (MITS)Risk in project changed it may change the exit criteria cannot run all tests that were planned prioritise test an run most important test first.

Document skeletonsTo record test activities use document skeletons (IEEE 829). Major Headings as starting point for building document templates (fits with the organization practise). It can be practical to combine documents It can be useful on small projects to use headings as a checklist It may be necessary to split documents or have bigger document family for a large project

1.3 Testing and the software life cycleTesting throughout the life cycle= V-model of testing It starts carry out testing activities early in the life cycle before coding startsTesting execute against the code.Existing code maintenance, evolutionary, prototyping late life cycle

Best practice Product is produced throughout the life cycle early products [requirements definitions and specifications]

are used to build later, final products. Errors in early products early test activities improve the development activity

Cost savings the earlier we find and fix faults the cheaper it is (each stage of the life cycle cost will rise 6 to 10 times)

Time savings the V-model allows us to get the requirements right before we build the system so less rework is needed and prevention is better than cure

Professional satisfaction doing a good job encourages communication between the supplier and the customer

Early life cycle techniques are effective quote static testing techniques such as inspection are more effective than dynamic testing

Late life cycle testing has a different focus early products can be correct, but mistakes still occur when building

Different types of testing may be applied at different stages depending on product and the requirements to be tested.

Working of ‘Test than code’ at each stage in the early part of the life cycle (contract and acceptance criteria definition, requirements definition, design and detailed design) a pair of test activities are associated with non-test activity. Two generic test activities are: Definition of tests to be run later in the life cycle (incl. Static test) Review / inspection a static test activity



Early life cycle testingCarry out the following testing activities before coding starts: Test management Strategy and test planning, risk assessment, setting of acceptance criteria for functional

and non-functional requirements, auditing and reviewing processes Test design using BB techniques: equivalence partitioning, boundary value, syntax, cause-effect, state

transition) to design tests based on specifications for functional tests. Static testing using review techniques (peer review, walkthrough, inspection) to test requirements,

specifications and other early products. Non-functional testing decide which non-functional test types are required and design these non-functional

tests, carry out static non-functional tests for some non-functional attributes, such as usability reviews Test Tools may support test management, requirements management, test design and review process

Late life cycle testingCarry out the following testing activities once code has been built: Test management incident management, reporting, configuration management, risk assessment, audit and

review of processes Test design using WB / structural techniques: statement, branch-decision, branch-condition, data-flow,

LCSAJ) to design tests based on code Dynamic tests run tests designed earlier, measure test coverage (WB or BB measurements) plus additional

ad-hoc testing Static Tests use review techniques as before. For code use static analysis to measure code complexity Non-functional testing at suitable points run the non-functional tests designed earlier. Reliability testing run

very late (after system test). Memory tests during component or integration test. Test tools test management, test design, test coverage measurement, testing dynamic attributes [memory

usages], static analyses, non-functional testing [performance, stress, security, usability], test execution, results comparison.

1.4 Test levelsWithin dynamic testing a number of different test levels can be distinguished

Component testing Is planned and designed early in the life cycle with detailed designs before coding starts Is run late in the life cycle, but before system testing Is based on the detailed design specifications Is based on the internal working of individual components, programs, objects and at extreme / failure conditions May be carried out by the specialist testers or by the development team within the project May require non-functional as well as functional tests Will use BB (specification based) and WB (code based or structural) design techniques May require tools to support the functional and/or the non-functional testing

Integration testing ‘in the small’ Is planned and designed early in the life cycle with the system designs Is run late in the life cycle, but before system testing Is based on the detailed design specifications Is based on the parts of the system that look at links between components and subsystems and at

extreme/failure conditions May be carried out by the specialist testers or by the development team within the project May require non-functional as well as functional tests Will use BB (specification based) and WB (code based or structural) design techniques Is sometimes referred to as integration ‘in the small’ or as link testing May require tools to support the functional and/or the non-functional testing

System testing Is planned and designed early in the life cycle Is run late in the life cycle, but before acceptance testing Is based on the acceptance criteria and requirements specifications Is based on the system as a whole, looking at business processes and at extreme / failure conditions May include interoperability (‘integration in the large’/ system to system integration) May be carried out by the specialist testers or by the development team within the project May require non-functional as well as functional tests Will generally use BB (specification based) design techniques May require tools to support the functional and/or the non-functional testing

Acceptance testing Is planned and designed as early as possible Is run late in the life cycle Is based on the acceptance criteria and a business view of the system May include interoperability (‘integration in the large’/ system to system integration) May be contractual Is carried out by any groups that needs to accept the system for operational usage, such as users, IT support, IT

security and database, network or operations administrators



May require non-functional as well as functional tests Will generally use BB (specification based) design techniques May require tools to support the functional and/or the non-functional testing

1.5 Traditional versus newV-model may be applied – in principle – to all types of life cycle, projects, technology and products. The essence remains the same, but the risks will be different specific techniques and types of testing In each iteration of prototyping or evolutionary life cycle apply V-model in miniature or move from formal to

informal application of processes or techniques Web testing additional: usability, security, load testing.

2 TMAP test processErik van VeenendaalTest process consists numerous activities.Process model map out various activities, sequence and interdependence. Number of phases subdivided into activities.Objective, input, process, output and tools of each activity are described in detail Process model cornerstone supporting structured testingTMAP provides answers to the what, when, how, where and who questions of testing.Four cornerstones: A development process-related life cycle and process model for the testing activities (L) Solid organisatorial embedding (O) The right resources and infrastructure (I) Usable techniques for various testing activities (T)

2.1 Testing as a processMain activities: planning (20%), preparation (40%) and execution (40%)During creation of the requirements specification MTP who performs which test levels, how and when.Ideally it covers all test levels component – acceptance testing or limited: higher (system / acceptance) or lower (component, integration, system).Larger projects more detailed phase test plans

2.2 Test process modelActivities: Planning, specification and execution divided in five phases: Planning and control Preparation Specification Test Execution Completion phase

For lower level testing too many activities select requires elements

The planning and control phase

Starts during specification of the requirementsProvides the basis for a manageable and high-quality testing processDiscussion about:Actual value of the development planning, the expected quality of the test object, the organization of the various tasks, the availability of staff, infrastructure and timeNecessary to agree on test assignment requirements specification, functionality and project organisationTest strategy is determined by means of risk analysis ascertain which parts of the system will get more attention during testing, depending on the risk involved.Defining test strategy basically a communication process, involving all parties (most important parts, most appropriate coverage) structuring the testing organisation and defining the test infrastructure.Other activities during the entire test process manage testing with regards to time and resources used.Reporting is done on the progress of testing and on the quality of the test object.Most important delivery: quality report describes the accompanying risks.Right from the start: developing a view of the quality.Well structured management information to avoid any misunderstanding

The preparation phaseAs soon as possible after the test plan has been drawn up and agreed upon.Training test staffSpecification 100% don’t start earlierDetailed study of the requirements specification and other documentation.Reviewing provides an insight on testability standard notations, understandability, recognizability.



Test basis will be divided into sub-systems that can be independently delivered and testedTest techniques are allocated to each of these test unitsChoice of test specification techniques will depend on the risks to be mitigated (derived from the test strategy)

The specification phaseTest cases are specified and accompanying test infrastructure is realisedTest cases logical and physical test designExpected outcome resultsLater logical test cases are translated into physical test cases (test procedures)Initial content of the test database is also defined. Test infrastructure is being set up.

The execution phaseStart: first testable component of the software product are availableCheck for completeness and installed on the test environment can run without immediate failures?Actual test Initial test database has to be set up. (important, accuracy)Pre-test to check main functions Is it sufficient Execution on basis of agreed test strategy, giving priority to the most important parts of the systemDifference between expected and actual test result is logged and analysed defect in software, defect in specification, test infrastructure or an invalid test case.Cause of observed difference is investigated further during checking and judgement activities.Rework completed tests are executed againAllowance for quick and reliable quality reporting

The completing phaseActivities to be done Mostly in a less structure way.Test execution under high pressure concessions to the control proceduresDefect tested without delay for the projectCompletion activities often get low priority

During execution large amount of testware prepare for maintenance regression testware

Test process evaluationData gathered and intermediate reports are combinedTesting process and quality of the product are evaluated

3 Testing and standardsStuart ReidTwo views are used to identify relevant standards: Those higher level standards that require software testing to be performed as part of a larger process A generic software testing model is presented and standards identified that activities within it.

3.1 StandardsWhat use are standards?Consumer viewpoint standards are generally useful / providing the expertise to a transaction that would otherwise be lackingProducer viewpoint existence of a standard detailing good practice (if there is one)In software: there is no kite-mark; testers have no single source of good practice.There are many standards that touch upon software testing Many standards overlap and contain what appear to be contradictory requirements. There are also large gaps in the coverage of software testing by standards. Integration testing no standards at all.

Many standards BS9725-1 & 2, IEEE 829, IEEE1008

3.2 Software testing in contextBS9725-1: mainstream definition process of exercising software to verify that it satisfies specified requirements and to detect errors. Software testing is one way of performing both software verification and software validation, another being static techniques, such as reviews. Verification and validation process: part of a larger process of software engineeringSoftware: rarely runs as a stand-alone process generally produced to run as part of a larger process (long before: embedded software).Process-oriented standards identified for each of the different levels ISO 15288, ISO 12207, IEEE 1012.ISO 12207: standard that defines a framework for software throughout its life cycle. as umbrella for IEE Standards for integrated testing.IEEE 1012 specific verification and validation processes, activities and task to be performed, based on integrity levelsDifferent perspective from Quality perspective Testing as a part of verification and validation can be seen as an integral part of software quality assurance.



Software as part of larger system software testing considered as part of overall quality management and assuranceIEEE 730: high level verification and validation X other testingISO 9126: lower level Quality Model for six characteristics: functionality, usability, reliability, efficiency, maintainability and portability. Each is broken down in sub-characteristics with corresponding metrics provided for their measurements. Especially useful for non-functional testing.ISO 9126 evaluation process now included in ISO 14598

Third model natural language Natural language at the highest level IEE 610 and BS9725-1Till now: software testing as BB. Further on model for WB testing

3.3 A software testing model This model covers test process, test improvement, terminology, documentation and incident management figure 3.4 (Generic Software Testing Model)

Test TerminologyCommon set of test terminology ensures efficient communication between all parties concerned software testing BS9725-1: a specialist vocabulary. IEEE610.12: good overall glossary of software engineering terminology.Shortcoming BS9725-1: is biased towards component testing

Test PolicyCharacteristics the organisation’s philosophy towards software testing standards ISO 90001, ISO 12207, IEEE 730 ISO 9001: Provides only a very high level requirement that suppliers perform testing as part of verification and document it.ISO 12207: defines the requirement for specific verification and validation process to support the primary process of development, maintenance etc.IEEE 370: requires the production of both a Software verification and Validation Plan (and corresponding Report) and documentation of other tests (by developers etc.)It is superfluous ISO 9001 en 12207

Test StrategyTest Phases on high level phases ISO 9000-3: guidance on the application of ISO 9001 and suggest unit, integration, system and acceptance testing, basing the extent of testing on the complexity of the product and risksIEEE 1012: verification and validation, activities and tasks to be performed based on software integrity levels ISO 15026: process for determining integrity levels based on risk analysis (according ICE 60200-3-9).

Project Test PlanDefines Test phases to be performed and testing within those phases aligned with test strategy. Any differences will be highlighted and explained within the document. ISO 9000-3 suggests a brief list of contents for test plansIEEE 829: comprehensive set of requirements for test planning documentation

Phase Test PlanProvides the detailed requirements for performing testing within a phase (component test, integration test plan)IEEE 829: comprehensive set of requirements for test planningBS9725-1: more relevant for unit / component testing phase detailed content of a software component test plan and provides an example set of documentation. There are no standards that cover other test phases specifically.

Test ProcessBS9725-2: component test plan (IEEE 829 unit test): Test Plan – Test Specification – Test Execution – Check and Report – Check Completion.Decisions made which test case design to use, which test completion criteria to apply should depend on integrity levels of the software based on some form of risk analysis.ISO 15206: Determining integrity levelsIEEE 1028: risk analysis BS0725-2: techniques to design test cases, coverage, completion

Incident ManagementAn essential adjunct tot the testing process ISO 11207 + IEEE 829: briefly covers incident reportIEEE 1044 : more detailed coverage defines anomaly classification process and classification scheme. Comprehensive guidelines IEEE 1044-1

Test DocumentationIEEE 829 comprehensive set of requirements for test planning, test specification and test reporting documentation

Test Process ImprovementISO 12207 + ISO SEI: capability maturity frameworksTMM: no specifically process improvement standards, but TMM available for software testing process improvement

Integrity levels and risk based testingMore a part of a system standard requires different parts to meet different requirements.It is sensible tot partition the system rigorous requirements fro some parts of the system Integrity determined on the basis of some form of risk assessment, and when used for safety-related applications this is obviously based on safety issues.Integrity levels determined corresponding requirements (methods, techniques) are selected



Application-Specific Standard No special testing techniques for specific products. Watch the risk.

3.4 Software testing frameworkThere will always remain some application-specific knowledge that is worthy of formalisation.

3.5 Final remarks ISO 9000: standards for compliance and marketingISO 12207: framework of life cycle processes within which testing is performedModel poorly supported in one main areaUnit testing phase is supported with two standard available best: BS9725-2Information of Test Improvement available proprietary

Use of integrity safety-related IEEE 1012: good standard of verification and validation (generic __> software testing, or economic or risk factors, rather than safety)Application-specific standards obsolete



Part 2: Test and Risk Management

4 Risk based testing Hans SchaeferOne has to test what is most important in the application functionality, frequency of use and possible cost of failureOne has to test where one may find the most problems identifying defect prone areas in the productUsing both one finds a list of areas to test more and those to test less.Defects clump together in defect prone areas later : focus on areas where defects have been found and one should generate more test aimed at the type of defect detected before.

4.1 The bad gameTesting may be done in the last minute and the deadline is fixed but there are options. Test plan is ready, product not ready and testers not available,

How to get out of the game?Present alternatives a product is going out of the door but the risks have to be understand. Strategies:Find the right quality level not all products needs to be free of defects cut down testing in less important areas.Most important defects first (= in the most important functions) analysing how every function supports the mission and checking which functions are critical. And which are not. Also test where more defects are expected (worst areas where more defects already found). Thus: most important and worst area priorityMaking testing cheaper in general automation of test execution (be cautious: it can be expensive without experience)Getting someone else to pay (= customer) another department. Option: require the product to fulfil certain entry criteria before you test (entry criteria: certain test coverage in unit testing, certain level of reliability). Problem: need high level support / entry criteria are skipped if the project gets under pressure and organisational maturity is low.Prevention: benefits only for the next project

4.2 Understanding necessary quality levelsFavaro 1996: Software is embedded in the larger, more complex business world. Quality must be considered in that context.Relentless pursuit of quality can dramatically improve the technical characteristics of a software product.Quality: the only of most important framework for strategic decision making?? The real issue: which quality will produce the best financial performance which qualities and functions are important.Avoid the fanatical pursuit for quality for its own sake.

4.3 Setting priorities in testingTesting is always a sample. You can never test everything need to make decisions about what to test and what not to test, what to test more and what to test less. General goal: to find worst defects first, and to find as many such defects as possible.Goal: To find the most important functional areas and product properties. Finding as many defect as possible testing more in bad areas of the product you need to know where to expect. Result: a list of functions and properties with an associated importance. Use a scale from 0 to 5. Five points for the most important / worst. 0 points not necessary to include them in the testing effort at all.

What is important?In this paragraph: a way to prioritise. In every product there may be other factors playing a role.Important areas functions/functional groups – properties such as performance / capacity / security.Major factors to look at include:

Critical areas (cost and consequences of failure)Analyse the use of software within the overall environment analyse the ways the software may fail. find possible consequences of such failure modes / at least the worst ones. Take in account: redundancy, backup facilities / possible manual checks.Software controls process analyse process itself.A possible hierarchy: A failure would be catastrophic Stop: large financial losses of human lives A failure would be damaging Program may not stop, but data may be lost or corrupted A failure would be hindering forced to work around / more difficult actions to reach the same results A failure would be annoying does not affect functionality (only less appealing)

Visible areasAreas where users will experience a failure, if something goes wrong. Factor in accounted: tolerance of the users to such problems. Software for general public needs careful attention to the user interface. Robustness major concern

Most used areasSome functions used every day X only a few times / many users X only a few users Give priority to the functions used often and heavily.



Possibility: cut out functionality which will only be used once per quarter / half-year/year.Possible hierarchy:Unavoidable A area most users will come in contact during average usage session (start-up, printing, saving)Frequent A area most users will come in contact with eventually, but maybe not every sessionOccasional A area a average user never will visit, but that deals with functions a more serious or experienced user wil need occasionallyRare A area most user never will visit and which is visit only if users do very uncommon steps to action. Critical failures are still of interest.See also Karlsson, 1997

What is (presumably) worst?Worst areas are the ones having most defects. Task: to predict where most defects are located. Analyse: probably Defects generators

Complex areasComplexity is maybe the most important defect generators. There are > 200 different complexity measuresMost complexity measures may indicate problematic areas examples: long modules, many variables in use, complex logic, complex control structure, large data flow, central placements of functions, subjective complexity

Changed areasKhoshgoftaar, 1998: Change is an important defect generator Understood as easy, but impact not thoroughly analysedChanges are often done under time pressure and analyses is not completely doneDebugging can be more detrimental than good to quality changes introduce more defects than they repair.Protocol of changes should exist parts with many changes may be designed bad from start (original design).Many changes symptom of badly done analysis (may not be correspond with the user expectations).

Impact of new technology, solutions, methodsProgrammers using new tools, methods and technology experience a learning curve. Any new tool or technique may give trouble; example: new GUI.Other factor: maturity of methods and models.Maturity: the strength of the theoretical basis or the empirical evidence (use established methods, like finite state machines, grammars, relational data models ….)Most software cost models include factors accommodating the experience of programmers with the methods, tools and technology. Import for test planning (cost estimation)

Impact of the number of people involvedThousand monkey’s syndrome more people more overhead / > change things will go wrongSmall group of highly skilled staff is much more productive than a larger group with average qualificationsAreas where relatively many and less qualified people have been employed, may be identified for better testingBeware: best people in more complex areas, less qualified people in easy areas.

Impact of turnoverIf people quit the job, new people have to learn the design constraints before they are able to continue that job. (Not everything will be documented, overlap between people). Areas with turnover will experience more defects than areas the same group of people has done the whole job.

Where there was time pressureTime pressure leads to people making short-cuts often try to skip quality control activities.Time pressure overtime work loose concentration. Look to: time lists, interviewing management / programmers.

Areas which needed optimizingOptimatization needs extra design effort (resources away from defect removal activities) or using less robust design methods (more defects).

Areas with many defects beforeDefect areas leads to changes, which lead to new defects, and therefore defect prone tend to persist. Experience: defect prone areas can be traced back to defect prone areas in the reviews and unit and subsystem testing.Khoshgoftaar, 1998 / Levendel 1991: Faults in the past are likely to have faults in the future

Geographical spreadPeople working together with certain distance between each other communication will be worse.Detrimental effect: People working in different floors, same building will not communicate People sitting more than 25 meters apart will not communicate enough Common area (coffee) improves communication Different buildings do not communicate as much as people in the same building Different countries have difficulties: culturally / language Different time zone communicate will be more difficult Geographical spread is not dangerous. Danger: communication

History of prior useMany users have used software before active group may be helpful in testing new versions (beta testing, prototyping)



Local factorsYou have to look out for possible local factors outside the factors having been discussed here.Who did the job, who is new in the project, who communicate best etc.New in project: run a rough test to find prone areas. First test will cover the whole system, but very shallow. Only cover typical business scenarios and a few important failure situations. Find areas of most trouble next round give priority to these areas deep en through testing of prioritised areas.

How to calculate priority of test areasAssign weights, and to calculate a weighted sum for every area of the system.Factor: For every factor relative weight 1, 3, 10Weight: For every factor chosen assign a number of points to every product requirement (function, function area, or quality characteristic) scale 0, 3, 5Number of points for a factor is then multiplied by its weight SUM

Testing assigned by the greatest number of tests to the areas with the highest number of pointsNumber of points: only a rough guideline good enough to distinguish the high-risk areas from the medium and the low risk areas.

4.4 More effective testingTo find more and more important defects in the same amount of time.Four phases: Test preparation Pre-test Main test After test

Test preparationSet up Areas to test, test cases, test programs, databases en whole test environment. Test environment: most trouble

Pre-testRun after the software is installed in the test lab. Test a few test cases. Gaol: to see if the software is ready for testing at all (or totally unreliable / incompletely installed) initial quality data.

Main testAll of the pre-planned test cases. Test run, failures recorded, defects found and repaired. Every new installation pre test

After testStart with every new release of the software. here optimisation Regression testing / Shift of focus Every defect is a symptom of a weakness of some design scan on the same time tot find similar defectsConcentrate more tests on the more common kinds of defects carefully analysis is needed.Location of defects may also be used to focus testing. Beware: high level of defects is not caused by especially high test coverage in that area.

4.5 Making testing cheaperDo work in a more productive and efficient way applying technology / personnel qualification

AutomationAutomation can probably do most in the area of test running and regression testing.Automation testing often find more defects. But repair defects cost time (delay of the project)Tools require investment in training and learning at startSometimes a lot of money is spent in fighting with a tool.Impossible without automation: stress, volume and performance testing (automatically or not to do it at all)Test management improvement tools for tracking test cases, functions, defects and their repairsAutomation keep cost for start-up and tool evaluation outside the project budgetTools can help if people are used to it use them effectively and efficiently

The people factor – Few but good people against many who don’t knowLargest obstacle to an adequate testing staff: ignorance on the part of managementTesting requires skill and knowledge application knowledge and experienceCheap: get a few highly experienced specialistsTest people: at least equally smart, equally good designers and have equal understanding of the functionality of the systemFull time testers

4.6 Cutting test workGet rid of part of the task get someone else to pay for it or cut it out completely.

Who pays for unit testingUnit testing is done by the programmers and never turns up in any official testing budget. Problem: unit testing is often not really doneTest coverage vendors 40 – 50% of the code are never unit tested



Test manager should require higher standards for unit testing

What about test entry criteria?If the supplier does not meet the contract, the supplier gets no acceptance and no money. Problems occur where there is only one supplier and when there is no tradition in requiring quality.Criteria can be applied if the test group is strong enough.Entry criteria: The system delivered to integration or system test is complete It has been run through static analysis and defects are fixed A code review has been done and defects have been occurred Unit testing has been done to the accepted standards (near 100% statement coverage) Any required documentation is delivered and is of a certain quality The units compile and can be installed without trouble The units may even have been run through some functional test cases by the designers Really bad units are sorted out and have undergone special treatment like extra reviews, reprogramming etc.

Less documentationTest by the book a lot of documentationA test log may be sufficient / checklist / test plan with a description of what is critical to test / test summary report describing what has been done and the risk of installation.

Cutting installation cost – strategies for defect repairEvery defect delays testing and requires extra cost. rerun test case, reproduce defect, document (for developers debugging), install new version and retest. Is not budgeted Some advice to keep them low:

When to correct a defect, when not?Every installation of a defect fix means disruptionOptimising defect repair work: Rule 1: Repair only important defects Rule 2: Change request and small defects should be assigned to next release Rule 3: Correct defects in groups! Normally only after blocking failures are found.

4.7 Strategies for preventionEverything is late & Where no professional budgeting has been done. Most organisation: no serious attempt to really estimate costs for development, testing and error cost in maintenanceImperatives are: You need a cost accounting scheme You need to apply cost estimation based on experience and models You need to know how test quality and maintenance trouble interact.Measure Size of project in lines of code, function points etc. Percentage of work used in management, development, reviews, test preparation, test execution, and rework Amount of rework during first three or six months after release. Fault distribution, especially causes of user detected problems Argue for testing resources by weighting possible reductions in rework before and after delivery against added

testing cost.A different way to prevent trouble is incremental delivery. small releases first: least commercially acceptable system (does exactly what the old one did, only with new technology). First version you can learn about cost, error content, bad areas.

4.8 Priority rulesManagement cuts budget and time you have to survive

Concentrate on the high-risk areas and worst areas Priority 1: Return the product as fast as possible to the developers with a list of as serious deficiencies as possible Priority 2: Make sure that, whenever you stop testing, you have done the best testing in the time available

5 Good enough testingPaul GerardJames Bach 1997: Good Enough understanding the risk-based testing test approach framework of decision making (at least in projects where risk are being taken)

5.1 The language of riskTesters communicate in terms of test plan, incidents, faults et son on; managers do not get the ‘big picture’.



One valuable by-product of risk based testing is that test plans and test execution logs reference the test objectives derived from risk analysis, and this traceability is maintained throughout the test process.Identify: risk that have been addressed (test passed) or because we know they are not yet addressed (test failed) you run tests, found some bugs confident about the features which have been tested some risks are cleared.Some other tests are still unmitigated wait for fixes, tests are not run.A decision to stop testing is made with clear knowledge of outstanding risk of release Talk to user management and project management in their language. they think in terms of risks and benefits.By looking at risks, testers act like the eyes of the project they search out potential problems some way ahead the project is now.Focus attention on the deliverables of most value to the project sponsors spent our time testing the most important things.Proposal: we should use the language of risk (and benefits) to plan our testing and to report progress throughout our test planning and executing. Using the language of risk, testers will be heard by management.Developers language sounds exciting and beneficial. Testers’ language sounds negative managers are bored and irritated. Testers have to raise their language to management level risk addressed and outstanding instead of tests, incidents and faults (to be used for developers).At the end testers have at least in a qualitative way, and even quantitatively (performance and sustainable user numbers) a very good idea of what the project is going to deliver managers will make bad decisions if they insight and data on the product’s tested capabilities and shortcomings. Testers have that information in large quantities and should provided the management

Difficult Questions for testersWe have never enough time and resources. However, the language of risk helps to answer these difficult questions. The four questions: How much testing is enough? When (and why) should we stop testing? When is the product good enough for release? How good is our testing anyway?

5.2 How much testing is enough?Manager at beginning of project: can you estimate for how many people and how long you need to do the system testing? Look at requirements feel complexity and scale of the system. Consult architects, designers, developers, users.It is never enough. The boss’ enough is less than yours. Mostly managers value developers more than testersWhy asking for an estimate when the decision has already been made? In mind of the managers testers always want more time and resources.Sometimes: time required for test execution is indeterminate you don’t know how many faults there are, their severity, how long fixes will take? You don’t developers don’t know.Testing is a sampling activity test planning must be about selecting the best sample of tests that we can run in the time available.Don’t be upset by limiting of test time. Testing is always time constrained. A good tester should identify the risks of concern to project and propose some activities (test) that can help to measure or reduce those risks. (Prioritize and include those tests that fit into the constraint imposed).Testers should take management ideas of addressing risks and for taking advantage of benefits.Only if the risks are visible can management recognize the risks they take by squeezing testing. So there is no kind of formula for right amount of testing. Important is: Consensus between project management, customer sponsors, developers, technical specialists and testers. De-scope tests if the risk are too low or the cost for testing too high.Enough testing should be determined by consensus, with the tester facilitating and contributing information tot the consensus process.

5.3 When we should stop testing?Test plans include acceptance or exit criteria define a set of target to be met by a test stage: All tests planned have been run All incidents raised have been resolved All faults found have been fixed, re-tested All regression tests run without failureThere no projects who meet all criteria.Nearly always: compromises are made – known faults remain in the system as it is shipped. faults deemed acceptableIf acceptance criteria are compromised, are they useful at all? Objective, inflexible acceptance criteria are sensible guide to ensuring things are done ‘properly’. Real world: if we release now, what are the risks of releasing?There is a balance to be struck and involves comparing the benefits to be gained by releasing with the risks. Tester should be able to identify the benefits of release, because he knows what tests have been run and passed and these are the evidence that certain benefits are now available.Tests that have failed evidence for certain features do not work yet en so some risks have not been addressed.. If tests can be tracked back to some risk Report of status of that risks / coverage of that risks. Answer the questions: What are the risks of stopping testing now? What are the benefits of releasing now?Recorded risk identify risks that have been covered and those that remain outstanding.Relate test tot benefits identify benefits that are ready to be delivered.



Risk not the main issue. Benefits to delivered and the residual risks. An imbalance in favour of the benefits allows us to make a positive release decision.

5.4 When is a product ‘good enough’?‘Good Enough (Bach 1997) caused some division in the consulting community.A cop out / a compromise too far / promotes shoddy work X As something that reflects what we do in real life where something less than perfect solution is inevitableThe Good Enough Approach is helpful to understanding the risk based test approach good framework for the release decision making.Mantras: Product perfection can only be achieved through continuous process improvement All bugs are bad, all bugs could be found, so you must do more testing, more rigorous testing, using the most

approaches and techniques available. Documentation is always worthwhile You can’t manage what you can’t count.There is no guarantee that process improvement will result in perfect products. Documentation frequently fails to get the right information to right readership at the right time. The main purpose of such mantras is to help consultants to sell their services process hypochondriacs.‘Good Enough’ is a reaction to this compulsive formalism it is not reasonable to aim at zero-defects.Compromise is inevitable.Don’t get upset when estimates are cut down. Guilt and fear should not be inevitable just because a project is constrained for budget and resources and has more than zero defects remaining.Definition of ‘Good Enough’ in the context of a system to be released is:

1. It is sufficient benefits2. It has no critical problems3. Its benefits sufficiently outweigh its non-critical problems4. In the present situation, and all things considered, delaying its release to improve it further would cause

more harm than good.5. All the above must apply.

This definition means that there is already enough of this product working (this system, increment or enhancement) for us to take it into production, use it, get value, and get the benefit. ‘It has no critical problems’ means that there are no severe faults that make it unusable or unacceptable. To try to make it perfect cost more moneyFirstly have sufficient benefits been delivered Executes test must demonstrate that the features providing benefits are delivered completely, so we have evidence of this.Secondly are there any critical problems? incidents reports provide the evidence of at least the critical problemsThere should be no critical problems for it to be good enough.Thirdly is our testing good enough to support this decision? sufficient evidence to say these risks are addressed and those benefits are available for release?

Who decides?Not the tester’s responsibility to decide the product is good enough.Main players: The accuses (the system under test) The judge (project manager) The jury (the stakeholders) Expert witness (the tester)Focus on the expert witness role explain complex evidence. The testers refuse to judge. It is for others to judge whether this makes a system acceptable.Tester is there to provide information for the stakeholders to make the decision.Testers present to their management and peers an informed and independent point of view. these benefits are available, but theses risks still exist.However, if a tester you are actually asked to make the decision, what should you do? you must help the stakeholders to make the decision, but not make it for them. The risk might still exist.The judgement on outstanding risks must be as follows: There is enough test evidence now to judge that certain risks have been addressed. There is evidence that some features do not work (the feared risk has been materialized) Some risks (doubts) remain because of lack of evidence (tests have not been run, or no tests are planned)Not ideal, but is preferable to unrealistic. You may be forced to give an opinion tale principal position as an expert witness you might raise your credibility with management. Future management give you the right responsibilities.

The classic Squeeze on TestingDevelopers deliver late into test and the go-live deadline remains fixed. Acceptance criteria determine what happens next (not always). Pressure acceptance criteria are set aside / downgrade severe bugs.If the decision to release is actually made, the stakeholders have explicitly chosen to accept a product before they have evidence that all risks are addressed, and that all benefits are available.This should not cause the tester a problem stakeholders have judged they have enough information to make that decision.Information to make decision first day of test execution.How good is it.



5.5 How good is your testing?Most common definition is based on numbers of defects found. Found many , users found la few in production --. Your testing is goodDefect Detection Percentage (DDP) = T / (T + P) expressed as a percentageT = Defects found by testingP = Defects found in productionT + P = the total known defects (so far).DDP is calculable metric after a project is finished and the users have had some time to find the residual faults on a system.In midst of testing DDP is incalculable.Good testing is finding a high proportion of defect but the ultimate target is impossible to achieve.Implicitly a tester is measured on a DDP scale. May be for future process improvements. Not useful during the activity it is mean to support.Another way of calculating test effectiveness coverage measures, against code or requirements, provide a measure of thoroughness, but coverage is only an indirect measure of the quantity of test evenly distributed according to some arbitrary model.Coverage provides a quantitative assessment of the completeness of testing measured against arbitrary targets.Ultimately, coverage is a one-dimensional measure of the quantity and thoroughness of testing, not the quality.

A definition of Good TestingAlternative definition of good testing It provides sufficient evidence of the benefits delivered It provides sufficient evidence of the current risk of release It does this at an acceptance cost and in an acceptable timeframe

Identifying and assessing product risks early in the project, the performance of the testers can be assessed without reference to ‘having enough time’ or bug counts Quality of the testing can be assessed very simple, before the projects endsKnowing the status of benefits is good testingKnowing the risk of release is good testingThe definition of good testing implies that the knowledge of risks is a prerequisite to doing any testing. The measure of test quality is the value of the information provided by the testersFirst day of test run Good testers should be able to report on the current risk of release.Good testing provides sufficient information on the risks and benefits throughout the project for the stakeholders to make decisions on how much testing is enough and to make the release decision in a balanced way. The testing is good if the stakeholders judge that: The testers have identified all of the risk of significance The tests planned address the documented risk The tests executed provide the stakeholders with enough information to be confident that the risks have been

addressed and the benefits are available for release.

Seems simple. Problem with this evaluation both qualitative and subjective. Getting the stakeholders to judge seems reasonable compared with selecting an arbitrary coverage measure.The reason we test to address the perceived risk of stakeholders and only they can judge whether the testing has achieved this purpose.

6 Measuring Software QualityRob Hendriks, Erik van Veenendaal and Robert van Vonderen

Quality requirements of software products are often described in vague and general terms difficult for test engineers to evaluate the quality of the software as no unambiguous and quantitative reference exist.This chapter a method to identify the most important quality characteristics by means of a risk assessment, to define completion criteria and to subsequently measure software characteristic using the ISO 9126 standard.

6.1 Project contextExample: The required quality level of the controller software was not specified. Measurable: Number of copies between failure & Average copies per repair. First goal: to find a base line. Océ

6.2 Quality model: ISO 9126McGall 1977: Breaking down the concept of quality into a number of quality factors. Many did so afterwards.Some elementary characteristics keep on reappearing International Organization for Standardisation (ISO)International Electrotechnical Commission (IEC) standard set of quality characteristicsISO 9126 six quality characteristics and the subdivision of each quality characteristics. Six quality characteristics: Functionality



Reliability Usability Efficiency Maintainability Portability

6.3 Selection of quality characteristicsNot all quality characteristics are of equal importance to a software product. Most important quality (sub-) characteristics need to be identified and selected by means of a risk assessment. interviewing stakeholders (project manager, software architect, Outside: various users)Interpretation of quality characteristics hard. It is difficult to communicate regarding these quality characteristics in an unambiguous way. Quality characteristics are usually not part of the stakeholders’ terminology.

Structured questionnaireQuestionnaire has been developed in ESPRIT-project SPACE-UFO. Methods relates the wishes and the needs of the stakeholders regarding to software quality characteristics. The stakeholders’ needs and requirements are explored and identified by using a structured questionnaire. so-called generic product characteristics intended to be stated in the stakeholders’ language. Usability X number of different users, their experience and their educational level. Answers given are used to deduct the most relevant quality (sub-) characteristics. See table 6.1

Quality profileMake a list of important quality characteristics. Include importance levels (from level D – level A) Quality ProfileQuality Profile: reflects the notion of quality for a certain software product and makes quality more tangible for all stakeholders (including testing). Initial Quality Profile meeting: to discuss the quality profile with the stakeholders. Result: a focus on the chosen quality characteristics primarily and to develop a set of completion criteria (metrics) for these quality characteristics.Example:

Quality Level- D C B A

Functionality X

Reliability XUsability XEfficiency XMaintainability XPortability X

6.4 Identification of quality metricsOnce the quality profile has been determined, it is time to start focusing on how to evaluate and measure the relevant quality (sub-) characteristics. ISO 9126 part 2: External metrics and Part 3: Internal metrics a large number of possible metrics per characteristic, including measurement scales and application guidelines.A selection of metrics is needed. Océ metrics possible and easy to measure (pilot project)Most of the metrics directly from the ISO-standard, some of them had to be fine-tuned to the project’s definitions. Mean Time Between Failure (MTBF) became MCBF A light change of one metric. Only controller Software was taken in account (no paper jams)Sixteen metrics were defined one was additional add to the ISO 9126 standardAvailability of documentation was a good indication for maintainability.

6.5 Defining completing criteriaCompletion criteria defined before starting actual measurement. Measured quality X completion criteria.Defining Completion criteria in advance forces people to start discussion when quality targets are not met.Completion criteria experienced engineers were asked to estimate on each metrics. Results were compared and discussed.Measure all criteria during the system test phase of each incremental development cycle.Collection of measurement data thorough administration was necessary during test execution.

6.6 Real –life measurementsFinding of software functional defects in relation with the large amounts of copies.Functionality, duration and performance

6.7 Release decisionProject during 2 years lot of experiences was gained on the metrics themselves. How to interpret them.Metrics were initially only applies to provide visibility on the software product quality. The next step was to submit changes in the development process to improve quality of the software itself and the process.Tension between time-to-market and product quality.Metrics by using this method major input for management tot support the release decisions. They allow a thorough discussion with all stakeholders on product quality and the identification of outstanding risks.



7 Test Point Analysis: a method for test estimationErik van Veenendaal and Ton Dekkers

TPA: objectively prepare an estimate for a system test or acceptance test. TPA only covers high level testingFPA low level testingTPA also be used to predetermine the test hour budget.TPA determine the relative importance of the various functions (using available testing time as efficiently as possible)

7.1 PhilosophyEstimate high-level test three elements are relevant: The size of the software product to be tested The test strategy (selection of system units and quality characteristics to be tested and the coverage of testing) Level of productivityFirst two elements together determine the volume of testing to be carried out (expressed in test points). Number of test point is multiplied by the productivity (the amount of time needed to perform a given volume of testing work) then the result is a test estimate in hours.

SizeSize of a software product is determined mainly by the number of function points. Derived from the FPA. Standard by the Int. Func. Point User Group (IFPUG , 1999) standard Huge difference between estimates from TPA and FPA is that in TPA the risk-based test strategy is taken in account. In FPA, high-level tests are included by increasing the productivity factor (testing takes x% of the development effort, regardless of the risks involved). FPA productivity factor covers the component and integration test levels; it lacks sufficient coverage of system and acceptance testing.In FPA, there some factors missing (Additions and adjustments need to be made). These are: Complexity: complexity relates to the number of conditions in a function. More conditions almost always means

more test cases and therefore a greater volume of testing work; Interfacing: the degree of interfacing of a function is determined by the number of data sets maintained by a

function and the number of other functions, which make use of those data sets. Interfacing is relevant because these ‘other’ functions will require testing if the modifying function is changed;

Uniformity: it is important to consider the extent to which the structure of a function allows to be tested using existing or slightly modified specifications, i.e. the extent to which the software product contains similarly structured functions.

Test strategyDuring development or maintenance, quality requirements will have been specified for the software product. The test activities are focused on determining the extent to which these requirements have been satisfied. In co-operation of the stakeholders, the system and/or subsystem quality characteristics to be tested are identified and their relative importance determined. The importance of each characteristic influences the thoroughness of the related test activities. The more important a quality characteristic, the more accurate and thorough the test have to be, and the larger the volume of work.The importance of the various characteristics should be determined in consultation with the stakeholders when the test strategy is being formulated; the information can then be used as TPA input. volume of testing work is calculated on the basis of the test strategy.Difference between the various functions. For each function, therefore, there are two (subjective) factors that influences the thoroughness of testing: User-importance of the function Usage-intensityThe test strategy specifies which quality characteristics are to be tested for each subsystem or function, and the relevant degree of coverage. TPA and strategy determination are closely related and in practice are often performed in parallel.

ProductivityProductivity is not a new concept to anyone who has produced estimates on the basis of function points. FPA productivity is an expression of the relationship between the number of hours necessary for a task and the measured number of function points. In TPA productivity relates to the time necessary to realize one test point, as determined by the size of the software product and the test strategy. Productivity has two components:Productivity number is based on the knowledge and skill of the test team and is therefore specific to the individual organization or projectEnvironmental factor indicates the degree to which the environment influences the test activities to which the productivity is related. Influential environmental considerations include the availability of test tools, the level of experience the team has with the test environment, the quality of the test basis and the availability of testware.

7.2 Basic procedureFigure 7.1

The number of test points necessary for dynamic testing is calculated for each function on the basis of the number of function points assigned to the function, the function-dependent factors (complexity, interfacing, uniformity, user importance and usage-intensity) and the quality requirements that will be tested dynamically. The sum of the test



points assigned to the individual functions is the number of ‘dynamic test points’. The number of test points necessary for static testing is calculated on the basis of the total number of function points for the software products and the quality requirements that will be tested statically. This gives the number of static test points.The total number of test points is the sum of the dynamic test and static test points. The number of primary test hours can then be calculated by multiplying the total number of test points by the calculated environment factor and the applicable productivity factor. The primary test hours represents the volume of the work involved in the primary testing activities, i.e. time required for completion of the test phases preparation, specification, execution and completion (chapter 2 TMAP life cycle)Finally the total number of test hours is obtained by adding an allowance for secondary test activities (test management, test planning and control) to the primary number of test hours. The size of the allowance, which represents the volume of work involved in the management activities, depends on the size of the test team and the availability of management tools. The total number of test hours is an estimate of the time required for all test activities, excluding the establishment of the test plan.

7.3 PrinciplesRegarding tot TPA, the following principles apply: TPA is concerned only with the measurable quality characteristics (according ISO 9126, 2001), which fall within

the scope of system testing and/or acceptance testing. A characteristic is considered ‘measurable’ if an appropriate test technique is available.

Consequence of the first principle, using TPA in its present form, it is not possible to allow for all the quality characteristics that might be addressed by system and/or acceptance testing. No allowance for no pre-defined test technique (yet) exist.

TPA is in principle analyst-independent two different people the same outcome. TPA depends on the availability of function point counts produced using IFPUG (gross function point count is

used as basis for TPA) For TPA purposes the best team’s domain knowledge is not treated as a variable. TPA estimates are made on the assumption that, on average, one complete re-test will be conducted.

7.4 The technique in detailInput and starting conditionsFunction design, consisting a detailed process description and a logical data model, preferably including CRUD-table. In addition a FP count made using IFPUG technique is necessary (or count dialect). Different methods should not be used. Choice of FP counting does not affect TP calculation, but it can influence the productivity factor.For TPA purposes, the function point count is amended as follows:The function points for various (logical) data sets, defined within the function point count, are assigned to the function(s) which provide(s) the input for those (logical) data sets. The function points for the various external data sets, defined within the function point count, are assigned to the

function(s) which use(s) those external data sets. The number of function points for a clone-class FPA function is the same as the number of points assigned to the

relevant original FPA function. A clone is a FPA function, which has already been specified and/or realized within the same or another user function in the same project.

The number of function points for a dummy-class FPA function is, if possible, calculated: otherwise, such functions are treated as being of average complexity and the corresponding number of function points are assigned. A dummy is a FPA function whose functionality does not need to be specified and/or realized, but which is nevertheless available because specification and/or realization has been undertaken outside the project.

If no FPA is available, the time to carry out the count can be determined as follows.The number of logical data sets is counted and multiplied by: 28 in case of a system with low data complexity (less than 10 LDS) 35 in case of a system with average data complexity (between 10 and 25 LDS) 42 in case of a system with high data complexity (more than 25 LDS)This is a very rough estimation of FPA. Approximation is divided by 400 to obtain the number of days necessary for the count

Dynamic test pointsNumber of dynamic test points is the sum of test points assigned for individual functions (individual functions, influential variables and factors) two categories:Function-dependent (Df) Quality requirements relating to the quality characteristics to be tested dynamically (Df)FPA function = unit of function –p-> user-importance and user-intensity is based largely on the function

Function-dependent (Df)User importance: 25% HIGH, 50% NORMAL, 25% LOWRating3 Low: the importance op the function relative to the other functions is low6 Normal12 High

Usage-intensityFrequency with which a certain function is executed by the users and the size of the user group that uses the function



Rating2 Low: the function is only used a few times per day or per week4 Normal: the function is used a great times per day 8 High: the function is only used continuously throughout the day

InterfacingAn expression of the extent to which a modification in a given function affects other parts of the system degree of interfacing determined by ascertaining first the logical data sets (LDS)Interface rating

LDS \ functions 1 2-5 > 51 L L A2 - 5 L A H> 5 A H H

L = Low interfacing etc.If a function does not modify any LDS’s, it is given a low interface rating. A CRUD table is very useful for determining the degree of interfacing.Rating2 The degree of interfacing associated with the function is low4 Normal8 High

ComplexityFunction is determined on the basis of its logical algorithm pseudo-code, Nassi-Schneidermann or ordinary test.The complexity rating of the function depends on the number of conditions in the function’s algorithm.

Compound conditions IF A and B , Then count twice. Case statement n-1 conditions Count the single conditions not the operators3 The function contains more than five conditions6 The function contains between 6 – 11 conditions12 The function contains more than 11 conditions

UniformityUnder the following circumstances, only 60% of the test points assigned to the function under analysis are actually counted: In the case of a second occurrence of an almost similar function: the test specification can be largely reused. In the case of a clone function: the test specification can be reused for clone functions. In the case of a dummy function (provided that reusable test specification have already been drawn up for the

dummy).A uniformity of 0.6 assigned in the circumstances described above, otherwise a uniformity factor of 1 is assigned. Software product contain functions that posses a degree of uniformity for test purposes.Unique = : A function which uses a combination of data sets which is not used by any other FPA function A function, which although it does not use a unique combination of data sets, does not use a unique processing

technique (e.g. a unique method of updating a data set).Even functions completely unique count them in TPA they have tot be tested.

Method of calculationDf factor is calculated by adding together the ratings of the first four function-dependent variables (User-importance, usage-intensity, interfacing, and complexity) and dividing the sum by twenty (the nominal rating). The result then should then be multiplied by the uniformity factor. Df factor is calculated for each function.

Df = (Ue + Uy + I + C) / 20) * U

Df = weighting factor for function-dependent factorsUe = User-importanceUy = Usage-intensityI = InterfacingC = complexityU = Uniformity

Standard functionsIncluding error messages, help-screen and/or menu structure. Standard number of test points:Function Fp’s Ue Uy I C U Df

Error message 4 6 8 4 3 1 1,05Help screens 4 6 8 4 3 1 1,05Menu structure 4 6 8 4 3 1 1,05

Dynamic quality characteristics (Qd)Related with ISO 9126-1 2001



A distinction is made between quality characteristics that will be tested by means of executing dedicated test cases (dynamic explicit testing common way) and those that will be tested otherwise.Looking for trends estimate the number of failures that are likely to occur during operation implicit testing.TPA, four quality characteristics (explicitly and dynamically tested): Suitability Security Usability regarding usability no distinction has (yet) been made in sub characteristics, since there are no usability

testing techniques available that have this level of accuracy Efficiently for the same reason as usability; efficiency is not split up into time-behaviour and resource-utilizationImportance of the requirements relating to each quality characteristic is related / done to each subsystemRating0 Quality requirements are not important and therefore disregarded for test purposes.3 Quality requirements are relatively unimportant but do need to be taken into consideration for test purposes.4 Quality requirements are of normal importance (Software related to support process).5 Quality requirements are very important (Rating for software related to a primary process).6 Quality requirements are extremely important

Dynamic, explicitly measurable quality characteristicsCharacteristic / rating 0 3 4 5 6Suitability (weighting 0.75)Security (weighting 0.05)Usability (weighting 0.10)Efficiently (weighting 0.10)

Use of techniqueRating 3 4 5 6SuitabilityProcessingScreen checks

EPError Guessing

Error Guessing

EP

Sample SEM and Error Guessing

EP and ECT

Sample SEM and SYN

ECT

SEM and SampleSYN

Security Error Guessing SEMSample user profiles

SEMUser profiles

SEM user profiles and overall system

Usability Error Guessing Use Case or PCT and SUMII

Use Case or PCT and SUMII

Usability laboratory test

Efficiently The thoroughness of the RLT is varaiable

ECT = Elementary Comparative Test = more thorough than EP (Equivalence Partition)

Method of calculation (Qd)The rating for each dynamic, explicitly tested quality characteristic is devided by four (nominal rating), then multiplied by the weighting factor. Implicit: 0.02

Dynamic test point formulaTPf = FPf * Df * Qd

TPf = number of test points assigned to a functionFPf = number of function points assigned to a functionDf = weighting factor for the function-dependent factorsQd = weighting factor for the dynamic quality characteristics

Static test pointsDepends on: FP count and influenced by the requirements regarding the quality characteristics the Qs factor ChecklistFor each quality characteristic Qs gets factor 16. For each subsequent quality characteristics added Qs (value 16)

TOTAL NUMBER OF TEST POINTSTP = Pf + (FP * Qs) / 500

TP = total number of test pointsPf = sum of test points(dynamic test points)FP total number of function pointsQs = weighting factor for the quality characteristics

Primary test hoursPrimary number of test points is multiplied by the productivity factor and the environmental factor to obtain the primary test hour count

Productivity FactorThe hours required per test point depends on experience, knowledge. Value between 0.7 and 2.0

Environmental factorTest tools



Rating1 During testing, supporting tools are used for test specification and test execution2 During testing, supporting tools are used for test specification or test execution4 No test tools are available

Development testingRating2 A plan for the preceding tests is available and the test team is familiar with test cases and test results4 A plan for the preceding tests is available8 No plan for the preceding tests is available

Test basisQuality of the documentationRating3 During the system development doc. Standards and templates being used. In addition: inspections are

organized6 During the system development doc. Standards and templates being used.12 System documentation was not developed using a specific standard and template.

Development environmentRating2 4 GL programming language and integrated DBMS containing numerous constrains4 4 GL programming language possible with .3 GL8 Only 3 GL language (pascal, cobol).

Test environmentRating1 Environment has been used for testing several times in the past2 Newly equipped environment similar to well-used environment within the organization4 Newly equipped environment experimental within the organization

TestwareRating1 A usable, generic initial data set and specified test cases are available for the test2 A usable, generic initial data set is available for the test4 No usable testware is available for the test

Primary test hours formula

PT = TP * P * EPT = number of primary test hoursTP = total number of test points assigned to the system as a wholeP = productivity factorE = Environmental factor.

Total number of test hoursNumber of primary test hours is raised (for planning and control) with Standard 10%. Depends on Team size and Management tools

Team sizeRating3 Team consists of no more than four people6 Team consists of between 5-10 people12 Team consists of more than ten people

Management toolsRating3 Both a time tracking and a defect management tool are available and used6 Either a time tracking or a defect management tool is available12 No supporting management tools are available

Method of calculationPlanning and Control percentage is obtained by adding together the ratings for the two influencial factors. Allowance in hours primary test hours multiplying with percentage en adding it to primary test hours.

Breakdown over phases Preparation 10 percent Specification 40 percent Execution 45 percent



Completion 5 percent

7.5 Test Point Analysis at an early stageIt keeps rough in the beginning. Not possible to determine factor as complexity, interfacing till detailed functional specification are obtained.Rough TPA assumptions



PART 3 REVIEWS

8 Formal review typesErik van Veenendaal and Mark van der ZwanInspection little room for discussion and detailed questionsTechnical Reviews and walkthroughs room for discussion and detailed questions

8.1 Document reviewsFormal reviewsThere are number of ways to improve a document. By Formal reviews a significant increase in productivity and product quality can be achieved (Gibbs & Graham 1993).Reducing the number of defects in the final product means that less time has to be spent on testing and maintenance.Inspection is most documented. There are more techniques 3 types of formal review (IEEE 1028, 1998).InspectionFormally defined and rigorously followed review process. Includes individual and group checking, using sources and standards, according to detailed and specific rules or checklists in order to support the author by finding as many defects as possible in the available amount of timeTechnical reviewObjective to reach consensus on technical, content related issues. Domain or technical experts check the document under review, prior to the meeting, based on specific questions of the author. In this meeting, the approach to be taken is discussed by experts, under guidance of a moderator or technical leader.WalkthroughThe author guides a group of people through a document and his/hers thought process, in order tot gather information and to reach consensus. People from outside software disciple can participate. In walkthroughs, dry runs and task scenarios are often applied.

Different review types different focus applicable at a different life cycle phase.Aim of all three improve the quality of the document / to find defects. Types of defects differ for the 3 review types. Using the right type of review at the right place in the software life cycle ensures a more effective and efficient review process.

IEEE 1028 other types of formal reviews:Management reviewA systematic evaluation of a project performed by / or on behalf of management in which the project status is evaluated against the plan (resources, costs, progress, quality) and appropriate actions are defined.AuditIndependent examination of a product or process against an agreed reference base. Is based on a problem definition and/or set of questions and carried about from a certain viewpoint (engineering, management, customer)Other focus and will not be discussed further

Informal reviewsMost common, various time in the life cycle. The goal: to help the author and to improve the quality of the document.They have value for an organisation. Other differences with formal reviews: No guidance or help on what and how to check the document-under –review Reviewers do not use different viewpoints to prepare same defects found by multiple reviewers (= negative on

efficiency) No limitations are set on number of pages to be reviewed many pages reviewed but a negative effect on

efficiency Some informal reviews do not have meetings comment simply sent to the author Meeting often over 2 hours with a lot of discussion on technical approaches influences both the efficiency

and effectiveness of the process No process data (effort, throughput time, results) no data stored benefits can not been shownReviews are a prescribed part of the development process in most organisations both formal and informal reviews With document strategy will improve efficiency and effectiveness of the review and development process.

8.2 Participants, roles and responsibilitiesParticipants of any type of formal review should have adequate knowledge of the review process. Important factors for participants Advantage for their own work during the process & properly trained

Five types of participants

ModeratorLeads the review process. He determines – with the author – the strategy and composition of the review team.He performs the entry check and the follow-up on the rework, in order to control the quality of the input and the output of the review processSchedule meeting, disseminate materials before the meeting, coaches other team members, paces the meeting, leads the possible discussions and stores the data that is collected on the process form



AuthorBasic goal: should to be learn as much as possible with regards to improving quality of the document to improve ability to write future documents. To illuminate unclear areas and to understand defects found

ScribeDuring the logging meeting the scribe has to record each defect mentioned and any suggestions for improvement, on the logging form. Usually author’s task. Write in own words

ReviewersTo check any material for defects, prior to the meeting. Level of thoroughness required depends on the type of review. Level of domain knowledge by reviewers also important.Documents: source documents, standards, checklist. The less documents more domain expertise is needed.

TraineeAdditional role gets a detailed look inside the organisation’s development process. Trainee could check whether the document-under-review complies with the standards.

8.3 InspectionsBackground, definition, goal and timingThe document-under inspection prepared and check thoroughly by inspectors. In Inspection meeting defects found are logged, any discussion is postponed until discussion phase efficient.Experience team / moderator will log 1 or 2 defects per minute Reason for carrying out inspections can be explained by Weinberg’s concept of egoless engineering self-justify actions.Inspection definition: Inspection is a structured review of an engineers’ work product against sources and other documents, carried out by his colleagues to find defects and to enable the engineer to improve the quality of his product (Fagan, 1968)Inspections can serve a number of goals efficiency (time to market), effectiveness (safety critical).Generally accepted goals of inspection are: Help the author to improve the quality of the document-under-inspection. Remove defects efficiently, as early as possible Improve product quality, by producing documents with a higher level of quality Create a common understanding by exchanging information among the inspection participants Training new employees on the organisations development process Learn from defects and improve processes in order to prevent reoccurrence of defects.When should an inspection be held?In general: should be held before a document transfers from one phase to another. Inspected and reworked document can be subjected to formal approvalInspection can be organised in all phases of the development process from requirements up to coding when the author thinks the document is ready.

Inspection procedure consists of seven steps

1. Initiation & planningBegin: request for inspection by the author to the moderator. Each document to be inspected moderator should take care of scheduling (dates, time, place and invitation).Moderator always performs an entry check document submitted is inspection-ready entry criteria. Too many mistakes de-motivate both reviewers and the author. Many minor defects conceal the major defects.

Some entry criteria: A short check does not reveal a large number of major defects The document to be inspected is printed with line numbers Spelling check performed References needed for inspection are stable and available The document author is prepared to join the inspection team and feels confident with the quality of the

documentAfter entry check author and moderator decide which part will be inspected Maximum 10-20 pagesComposition of the inspection team by the moderator with help of the author normally 4 – 6 person (incl. Moderator and author) different roles are assigned (focus on particular types of defects reduce change of finding the same defects by different reviewers. The moderator assigns the roles of the reviewers.Following review roles: Focus on higher documents does the design comply to the requirements Focus on standards internal consistency, clarity, naming conventions, templates Focus on related documents at the same level interfaces between software functions Focus on use feasibility, maintainability and/or testabilityAuthor: additional specific roles. Moderator can also add a role beside his task as inspection leaderModerator type 2 role (compliance to standard) more objective

2. Kick OffOptional step Goal: to get everybody on the same wavelength and to commit to the time. Recommended because of the effectiveness of the inspection Short introduction on the objectives and documents. Relations between the different documents are explained.



Further: role assignments, checking rate, the pages to be checked, process changes, questions.Rate of finding defects after a kick-off meeting is higher

3 PreparationParticipants work individually on the document-under-inspection using related documents, procedures, rules and checklists provided. Identify defects according to their role. All defects are recorded on a logging form. Spelling mistakes recorded but not mentioned during the meetingChecking rate: 5 –10 pages pr hour. Document size is important.

4 Inspection meetingThree sequential parts: Logging phase Discussion phase Decision phaseAd 1 Individual defects are logged by the author (page-by-page, reviewer by reviewer). No real discussions (efficiency). Discussion needed postponed till the discussion phase.Every defect and severity should be logged. Severity classes are: Critical: downstream damage the scope and impact of the defect is beyond the document-under-inspection Major: could cause a downstream effect fault in design could result in an error in implementation) Minor: not likely to cause downstream damage non-compliance with standards and templatesSpelling remarks: not part of the defect classification

Focus: logging as many defects as possible within a certain timeframeAd 2 DiscussionBringing forward their arguments; Moderator: prevents discussion getting too personal, prevents heated discussions, rephrases remarksReviewers are allowed to leave (not their subject). Moderator: time box, note action points (if could not be solved). Outcome must me written down for future referenceAd 3 DecisionDecision based on exit criteria average number of critical and major defects found per page (no more than 3 criticals/major per page). If number of defects exceeds a certain level document must be inspected againDocument complies rework check by the moderator and one or more participants.Time pressure quantified exit level criteria may help the moderator to make firm decisions at all times.Possible options for decision are: The document fails the exit criteria which means: a new inspection of the updated document must be scheduled The document passes the exit criteria and the follow up will carried by either the:

o Moderator checks updated documento Participants check updated document to determine whether their commands are addressed

5 Causal analysisUsed to immediately analyse defects, in order to find common causes and to define possible actions for improvement leads to bottom up and practical, process improvementPareto analysis meeting on a regular basis:1. Select 3-5 critical defects2. Categorise each defect to focus the analysis. Categories: organisation, people, architectural, technology and

process3. Every participant provides a number of possible causes in a brainstorm manner. One or two possible causes

are selected4. Every participant provides a number of possible cures, again brainstorm manner. Cures are prioritised.5. Improvement actions are defined by the end of the meeting, based on the cures mentioned6. The improvement actions are communicated to the QA-department or a similar group within the organisation Causal analysis: immediate improvements, based on inspection cycle from detection and removal to prevention

6 ReworkBased on the defects logged, the author will improve the document-under-inspection step by step. Also other improvements – not based on inspection’s defect log. Not every defect is found leads to rework Write track changes easy to identify changes

7 Follow up and exitThe moderator is responsible for ensuring that satisfactory actions have been taken on all logged defects, improvement suggestions and change requestsModerator checks (not in detail). All participants check moderator responsible for distributionNumber of measurements to control and optimise the inspection process

8.4 Technical reviewsBackground, definition, goal and timingTR: discussion meeting that focuses on achieving consensus on the technical content of a document.Compared with inspections: less formal, little/no focus on defect identification, intended readership and rules.Defects found by experts. Needed experts: architects, chief designers and key-users. TR = peer reviewTR definition: A TR is typically a peer group discussion activity that focuses on achieving consensus on the technical approach to be taken. A technical review is also known as a peer review.Goals:



Asses the value of technical concepts and alternatives in the product and project environment Establish consistency in the use and representation of technical concepts Ensure, at an early stage, that technical concepts are used correctly Inform participants of the technical content of the document

When should a technical review be held? early stages in the life cycle of a document whenever issues arise during specification, design or coding that need to be discussedFocus on: approach taken, technical feasibility.Inspections and TR are complementary. TR can be done on a 70% complete document.The TR procedure consists of five steps:1. Inspection and planning2. Kick-off3. Preparation4. Technical review meeting5. Rework, follow-up and exit

Ad 1 Inspection and planningModerator should be assigned for each document to be reviewed. Moderator: organisation and schedulingModerator (and author): specific questions that need to be answered.No formal entry-check possible 70% complete documentModerator and author will select participants domain / technical knowledge (only experts) 5 –6 persons (incl. M & A)Project leader could be one of the reviewers (Planning, expert knowledge)Recommendation: Technical leader as moderator sufficient knowledgeTR: no strict criteria for checking rate and document size. Appropriate upper limits should be determined collecting metrics

Ad 2 Kick off Optional, highly recommended same wavelength and to commit to time. Instructions skip sections that are not yet ready. Different viewpoints

Ad 3 Preparation: Participants work individually technical skills and domain knowledgeFocus: specific questions, as many defects as possible critical and major defects. Inspection on the document itself or relate to the content and approach in the document

Ad 4 Technical review meeting: Two sequential PartsDiscussion and logging phaseDecision phaseAd 1: TR no distinction between logging and discussion. Discussion allowed Questions raised need to be answered. Additional defects identified during individual preparation are discussed (defect by defect, page by page)Only the more severe defects (critical or major) are discussed and logged (before: classification). No sense in logging, nor mentioning of minor defects.Participants bringing forward their arguments. Moderator keep meeting focussed on providing answers on the questions & find as many as possible defects. Moderator, authors & reviewers will decide if a defect needs further discussion in or outside the meetingEnd moderator final check. All initial questions raised by the author have been answered with the expected level of detail

Ad 2 DecisionDecision have to be made by the participants. Focus: how improve the document.Logged defects = list of action items decision must be made. No strict exit criteria. Decision is less formal and more expert based than the decision in an inspection process.Three possible outcomes:1. The updated document will re-enter a new TR, which must be scheduled2. The updated document will be checked by the moderator (tech. Leader)3. The updated document will be inspected.

Ad5. Rework, follow-up and exit Based on action items author will improve the document-under-review step by step. Also improvements and corrections which were not logged.Not as formal as with inspections Moderator checks (new review needed procedure starts all over again)In general: no change request on other documents. Possible improvement suggestions templates

8.5 WalkthroughsAuthor of the document-under-review (d-u-r) guiding the participants through the document and thought process to achieve common understanding.Definition: A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content (Freedman & Weinberg, 1990).Author: most of the preparationParticipants: different departments and backgrounds, not required detailed study in advance .



Large number of people number of viewpointsIn general the goals of a walkthrough are: To present the document to stakeholders both within and outside the software discipline, in order to gather

information regarding the topic under documentation To explain and evaluate the contents of the document (knowledge transfer) To establish a common understanding of the document content To examine and discuss the validity of proposed solutions and the viability of alternatives: establishing consensus

When should a walkthrough be held?Most useful during two particular stages of a document:early stage of its development (working stage)or just before the document is to be released (draft document)Walkthrough: 4 hours – whole day entire document must be explained and evaluated

Working documentExample higher level document in early development stage Software Requirements Specifications (SRS). Input: customer requirement specification, software prototypes, customer visit reports, and any other form of documentation / information.Goal: to get feedback and discussion on the documented interpretation of these inputs.Part of the validation process information is gathered in order to establish if the right product is being developed

Draft documentFocus: common understanding of the presented document, which contains the solution to the earlier defined opportunity or problem. Is like a presentation, focus on achieving consensus.Invited: 2 – 7 project members (incl. Members outside software discipline).No study in advance; reading without understanding every detail is sufficientFour steps of a walkthrough:1. Initiation & planning2. Preparation3. Walkthrough meeting4. Rework, follow-up and exit

Ad 1 Initiation & planningStart: ‘Request for walkthrough’ by the author to the moderator.No special entry check (like by inspection).Author (and Moderator ) identify special topics that need clarification / select participants based on their domain knowledge and background. Moderator distributes materialsAuthor guiding / moderator chairing scribe is needed: record each defect discussed, any suggestions and/or alternative approaches

Ad 2 PreparationNot extensive participants need to familiarise themselves with the d-u-r Reviewers: Read document and prepare a list of questions and defects recommendation to improve the document

Ad 3 Walkthrough meetingDivided in two sequential steps:1. The Kick-off2. The WalkthroughKick-offModerator states the objectives of the walkthrough; author: presents overview of the document. Participant general remarks and ask questions. Agenda is determined and agreed on The walkthroughThe author walks through the document page by page members can ask questions, raise issues, recommend improvements. Author step by step Focus on most important parts Task scenarios and dry runs could illustrate the documentOutput meeting: list of recommendations and action items with respect to the d-u-r

4. Rework, follow-up and exitBased on action items, recommendations and defects logged the author will improve the d-u-r step by step (also improvements and corrections not based on defects logged)Moderator checks whether the action items are closed, using the updated document.Working document most likely to be reviewed as TR in a later stageMore final document new defects changed requests. Moderator has to check whether these change request are recorded.Document updated, action items are closed, change requests are documented doc may exit the walkthrough process

8.6 Combing review typesReviews are time consuming activity that must be applied with care.Testing is setting priorities strategy is needed to ensure the most feasible and applicable document coverage review strategy should aim at optimising the total amount of undetected defects. Also here risk assessment and the probability and impacts of defects, both detected and undetected during review, needs to be assessed.



An important distinction: product and project documents. Thorough discussion on the content of the project documents is often more effective than whether a plan is written in accordance to rules and standards.Focus on higher level documents.Informal review – walkthrough - (Informal review) – TR Inspection – walkthrough they are complementary to each other.

Comparing the review typesWalkthrough Technical review Inspection

Objectives Gather information, evaluate content, establish consensus, forum for learning

Discuss technical (alternative) concepts, establish consistency, inform others

Find defects, verify product quality, exchange information

Phase Start of – and released document

70% document Almost finished document

Participants Peers, technical leaders and participants from outside the software discipline

Experts (technical leaders), peers and sometimes project leaders

Peers

Preparation Not extensive. Defects are found during the meeting. Roles are not needed

Roles based on specific question of the author. Not too many pages

Formal preparation using roles, based on rules, at a slow checking rate

Meeting Presentation like, discussion. Author is the leader

Discussion. The technical leader (or alternatively the moderator) is the meeting leader

Formal, focus on logging, no discussion. Moderator is the leader

Logging Separate scribe Author Author

Follow-up Moderator checks if action items are closed

Expert-based decision regarding the follow-up

Formal check by the moderator and/or the participants

Materials needed Document-under-review Document-under-review Document-under-review, source documents, standards, rules & checklists

Process leader Moderator Technical leader will act as moderator

Moderator

Team size 5 – 8 4 – 6 2 – 6

Different responsibilitiesThe reviews described are applied to help the authorModerator: responsible for ensuring that these goals are met (Co-ordinate and controls the review process to achieve efficient and effective document quality)Approval: Management / allows to be used as a basis for the next development stepRecommendation: results of formal review as a basis for approval decision (risk of approving document are made visible by the review process)

8.7 Success factors of implementationImplementation formal reviews is not easy. How do you start formal reviews?: Find a champion will lead the process on a project or organisational level needs expertise, enthusiasm and

practical mindset in order to guide moderators and participants. Authority should be clear in the entire organisation.

Pick thing that really count most important documents (highly critical / upstream documents) Explicitly plan and track formal review activities make hours visible in project plan (also hours for

preparation and rework) Follow the rules but keep it simple in the beginning formal afterwards: modify them Respect optimum checking rates to high: too many minor but critical are not found (be ware of that) Collect data and provide feedback no data process will vanish Data and feedback are essential Continuously improve process and tools checklist, ideas of participants Use a thorough entry check only documents that are ready Reports results report quantified results and benefits incl. Consequences if they had not been found this early Just do it experience is needed to execute correctly. Practise with experienced people.

9 Reading techniquesForest Shull & Victor Basili



Remark: Too little attention has been paid to the development of effective procedures for the individual defect detection phase of a inspectionReviewers: often on their own (own implicit procedure and experience)Focus: Reviewers on the perspective of a downstream user of a document being reviewed, reading techniques represent an improvement over ad hoc approaches

9.1 Current practicesCommon practice software developers rely on ad hoc techniques for individual review. Ad hoc = nonsystematic, nonspecific, nondistinct.Nonsystematic = not set procedure or rules to follow / use whatever processes or tools seem useful. No set procedure difficult to provide training to inexperienced inspectors / difficult to learn. Also difficult to improve a process which is informally defined.Non-specific = inspectors are required to find defects of all types. a lack of a specific gaol makes defect detection process harder / looking for many different thingsNondistinct = inspectors have the same goal and responsibilities duplication / same things. Distinct responsibilities different inspectors become ‘experts’.

Inspectors using rules and checklists improvement over ad hoc approaches more specific and systematic. Providing focus distinct roles of responsibilities = what. “How’ is still be missingReview process explicit reading techniques allow inspections to be adapted over time to better meet the needs of the organization extra attention for defects consistently missed or less for that a type never lead to defects

9.2 A reading procedure templateEach reading technique captures a different perspective on the information all to gather the family of techniques cover all perspectives of interest Each inspector a particular technique from a particular perspective. Team members have unique responsibilities.Use of perspectives belief: correctness of software artefacts must be evaluated not to a global / static definition of quality, but according to how such documents will be used during software lifecycle (in order to be used by some stakeholder at some later stage of software development) greater confidenceNeeds of stakeholder not met deficiency in the quality of the requirements / negative impact (requirements defect).Perspectives: to minimize the overlapping responsibilities among inspectors, while achieving a high level of coverage of defectsAd hoc: reviewers inspect whole document typically focus on some subset (their own experience)Idea behind reading techniques: that each reading technique provides the reviewer with a process for reviewing the document from a certain perspective, and thus focuses the reviewer on a certain subset of defects.Perspectives are chosen with the goal that all possible defects of interest are the target of at least one technique, with minimal overlap between them.Following perspectives were found to be useful:Perspective of designer who uses the requirements as the basis for the system designPerspective of tester who uses the requirements to determine whether the system exhibits proper behavior for each test case.Perspective of designer who has certain needs for the system functionality, which are described in requirements.Each reviewer assigned only one perspective from which to inspect the document (= distinct responsibilities) and focus on the particular defects relevant tot that perspective (specific focus). Target: union of inspectors covers all defects with a minimum of overlap.Inspectors receive guidance for staying within their perspective by means of an operational procedure, a set of process steps, or guidelines that keep them focused on just the information in the artefact that is important for their perspective.First: identifying an abstraction of the document / a way of modelling only the information in the document that is important for the perspective What information should be important & How can that information be used to detect defects necessaryAbstractions of Information: initial procedures for identifying information in this document what is relevant)InfluencesUses of Information: initial procedures for using information to accomplish the taskAre combined in:Reading technique: Tested practices for accomplishing a particular task

Abstraction procedure how the abstraction should be built traversal algorithm that instructs the reader how to read through the document and find information necessary for building the abstraction. Creating abstraction procedure usually involves determining what are the relevant organizing structures for the given document from the reviewer’s perspective. concentrate on just relevant pieces of the document.Example: perspective of tester needs not to worry to make each requirement testable.The organizing structure is thus the requirements themselves to identify relevant aspects, such as expected inputs, different classes of input values, and expected outputs.Perspective of user abstraction: larger functionalities involved and to abstracts modes of behaviour from the requirements (easier than a list of individual requirements).Second: use procedure should provide a step-by-step way to check that information for defects (use information to identify defects).



9.3 The requirements customer perspectiveExample create a set of use cases step 1: identifying participantsProcedure: create use cases to document functionality that users of the system should be able to perform Listing the functionality that the new system will provide & external interfaces that are involved.Identify actors set of users who will use the system for specific types of functionalityAbstraction step: Read through the requirements once, finding participants involvedParticipants other systems and users that interact with the systemQuestions:Which user groups use the system to perform the task?Which user groups are needed by the system in order to perform its function? These functions could be its major functions, or secondary functions such as system maintenance and administration Which are the external systems that use the system in order to perform the task? External systems are managed Which external systems send information Which external systems receive informationEtc.

Use stepsQ1.1 Are multiple terms used to describe the same participants in the requirements?Q1.2 Is the description of how the system interacts with a participant inconsistent with the description of the

participant? Are the requirements unclear or inconsistent about the interactionQ1.3 Have necessary participants been omitted? That is, does the system need to interact with another system,

piece of hardware, or a type of user that is not described.Q1.4 Is a external system or a class of ‘users’ described in the requirements, which does not actually interact with the

system?

Abstraction step 2: Read through the requirements a second time, identifying the product functionsa. Functionality/ What Activities do the requirements say the system must do?b. Consider how a user of the system will view it(to achieve some goal / not individual activities) use

casesc. For each use case, decide which of the system activities you previously recorded are involved sketch the

steps / exception functionality Check: details of processing and flow of control are correct and within the appropriate functional requirements

d. For each use case which classes of users would use the functionalityUse steps:Q2.1 Are the start conditions for each use case specified at the appropriate level of detail?Q2.2 Are the Class(es) of users who use the functionality described, and are these classes correct and appropriate?Q2.3 Is there any system functionality that should be included in a use case but is described in sufficient detail or

omitted from the requirements?Q2.4 Has the system been described sufficiently so that you understand what activities are required for the user to

achieve the goal of a use case? Does this combination of activities make sense, based on the general description and your domain knowledge? Does the description allow more than one interpretation as to how the system achieves this goal?

Q2.5 Do the requirements omit use cases that you feel are necessary, according to your domain knowledge or the general description?

Abstraction step 3: Match the participants to all of the use cases in which they are involved two participants in all of the same use cases might represent a single unique actor and should be combinedUse steps:Q3.1 Is it clear from the requirements which participants are involved in which use cases?Q3.2 Based on the general requirements and your knowledge of the domain, has each participant been connected to

all of the relevant use cases?Q3.3 Are participants involved in use cases that are incompatible with the participant’s description?Q3.4 Have necessary participants been omitted (e.g. are there use cases which require information that cannot be

obtained from any source described in the requirements)?

9.4 The requirements test perspectiveGoal of this perspective: to evaluate whether the functionality contained in the requirements is well specified (test cases can be constructed) regardless of quality aspects for other stakeholders (complete set of functionality)Abstraction step 1: For each requirement, read it trough once and record the number and page on the form provided, along with the inputs to the requirement.Use stepsQ1.1 Does the requirement make sense from what you know about the application or form what is specified in the

general description?Q1.2 Do you have all the information necessary to identify the inputs to the requirement? Based on the general

requirements and your domain knowledge, are these inputs correct for this requirement?Q1.3 Have any of the necessary inputs been omitted?Q1.4 Are any inputs specified which are not needed for this requirement?Q1.5 Is this requirement in the appropriate section of the document?



Abstraction step 2: For each input, divide the input domain into sets of data (called equivalence classes), where all of the values in each set will cause the system to behave similarly Input condition: specify rage at least one valid and two invalid equivalence sets (less and greater than extreme) Input condition: specify member of set one valid (set itself) and one invalid equivalence class Input condition: specific value one valid (the value itself) and two invalid equivalence classes (less and greater

class than the value)Note: Each equivalence class should be recorded with the appropriate input.Use step:Q2.1 Do you have enough information to construct the equivalence classes for each input? Can you specify the

boundaries of the classes at an appropriate level of detail?Q2.2 According to the information in the requirements, are the classes constructed so that no value appears in more

than one set?Q2.3 Do the requirements state that a particular value should appear in more than one equivalence class (that is, do

they specify more than one type of response for the same value)?Q2.4 Do the requirements specify the result for an invalid equivalence classes?

Abstraction step 3: For each equivalence class write test cases, and record them beneath the associated equivalence set on the formUse step:Q3.1 Do you have enough information to create test cases for each equivalence classes?Q3.2 Are there other interpretations of this requirement that the implementation might make based upon the

description given? Will this affect the test you generate?Q3.3 Is there another requirement for which you would generate a similar test case but would get a contradictory

result?Q3.4 Can you be sure that the sets generated will yield the correct values in the correct units? Is the resulting

behaviour specified appropriately?

9.5 Customizing reading techniquesTo be affective in practice, reading techniques must be tailored to the needs of the project and the organization. Reuse prior knowledge whenever possible tools, techniques uncover particular types of defects.

Tailoring PerspectivesIdentify properly all of the stakeholders in the later development phases.In what other phases of the software lifecycle is this document needed?In each phase, what other specific software tasks does the document support?How / yield the following As a description of the needs of the customer functionality and performance constraints must be met by the

final system As a basis for the design of the system design has to achieve the functionality described by the requirements

within the allowed constraints As a point of comparison for system test system test plan has to ensure functionality ad performance

requirements have been correctly implemented.Designer: needs set of requirements that are correct and described in sufficient detail for major system components to be identified and correctly designed.Tester: requirements are testable and described in sufficient detail test plan / test casesCustomer: functionality system to have be completely and correctly captured by the requirementsLife cycle model should be used as a tool to help identify the right perspectives for a given environment.Other stakeholder possible integrate functionality or maintainer perspectiveRight mix expected uses of the document being reviewed in the organizational environment

Tailoring abstraction ModelsUsing the right abstractions for a set of reading techniques crucial. Two ways:They have to support defect detection as they are created (abstraction as simple as possible effort not distractsabstractions also be useful later in the lifecycleTranslate requirements in higher level design sketch / high level test plan / set of use casesAbstractions should be chosen with which the likely reviewers have some expertise.In beginning more likely in creating the system abstractions (test plan) then in looking for requirements defects. Using the test cases as an abstraction for a reading technique helps reviewers leverage their existing experience for a new goal.Identifying the right abstractions: What information must the document contain to support each perspective identified? What is a good way of organizing that information, so that attributes of interest are readily apparent to the inspector? What artefacts are being created in other lifecycle phases to support downstream development?

Tailoring the Defect TaxonomyQuestions asked about abstraction correspond directly to the types of defects being sought create a taxonomy of the important issues analyse of exactly what types of issues are of most concern to the project and organization. What constitutes a defect is largely situation-dependent.Generic taxonomy:System description increasingly formal notations / adding increasing levels of detail.



Wrong information in the next phase Extraneous information being enteredInformation ambiguously multiple interpretationsPossible defects can occur. Some categories: Omitted information: some significant requirement related to functionality, performance, design constraints,

attributes or external interface is not included, or the response of the software to all realizable classes of input data in all realizable classes of situations is not defined.

Incorrect Fact: a requirement asserts a fact that cannot be true under the conditions specified for the system Extraneous Information: information is provided that is not needed or used Inconsistent Information: two or more requirements are in conflict with one another Ambiguous Information: a requirement has multiple interpretations due of to multiple terms of the same

characteristic, or multiple meanings of a term in particular context.Important point: not the defect taxonomy itself, but used instead to create a set of questions aimed at eliciting defects, for each step of the abstraction procedure.It is not a definitive taxonomy of defects (it can be augmented or shortened or replaced altogether)

Tailoring the level of DetailDimension customize to a particular organizationLevel of detail needs to vary with the expertise of the inspectors (step-by-step till use their own hard-won experience).Tradeoffs to keep in mind:No procedure, no matter how detailed will be effective if the reviewers don’t have enough training to make effective use of the abstractionChoosing a very detailed abstraction means reducing flexibility not every requirement is well suited to the equivalence partitioning approach. The lack of applicability of the more detailed procedure tot some requirements has to be balanced against how well the inexperienced reviewers could do if allowed to select their own test approach on a requirement-by-requirement basis.Important to keep the mind of the model-building. Not he model are important themselves but as tools to find the defects.

Tailoring Team CompositionMore advanced level of tailoring: changing the composition of the inspection team. one member of one perspective. But it can change . extra reviewers on customer perspective if necessary

9.6 ExperiencesPerspective Based Reading (PBR) focuses requirements inspection on catching inconsistent or incorrect requirements before they form the basis for design or implementation, necessitating rework or on verifying that information is complete and correct enough to support all stakeholders in downstream phases of development.Improves average of finding defects.

10 Making inspections workTom Gilb

10.1 Basic definitionsInspection is a proven technique for achieving quality control of specifications and identifying associated process improvements. In fact, inspections can be applied to any discipline that produces documentsTwo main processes:The defect detection process concerned with document quality, especially identifying and measuring defects in the documentation submitted for inspection and using this information to decide how best to proceed with the main (product) document under inspectionDefect prevention process concerned with learning from the defects found and suggesting ways of improving process toe prevent them from reoccurring. Causal analysis = simply brain storming. Major part of defect prevention Out of line the day-to-day inspection.Major defect = if not dealt with at the requirements or design stage, will probably have an order-of-magnitude or larger cost to find and fix when it reaches the testing or operational stages Gilb & Graham 1993: On average find and fix in this stage (upstream) 1 hour, and 9 hours downstream.Page = logical page unit of work on which inspection is performed.

10.2 Benefits and current best practiceGiven adequate management support, inspection can quickly be turned from initial chaos phase (>= 20 defects per page) to relative calmness (2 of 3 major defects per page). E.G. Eurofighter Project in 18 months from 20 till 1 – 1,5 defects per page.Gilb & Graham 1993: detection process can find up to 88% of existing major defects in a document. Greater benefit achieved by teaching effect of inspection attending inspections rapid learning process by software engineers.Defect detection process can and should be extended to support continuous process improvement by including the associated defect prevention process 50% in first year, 70% in second year till 90%.At least 13 to 1 ROI downstream (rework saved by inspection) compared to the costs of inspection.



Defect Prevention Process is the model for the Software Engineering Institute’s Capability Maturity Model (SEI CMM) level 5.Rework cost are reduced from 40% to 5%.

10.3 Improving InspectionsTips for improving inspection

Inspection Strategy Don’t misuse Inspection as a clean-up process. Use it to motivate, teach, measure, control quality, and improve

process.Inspection is not cleaning up bad work. It is Improvement of future workContinuous process improvement integrate process into conventional inspections.

Use inspections on any technical documentation (not only code inspection) Inspect upstream first (Fagan, 1970: defect actually lie upstream in requirements and design areas)

Bellcore: 44% of all defects were due to defects in requirements and design reaching the programmers. Inspection have to start with contract, management and marketing plans. Don’t exclude Managers from

inspection of management-level (requirements or contracts). They also can support in the experience phase ‘No managers present’ is a rule from the past.

Make sure there are excellent standard to identify defective practices Juran 1995: Inspection requires good standards be in place. Standards provide the rule. Standards must be built

by hard experience brief, to the point, monitored for usefulness, respected by the troops, not be built by outside consultants of dictated by management. They must be seen as the tool to enforce the necessary lessons of professional practice on the unwary or unwilling.

Give inspection team leaders (moderators) proper training, coaching after initial training, certification, statistical follow-up, and, if necessary, remove their ‘license’ to inspect.Certification – like doctors – based on demonstrated competence and training. Enough trained inspection team leaders 20%

Entry ConditionsUse serious conditions, such as minimum level of numeric quality of source documentsLack of discipline and lack for entry conditions wastes time. Most important entry conditions: mandating the use of upstream source documents.Mistakes: use op expert’s memory, use source uninspected source documents with >= 20 defects per page, poor quality source documents.Use cursory checks on the product documents to be returned if it has too many defects (15 minutes check is enough).Learn which entry conditions have to be set and take them seriously. Management has to take the lead on this. Managers are often responsible for overriding entry criteria.

Planning Plan inspections well using a formal review process form

Use one-page review process form, document supporting documents, assign checkers special defect-searching roles, carefully manage the rates of checking and total checking time needed. Establish formal purpose(s) of each specific inspection. Team stretch goal established and specific strategy to help themPlan inspection to address the inspection purposeThe are more than 20 distinct purposes for using inspection: document quality, identifying defects, removing defects, motivation, helping the author, improving productivity, and reducing maintenance costs.

Inspect early and often while documents are still being writtenIf the process generates the document is faulty, discover it early and fix it.

Use sampling to understand the quality level of the documentThe main purpose of inspection is economic to reduce lead time and people costs caused by downstream defects. Defects should be cleaned up or avoided Harlan Mills’ Clean Room (1987)Cleaning is unnecessary and sampling provides the information to decide if it is economically safe to release the document Perfection is not required.

Use an optimum number of people on a team to serve the current purpose of inspection, for example effectiveness, efficiency and training.Best effectiveness: 4-6 people, efficiency (effect over costs) 2-4 people and only teaching justifies a larger number. Optimum per document.

Allocate special defect-searching roles to people on the teamEach person on an inspection team should be finding different defects time and money risks, check against corporate standards for engineering documentation, check against security loopholes.

Individual preparation Check against source and kin documents; check them for defects, too

Focus on the major defects that still persist in source and kin documents. Source documents = input documents. Requirements document can be a source document and used to produce a design specification.

Check the significant portions of the material – avoid checking commentaryMost organizations waste time checking non-significant document areas. Optimum rates find major defects.Non-commentary (or meat) text is text where any defects might translate into serious downstream costs (major defects could be found). Commentary text (fat) only contain minor defects and so is less important.

Define a major defect as ‘possible larger costs downstream’



Is does not matter if a defect is not precisely a non-conformance or if it is visible to a customer. If it could lead to significant costs if it escapes downstream, classify it as a major defect, and treat it with due respect. Concentrate on major defects.There are at least 18 different tactics that shift from minor to major (figure 10.2)Use checklists to interpretate the rules, log only major

Check at an organization’s optimum (coverage) checking rates to find major defects.Optimum checking rate is not a reading rate. Checking in real inspections could involve 10 – 20 individual documents to check (source documents, standards, checklist). Requirement is to check a single product document line against many sources, and it takes time.Expected optimum checking rate range between 0.2 and 1.8 pages or 300 non-commentary words per checking hour. In practice: 10 – 20 statements per hour per individual checker.Checking speed to optimum more defects found. Seems slow. Team meeting will produce 15% additional defaults

Use the individual checker’s personal notes instead of proper meeting defect logs when major-issue density (non-exit level) high, or when there are many minor defects.Checkers should not be required to use any particular method to make notes during the checking process. Important note against the defective words which rule was broken and classify the decision.More issues hand over notes to the author instead of logging everything. Author: rewrite and submit their documents

Logging meeting Decide whether it is worth holding a logging meeting, based on checking phase data

Use checking phase data are collected at the beginning of a logging meeting or before.Make decisions about the logging meeting and the rest of the inspection most critical meeting is necessary

At logging meetings, avoid discussions and suggesting fixesMaking maximum, meaningful progress for professionals

Causal analysis Use the defect prevention process on inspection itself for continuous improvement

Systematic continuous improvement of inspection process is necessary improve process, implement of correct inspection process and tailor it to the organization

Exit Criteria Use serious exit criteria

Maximum 0.2 defect per page. There are still a number of remaining unfound defects.Maximum effectiveness between 30 – 88 %of existing defects. O.2 defects per page is clean enough, going on is uneconomic.Uneconomic: 10 defects per page (at 50% effectiveness) 10 defects in 100-page document 9 * 10 * 100 hours of additional work.Prevent is better than cleaning up

Inspection metrics Build or buy an automated software tool to process inspection basic data capture data-summary data and

to present trends and reportsKey distinction between inspection and other review process is the use of data to manage inspections optimum checking rate has to be established early and updated as they change. Inadequate exit levels (too many major defects floating downstream) must be caught with expensive testing process

Put inspections artefacts on a company web siteStandards, statistics, experiences, problems

Measure the benefit from using inspectionsHighly profitable 10 to 1 ROI. If not stop it. Inspections must be evaluated

Inspections floated from clean up to sampling measurement, exit control and defect injection prevention.Inspections will continue to be a powerful software process tool.



Part 4: Test Techniques

11 Static AnalysisLes HattonThere is much to be done examining system artefacts without actually running the system. This chapter different kind of review: examine digs and code, usually with some automated assistance, to ferret out additional problems before the code is actually run static analysis (= just another form of testing)Static analysis is an examination of design and code that differs from more traditional testing in a number of important ways: Static analysis is performed on a design or code without execution Static analysis is performed before the kind of peer reviews Static analysis is unrelated to dynamic properties of the design and code, such as test coverage The goal of static analysis is to find faults, whether or not they may cause failure

11.1 Static fault versus dynamic failureStatic no actual run time behaviour involved. Examining the code either by eye, with a tool, with two things in mind:The nature of the design or code is studied to see if there is something clearly amiss, such as reference to a non-existent item (review is a form of static code analysis).Tools and techniques are used to examine characteristics of the design or code, to see of these characteristics warn us that the design or code may be faulty (e.g. levels of nesting).Addressing to both goals design or code are intellectually interpreted, predicted behaviour is inconsistent with the understanding of which the program will actually do at run time fault.Solid software static analysis is used in combination with other techniques.Contrast with dynamic analysis program is actually run and an observation is made on what happens program exhibits different from what which was expected software failure.

11.2 When faults become failureIt is very important tot distinguish faults from failures and to understand the relationship between the two.Ultimate goal in building solid software reliable and dependable. Keep from software fails, or at least no physical, environmental or financial harm.

Failure is the result of one or more faultsBut not all faults cause failures during the life-cycle of the code the number of failure-causing faults is actually significantly smaller than the number of all faults in a typical piece of code. A few lines of a program that are unreachable (could be full of faults, but was not noticed, because it was not reachable).Some faults are latent, causing failure only when certain conditions are met.Adams (1984): a third of all detected faults led to the ‘least frequent’ types of failures. 2% of the faults cause failure within 5 years. Most faults in the sense of a given time will not lead to failure Mean Time To Failure (MTTF)Products with a very large number of faults can in fact fail are very rare lack of failure (not fault counting or fault density). Using fault rate or fault density as a predictor may be misleading, the code may be better than we think.

11.3 Early versus late detectionBoth static and dynamic analysis are used to get a snapshot of quality at different times. Static analysis is applied in a much earlier stage of development than dynamic analysis. Dynamic only when enough of the product has appeared to allow it to be compiled, linked and executed in various test scenarios.Why do static analysis as early as possible cheaper to detect faults early then late. Chart of Boehm (1981). High and Low critically.There is a strong financial incentive for finding faults and fixing them early in the process.

11.4 Measurement for static analysisPerforming static analysis information is calculated about structural code, depth of nesting, number of spanning paths, cyclomatic number, number of lines of code, unreachable statements. For design, code and also as changes are made to see if the design or code is becoming bigger, more complex, more difficult to understand and to maintain.Measurements can also help decide among several alternatives, especially when redesigning portions of existing code.Many different kinds of structural measures to write, to understand Several aspects of code structure to consider: control flow structure sequence in which the instructions are executed (reflects to iterations, loops) data flow structure follows the trail of a data item as it is accessed and modified by the code (many times

transactions applied to data are more complex than the instructions that implement them) data structure refers to the organization of the data themselves, independent of the program (data arranged as

lists, queues, stacks algorithm is more likely to be well-defined too). Data structure provide much information about the difficulty in writing programs to handle data, and in designing test case to show program correctness. Program complex, because data structure is complex.

Fenton & Pfleeger (1997): static code measures discussed in great detail. A tester can ask to see measures for: Problem complexity complexity underlying problem Algorithmic complexity efficiency of the solution implemented by the software



Structural complexity how code structure affects overall quality Cognitive complexity the effort that is likely to be required to understand and maintain software.Type of measure ExamplesProblem Complexity Number of requirements, lines of code, function pointsAlgorithmic complexity O(n), number of comparisons

Structural complexity Control flow Depth of nesting, cyclomatic number Data flow Number of transformations Data structure Number of data items, number of linksCognitive complexity Gunning for index (applied to written text)

11.5 Coverage: how much is enough?How do we know how much static analysis to do?

Coverage for static analysis has nothing to do with coverage of dynamic testingTesting measures crude level to determine how many of the functions present in the system are exercised.Myers, 1979: loop and path coverage unachievable in any reasonable time. There are serious trade-offs between test coverage and business needs getting the product out of the doorProgram can never be exhaustively tested dynamically.FailureStatic analysis failure to cover part of the system adequately is essentially a management or an educational issueDynamic analysis failure to cover part of the system is a architectural issue strongly related to the ability of the test engineers to design suitable test casesStatic & dynamic testing must be designed and organised.

11.6 Approaches to static analysisDesign and code review, walkthrough and inspection. Also Static analysis component calling tree

Static analysis of designsSophisticated design techniques and uniform notations = utopia. Reality: vary dramatically per project. Design process very creativeLack of uniformity and formality difficult for static design analysis.Source code is more standardized than design easier to define relationships between code expressed in one way and another.

Using automation to find code faultsMany tools numbers, depth of nesting, graphic description of control flows, data relationships, number of distinct paths.Engineers rely on their compilerProgram languages rich source of static detectable faults: Programming languages contain well-known errors These faults are often missed during testing and so appear in released products Then the product fails

Programming languages contain well-known errorsTwo reasons: politics and mistakes The standard is so badly written as to be confusing The standard does not say anything at all The standard is inconsistentStandardization does not usually lead to fewer failure modes because of new featuresCapability expansion and backwards compatibility creates a lethal cocktail20 % of code is generally used / 80% of the code / functions is rarely used

These faults are often missed during testing and so appear in released productsFault mode exist 8 faults in 1000 lines of codeWhen systems fails root-cause analysis why the fault was missed.

Then the product failsA percentage of the faults do in fact fail in the life cycle of the code

Code faults can not be found by automationThere will be always faults which can not be found by automationStatic noise:Overdoing static analysis is still much cheaper than waiting until testing to find faults.Static analysis finds faults; dynamic analysis finds failure.Discovery of faults can be considered that are never likely to fail inherent noise in the static analysis process. It is not possible to assign a likelihood of failure to each fault discovered.Inspection data alone still overwhelmingly favour static analysis techniques careful control of the noise.Static noise manifests itself in to important ways: Static noise greatly complicates manual code reviews.



In the output of static analysis tools, static noise frequently hides faults more likely to fail from faults less likely to fail.With inspection many items turn out to be false positives dismissedControlling static noise in tools and with tools requires considerable language and system expertise control: use rules and tools in the form of checklists. Techniques and tools targeted at narrow subsets of problems we know right away what is happening in the code.

After fixes new faults could be introduced. Configuration management is still important.

12 Testing Techniques; why bother?David Hayman

Recent focus on risked based testing many techniques of the testing activity. Many sources of information with respect to test techniques, their methods and coverage measures. This chapter Why we should consider the use of test case design techniques as a part of our everyday testing activities and who should be looking to use them.

12.1 What are techniques?Techniques: Mode of artistic execution in music, painting etc. testers use their own imagination and experience to produce

executable tests Mechanical skill in art to best suited to the application, environment, or test phase / repeatable amongst other

things measures objective and result = have control over their tests Means of achieving one’s purpose, esp. skilfully objective of the test phase or level. Objectives: scope, coverage,

riskThe art of testingTesting techniques make a part of the exercise mechanical in that the production of the test case becomes formalised.Techniques to pr0vide objective and measurable tests and results to allow users to make an objective decision about the likely impact of taking the system into live operation.Meyers, 1979: What subset of all possible test cases has the highest probability of detecting most errors? tester cannot / will not / should not even try to run every possible test case.Testing techniques many shapes and sizes, some formal some not, some dynamic and some static. Overall objective discuss the issues surrounding the use, selection, advantages and disadvantages of testing techniques as a set of tools.Many dynamic testing techniquesStatic testing techniques prevention will always be better than cure.

12.2 An overviewDynamic vs. StaticKey difference point in the life cycle / V-model dynamic test case design activities at the same with static ‘review’ activitiesStatic testing (reviews and static analysis) primarily techniques aimed at preventing faults being propagated into the next phase or into the test environmentDynamic techniques (either black or white box) are intended to find faults in the translation of the specifications into the actual system.

Systematic vs. Non-systematic techniquesDistinction is formalitySystematic techniques are based around a set of rules that make the activity mechanistic, repeatable and measurableNon-systematic techniques are based on the experience of the tester, either at product-level or within the testing function. but also techniques such as extreme programming (XP) experienced testers and designer. Lack of documentary evidence and therefore repeatability of the test scenarios. Not a cure for all testing functions.

Functional vs. Non-functional techniquesNon-functional techniques focus on establishing how the system does what it does i.e. is it secure? How quickly does it respond? Is it reliable?There is little information relating non-functional techniques creating test cases tends to be more of a strategic activity based on a series of inter related requirements. Often use tools to execute tasks with supporting tools analyzing the dynamic system activity (memory use, concurrent threats etc.)

12.3 Are techniques for testers only?Karl Kraus: The development of technology will leave only one problem: The infirmity of human nature.Development, implantation, users has a vested interest in establishing the quality of the system to be delivered.



Look for specific types of faults tester should be taught techniques that will reflect the phase of system development in which that the tester is involved techniques for reviewing system documentation, other techniques.It not only depends on the testes.Everyone in the IT should be aware apply basic aspects of test case design techniques many, many faults would either not happen in the first place or be found before they could have an impact.Allowing the current situation no improvement in the quality of the systems

12.4 Why use techniques at allPablo Casals: The most perfect technique is that which is not noticed at allWhy use test cases design techniques at all Objectiveness guarantees a certain level of coverage linked to an identifiable process of achieving it Increase in the defect finding capability The ability to reproduce tests Long term system maintainability and change controlThus need to ensure the activities are measurable, controllable and repeatableMany techniques and supporting tools have been developed to allow testers to meet those objectives take advantageTest techniques provide an understanding of the complexities imposed by most techniques forces testers into thinking about what they test and why they are testing it. test coverage objective decision could be made which tests could be leave out.Fundamental part of function testing: Risk identification, assessment (focus), management (test process control) and mitigation (correct coverage)Use of test techniques is methodical and mechanical for the core tests but used properly they are flexible. Used together formal and informal techniques create an entire re-usable test package new function and a few bug fixes in new release use some existing scripts. Not all is newly written down. But techniques are used (EP, BVA and test cases which are not formally documented)Question should be: ‘which techniques should we use?’ instead of ‘why use techniques?’

12.5 When should techniques be used?Henry Dumas: Is it time to make time?Often: we don’t have time to use techniques, we just need to get on with the testing. Those who are working in a formal framework are in the minority.It has to be understood and appreciated that using techniques has an initial cost and time overhead The log-term benefits must also be factored into the tester’s planning and cost benefit analysis.Early design of dynamic testing techniques could be used as static testing technique in this stage many defects could be found.Testers must always consider the tests they are going to run.Time is always the criterium time available and test stages will dictate which technique and how much detailed documentation can be completed.Most used / simplest techniques: EP and BVA (together with error guessing)Testers should use test case techniques all the time at every stage. Question: Which techniques are the most appropriate? should be focused on the type of bugs that the testers are expecting to find, or should be looked for at any given stage of the system delivery cycle. See also Beizer 1990, The Taxonomy of Bugs

12.6 Software and system modellingEvery picture tells a storyThe method of ‘documenting’ system has been prevalent for many years and is still in existence today. new techniques has exacerbated the situation. Often documentation afterwards. That is why XP was introduced usefulness or otherwise producing diagrams.Models are used in form of flowcharts, data flow diagrams etc. represent some part of the physical or logical system.Modelling software or interfaces within the system will provide a ‘picture’ to some degree of how that part of the system is intended to operate.Vast majority of formal dynamic test techniques are identified in BS7925-2 not to provide a objective coverage measure for establishing paths, decisions, branches and statements, but also to document the decision making process, whilst making the component clear to both technical and non-technical staff.Documentation of test coverage is critical in a function where objectivity is the be all and end all. Level and detail dictated by the technique chosen and the models are not feasible without the use of automated tools.They must ensure they use models to the best advantage to provide this objectivity hand in hand with a deeper understanding of how the system works.

12.7 New technology and techniquesTechnology is changing does not mean the techniques used to test it must change as well. It is still the same type of bug (may be just another focus, use of new technology to create a component)Internet first code HTML, interfaces dynamicHTML reason: to be used by every browser. Test: static test tool configured to adhere to a specific HTML standard.Interfaces they shear volume (hyper links) to be tested with tool of expediency (link control)Risks: Usability, Performance, Reliability, Availability, Security etc.XP different technique different people, different tools test techniques remain the same. Everyone should be aware of and trained in at least the most common test techniques.



12.8 Advantage / disadvantagesFormal – informal test techniquesAdvantage DisadvantageObjectivity Require training to some degreeFormal coverage measures Time to implement – culture changeEarly defect finding Everyone must be ‘bought’ inTraceability Not seen as useful for all applicationCoverage independent of the tester Take more time than less formal designWay to differentiate test depth based on risk using different techniques

Do not cover all situations (error guessing still useful)

High level of re-use (re-usable testware) Little use of domain and product knowledge of testerRepeatability and ReproducibilityAudit trailsHigher defect finding capability

Test case design techniques integral part of everyday testing life.In order to remain objective the level of testing must be measurable coverage and formal test case techniques, static testing methods

13 Black Box TechniquesErik van Veenendaal and Jurriën SeubersFormal BB test specification standardised procedure for deriving test cases from requirement specification and design documents.

13.1 CharacteristicsBB techniques are based on the analyses of the specification without reference to its internal workings.

BB techniques: Equivalence partitioning Boundary value analysis Cause/effect graphing (cause/effect analysis of decision table testing) Classification tree method Data cycle test Elementary comparison test Process cycle test Semantic test State transition test Syntax test Test uses casesFor each technique a short description is provided and characteristics, such as: Test levels Test basis Coverage Application area Quality characteristics

Test levelsIndication at what level the techniques are most appropriate and often used

Test basisBB often based on a specification (from high level – very specific information). Data cycle: CRUD; PCT: process flows. Without almost impossible tot apply the technique.

CoverageBB techn. coverage criteria do not relate to software code, but to level of thoroughness with which the requirements are tested. Decision coverage: test cases are generated to exercise all possible outcome of every decision Condition coverage: test cases are generated such that all atomic conditions (in a decision) are tested once

being true and once being false; Modified decision/condition coverage: test cases are generated such that every atomic condition determines

twice (once true and once false) independent of the other conditions, the result of the decision; Branch condition combination coverage: test cases are generated to exercise all possible combinations of

true and false outcomes of atomic conditions within a decision.

Application areaTest interaction user – system (user interfaces, reports), relationship between business processes and the system or to test batch processing. Or integration of components. Suitability related to type of defects (incorrect input validation, incorrect processing, integration defects).



Quality CharacteristicsUsability, functionality, interoperability, security

13.2 Equivalence partitioningMost used BB-technique and is applicable to all levels of the V-modelBasic idea: To divide the input domain into equivalence classes (EC) or partitions of data, which according to the specification, have the same behaviour. Reducing input: any value from an equivalence partition (or class) is as valid as any other.Invalid data outside a specific partition (does not mean it is incorrect) / negative testing.EP reduces the number of input conditions to be tested by identifying classes of conditions that are equivalent to many other conditions.Can be extended / combined with Boundary Value Analysis.

CharacteristicsTest levels All test levels, especially recommended for integration testing, system testingTest Basis Requirements specification and design documentsCoverage Condition Coverage (Decision coverage)Application Area All types of systemsQuality characteristics Functionality, interoperability

Design procedure1. Identify attributes2. Identify the accompanying equivalence classes (Kit, 1995)

a. Boundary value 1 valid EC (within boundary) and 1 invalid EC (outside boundary);b. Boolean 1 valid EC (true) and 1 invalid EC (untrue/false);c. Range 1 valid EC (within range) and 2 invalid ECs (one outside each end of the range);d. Number (N) 1 valid EC (the exact Number) and 2 invalid ECs (<N and >N );e. Set of valid values 1 valid EC (from within the set ) and 1 invalid EC (outside the set);f. If the elements in a EC are not handled in an identical manner by the program, subdivide the EC in

smaller EC’sg. For compulsory input also test with an empty input.

3. Test cases are identified and described First all valid and afterwards invalid test cases. Invalid: only one4. Partition the output if needed, to cover output messages

13.3 Boundary value analysisUses a model of the component that partitions the input and output of that component, e.g. a test item (function, component) with numeric inputs that can be divided in equivalence classes. The values at and around the boundaries of the equivalence class are referred to as boundary values most defects to be found.Can be applied at all test levels especially component testing. BVA extension of equivalence partitioning

CharacteristicsTest levels All, but especially recommended for component testingTest Basis Requirements specification and design documentsCoverage Condition Coverage (as EP but now with boundary focus)Application Area All types of systems, especially usable for testing components (functions) with numeric

equivalence classes Quality characteristics Functionality

Design ProcedureFor each input that concerns a range, test cases for the end of the range and invalid –input test cases for conditions just beyond the endsIf the input is a number of valid values, test cases for the minimum and maximum number of values and one beneath and beyond these valuesFull boundary: three values per boundary are selected.Extreme boundaries minimum and maximum value permitted

13.4 Cause/effect graphing (cause/effect analysis of decision table testing)Is a specification-based technique which aims to create ‘interesting’ combinations of inputs for testing.Causes (inputs) X effects (intermediate results, messages, output). Causes and effects have to be stated in such way that they can either be true or false (Boolean).Creates combinations of input.Disadvantage: large set of inputs very complex. Should be applied to reduce the complexity and high-risk components.

CharacteristicsTest levels Component testing, (integration testing), system testingTest Basis Requirements specification and design documentsCoverage Branch Condition Combination Coverage Application Area Test items with Boolean inputs. To be useful for critical components, state testing and



complex processing Quality characteristics Functionality

Design procedureAnalysing inputAnalysing outputNumber of test cases = 2 to the power of N

SimplificationStandard rule: for two test cases only one cause (input) differs and the resulting effects (action) are the same these test cases (columns) can be joined. Remove one the right one

13.5 Classification tree methodClassification tree method (CTM) a special approach to partition testing by partly using and improving ideas from the equivalence partitioning techniques.Input domain of a test object is regarded under various aspects assessed as relevant for the test disjoint and complete classification.Test cases are formed by combining classes of different classifications

CharacteristicsTest levels All, although mainly used during system testingTest Basis Requirements specificationCoverage Condition coverage, but can be extended to branch condition combination coverageApplication Area All, but especially used in technical automotive, and embedded software environmentsQuality characteristics Functionality and portability

Design ProcedureIdentify aspects of the test object that influence the functional behaviour of the component (shape, colour)Selecting class for each aspect

Combing these classesMinimum number of test cases: when every class is covered by at least one test casesTheoretical maximum: every possible combination of the classes of all aspects must be covered by test cases (branch condition combination coverage).

13.6 Data cycle testTo test the life cycle of data elements CRUD. It is verified whether sequences of functions process the data correctly, and whether the referential integrity checks (data model consistency checks) are compiled with. Tests ‘nearly’ the entire information system final stage of the acceptance test (testing the system integration)

CharacteristicsTest levels System testing, acceptance testingTest Basis Data model, requirements specificationCoverage No formal coverage measure applicableApplication Area Data base related systemsQuality characteristics Functionality, Reliability

Design ProcedureDetermining which functionality are presentFor each function elementary actions are executed by the function with regard to what entitiesPlace in CRUD-matrixTest cases are created for each entity1. Create, read2. Update, Read3. Delete, readTest cases1. Entering data (one test case for each create option)2. Changing data (one test case for each change option)3. Deleting data (using each delete option)4. Checking data after each action using the read option)Further additions and variations will emerge from the referential integrity checksIncluded also: All restrictions (consistency checks of the data model) on entering, changing and/or deleting have been identified,

described and tested. A test action has been identified with respect to all functions (including the batch functionalities) that makes use of

the entity according to the CRUD-matrix.

13.7 Elementary comparison testECT is aimed at testing in detail all functional paths of a function or component.



CharacteristicsTest levels Component testing, (integration testing), system testingTest Basis Thorough requirements specification and design documentsCoverage Modified decision condition coverageApplication Area Complex processingQuality characteristics Functionality

Design ProcedureMaking an inventory of all available (functional) conditions, which are defined in the requirements or designTranslated into lines of pseudo codeAnalysis pseudo-test code cases are designed using Modified Decision Condition CoverageHigh degree of completeness (complex, time-consuming, not suitable for end-users). For high risk, complex calculation.

ECT is based on the fact that it is not useful to identify all possibilities as situations to be tested, but that sufficient coverage can be achieved with a smaller number of test situations.Atomic conditions once true once false.Compound conditions modified condition coverage every atomic condition determines twice (once true and once false) the result of the compound condition. situations OR = true-true & AND = False-false will not be tested

Establishing Test casesFinding the paths in the pseudo-code in which each situation of each condition will be tested at least once traceability matrix possibilities in columnsNext step: logical test cases

13.8 Process cycle testTo check whether the automated part of the information system fits in the information system fits in the business process, particular attention to the interfaces between the automated processes and the manual procedures. Match of automated process and manual procedure: Do the automated process provide sufficient sat in order to perform all manual procedures? Does the output of the manual procedures provide sufficient data in order to start all automated processes? Do employees have sufficient authorisation to perform the procedure?PCT suitability (usability, security).Not a specification test, but more a structural test consists of a sequence of actions that together form a path through the procedure flow.

CharacteristicsTest levels Acceptance levelTest Basis Business procedures, user requirements specification and possibly user-manualCoverage Decision coverage (also related with switch coverage state transition )Application Area Information systemsQuality characteristics suitability (usability, security)

Design ProcedureTest measure:In-depth PCT Measure N: all dependencies of actions before a decision point and after N-1 decision points are verified by including all possible combinations regarding the actions in the paths N2 highly related to 1-switch coverageNormal PCT. N1 highly related to 0-switch coverageTest measure 2Make an inventory of the decision points in the flow chartDetermine each path combinations per decision point

13.9 Semantic testTo verify the relations between data items during entry (data within a screen, between data on several screens etc.)Especially for testing user-interfaces during a system of acceptance testingFocused upon functionality (security user –id / password)

CharacteristicsTest levels Integration testing, system testing, acceptance testingTest Basis Data model, user-interface specification, security requirementsCoverage Decision coverageApplication Area User-interfaces, internal and external interfacesQuality characteristics Functionality, Security

Design ProcedurePoints within the system are determined at which data relationships with corresponding validations are available and can be tested relationships found subsequently elaborated into unambiguous pseudo code using only IF / THEN / ELSEA And B If A Then if BA or B If A Then …. Else if B



13.10 State transition testUseful for both procedural and object-oriented development. Based on the concepts of states and finite-states machines and allows the tester to view the developing software in terms of its states, transitions between states, and the inputs and events that trigger state changes. State transition testing uses a model of the states the component may occupy, the transitions between these states, the events which cause those transitions, and the actions which may from those transactions. The states of the model are separated, identifiable and finite in number.In embedded softwareTesting screen-dialog e.g. with internet applications

CharacteristicsTest levels Component testing, integration testing, system testingTest Basis State transition diagramsCoverage Switch coverage (various levels)Application Area State based system, e.g. technical automation, embedded software, and internetQuality characteristics Functionality, Resource behaviour (memory)

Design procedureState transition diagrams state, transition, inputs, events, actions and outputs.Test cases exercising transitions between states.For each expected transaction within a test case: The starting State The event which causes transitions to the next state The expected action caused from the component The expected next stateExemp. STD 3 states, 5 transitionsN-switch coverage: A form of state testing in which all test cases are designed to execute all valid sequences of N+1 transactions.Practice: mostly 0-switch (every state is passed once) or 1-switch coverage (two subsequent transactions than can be executes one after another.For each test case the following must be specified: The starting state of the component The input(s) to the component The expected outputs form the component The expected final stateExample STD (set of test-cases for 0-switch), see page 2453 states and 5 transitionsTest Case 1 2 3 4 5Start State S1 S1 S2 S2 S3Input A E B C DFinish state S2 S3 S1 S3 S2Test cases

State table valid or invalid (first column all states)A B C D E

Start State S2 Invalid invalid invalid S3Input Invalid S1 S3 invalid InvalidFinish state invalid Invalid invalid S1 InvalidState table

1-switch coverageTest Case 1 2 3 4 5 6 7 8Start State S1 S1 S1 S2 S2 S2 S3 S3Input A A E B B C D DNext State S2 S2 S3 S1 S1 S3 S2 S2Input B C D A E D B CFinish state S1 S3 S2 S2 S3 S2 S1 S3

13.11 Syntax testTo find defects in primary input validations, user-interfaces, screen navigation, error messages and layout of screens.Test basis, see below

CharacteristicsTest levels Component testing, integration testing, system testingTest Basis User-interface standards, specifications of user-interfaces, error messages and reports,

interface specifications and data modelCoverage No formal coverage measureApplication Area User-interfaces, internal and external interfacing



Quality characteristics Fault tolerance, Functionality, Interoperability, Usability

Design ProcedureGeneric checklist which tests regarding the various data types, interfaces, screens and outputsWhat types of items are there: numeric, alphanumeric, datesWhat options are possible for the screens and/or each item: help option, function keys, scroll bars, short-keysFor each item type and navigation option must be determinedMake Selection to Perform tests selection based amongst others frequency of use, business critically, screen types (input, query, selecting) and/or experience of the development team.Assemble the detailed test procedureLarge number of defects in sample during execution expand the number of screens and interfaces.Combination with semantic test is possible

13.12 Test uses casesA sequence of action performed by a system, which combined together produce a result of value to each system user.Each case pre-conditions which need to be meet for a use case to work successfully. Withdrawal from a ATM-machine (you have to have a bank account with money upon it).Uses cases and test cases work well together in two ways Use case complete, accurate and clear test case is straightforward.Use case not in good shape attempt will help to debug the test case.Use cases describe process flows

CharacteristicsTest levels Integration testing, system testing, acceptance testingTest Basis Requirements specification (use cases), user manualCoverage No formal coverage measureApplication Area On-line / interactive systemsQuality characteristics Usability, Functionality, Security, Interoperability

Design Procedure Identification of users of the system (human or other systems) Identification use cases for every user group Identification scenarios for every use Select use case for testing

o Template Name of summary User group Business importance (priority) Pre-condition e.g. other use cases Basic flow Deviation flows Alternative flows Influencing use cases Post-conditions Additional attention points e.g. understandability of messages, performance issues

BB-Techniques / Test SortBB Techniques Unit /

component test

Integration System Acceptance

Equivalence partitioning

X X X X X

Boundary value analysis

X X X X X

Cause/effect graphing (cause/effect analysis of decision table testing)

X (X) X

Classification tree method

X X X X X

Data cycle test X XElementary comparison test

X (X)

Process cycle test XSemantic test X X XState transition test X X XSyntax test X X XTest uses cases X X X



BB-Techniques / RequirementsBB Techniques

Functionality

Interoperability Portability Reliability Suitability usability Security Resource

Equivalence partitioning

X X

Boundary value analysis

X

Cause/effect graphing (cause/effect analysis of decision table testing)

X

Classification tree method

X X

Data cycle test X XElementary comparison test

X

Process cycle test

X X X

Semantic test X XState transition test

X X

Syntax test X X XTest uses cases

X X X X

14 Exploratory testingJames BachPowerful approach, yet widely misunderstood.It can be orders of Magnitude more productive than scripted testing. Every does it once and a while.

14.1 IntroductionET: simultaneous design and execution of tests. To casual observer it has no structure at all.Et is also known as ad hoc testing (often synonymous with sloppy and careless work) Cem Kaner (Testing Computer Software, 1993) emphasis dominant thought process involved in unscripted testing into teachable discipline.It all becomes to skills skills to listen, read, think and report rigorously and effectively.Some tests that must be executed in just the same way, every time, in order to serve as a kind of benchmark scriptingEP is not against the idea of scripting. In some context your mission will benefit more from the ability to create and improve tests as you execute them. Most situations: mix of scripted and exploratory approaches.Two definitions1. EP = any testing to the extent that the tester actively controls the design of the tests as those are performed

and uses information gained while testing to design new and better tests.2. CAM Kaner EP is any testing to the extent that the tester actively controls the design of the tests as those are

performed and uses information gained while testing to design new and better tests, and where the following conditions apply:

i. The tester is not required to use or follow any particular test materials or proceduresii. The tester is not required to produce materials or procedures that enable test re-use by another tester or

management review of the detailed of the work done.EP = tactic to employ at particular times on a project, rather than a thought process that is more or less always present.These two views represent simply two styles of test management

EP is a Profoundly Situational PracticeJigsaw puzzle = EP concentrate on a particular part of the puzzle / test. Puzzle changes the puzzlingSpecifies affect EP: The mission of the test project The mission of this particular test session The role of the tester The tester (skills, talents, preferences) Available tools and facilities Available time Available test data and materials Available time of other people Accountantability requirements What the tester’s client care about



The status of other testing efforts on the same product The product itself

o Its user interfaceo Its behaviouro Its present state of executiono Its defectso Its testability

What the tester knows about the producto What just happened in the previous testo Known problems with ito Past problems with ito Risk areas and magnitude of perceived risko Recent changes to ito Direct observations of ito Rumours about ito The nature of its users and user behaviouro How it’s supported to worko How it’s put togethero How it’s similar to or different from other products

What tester would like to know about the productEP Tester: what is the most powerful test I can perform right now?

14.2 Practising Exploratory testingOver a period of time a tester interacts with a product to fulfil a testing mission, and reporting results.EP begins with a charter mission and tactics. Chosen by tester or test manager. Sometimes writer down Explore and analyse. Produce a coverage outline Identify and test all claims in …manual …. We need to understand the performance and reliability characteristics of Product as decision complexity increased.

Realistic scenarios coverage of each primary function of the product. Test all fields that allow data entry Check GUI against Windows standards

….Ambiguous correct. Charters are intended to communicate the mission of a test session clearly and succinctly to testers who have already been trained in the expectations, vocabulary, techniques and tools of the organisation.Only result in EP: bug reports.Basics of EP (inside the tester) Test Design: tests that systematically explore the product skills to analyse, evaluate risk, use tools, and think

critically among others Careful Observation watch for anything unusual or mysterious, distinguish observations to inference Critical Thinking looking for errors in their own thinking (especially in defects) Diverse ideas use of heuristics guidelines, generic checklists, mnemonics, rules and thumb. Rich Resources inventory of tools, information sources, test data an d friends. Alert for opportunities to apply

those resourcesManaging Exploratory TestingTest lead EP managed in two ways: delegation and participationDelegation lead specifies the charters. Testers design and execute, report back. Reports are oral benefits from each other (What is the most interesting bug for me?). Each tester as an executive who manage the value of its own time they have to earn credibility (coach or semi-independent creative agent)Participation lead tests right alongside the rest of the testers delegate administrative and meeting attendance to other people. To direct the strategy & continuously demonstrate the behaviours he expects from his teamTeam exploratory can be extremely powerful. Testers into pairs (driver and watcher) blockbusting tool to get effort

14.3 Where exploratory testing fits You need to provide rapid feedback on a new product or feature You need to learn the product quickly You have already tested using scripts and seek to diversify the testing You want to find the single most important bug in the shortest time You want to check the work of another tester by doing a brief independent investigation You want to investigate and isolate a particular defect You want to investigate the status of a particular risk, in order to evaluate the need for scripted tests in that areaBroad definition of EP: Improvising on scripted tests Interpreting vague test inscriptions Product analysis and test planning Improving existing tests



Writing new test scripts Regression testing based on old bug reports Testing based on reading the user manual and checking each assertionPowerful from testing to re-designing themFeedback loop is weak, slow ET loses power fall back on pre-designed scripting (also by extreme retrospective scrutiny)Combine exploratory testing and scripting strategy get the best of two way.

14.4 Exploratory testing in action Find violation of the compatibility requirements.General plan of attack; allow yourself to deviate from it for short periods of time tour busPrevent itself from executing if there is not enough memory to perform the operation.

14.5 ProductivityThere are no reasonable numbers that compare EP with scripted testing.JB: test scripts produced were far less powerful than were the exploratory tests.EP = Martial arts of the mind

15 Usability testingIsabel EvansUnderstanding Usability and usability testing. Importance and use of accessibility and how usability acceptance criteria are measurable.

15.1 What is usabilityIs the quality of a product (software or other) which addresses its suitability for the people who will use it.Usability and usability testing have aspects in common with other types of testing, but here are specific differences with affect how usability testing is set up, executed and reported on.Usability requirements: Include consideration of the context within the product is used, the people who use it and the environment within it will be used.Narrow scope: ease-to-use perspective determined by software itself user interface) ISO 9126-1Broader scope using the product in its (operational) environment type of users, tasks to be carried out, physical and social aspects that can be related to the use of the software products are taken into account. quality-in-use (ISO 9241-1)Product quality influences Quality in useQuality in use depends on Product quality

Narrow focus definition of usabilityISO 9126-1: The capability of the software to be understood, learned, used and liked by the user when used under specified conditions.Attributes: Understandability: bear on the users’ effort for recognising the logical concept and its applicability Learnability: bear on the users’ effort in learning the application Operability: bear on the users’ effort for operations and operation control Attractiveness: capability of the software to be liked by the userConsider Surrounding manual process within this context

Broad focus definition of usabilityISO 9241-1: The effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments Effectiveness: The capability of the software product to enable the people who use it to achieve specified goals

with accuracy and completeness in a specified context of use Efficiency: The capability of the software product to enable the people who use it to expend appropriate amounts

of resources in relation to the effectiveness achieved in a specified context of use Satisfaction: The capability of the software product to satisfy users in a specific context of use

Good usability is vital to delivering products that can be used successfully and which are appealing, encouraging people to use and re-use the product.

Usability of an ATM Tasks withdrawing money, checking bank accounts People who use it: anyone with a bank card Environment; outside and inside a bank, fixed, used at night and in sunlight Human Computer Interface screen and push buttons Decisions to make on Colour of screen Font size



Complexity of language Help Height of screen Privacy, safety ….

15.2 Contexts of useConsidering many viewpoints contexts of use

Understanding the contexts of useWho will be using it, how they will be using it and where they will using it

The people Types of people will use the product their interests, attributes Types of people will use the product their skills and knowledge Changing level of ability as people become accustomed to the product Physical and mental demands

The tasks: Goals of the system Tasks to performed, including steps, frequency, task goal, outcome, duration Flexibility of the product different approaches for carrying out tasks

Environmental context Environment of use – home – office – outside - inside Computer environment – control of, knowledge of, private, public, extremes of environment Dependencies on other products

15.3 Designing for usabilitySelection of methods that gather information as a system in use (observation of the user or asking the user to comment) Contextual design discovery process (more like learning than testing) carried out to learn the context Ethnographic study / field observation how people behave in their work environment (actual work / actual

reaction) Interviews and focus groups who will use this products (opinions and experiences Nielsen 1993) Prototyping models Affinity diagrams views about system, interface. Used by a team to organize a large amount of data according

to the natural relationships between items

15.4 Planning and managing usability testingWhat is usability testing?SIGiST, 2000: measures the suitability of the software for its users, and is directed at measuring the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments or contexts of use.Other definition / Van Veenendaal 1998: A process of planning, preparation and measurement regarding a software product and related (work) products with the objective to determine the effectiveness and user satisfaction.Assessment of risks associated with each attribute is required:

How likely is it that this will cause a problem? How large is the impact if it goes wrong?

Usability test have other aspects in common with other types of testing.

Usability requirements may need evaluation as well as testing Measurements against defined requirements and measurable acceptance criteriaHigh level Goals for effectiveness, efficiency and satisfaction X but they can not be measured or precisely specified.Static and dynamic testingNarrow and broad usability testing

Building a strategy for usability testingThree steps in Usability SIGiST Draft Standard (Four step approach overlaid of the V-model): Establish and validate usability requirements Inspect or review the specification and designs from a usability perspective Verify the implementation against the requirements (narrow focus usability testing) Validate the implementation (broad focus usability testing)Usability testing starts as early as possible. In use evaluations from survey techniques as well from information gathering during requirements gathering such as focus groups and ethnographical studies

15.5 Why is usability testing different?Usability testing is planned and managed in the same way as other types of testing (test strategy, high level and detailed plans). Significant differences: Usability testers have a special role in observing the test rather than executing the test



Sampling and sample size, as well as choice of participants to meet the context of use requires considerable planning

Designing a fair test requires additional thought compared with other tests. Usability metrics are not necessarily easy to define quantitatively The test environment must closely match the environment in which the product is to be used (this is not just the

computer environment but also the surroundings). There is a specific industry standard for documenting usability testing Usability issues may have a wider scope than testers normally deal with.

Role of the usability tester set up and control (not execute)

Sampling and sample sizes reliable: 8 users, representative of users: 5 users

Designing a fair test Instructions for the participants Allowance of time for pre-test interviews for giving instructions Allowance of time for post test interviews for receiving feedback Logging the session Observer training An agreed protocol for running the sessionsProtocols: description how the test will be carried out welcome, confirmation that this is to correct user, timings, note taking and session logging, interview and survey methods are used. Test run against agreed level of help and information.The way of instructions: The tester should give the same instructions to all participants the way instructions are given will affect the

attitude of the participants. The participants all need the instructions immediately Only one participant will be using the project at a time Examples of factors that may affect the outcome include the usability tester’s body language, tone of voice,

degree of enthusiasm, the wording which is used.

Deciding what is reasonable to measureUsability measurements must be agreedEffectiveness Completion rate (% of participants who completed each task correctly) Mean goal achievement each task completely and correctly achieved, scored as a percentageEfficiency Task time: mean time taken to complete each task Completion rate efficiency: mean completion rate/mean task time Goal achievement: mean goal achievement/mean task timeSatisfaction Measure using questionnaire, at the end of the session, giving scores for each participant’s perception of overall

satisfaction, efficiency, affect, controllability and learnability

Environments for usability testing‘In real life’ environments supports the execution, observation and recording of tests (example. Usability lab)

Reporting on and documenting usability testingCovered in SIGiST Draft standard Common Industry Format (CIF) developed by Usability Reporting project IUSR report including test purpose, product types, the quality model, contexts of use for the product and for the test (incl. Goals, and environment), metrics (normally at least one for each effectiveness, efficiency, satisfaction and where relevant safety), assessment criteria and an interpretation of the usability measures of the usability.

Usability problems and issuesProblems should be logged and tracked in the normal way Documentation, installation, messages, return codes. Function OK, but spelling not look into it.How test participant’s comments will be considered .Only one person still be regarded as a problem agreement will be needed

15.6 Designing usability testsConsiderations with usability tests:Many techniques, clear definition of the goals for the test.Appropriate narrow and broad focus measures must be agreedDecision about selection of techniques, depending on costs and risk

Specialist organisations

Complexity Costs for usability labs



15.7 Early life cycle techniquesHeuristic evaluationSystematic inspection of a user interface design for usability. To find usability problems in the design so they can be attended to as part of an iterative design process. Small set of evaluators examine the interface and judge its compliance with recognised usability principles (the heuristic)Nielsen’s heuristics: Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognise, diagnose, and recover form errors Help and documentation

Cognitive walkthroughExpert evaluators construct task scenarios from a specification or early prototype.Play the role of the user working with that interface walking through typical tasksEach step the user would take is examined blocked by the interface indicates something is missing.Convoluted and circuitous interface and software need simplifying.

Pluralistic walkthroughsMeetings with users, developers and usability professionals step through a task scenario, discussing and evaluating each element of interaction (diverse range of skills and perspectives)

Standards inspectionsTo ensure compliance with industry standards

Feature inspectionsAnalyse only the feature set of a product, usually given end user scenarios for the end result to be obtained from the use of the product.

15.8 Late life cycle techniquesThinking aloud protocolParticipants talk while using the product, to describe what they are doing, why they are doing, and their interaction to the system – they ‘think aloud’. Is recorded, given instructions before. Are observed

Co-discoveryTwo participants attempt to perform tasks together while being observed, allowing interaction and mutual help during exploration of the product. Questions asked testers ask them directly about the product. Ability or lack of it can show which parts of the product interface were obvious and which were not

SUMITo use survey techniques and attitude questionnaires both during testing and once the product is used live.

Software Usability Measurement InventoryWebsite Analysis and MeasureMent Inventory = WAMMISUMI = brief questionnaire that is marked against a benchmark of responses to surveys of systems. Proven method of measuring software quality from the end user’s point of view.

Functional testing techniquesFunctional test focus of outcomes are redefined to describe the affect of the system to the user (not functional correctness, but in terms of users’ responses to the system Can the person using the system get the level of help they require? How efficiently can they work? How attractive is the system? Are they going to use it again by choice? Did it help or hinder?Test the syntax EP, BVA make it easier to define tests across the range of context without repetition or missing context.

15.9 Additional sources of informationMany sources, many organisations, special interest groups

16 Performance testingDirk MeyerhoffGuidelines and recommendations for planning, designing and carrying out performance tests



16.1 Performance testingOften load dependenciesBenchmark for performance application or the scalability of an IT infrastructureImportant: client-server and e-business systems.Long system response dramatically for productionEfficiency: The capability of the software to provide appropriate performance, relative to the amount of resources used, under stated conditions.Time behaviour: The capability of the software product to provide appropriate response and processing times and throughput rates when performing its function, under stated conditionsResource utilisation: The capability of the software product to use appropriate amounts and types of resource when the software performs its function under stated controlsEfficiency compliance: The capability of the software product to adhere to standards or conventions relating to efficiencyPerformance issues often arise in software development projects, which assign too low a priority to performance in the early stage project phase. Recommended to begin during requirements analysis1. Start with realistic estimate of the data volumes and user behaviour to be managed2. In addition specification of acceptance response times from the end user’s point of view is needed define

measurable set of performance quality requirements3. During conceptual phases, lean throw-away prototypes can accompany key decisions in the functional model

of an application testability against standard load-generation tools will be decided4. Once the functional specification is available, performance testing activities can start

16.2 MotivationWhy conduct a performance test?Performance testing became relevant only when performance issues disturbed or even blocked production. Today: system’s architectural components are very new and there is little or no experience in integrating these components into larger system. There is usually a multitude of potential bottlenecks for system performance. Performance becomes one of the key risk factors performance testing helps to avoid such risks.Additional important reasons to asked for: What are the major technical and functional performance characteristics of the application How does the

system react to additional concurrent users / invest in more bandwidth? How can the application’s performance behaviour in produced be estimated? will call centre agents be able

to access needed information in seconds Is the current hardware-software configuration capable of managing future data volumes and what must be done

to keep application productive until another solutions becomes available? more powerful hardware? Large database into several smaller ones?

What, if any, are the differences in performance between this version and the previous one? Have software changes improved the performance?

Specified-related questions compiled beforehand and included in the project or phase test plan as major goals of performance testing. Assessment

When should performance testing be conducted and by whom?Performance issues mostly because of problems in production. Performance testing often pushed or demanded by production departments.Performance testing in production is definitely the latest possible test, also most expensive.PT as early as possible as early as functional system testing, or even better a couple of weeks before (order hardware). Even more cost effective to conduct a performance analysis and create a performance test design based on the first architecture descriptions available. Also: reviews of architecture and design documents to find and address bottleneck candidates early in the development process.Basili, 1986: Performance tests are similar to scientific experiments.PT high skilled staff with thorough technical expertise and experience in testing and tools.Most companies personnel with such profile very scarce. External consulting companies PT as a small project or outsourcing to external test laboratories. At a fixed price.Knowledge of the functionality required for PT is relatively low and any additional information needed can be acquired in interviews/review meetings/ Limiting factors are then whether: The company wants to distribute core know-how to an outside party The test environment can be set up with a reasonable effort Support from in-house experts can be guaranteed, for example via a hotline

16.3 Conducting a performance testPrerequisitesBefore begin all parties should clearly agree upon the major objectivities of the test. Better to reduce test focus then to overwhelm all participants. Few questions must be answered before PT can begin: What are the business departments’ requirements in detail (business transactions, test loads)? How many users

log in at … How can the performance test expertise that currently exists in the company be expanded? two other employees What sort of test data is needed in order to run the test? names, passwords, account numbers



Which tools can be used for PT and how can a test environment be set up to automate regression tests? capture and playback, load generation

Detailed functional and technical specification is necessary to begin preparation for Performance testing. Minimal requirements for response times, maximum data volumes, test data must be defined.Set up tools, production like environment.Core functionality minimal maturity with respect to stability and correctness

Tool evaluation and selection guidelinesUse of tools in PT is essential for an efficient test execution.Capture-playback tool at least.Load-generation tool multi-user tests.Monitoring tools measurement on the client and server.Test Management tool remote execution of processes, archiving and script organisation.For evaluation and selection tools apply standard process: Analysis of business requirements and needs Documentation checks (basic usefulness of tool for required purpose is checked using available tool documentation) On-site evaluation (most tool providers offer evaluation licenses Decision after first successful test using the toolTools offer support as part of the evaluation license . Trial of tools recommendedImportant questions for capture-playback tools: Does the capture-playback tool recognise the GUI controls (GUI objects) of the application under test? Can the recorded scripts be easily parameterised and how reliable is the replaying of scripts? What are the hardware requirements of the tool and what is their impact on the performance behaviour of the

application under test? What initial effort is required for learning how to use the tool? Can the tool access system resource information (CPU time, memory usage etc.)?Important questions for load generation tools: Does the load-generation tool support the client-server protocol of the application? How does the tool support script parameters Can scripts from a capture-playback tool be imported into the load-generation tool? How many virtual users can the tool simulate? What are the hardware requirements of the tool and what is their impact on the performance behaviour of the

application under test? Can the virtual-user script run directly on the server? Is the communication between client end server encrypted? If yes, can the tool handle this?Important questions for monitoring tools: Is it possible to measure key parameters such as network traffic, client server CPU and memory usage? Can the results be stored as protocols in the test archive? Can the results be visualised as graphs What is the accuracy and resolution of the measurement? What is the measurement error caused by the tool itself?

Test items and test cases in performance testingEntire test process remains goal-oriented performance critical functional transactions should be identified together with key representatives from the business units. Log inPerformance test results based on critical transactions are easily understood because end-users can compare the test results with their intuitive expectations. recommendations on performance improvements.50% - 70% of transactions performance relatedSystem analyses performance parameters – database configuration, simultaneous sessions – could be identified.Each test item – login, buy transaction – should be covered by at least one business workflow executed during the performance test test scenarios. Production mix all test scenarios mixed with expected operational load profile.Performance testing does not replace functional testing as only a few business process are considered. Performance test scenarios should be kept as simple as possible, not different functional, unless additional insights regarding performance are expected. Not content, rather the number and type

Test executionValidation and documentation important. Much output Immediate test result validation is therefore mandatory.Prepare test environment, tools , background test data.Automated logging is strongly recommended.

Test evaluationUsing statistical methods dependencies between technical key parameters and measured runtime data can be proved. Stability is highly recommended to observe distribution of the measured runtime data Check time and accurateness of the results (differences between functionality under high load and low load)Automated test evaluation at least import as automated test execution.For evaluation two experts: The application expert verifies whether the transactions and functions have operated correctlyThe performance measurement expert validates the measured runtime data against expectations / verifies exact performance requirements using the measurements taken.



16.4 Risk managementEstablished risk assessment helps to identify and avoid potential pitfalls during test execution. Four typical problems might occur: functional, technical, organisational and methodological issues.

Functional issues Missing or misinterpreted functional information leads to incorrect use of the application. Data, for example input data from the external systems, is missing or dependencies between data are hidden. Error messages, warnings and software outputs are misinterpreted Runtime errors and even data corruption occur due to mass data and functional defects.

Technical issues Installation of new software versions or technical components leads to significant downtimes. Capacity of (disk) storage is exceeded Additional hardware is required or hardware that is ordered is not delivered on time Load data machines are not capable of generating the required load.

Organizational issues The work of different departments adversely influences the test environment. For example, server administration,

client-administration and network experts change system settings without co-ordination. Performance test resources, e.g. network bandwidth, are shared with external parties. Key persons are not available to clarify technical and functional questions. The time needed to execute certain test exceeds the time planned and thus blocks the test environment so that for

example a conversion step for the production data is also delayed.

Methodological issues Using of an unrealistic load profile to false results. Use of an inappropriate testing environment makes reliable predictions almost impossible. Typical pitfalls for testing

environment are: Significantly less powerful machines than in production, Significantly smaller database than in production, Configuration settings different from those in the production environment. Runtime data is misinterpretedSolutions: careful planning, transparent test organisation, validity check ed regularly.Performance test can not always be transferred to reality The CPU of the application server should run at more than 90% idle time (to be used for PT). The network transfer rate should be less than 800 KB (bandwidth required for a reasonable load test is actually

available)

16.5 Experiences from real lifeServer enough resources, but application slowed down severely. BecauseThe production environment was tuned by each administration team individually.Database connection pooling was configured but not enabled type of customerMost important factors for the success of this performance test were: Do not overwhelm the testing focus. Placing too many demands on a performance test makes test design as well

as test evaluation complicated and expensive. Concentrate the focus in an interative top-down manner. A very general performance test is often much cheaper

to implement then a test designed to reveal very specific performance problems. Therefore, specific performance tests should only be carried out when performance is identified as having problems on the general level.

Facilitate communications between involved parties. User and IT departments should be involved in performance test-design and execution. Often performance requirements are defined but not generally made known to all project members. Well-prepared performance test meetings with users and IT-staff often answer all questions pertaining to performance testing.



Part 5: Incident management17 The bug reporting ProcessesRex Black (reprint of Chapter 11 from Critical Testing Process, Pearson Education, 2002).

There is an internal test process that produces results that are very visible: the bug reporting process.Bug reports are the only tangible product of the testing process itself, solid technical documentation that describes a behaviour or a potential behaviour in the system under test that could negatively affect the customer’s experience of quality.Bug reports also give management the information the need to decide the priority of the problems and provide developers with the information they need to fix them.Why is bug reporting process so critical? Bug reports are the vehicle by which we as testers influence increased product quality. Bug reporting is something that we do every day, and many times a day, usually during test executions periods. Bug reports are often highly visible, read by developers and senior managers alike.

17.1 DefinitionsBug: can be defined as a problem present in the system under test that causes it to fail to meet a reasonable user’s expectation of behaviour and quality.Reasonableness of an expectation usually decided by iterative consensus.Rephrased definition: a bug exists when the product passively fails to satisfy the customer or actively causes customer dissatisfaction.Bug = either a defect in the system under test that leads to quality-reducing behaviours or the absence of support in the system under test for quality-increasing features (that a reasonable user would expect). it is not a bug, it’s an enhancement request. Quality issue to solve in this release or the next one up to the management Word ‘bug’ is not ubiquitous. Symptom failure, failure mode, event issue.Bug report technical document written to describe the symptoms of a bug for the purpose of: communicating the circumstances, effects, and classifications of a quality problem prioritising the bug for repair helping the developer locate the underlying defect and fix it.

17.2 IEEE 1044 bug processIEEE 1004 includes four major steps1. Recognition2. Investigation3. Action4. DispositionWithin each step, three activities take place: identifying, classifying and recording.Testers are primarily responsible for step 1 and final confirmation of repair for bugs which are fixed. Step 2,3 and 4 involves programmers, system engineers, hardware engineers, project management (prioritise)

17.3 Bug reporting processA number of generic rules for a thorough reporting process:

StructureGood bug reporting begins with solid, organized testing (manual or automated, scripted or exploratory)

ReproduceBugs almost always take us by surprise. Reproducing the problems sharpens our understanding of the issue and allows us to document a crisp set of steps that will recreate the failure. Three tries is a good rule of thumb.For bugs that do not reliably reproduce their symptoms, the tester will still writes the report, but notes the frequency of symptom occurrence.

IsolateMany factors can affect the behaviour of software, and some of those factors will affect bugs change of the environment. By changing key variables, one at the time, the tester can write more insightful bug reports.Isolating a bug is not easy how the systems under test works & think through some experiments. Beware of voyage in the land of debugging or spent inappropriate amounts of time on trivial issues.

GeneralizeNot to generalize to the point of absurdity. Nevertheless, trying to find the general rule how the bug occurs deepens understanding and help convince developers and managers that the problem is not an isolated one

CompareTesting often covers the same ground run same tests against a later version of tests that cover a similar condition. Is the failure a regression? Does the same feature work in other parts of the product? When a test was blocked against a previous version, it can save the developers a lot of time if the tester can find this information.



SummarizeReason to summarize the bug to communicate to managers in a single sentence the essence and the significance of the problem testing has found.Summary is invaluable significance when it comes to time to prioritize the bug. Also a ‘handle’ or name for developers.Spend time to write good summaries, because it’s the most important sentence in the bug report.

CondenseThe best reports are neither cryptic commentary nor droning, endless, pointless documents. Use just the words needed and describe only the steps and information that the actually pertain to the bug being reported.

DisambiguateTo disambiguate is to remove confusing or misleading phrases that might make a reader wonder what is meant.

NeutralizeAs the bearers of bad news, testers needs to express themselves calmly and impartially in bug reports. No attacks, no sarcasm. Be temperate and fair-minded confine bugs to statements of fact, not conjectures or hyperbole. Write only exactly what you mean.

ReviewPeer reviews, inspections and walkthrough are powerful tools for early detection of defects. The same applies for bug reports. Try to avoid waste time of the developer by trying to reproduce a bug or when a developer can’t figure out what the bug report is about and returns it as ‘irreproducible’.

Identify, Classify and RecordThree point about the process: 1. writing is creative two good bug reports on one problem can differ in style and content. Purpose of bug

reporting process not to standardize bug report, but rather to ensure quality.2 Process works best when thought of as a checklist rather than a strict sequential process (first and lasts,

structured testing and peer review, are logical bookends, but proficient bug reporters practice all eight intervening steps in parallel)

3 Most testers use a Tool set of screens, queries, an reports (with business rules implementing a workflow) all built on top of a database

17.4 A hypothetical case studyIllustration of the bug reporting process

17.5 Classification: beyond the failure descriptionSo far focus has been on failure description, which is the heart of the bug report. Other pieces of data classifying information and other supporting data. text, identifiers (code with special significance), measurements (number, dates and other metrics), pointers or references to other information sources, and administrative information.Classification information (IEEE 10044 refers to recognition and generic anomaly process):

Project activity (Mandatory)What was the person who reported the bug doing when the report was recognized? Analysis, review, Audit, Inspection, Code/Compile/Assemble, Testing, Validation/Qualification testing, Support/Operational, and Walk-through

Project Phase (Mandatory)At what stage in the system lifecycle was the project when the bug was recognized? Requirements, Design, Implementation, Test, Operation and Maintenance, and Retirement. Some of these classifications include sub classifications as well.

Symptom (mandatory)What kind of problem does the person reporting the bug observe? Operating System crash, Program Hang-up, Program Crash, Input Problem, Output Problem, Perceived Total Product Failure, System Error Message, and Other. Some of these classifications include subclassifications as well.

Suspected Cause (Optional)What does the person reporting the bug think might be failing or causing the failure? Product, System Test, Platform, Outside Vendor/Third Party, User, Unknown. Some of these classifications include subclassifications as well.

Repeatability (Optional)As mentioned in section 16.3, how often can the person reporting th bug get the symptom of the bug to recur? --? One Time Occurrence, Recurring, Reproducible, and Unknown.

Product Status (Optional)What state the system in due to and/or following the failure observed by the person reporting the bug Unusable, Degraded, Affected (use Workaround) and Unaffected.IEEE 1044.1 additional classification data Two additional classification items: severity and priority.

SeverityThe technical impact on the system under test. IEEE 1044.01 scale of severity:1. Urgent: The failure causes a system crash or unrecoverable data loss or jeopardizes personnel.2. High: The failure causes impairment of critical system function and no workaround solution exists.



3. Medium : The failure causes impairment of crucial system functions though a workaround solution does exist.4. Low : The failure causes inconvenience or annoyance5. None: None of the above, or the anomaly concerns an enhancement rather than a failure.

PriorityThe business importance, such as the impact on the project and the likely success of the product in the marketplace.. IEEE 1044.01 scale of priority:1. Urgent: Extremely urgent, resolve immediately.2. High: Resolution requirement for the next external release.3. Medium : Resolution requirement for first customer ship or deployment 4. Low : Resolution desired for the first customer ship or require for the subsequent release5. None: Resolution not required.Priority and severity are distinct concepts and priority is the more important one. One should focus on how the associated symptoms, behaviours, and failure modes affect the customers.

One of the key pieces of information for a bug report is the tested version of the software.Integrated test management systems bug and test repositories are linked. If not test identifier in the failure description. Traceability also should be toed to specific requirements, business rules , use cases or quality risks.Also not specific configuration tested. Capture of affected subsystem, component or function Examples: User interface, Edit engine, Installation or

configuration. Capturing this information is useful for various metrics tuning on-going and future testing to focus on those areas

of the product that create most problems. Developer, release manager enter fields as well investigation, action, disposition steps. Testers could use these

fields also for metrics.

17.6 Process quality indicatorsClear, concise bug reportsTesters should write clearly and to the point. connects with their audience in their terms (jargon X well-understood project terms). Testers should produce clear, concise bug report that everyone understands.

Bugs fix ratioKaner et al, 1993: The purpose of testing a program is to find problems in it The purpose of finding problems is to get them fixedTest team must have a positive return on the testing investment in terms of increased product quality.When a bug reporting process provides managers with the information they need to prioritize bug fixing efforts and provides developers with the information they need to fix bug reports assigned to them, that bug reporting process will have a better bug fix ratio than one which does not meet these criteria.

Duplication rateBug report are about symptoms – observed anomalous behaviour – while bug fixes address defects in systems. It is certainly the case that some disparate bug reports lead to the same underlying bug. However, typically this number is very low..Most test and software professionals prefer have two bug reports on a must-fix than no bug report e.g. a test escape. Rule: Don’t spend more than five minutes to find an existing bug report. Good practice: circular bug reports via e-mail to the entire team, either as part of a review process or just for general information. Level of duplication rate down or around 5 percent is tolerable.

Bug report ‘pin-pong’Game between tester and developer where bug reports bounce unproductively back and forth. Developer it is irreproducible (better description) or inappropriately assigned to his component or piece of code (manager problem)

Delineate a clear boundary between testing and debuggingTesting is seen as a distinct set of tasks devoted to locating bugs and managing quality risk.Seven steps collaborative process to improve the quality of the system under test:

Testing / Test team 1. Can I reproduce the failure?2. Does the failure indicate a test bug or a system bug?3. What factors influence the failure

Debugging / Development team 4. What is the root cause of this failure?5. How can I repair the defect without introducing new problems?6. Is my fix properly debugged?

Testing / Test team7. Is the problem fixed? Does the system now pass the same test it failed before? Does the rest of the system still

behave properly?First three steps test activities (test team) and occur as part of or in parallel with the bug reporting process. Next three steps are debugging activities (development team) and happen as part of fixing the problem described in the bug report



of the test team. Final step confirmation and regression testing of the fix test team. These seven steps collaborative process that improves the quality of the system under test though closed-loop corrective action driven by bug report. Collaborative test team and development team.IEEE 1044 distinction between recognition and investigation and action. Disposition involves in part management guidance in terms of which bugs to defer, which bugs to fix, and who should do the fixing. Also the final step conformation and regression testing of the bug fix and if appropriate, the test release containing the bug fix.A good bug report reporting process supports this separation of duties good bug report gives the developer all the information he needs to reproduce the problem and start tracking down the bug. The bug reporting and bug management process are the only vehicles required for communication between the two teams.The hand off (figure 17.1 find /debug/confirm process) metaphor inquiry / team activity good bug reporting and management process protects the developer from having to ask the tester questions about what their bug reports means and protect the tester from having to spend time duplicating effort reproducing the problem for the developer.Tester don’t want to debug. Test manager’s position:1. The test manager has a lot of testing work for his testers to do -. He ‘s held accountable for the work getting

done. When testers help developers fix problems, they’re not testing2. Test managers can’t manage their test operators when people loan themselves to the development

organization at their discretion. Part of managing is allocating human resources to meet corporate objectives. Testers have no insight into the evolving priorities

3. Debugging assistance by testers is seldom a wise use of resources. Even he is a competent programmer, he/she is more valuable in a testing role.

4. Working side-by-side with developers on debugging may boost egos and feel like teamwork, but doesn’t build careers. Adding to developers test toolsmith and test automation tasks instead of debugging.

5. As hinted above, these activities often indicate a poor job of bug reporting being done by the tester. If developer can’t figure out what’s going wrong based on the bug report and must ask the tester, then the solution is a crisper reporting process, not tester participation in debugging.

On some occasion testers should or even must participate in debugging, especially when unique test configurations or tools are required to reproduce the problem. It’s really exception of the rule, though, and a good bug reporting process minimizes the occurrence.

No bug reports for management problem escalation Bug reports to communicate about a problem in the system under test (prioritized and possibly fixed) testing uses the communication channel the bug reporting and bug management process exclusively for that purpose. Submitting bug reports that report failures of the process rather than the product should be avoided. Talk to release manger about this and if necessary escalate it to the management.

Distinguish between test problems and SUT problemsObserved anomaly might actually be a testing problem. Such testing problems could arise with automated testing when the software driving the test fails, and it could arise for both automated and manual testing when the expected result is defined or predicted incorrectly. There is no magic bullet for eliminating testers mistakes.

Metrics support (project dashboard)The test execution process needs to support gathering key metrics for the managing testing and the project as a whole. Some metrics arise from the bug tracking system Write down all important information for these metrics. Categorize bugs appropriately in terms of affected bugs. Bugs must be complete and honest. Make sure that the process supports gathering all the data as you do so. IEEE 1044 further information on metrics to gather.

17.7 Handling challengesWriting good bug reports & managing those report to closure present the test team / test manger with a number of challenges technical (related how hard it is intrinsically to report about the unknown) and political (related to the need to manage the interpersonal and team dynamics of criticizing the work of others).

Bug or featureAmbiguity in, or a complete lack of, requirements or specifications can complicate the bug reporting process no discussion but how to resolve the ambiguity trough discussions with the various concerned parties (technical support manager, development manager, others on the management team). So long you don’t escalate, they are willing to help.

Bugs that get fixed by accidentDaily releases disturbed test execution process. To much time was spent on regression test for one bug.. Bugs fixed by accident? developers must try to reproduce the problem. Bugs don’t get typically get fixed by accident. Retest every bug by every release is off change that somehow the bugs will magically went away. Develop manger squid ink (rookgordijn) designed either:1. to obfuscate the fact that the product’s quality is absolutely bad;2. to slow down the onslaught of bugs reported by the test team.Don’t waste time of the test team by retesting bugs that no has tried to fix. Better: IEEE 1044 Recognition, Investigation, Action and Disposition. Until the proper entities have undertaken investigation and, if necessary, action on a recognized bug, then disposition work by the test team is seldom appropriate.

Irreproducible bugsSome bugs just won’t produce the same symptom again and again on command like a trained seal. Memory leaks depend on specific conditions arising repeatedly over an extended period of time before the system crashes or the application locks up. Other examples: modem connections drop out, LAN connections, terminal mishandling of inputs disrupt a user’s work and even destroy their data.



So be sceptical of claims that a bug ‘just went away (see above). Keep such bug reports open for a few test cycles (2-3 weeks) to make sure the symptom is not seen again. Solution of development team: reboot usually destroys the state data). Make sure to keep a record of every time the failure does rear its ugly head again. Some sense of the kind of system unreliability that the bug will cause is crucial to intelligent prioritization to fix effort, and means that testing need to be able to report on its frequency.

Deferring or creating test escapes?Sometimes bugs are reported that even testers consider nit-picky or insignificant. Skip this reports? Is arrogating the right to make decisions about what quality means for this product; that decisions really lies with the customer whose needs and requirements the testers usually understand only imperfectly. Testing must strive for a bug prioritization process that sees the impact of the problem through the customer’s eyes, and make decisions about which bugs to fix and which to ignore based on that.

Build trust with developersSome testers enjoy gotcha. Can create real problems with bug reporting. If developers see testing as a game of tag, then ping-pong issues are sure to arise. In addition to keeping a team-oriented mindset during testing, four specific bug reporting actions support a cooperative relationship between testers and developers:1. Keep your cool during discussions about bug reports with developers.2. Discuss bug reports with an open mind, considering the possibility that you are mistaken about correct

behaviour.3. Submit only quality bug reports and be open to suggestions for improving the quality of your bug reports.4. Be flexible on the delivery and reporting of bugs, e.g. if a developer wants a particular file or screen shot

attached to a bug report, assuming the bug tracking system support such attachments, be cooperative.In extreme cases: Developers and developers managers can overview and approve bug reports before the reports are entered into the bug tracking system.

Let the development manager drive the bug fixing processSometimes test managers fall into the trap of haranguing individual developers about when a particular bug will be fixed. Bad idea:1. Since developers don’t usually report to test managers, the test managers don’t have a lot of insight into what

other tasks the developers might have assigned to them. Interfering with their priorities won’t endear the test manager to them or to their manager.

2. Test managers have little if any authority over developers, so one likely outcome is that the managers will irritate people without achieving any useful result. Test manager should not take over one’s other team.

Testing should also avoid using the bug report data to create charts or circulate reports that make individuals look bad. Example: score of not finished ‘estimated fix date’.Developers must trust with the bug tracking system, otherwise dysfunctional events occur.

17.8 Implementing changesOther Suggestions to keep your bug reporting process under control:1. Testers have usually strong opinions about bug reporting. Be careful not to let your opinions blind you to the fact

other people are customers of these bug reports. Talk to customers, technical support people, developers, fellow mangers, test team find out what about the current process the like and dislike, as well as how they’d like to see the bug reports improved.

2. It is important to have a bug tracking system. Remember, we want to support gathering metrics so including with a good database which is able to generate metrics / analysis data

3. Exiting bug tracking system in place try to adapt it to support this process or some useable variant of it before deciding you want to replace the system. This is a big decision. Bug tracking systems are often shared with other groups release management, developers.

4. In some cases rigidity and consistency of the process are critical, but in the case of bug reporting, flexibility and sensitivity to priority are more important. Rule: every bug report undergoes a peer review. Some times reproducing, sometimes you don’t do it (irreproducible).

5. Some people have commended that they don’t see how to make the peer review work try it at least. Bad reports can waste a lot of time. Tester are in the information business, we deliver an assessment of quality and risk, so the last thing we want is for our primary, highly visible information products to be inferior.

6. The bug reporting process suggested is a lot of work. It might help to come up with a metric that can show the improvement of the process. Example: what is the reopen rate on bug reports? how many come back? Other example: closure period good bug reporting process should help drive both the numbers down.

7. What ever changes you implement patience and perseverance are important. People develops habits of working, especially it comes to tasks they do over and over again. Bug report fix rates have been achieved, e.g. closure of the bug report with an associated code change of between 70% to 80%.



Part 6: Test Process improvement18 Testing Maturity ModelErik van Veenendaal and Ron Swinkels

18.1 History and Background

18.2 TMM Maturity levelsLevel 1: Initial

Level 2: Definition

Level 3: Integration

Level 4: Management and Measurement

Level 5: Optimisation

18.3 TMM StructureMaturity level

Process Area

Key Practices

Subpractices

18.4 TMM Level 2: definition Test policy and Goals

Test Planning

Test Techniques and Methods

Test Environment

18.5 From Detection to preventionIntroduction

For development and test process improvement there are several reference models. These models are: Reference models for determining maturity Guideliness for your improvement process Proven track record

Pre-requisites for models like CMM and TMM to be implemented are: Management awareness and commitment Resources Realistic targets, based on business targets Focus not only on the process, but also on change management Commitment on all organisational levels

Also see the list of successes and failures in the Test improvement reader §6.3 page 18-19!

CMM does not say much about testing. Therefore TMM was invented. TMM is a Test maturity model related to CMM.

TMM adresses: Static and dynamic testing For dynamique testing: low level and high level testing The corner stones of structured testing: life cycle, techniques, infrastructure, organisation.

The basis of TMM consists of the terms: Maturity levels Process areas Key practices Sub-practices

TMM consists of 5 maturity levels, indicating the test maturity of an organisation.

Level Process area



Level 1: Initial naLevel 2: Definition Testpolicy and goals

Test planning Test techniques and methods Test environment

Level 3: Integration Test organisation Test training program Test life cycle and integration Control and monitor

Level 4: Management and measurement Peer reviews Test measurement Software quality evaluation

Level 5: Optimization Defect prevention Test process optimisation Quality control

For a brief description of each level, see page 336-338 of The testing practitioner.

Notice: level 2 and 3 are the ‘normal’ maturity levels that each organisation should comply with. Level 4 and 5 are mostly an extension of the scope and the use of metrics.

For each maturity level process areas are defined in which an organisation can make improvements in order to reach a certain maturity level.

Process areas are clusters of related activities within the test process, e.g. test planning. Within the process areas maturity goals are set. Each goal within a process area must be achieved to consider the process area to be achieved.

Key practices describe the activities that contribute to the effective implementation and institutionalization of a maturity goal regarding a process area. The key practices address the tasks, activities and responsibilities for management, development, testing and users/customers. Sub-practices describe activities that may be implemented in establishing the key practice. They are only informational.

How CMM and TMM are linked: table 18.1 of The testing practitioner.

As a very useful illustration: level 2 is described. This desciption is also a good overview of a ‘should-have’ setup of testing in an organisation. See The testing practitioner, § 18.4 page 339 – 343.

See also the sheets of module D-03 on this subject.

TMM is quite different from TPI:

TMM TPICMM (-I) related Tmap relatedAll test levels (incl. Static testing) Focused especially on higher test levels (ST/AT)Testing = evaluation Testware engineeringTop-down Bottom-upEspecially commitment of managament required Escpecially commitment of test engineering req.



Highly focused: only specific process area for each level

20 key areas spread across 13 (14 incl. 0) levels

Detailed Highly detailedResearch product Commercial productBook and guideline Book in public domain

Further comments on TPI: Great practical model – everyone can work with it using that thin book and key areas checklist Foundation uncertain – invented from the desk, while TMM is developed through research Especially used for a project focus – solving today’s problems quickly

19 Test Process improvementTim Koomen and Martin Pol

19.1 How good is your test process?

19.2 The modelKey areas

Levels

Checkpoints

Test Maturity Matrix

Improvement suggestions

19.3 The Improvement processObtain Awareness

Determine target, area of consideration, and approach

Execute assessment

Define improvement actions

Formulate plan

Implement improvements actions

Perform evaluation

19.4 Conclusions and remarksTPI – Test process improvement

AlgemeenDit hoofdstuk geeft een globale impressie van het model Test Process Improvement (zie: “Test Process Improvement”, 1998, Koomen/Pol).

Het model is primair bedoeld om testverbeterprojecten uit te voeren, met als doel teststraten op te zetten of te verbeteren.

TestprocesverbeteringOm te beginnen, wordt hier een definitie gegeven van testprocesverbetering; “het optimaliseren van de kwaliteit, kosten en doorlooptijd van het testproces”. Het begrip kwaliteit betekent hier; “de mate van inzicht die het testproces levert in de kwaliteit van het geteste object” en dus niet de kwaliteit van het testen systeem.Bij testprocesverbetering is het van belang het testproces te zien als onderdeel van het totale ontwikkelingsproces. Verbeteringen in het testproces zullen meestal ook verbeteringen vereisen in de andere onderdelen van het ontwikkelingsproces, zoals het ontwerpen en bouwen. Gebeurt dat niet, dan ontstaan er verschillen in kwaliteit van de diverse processen, met als gevolgen: Het rendement van de testprocesverbetering zal beperkt blijven. Een verbeterd testproces zal eisen stellen aan het

ontwerpproces, waaraan niet kan worden voldaan. Er ontstaat onduidelijkheid over de oorzaak van problemen. Wanneer er, ondanks de testprocesverbetering, nog

steeds fouten optreden in productie is dit dan het gevolg van slecht testen, of van een gebrekkig ontwerpen programmeerproces?

Een verbeterd testproces geeft een beter inzicht in de kwaliteit van de geteste objecten. Dit kan het testproces een negatief imago geven, omdat het lijkt alsof het testproces alleen maar aantoont hoe slecht de andere processen werken. In een ongunstig politiek klimaat is het dan soms beter voorlopig geen verdere procesverbetering te doen.



TPI modelHet TPI-model draait om de volgende vier termen: Key areas, Maturity level, Checkpoints en Improvement suggestions.

Key areasOm een testproces te verbeteren, moet worden gekeken naar alle aspecten in een testproces. Deze aspecten worden Key areas genoemd. Voorbeelden zijn specificatietechnieken, fasering en rapportage.Elk Key area heeft een aantal mogelijke niveaus. Hoe hoger het niveau, hoe beter het testproces functioneert op dat Key area. Voor het Key area specificatie-technieken worden de volgende niveaus onderkend:

Niveau Beschrijving Globale eisenA Informele technieken (A) De testgevallen worden meteen beschreven techniek opgesteld.

De techniek bestaat tenminste uit een beschrijving van: a) beginsituatie, b) veranderingsproces, c) verwachte eindresultaat.

B Formele black-box technieken (B) Naast informele technieken worden in de black-box tests ook formele technieken gebruikt, waarbij op eenduidige manier van testbasis naar testgevallen te komen is.Er is voor de black-box tests een gefundeerde uitspraak mogelijk over de dekkingsgraad van de testset (t.o.v. de testbasis).De testware is overdraagbaar door de uniforme werkwijze.

In de volgende tabel staan alle Key areas van een testproces. Voor de beschrijving van elk van de Key areas wordt verwezen naar The testing practitioner.

Teststrategie Commitment en motivatieToepassing fasering Testfuncties en opleidingenMoment van betrokkenheid Toepassingsgraad van de methodiekBegroting en planning OverlegSpecificatie technieken RapportageStatische testtechnieken BevindingenbeheerMetrics Testware beheerTesttools Testproces beheerTestomgeving ToetsenTestwerkplek White-box testsoorten

Maturity levelsHet totale testproces wordt geclassificeerd aan de hand van Maturity levels. Hoe hoger het niveau, des te beter het testproces in termen van kwaliteit, kosten en doorlooptijd.Er zijn 13 mogelijke Maturity levels. Elk daarvan stelt een aantal eisen aan de Key areas. Voor een hoger Maturity level moet worden voldaan aan de eisen van het gewenste niveau èn aan de eisen van de voorgaande niveaus.Nu is niet ieder Key area even belangrijk voor het rendement van de procesverbetering. Dit als gevolg van afhankelijkheden tussen de Key areas. Zo levert het weinig tot geen rendement op wanneer testtools worden ingezet wanneer er onvoldoende gekwalificeerd personeel is of specificatietechnieken gebrekkig worden toegepast.In de Testvolwassenheidsmatrix zijn de niveaus voor testvolwassenheid weergegeven:

Key area/Maturity level 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Test strategie A B C D

Toepassing fasering A B

Moment van betrokkenheid A B C D

Begroting en planning A B

Specificatietechnieken A B

Statische testtechnieken A B

Metrics A B C D

Testtools A B C

Testomgeving A B C

Testwerkplek A

Commitment en motivatie A B C

Testfuncties en opleidingen A B C

Toepassingsgraad van de methodiek A B C



Key area/Maturity level 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Overleg A B C

Rapportage A B C D

Bevindingenbeheer A B C

Testware beheer A B C D

Testprocesbeheer A B C

Toetsen A B

White-box testsoorten A B C

Om een gewenst test Maturity level te bereiken, moet dus het juiste niveau worden gehaald van ieder van de relevante Key areas. Het te behalen niveau is echter minimaal. Dat wil zeggen, bij het streven naar een bepaald Maturity level, kan er voor worden gekozen om bepaalde aandachtgebieden te verbeteren tot boven het gestelde niveau, bijvoorbeeld omdat daar duidelijke knelpunten te zien zijn.In het volgende voorbeeld wordt extra verbeterd bij Teststrategie en Toepassing fasering:

Key area/Maturity level 0 1 2 3 4 5 6 7 8 9 10 11 12 13Teststrategie A B C D

Toepassing fasering A B

Moment van betrokkenheid A B C D

CheckpointsElk niveau van elk Key area heeft bepaalde Checkpoints. De Checkpoints worden gebruikt om objectief vast te stellen op welk niveau het Key area zich bevindt. Wil men bijvoorbeeld vaststellen of het Key area Teststrategie voldoet aan niveau A, benodigd om Maturity level 1 te kunnen halen, dan gelden o.a. de volgende Checkpoints:

Er vindt gemotiveerde risicoafweging plaats, waarvoor kennis van het systeem en het gebruik en beheer daarvan is ingebracht.

Er is differentiatie in diepgang van de tests, afhankelijk van de gelopen risico’s en, indien aanwezig, van de acceptatiecriteria

Enzovoort.Wanneer een Key area, voor het gewenste Maturity level, bijvoorbeeld een C of D moet scoren, gelden niet alleen de Checkpoints bij niveau C of D, maar ook de Checkpoints van A en B. Met andere woorden, de Checkpoints zijn cumulatief.

Improvement suggestionsImprovement suggestions zijn tips voor het realiseren van een bepaald niveau van een Key area. Feitelijk zijn deze tips actief geformuleerde Checkpoints. Bijvoorbeeld, om het Key area Teststrategie naar niveau A te krijgen, kan de volgende tip worden gegeven: “Test niet alles met dezelfde diepgang, maar zorg voor een gemotiveerde differentiatie van de testdiepgang.”

Testprocesverbetering Om met het TPI-model testprocessen te kunnen verbeteren, moet een aantal stappen worden doorlopen. Deze stappen vormen in feite een veranderingsproces, waarbinnen het TPI model wordt toegepast.



Ieder van deze stappen zal nu in het kort worden besproken. Meer informatie is te verkrijgen in het boek “Test Process Improvement”, 1998, Koomen/Pol.

Stap 1: Obtain awarenessVoor iedere verandering is het nodig dat de betrokkenen zich bewust zijn van de noodzaak van de verandering. Het is aan de testcoördinator (projectleider TPI) om de betrokkenen te overtuigen van de noodzaak het testproces te verbeteren, hetzij voor de hele organisatie, hetzij voor dat ene project. Tot de betrokkenen worden ook de testteamleden gerekend, die actief zullen moeten participeren.Stap 2: Determine target, area of consideration and approachAls eerste wordt het doel van de testprocesverbetering bepaald. Doelen moeten helder, concreet en meetbaar zijn. Dit kan door ze op te stellen middels de Balanced Scorecard waar wordt gesproken over KSF (Key Succes Factor) en KPI (Key Performance Indicator).Voorbeelden zijn; meer inzicht in de kwaliteit van de testobjecten of betere overdraagbaarheid en herbruikbaarheid van de testware.Vervolgens wordt het beschouwinggebied bepaald. Gaat het om één testsoort voor één project of voor meerdere testsoorten voor de hele organisatie? Het is hier van belang het beschouwingsgebied af te wegen tegen de haalbaarheid van de doelen.Tot slot wordt op basis van de doelen en het beschouwinggebied de benodigde aanpak bepaald. De onderdelen van het TPI-model, zoals Key areas, controle- en Improvement suggestions, bieden hierbij ondersteuning.

Stap 3: Execute assessmentEen assessment wordt uitgevoerd om vast te stellen welk Maturity level het testproces nu heeft. De Checkpoints uit het TPI-model gelden hier als referentiekader. Na het assessment worden de sterke en zwakke kanten van het huidige testproces gevisualiseerd in de Testvolwassenheidsmatrix.

Stap 4: Define improvement actionsOp basis van de gestelde doelen en de resultaten uit het assessment worden de verbeteracties bepaald. Per Key area wordt bepaald naar welk niveau dit moet worden gebracht. De Improvement suggestions uit het TPI-model kunnen hierbij ondersteuning bieden. Wel zullen zij moeten worden toegespitst op de situatie waarin de testprocesverbetering plaatsvindt.Criteria om tot verbeteringen te komen zijn o.a: Snelle en zichtbare resultaten. Lage kosten. Hoogste risico’s verminderen.

Stap 5: Formulate planEen plan wordt opgesteld om op korte termijn de verbeteracties te implementeren. In dit plan wordt vastgelegd het wat, wie en wanneer van de verbeteringen, die moeten zorgen voor de realisatie van de gestelde doelen.

Stap 6: Implement improvement actionsHet plan uit stap 5 wordt uitgevoerd. Belangrijk bij deze stap is de communicatie van de maatregelen en de voortgang, en de verankering van de verbeteringen door te zorgen voor een structurele wijziging in de processen.

Stap 7: Perform evaluationBij deze stap wordt bekeken in hoeverre de verbeteringen succesvol zijn geïmplementeerd en in hoeverre de doelen zijn gehaald. Ook hier kan het TPI-model weer als referentiekader worden gebruikt.



Part 7: Test tools20 Test tool overviewErik van VeenendaalThe quality of test tools has matured during the past number of years. CASE & CAST Computer Aided Software Testing list of available test tools, classified according to application and platform.Test tools considerable improvement in the productivity of the test process. Bear in mind: Changing and evolving market

20.1 A closer lookTime to market testing is on the critical path / effective test process is necessary to ensure the deadlines are met. Tools needed to provide the necessary support.Critical success factor for the use of tools existence of a structured approach of testing.Problem for tool test tools do not always sufficiently fit in with an organisation’s structured test approachUnstructured tools counter productive (does not comply to level of repeatability and standardisation regarding activities) so : structure and toolIntroduction of methods preceded arrival of supporting tools.Activities need to be carried out accurately while they are relatively routine (comparing reports after repeated execution). Routine activities suitable for automation.Programmers have tools to generate code etc. Test designs may not be preserved between projects or shared across the organisation, and the team reinvent test designs for each project. Test cases manually and assess the results manually. Application of test tools step further in the direction of high quality testing and software.Present status of test tools support during testing is mainly possible in test management and test execution and to a lesser extent in test design. Useful tools to support test design techniques are still not available.Test tools focus on the second half of the testing life cycle model (there are exceptions). Distinction between WB and BB tools (more WB Tools).Basic every tester. Automation of test process requires specialist in depth knowledge of automation and tools test automation specialist.

20.2 AdvantagesApplying tools less effort being spent on routine based tasks. Test tools within a structured test approach advantages.Advantages of automation test process: A large number of tests can be carried out unattended and automatically e.g. at night Automation of routine and often boring test activities leads to a greater reliability of the activities than when

carried out manually, and to a higher degree of work satisfaction in the test team. This results in higher productivity in testing.

Regression testing can, to a large extent, be carried out automatically full regression tests Test tools ensure that the test data are the same for consecutive tests so that there is a certainty about reliability of

the initial situation with respect to the data Some tools e.g. static analysis, can detect faults that are difficult to detect manually. Using such tools, it is in

principle possible to find all incidences of these types of faults The automatic generation of large amounts of test data can be carried out automatically with the support of a test

tool. entering initial data sets only once. Still needed to make a focused set of test data which addresses the risksTest Tools: More activities in less time, increase quality an productivity of the test process

20.3 ConsiderationsTest automation as the solutionAutomation of the process higher level of quality / product & test process higher level of productivity.Code generators defect free software ???? answer to solve this Component Based Software (CBS) and Rational Unified Process (RUP).Best solution: combination of more than one measure. Latest technology is not always the best choice. Automation of the test process is no exception of this rule. At start (business) objectives need to be defined. Tool in testing process only if it provides improvements regarding the objectives.

Management commitmentManagement needs to aware of that applying a test tool is an investment both in terms of cost and effort, that often pays off after a period of time in terms of better and more efficient testing. Not aware usage of tool is stopped when the first problems occur.Another aspect of management commitment is the level of support it provides to structured testing.

Interpretation of tool resultsAutomation higher number of test cases. 100% statement coverage does not mean the system has been tested exhaustively. Analysing is still important to prevent misinterpretations.

20.4 Test tool overviewTest tools classified according to the activities they support



Test managementPlanning and control process, spreadsheets, word processors, risk analysis packages, automated version of TPA (test estimation) Configuration management

deliverables to be controlled management of configuration items and their changes Incident management

registering defects and tracking and monitoring their life cycle (also possibilities producing reports and statistics) Test management

control of creation, management and control of the test documentation. (test plans, specification, results). Incident management Workflow-oriented facilities

Test preparation Requirement management

automated support for the verification of requirements models, such as consistency checking and animation Static analysis

provide information about the quality of the software by examining the code, rather than running test cases through the code. Usually include checking against coding standards, structure analysis (call graphs, flow charts) and quality metrics (cyclomatic complexity)

Test specification Test design

generate test inputs from a specification that may be held in a CASE tool repository or from formally specified requirement held in the tool itself. Also generate inputs from an analysis of the code.

Test data preparation enable data to be selected from an existing database or created, generated, manipulated and edited for use in tests. Sophisticated tools with a range of database formats. Particularly useful for performance tests.

Test execution Comparators

used to detect differences between the actual results and expected results. Range of database, built-in filters to ignore rows or columns of data or areas on screen

Coverage tool provide objective measures of structural (WB) test coverage when tests are executed. Instrumented for compilation. Instrumentation code dynamically captures the coverage data in a log file and necessarily slows the program, which may affect the behaviour of the program under test. After execution log file analysed, coverage statistics generated

Dynamic analysis run-time information on the state of executing software. To identify unassigned pointers, check pointer arithmetic, monitor the allocation, use and de-allocation of memory to flag memory leaks and highlight other errors that are difficult to find ‘statically’.

Hyperlink testing tools to check no broken hyperlinks are present on the web site

Monitoring tools to gain insight into aspects such as memory usage, CPU usage, network load and performance. Also for e-commerce, e-business, web sites. Are available for customers and produce a warning if the service begins to degrade or has failed.

Performance tools (load and stress) load generation and test transaction measurement (multiple users, response time) test logs, and graphs of load against response times.

Record (or capture) & playback user-entered keystrokes, capture screen responses, GUI mouse movements, button clicks, recognise GUI objects (windows, fields, buttons and other controls). Object states and bitmap images. Programmable script language.

Security tools are typically used for testing e-commerce and e-business applications as well as web sites. Check any aspects of a web-based system that are vulnerable to abuse by unauthorised access.

Simulators to support tests where code or other systems are either unavailable or impracticable to use. Imitates the operation of the environment of (part of) the system to be tested.

Stubs, test harnesses and drivers Used to execute software under test which may not have a user interface or to run groups of exiting automated test scripts which can be controlled by the tester.Stubs: to replace a program that is called upon / called from the program to be testedDriver: calls a program to be tested.Stubs and drivers are almost indispensable during component and integration testing



21 Tool evaluation and selectionJohn WatkinsFacing the paradox of ever-shorter development and testing timescales combined with the need for software of ever increasing quality. Improvements in the management of testing process provide the means making testing as effective and efficient as possible. Another possible strategy Automated software saving time, effort and costs and improving quality.Simplify the selection of an automated tool that matches the particular testing requirements

21.1 Do you really need a tool?Consider the following: Tools cost money (could spent more effectively) Are you managing your approach to testing efficiency – and could better management techniques be a solution? Could adopting an effective development and testing process be of benefit? If you have a short term or infrequent requirement for testing, would it be more cost effective to outsource the

testing?Do not rush t o buy a test tool if your project is in crisis worst time.Benefits to be gained through the use of tools only planned and managedAppropriate to use automated test tools: Frequent builds and releases of the software you are testing A requirement for thorough regression testing, and particularly for business critical, safety critical, and secure or

confidential software systems Software involving complex graphical user interfaces The need to deliver the software across many different platforms. The need to reduce timescales, effort and costs A requirements to perform more testing in shorter timescales

21.2 The need for a formal evaluationTool must make the selected requirements and satisfy to that.5 issues for tool purchasers: Reliability of the tool’ Good match to requirements Adequate performance Ease to use Good documentationAdopting tool significant investment in time, effort and money. Tool will match your testing requirements formal means of identifying the right tool.

21.3 Identify and document requirementsFunctionality, process support etc.Determine just how important each one is and assign weighting to it to indicate its significance (Essential, Important, Desirable)Relaxing and strengthening the weighting of specific requirements

21.4 Conduct market researchIdentify these test tools that most closely match your requirements. Two-phased activity: identify a collection of candidate tools which loosely match your high level or ‘Essential’ requirements. review the candidate tools more rigorously to reject those products that fail to match your ‘Essentials’ and ‘Important’

requirements. contacting the supplier, obtaining product brochures and visiting the supplier Web sitePerforming research: testing trade magazines, special interest groups, testing exhibitions, analyst publications, internet.Output: short list Contact for details the supplier

21.5 Organise supplier presentationsCopy of requirements is there a match with the tool. Sample of your application they can demonstrate their tool using your software.Questions should be answered adequately.Review outcome

21.6 Formally evaluating the test toolMost closely match requirements chosen tool final evaluation. Proof of concept evaluation copy of the tool. Also level of support.Run evaluation as a formal project and ensure that adequate commitment is available from the senior management in terms of timescale and resources.After evaluation document the results of the project evaluation report (record of requirements, their weightings and evaluation score). Supplier can review report and give advise for misunderstood parts of the tool.Also .a business case document for senior management recommendations on the acquisition of the tools as well as return on investment calculations.



Review results decide next actions

21.7 Post evaluation activitiesNumber of licensesDiscountsSupplier offering training, support, mentoring. Beware of price slashes Consultancy visit in later stage health checks to ensure continued effective and efficient use of the toolUse the tool otherwise you will not realise benefits of it.

22 Test execution automationChris SchotanusAutomated test execution Because of modern applications with GUI (client/server and browser based applications) are very complex and contain many characteristics that require extensive testing. New development techniques and methods Object Orientation (OO) and component based development (CBD).demand high quality regression testing since applications are built during iterative processes.

22.1 AdvantagesPC ideal platform for Computer Aided Software testing (CAST). Tools / applications under tests run elsewhere.General advantages of automated test execution: Better use of available time easy repetition of the test Night, tester other tasks) Less boring activity manually =a activities become boring Consistent test execution human factor is eliminated More reliable results human testers can make mistakesImproved test planning test project less dependent to the availability of testersHigher regression test coverage testers tend to test only the changes (because lack of time). Automated run out of working hours

Four main techniques available for using test execution tools: Record and playback Test programming Data Driven Testing (specific use of test programming) Testing with Action Words

22.2 Record and playbackRecords key strokes and mouse movesFirst time test actions are carried out manually and recorded by the tool. Checks on screen. Script file technical nature (not easy to understand by non-technical testers). Mixture of test data with technical information and the flow (sequence of actions)Reasons for using R&Pb: Easy to create no need for technical knowledge of the test tools by the testers Once a script has been created it is easy to repeat one keystroke is enough provided the application does not

change Tests can be executed outside ‘office hours’ Disadvantage: test script file is difficult to maintain. All action recorded in one file containing all information (both technical and test) in one large sequence screen changes also script changesAlso recording test script only after delivery of a correctly working system only applicable as regression test method

22.3 Test programmingReduces the maintainability problem and increases the readability of the script. Use of R&Pb toolScript language possible to create routines that handle standard actions. Possible to separate technical issues from test dataRecording facilities of the test tool are used initially to create the script. After recording editor to change script.More readable and maintainable than a script created during record and playback. Al technical items, that are sensitive to changes, are programmed in separated functions. In case of changes in the application only the functions may need to be adapted and the calls of these functions in the main script remain the same which reduce the maintenance effort.Test data and test flow are still stored in the test script itself, easily accessible by non-experts like end-users.

22.4 Data driven testsSpecial implementation of the test programming. Scripts are programmed and contain all the functions for entry and checking information. Test data needed for test is stored in separate files either in spreadsheet or ASCII format.In the script a main loop is programmed that will read the file record by record and process the dataLook a like test programmingAdvantage: separation of test input data from the script increases readability and reduces maintenance and data dependency. Data changes only in data file. Various files can be used (reusability of the script)



Only data in the pre-defined layout can be used. Different record lay-out create another script.Useful for large data entry but less usable for testing based on user scenarios.

22.5 Testing with action wordsCan lead to improvements in test execution process especially for regression testing (handle with care).ProblemsLack of maintainability R&Pb and test programming there is a lack of maintainability. Menu structures changes imply substantial effort to keeps the tests running or for R&Pb recorded again Accessibility of both the tests and test results action and data are mixed hard to understand what exactly is tested. Data driven testing is improvementResult checking is done by special functions of the tool screen contents are compared with the file that contains the original one/ Comparing pre-defined results with actual outcomes cannot be doneR&Pb only if the system is running

Testing with action words testframe. Tests are stored in spreadsheets test clustersTest language specifying actions to be taken and the test data.Test data input or (expected) test outcomes. Actions action words(short commands to the test tool)

Test clustersContains the data that will be used during the test input data and the expected results. Difference from data driven: test data but also the test instructions are specified in he cluster (column A: action word En column B: data used with the action wordSpecial &-function to retrieve a unique item.Action words usually specific for the application that is tested.

Navigation Test automation is separate activity, completely separate from the test design.To interpret and execute the commands in the test cluster a special script is written, so called navigation script written in the script language or R&Pb tool.Scripts: several components general (functions for reading the test lines from the tab separated ASCII text file). Specified procedure for action words

Test reportsConsist of a header, a detail part and a summary. Content dependent of the project (failed, succeeded). Failed are marked and analysed not necessarily wrong Reports results, data, version information etc.

22.6 ConsequencesTesting with action words has been applied in number of large and small projects, both on-line and batch systems. Number of efforts: Flexibility test set, test clusters, navigation script more maintainable. Changes in most cases only the

navigation has to change. Change in menu only change action word. Accessibility of test clusters and the reports end users can prepare test cases by setting up the spreadsheets Planning can start in early stage of a project, without a knowledge of all functional details of the system.

Business oriented test cases as soon as requirements are available. Test: in phase of detailed design or implementation

Benefit in cost effectiveness. Implementation of navigation is an additional activity in a project can be compared with the manual testing effort for the first test. Here first test is already automated (instead of R&Pb)

Increase in motivation. Less interesting parts are carried out by a computer and separation of test design and navigation people work on the job they like most and they can do best (business X technically)

Test automation: reliable, run all tests, catching notorious unexpected faults resulting from unexpected changes in other parts of a system.Introducing test automation not lightly. People need to be trained, technical infrastructure needs to be implementedTools have to be supported. Ease planning, programming and data driven testingTesting with action words provides the most flexible solution, but requires the largest investmentWhich techniques depends on organisation / project dependencies business objectives, risks of the system-under-tests, system life cycle, skills available.Main goal test automation: reduce effort needed for test execution and increase the accessibility of the test by non-technical personnel.



Part 8: People23 Team BuildingChris ComeyTesting is recognised as profession (training, career)

23.1 Steaming along or just ploddingDeliverer teams are as effective and efficient as possible. Having the right skill, skills are applied at the right time and place in the project and that the member of the team understand their role and what is required from them. Puul together and communicate effectively, sharing success as nay issues. To succeed confident team that want to achieve, understand the difference that they make the project, and can be proud of what they do.

23.2 What makes a successful team? The correct combination of skills and experience The correct combination of people The correct leadership and management The correct process and controls

Skills in a test teamImpossible to list the specific technical skills for all testing IT projects. Information detailing these variables is made available to the development en testing teams as soon as possible in the project lifecycle

Specific Testing Skills Will need good general knowledge of testing and the influence of testing throughout the development of software

systems. Knowledge of how testing fits into the different lifecycles Knowledge of different test levels and approaches from component testing to acceptance testing and the related

entry and exit criteria for each phase The testing professional should be prepared for the iterative nature of regression testing Knowledge about change control and configuration management and the risks associated with failure of any

part of the development or deliverables from development The tester will need knowledge of how to help management to create a suitable policy for quality in general and

testing in particular knowledge of how to establish strategies for traceability of test objectives to user required functionality, formal

test specification and design techniques. The test professional should be familiar with both external and site-specific standards.Additional (specific skill for an individual to be a good tester): Experience of development / coding is an advantage Experience of Static and dynamic test methods Technical support experience is useful for the test environment support Data analysis experience for production of test data Business analyst experience Automated testing tools experience (CAST-tools) Experience of metrics Knowledge of improvements models such as CMM Knowledge of test process improvements (TMM and other models)

Team: diversity of skills

Phase Specific Skills

Component Test Knowledge of coding language Knowledge of the code under test Experience with WB testing techniques path analysis, statements & branch coverage, LCSAJ etc. BS-7925-2 Knowledge of static analysis techniques (complexity measures, paths) Experience with static tools (complexity measures, syntax testing, pre-compilers) Knowledge of code inspection

Integration Testing in the small All the skills for component test will also apply for integration testing in the small. Other experience that may be

useful would be: Experience of integration strategies Experience of testing component interfaces Experience of testing using variables and data exchanges

System Test Knowledge of WB and BB test techniques Knowledge of functional and non-functional testing Experience of dynamic test tools (capture playback tools)



Experience of test management tools (test planning and tracking, incident management and configuration management tools)

Experience of test automation

Integration Test in the large Experience of interfaces and files Knowledge of the business processes from end tot end An understanding of operational profiles.

Acceptance Test An understanding of the principles of testing Expert business knowledge from each business area such as operations, accounts, finance, compliance, live

support etc. An understanding of requirements test coverage Experience of test case design and execution methods

Useful skills for all stagesKnowledge of static analysis techniques (reviews and inspections etc.)Knowledge and experience of static and dynamic test toolsAn understanding of project management, incident management, configuration management, delivery management, and implementationKnowledge of test environments and test data requirements at high level (depends of which level component testes needs to understand stubs and drivers)

23.3 People in a teamBelbin, 1993: Finite number of behaviours / team roles compromise patterns of behaviour adopted naturally by the various personality types found among people at work.Belbin helps to identify the personal characteristics in each individual to identify their natural strength and weaknesses. Identifying Each members’ strength and weaknesses they will know what they are good at and, more importantly, where they need to develop.Once knowing people’s natural roles their strengths and weaknesses are known.

23.4 How to build a more effective teamEveryone is different and each individual is motivated differently. The more each individual’s needs are met by the team, the more each individual will feel part of the team More individual input and a better overall team achievementIf you won’t paddle then don’t led them on the boat; leave them behind. If they paddle slowly then everyone else has to paddle that much harder. Not prepared to paddle then we are going to drown.

23.5 ManagementYou need to do what you are doing, believe in what you are doing, and you must take the initiave in all the things that involve your team.Some attributes of a good manager Communicate and encourage communication Lead by example Hold progress meeting and share successes as well as problems Encourage people to take part in team meetings and canvas the opinions Involve people in the work allocation process Get individuals to take ownership of tasks and be accountable to you for their completion Encourage group problem solving Identify individuals strengths and weaknesses diplomatically but openly Identify individual career paths, training requirements, and personnel development plans Introduce mentoring, shadowing, or buddies to promote cross fertilisation of skills and reduce single points of

failure in individual team members. Foster open and honest communication in both directions Maintain individual’s self esteem Encourage team members to take responsibility and make tem accountable where appropriate Do something a bit different every now and then, a curry lunch maybe, a night out somewhere, try and make it

mandatory If they are working in accordance with your guidelines ensure they know that you will always back them up and

fight their corner for or with them.Remember a good leader is a skill too, and not everyone is a natural leader.Can be trained; open for constructive criticism. Poor leader Bad experiences /practices Team leader not contributing fully take him/her to one side and ask what is the problem. Always ask first before taking any from of action. Come to understanding.Different characters in team confrontation between team members is possible make decision carry out as a sort of peace brokering. Successful team should be able to function without any single individual being present.

DelegationYou can delegate the responsibility but you cannot delegate the blame.



Regardless the excuses:Too busy check workload first and agree ownership Didn’t understand Ensure the task is clear Some else came up need them to inform you Didn’t know how you chose a person with insufficient skill yet Thought you meant ‘next’ Monday communicationChoosing which task can be delegated and to whom is a fine art.Micro manager who cannot delegate anything to anyone have no confidenceAbsentee manager always busy doing something else, never has time for meetings with his staff and pushes everything down the line onto his team. poor communicator, hard to pin down, seldom returns his calls. He doesn’t know what to do first.Fortunately most managers between them. Do not aim to perfection. Apply 80/20 rule

Individuals versus TeamsDo you believe that teams are better than individuals?Experiment: Take twenty tricky questions on general knowledge and ask them to individuals and teams

23.6 Retaining test expertise Testing is not dull.Key points of he process Position testing as a multidisciplinary discipline stress the variety of the work and the opportunities for

assignments and personal development Show clearly what career opportunities are, as this will demonstrate what a position or career in testing has to

offer. Put supporting training courses in place, this will motivate employees and show them what career opportunities

are and facilitate the achievement of their goals and ambitions Ensure the right man/woman is in the right job: in consultancy this means greater focus on matching the right

test consultant with the right assignment Implement internal marketing activities and PR, to improve the image that testing has However the most important task is still: talk to your employees, ask them what they want to achieve and help tem

to achieve it.Taking action to improve situations, and monitoring the success of the process

Rewards – Motivators or De-motivators?Rewards = recognition of outstanding contribution. But actual effort has been recognised that motivates people (not bonus payments).Beware! This subject could be a minefield. Recognising of an individual be sure you are not de-motivating other team members or damaging any individuals standing within a team by singling them out.Who makes the significant contribution? Reward self-promoters You will not encourage them and will de-motivate the hard workers.Team bonuses are a much more productive approach. Peer pressure is strong force. Bonus: cash or a special day

23.7 Your career belongs to you!Develop technical and soft skills. Ask what your weaknesses are. Mentoring and shadowing. TrainingOther ways to Increase your knowledge and awareness Read the books that have been written on the subject of testing Subscribe to the testing magazines and circulars Join the appropriate professional organisations for testers Attend the relevant testing focused conferences Achieve formal qualifications in testing Listen to others. Even if you don’t agree with what they are saying, they may just point out another angle that you

have not considered.A testers we need a wider range of skills to allow us to handle the ever-increasing rate of change in our industry. Learn strengths and weaknesses, learn to integrate with their colleagues, and learn to work more closely.

23.8 How do we spot a good tester?Testers need to Be skilled and experienced Be confident Be good communicators Be conscientious Be self motivated Be reliable Be tenacious Be pedantic Be diplomatic Pay attention to detail Stay calm in a crisis Have a sense of humour



Be stress resistant Be accurate

Individual’s character by asking a few simple questions: When you interview potential employees take some time to try and assess their character, what their strengths

are and how they will fit into the team. Look not only at the answers that they have given to any technical questions but also the way in which the

questions were answered. Did they think about the answer or offer an answer straight away, did they ask for further information when the questions was not clear, did they confirm their understanding before answering the question?

You could ask them a few opinion questions to see if they have views or strong feeling on certain subjects. ‘How do you feel about testing without any CM in place?’ Should be enough to render a strong response if indeed it does not send them scurrying for the door.

Offer the candidate the change to ask you any questions that they may have, listen to what they want to know. It can give you information on their level of experience or what is important to them.

Informal questions hobbies, worst boss, greatest achievement at work?Produce a small Visual Basic application to be tested.

23.9 Communications7% is spoken works, 93% of the communication is relayed from the context of the conversation, the emphasis placed on the words by the speaker, the tone of voice, body language, facial expressions. So beware the written word.Communication is of course a two-way thingThird person avoid jargon, dreaded three letter abbreviationsIn Reasonable proximity to each other Personal relations break down main contributing factors: communication breakdown

Making decisionsTrained experienced professionals rarely make a wrong decision based on information available to the person making that decision.The problem arise where information is available but not distributed.Rapid changes in ITTest team involved is involved with every aspect of the project from requirements capture stage to support for the live operations team once the product is alive.Test team interface with all parties: Customer (requirements analysis , environment, acceptance testing, test execution) System users learn from the system users how the system is used in real life, what problems exists Business analysts assist the customer (requirements, acceptance test cases) Project managers ultimate responsible Design team key role/ large interaction with them Development team Interact regarding faults, change management Technical support team regarding live faults, quick fixes, potential workarounds Technical authors system operation and performance Configuration management team Change control Board Fault management area will decide the action to be taken for each individual fault (prioristised etc.)They work together

Keep talking even if it means disagreeingConstant communication is what keeps surprises at a minimum

Don’t confuse meetings, memos, and e-mail with effective communicationThey are vehicles not the message. Communication: not only what you say, but also how you say it.

23.10 The end gameKeys to a successful team: Have a clear, common understanding of the team goals Have the technical skills to enable the team to carry out all composite tasks Have the personal skills to allow the team work together Have the management, processes, and controls to allow the team to function efficiently without excessive

constraints Communicate effectivelyCareer path for testers train them, increase their skills, motivate them and reward them

Setting up balanced teams is a lengthy task, but it will pay dividends in the long run, for the individual, the team and the company

24 Test career pathsRuud TheunissenSkills and attitude are equally important. Keep them motivated, fascinated, enchanted



24.1 Three dimensions

Career perspective must be offered. From test engineer to test manager (first dimension). Career cube allows testers to differentiate technical, methodical and managerial (second dimension). Final dimension is the backbone for each successful career theoretical background, training, coaching and (social) skills.

The first dimension: GrowthGrowth tester to test manger, test engineer tot test advisor. Growth directly related to responsibilities and the involvement an employee has within a project or the organisation. Test engineer execution to preparation to co-ordination to planning.Vertical growth and horizontal expansion.

The second dimension: DifferentiationTesters are individuals. Same basis: preparation and execution of test scenariosThree different career paths:Team and project leadership leading a test projectMethodological advice give advice in test strategy, selection and application of test specification techniques etc.Technical advice testing opt utilisation of tools or set up and management of a test environmentD General test managementC Test project leadership Methodological test advice Technical Test adviceB Test tea leadership Methodological test specialisation Technical Test specificationA Test executionFunctions and levels

The third dimension: Knowledge and skillsFollowing components are distinguished:(test) trainingsocial skillsexperiencecoaching and supportCoaching and support essential conditions for a sound career path, especially for the lower function levels

24.2 The test career cube Testing is a profession and companies should treat it as one career opportunitiesKeep testers motivated ‘test’ your testers continuously and appreciate them by offering them a career and paying them. The right salary.



AdditionalRisked based testing Testing is Risk Management Evaluate where the areas of high risk are:

What is the likelihood? What is the impact?

Spend the available time and resources testing to mitigate the highest business risks

Test levelsEach test level has the following characteristics Objective Responsibility Entry criteria Exit criteria Test deliverables Techniques Tools Standards Typical non-functional test types

Test Strategy detailsThe test Strategy details The following high level description of: Objective, Responsibility Level of independence (test organization) Entry criteria and Exit criteria for each phase Test process and milestones deliverables Approach (top-down, bottom-up, priority driven) Test case design Techniques Type of tools that will be applied Standards that must be adhered to The environment in which the test will be executed. Typical non-functional test types

Key Learning PointsA series of test planning documents Test Policy – the organization philosophy Test strategy – the high level document defining test approach Project Test Plan – detailed project document defining test phases and content Phase Test Plan - detailed project document providing all testing and detailed schedules for a phase Test Specification

o Analysis – Identify test units and test coverage itemso Design – Create test units that exercise test coverage itemso Build – Organize test procedures / scripts, test data and environments

Test cases – measurable, objective, repeatable, risk prioritized A source for test specifications: BS7925-2, 1998 Note: Test specifications will also be needed for non-functional testing (different techniques / skills)

Test plan outline1. Test plan identifier2. Introduction3. Test items4. Features to be tested5. Features not to be tested6. Approach (wat is haalbaar?)7. Item pass/fail criteria8. Suspension criteria and resumption requirements9. Test deliverables10. Testing tasks11. Environmental needs12. Responsibilities13. Staffing and training needs14. Schedule15. Risks and contingencies16. Approvals



Test plan outline (details) Test plan identifier Introduction

o Short description of what to be tested and will not be testedo Reference documents (main plan only) --> project assessment, project plan, quality assurance plan etc.o May act as a management summaryo Derived from risk analysis and test strategy

Test itemso Test items including their versiono Include risk levelo Specify characters of transmittal media o References to the documentationo Reference incident reports related to test itemso Items that are to be specially excluded from testing should be identified

Features to be testedo Detailing test itemso Identify all software features and combination of software features to be tested o Define the features that not will be covered (reason)o Include risk levelo Often acts as non-functional attributes to be testedo Reference design documentation

Features not to be tested Approach (wat is haalbaar?)

o Specify major activities, techniques, tools that are used to test the designated test items and/or groups of features

o Level of impendenceo Define metrics to evaluate coverage and progressiono Approach should be different per risk levelo Identify significant constraints regarding approach

Item pass/fail criteriao Specify the criteria to be used to determine whether each test item (or group of item) has completed

testingo Test Processo Number of outstanding defects (per priority)o Use standards (ISO 9126, part 2 & 3)o To provide unambiguous definition of the expectations

Suspension criteria and resumption requirementso Suspension: waarom stoppen (al vastleggen aan het begin --> te veel fouten, niet-reproduceerbare fouten)

At start and during testing To define when to temporarily stop testing

o Resumption: Specify the testing activities that must be repeated, when testing is resumed Test output (development), built log, regression test To define tasks to execute when resuming tests

Test deliverableso Test plan (and sub-plans if present)o Test design specifications (one or more documents)o Test case specifications (one or more documents)o Test procedure specifications (one or more documents)o Test scenarioo Test logs (milestone)o Test incident report (milestone)o Test summary reportso Test evaluation reporto Test software

Testing taskso Direct and taskso Identify all intertask dependencieso Identify any special skills required

To prepare requirements for resources To verify all deliverables can be produced

Environmental needso Hardwareo Toolso Any other testing needs (office)

Responsibilities



o Identify the groups responsible for managing, designing, preparing, executing, witnessing, checking an resolving

o Identify the groups responsible for providing the test items identified in 3 and the environmental; needs identified in 11

o These groups may include the developers, testers, operational skills, user representatives, technical support staff, data administration staff, and quality support staff

Staffing and training needso Specify test staffing needs by skillso Identify training options for providing necessary skills for

The level of technical (testing) expertise required The number and type of resources needed The hardware and software required

Scheduleo Include milestones identified in the project schedule as well as all item transmittal eventso Define any additional test milestones neededo Estimate he time required to do each testing tasko Specify for each testing task the test milestoneo Specify the schedule for each testing task and test milestoneo For each testing resource specify its periods of useo Enable other teams to plan accordingly

Risks and contingencieso Identify the high risk assumptionso Include likelihood, impact and triggero Specify contingency plans for each (delayed delivery)o For each component, task, resource, schedule, look at the risk if it becomes unavailable , is delivered lateo To be prepared if problems arise

Approvalso Specify the names of all persons who must approve this plan. Provide space for signature and dateo To have everybody buy into your test plano To avoid discussions when defects are found or testing is doneo Contact each of the stakeholders

Test assignment Test objectives Expectations Sponsor Responsibilities test team Scope of test Constraints

o Project planningo Policy, strategy and standardso Time, resources

N.B. Input for test plan

Risk management Risk identification Risk analysis Mitigate the risk Monitor and track the risk

Risk Identification Techniques Expert Interview Independent assessment Risk templates Lesson learned Risk workshops Brainstorming Checklists Failure Mode and Effect Analysis (FMEA)

FMEA risk identification1. Defining FMEA field of application in Process and Product2. Preparation of FMEA workshop3. Failure Mode and Effect Analysis



4. Critically Analysis to prioritize the identified failure modes5. Definition of measures to elaborate confidence w.r.t. identified key-areas6. Reporting and follow-up

Other method: Functional risk matrix --> determine importance components

Risk (importance = likelihood * impact

Likelihood on defects Complexity New development (level of re-use) Interrelationship (# interfaces) Size Technology Inexperience of the development team

Impact User importance (selling item) Financial (or other) damage (e.g. satisfy) Usage intensity External Visibility

Compl. New Interfaces Size Techn Inexep Import Intens VisFunction 1Function 2Function 2

9 = critical5 = High3 = Moderate1 = Low0 = None

Functional risk matrix

Could Test Must TestLI 54kelih 27ood

013 27

Won’t Test Sould test


I II

IV III