2012 q1 Reid Mette

7/31/2019 2012 q1 Reid Mette

1/32

ISO/IEC 29119The New International

Software Testing Standard

Stuart Reid

Testing Solutions Group

117 Houndsditch

London EC3

UKTel: 0207 469 1500Fax: 0207 623 8459

www.testing-solutions.comStuart Reid, 2012
http://www.testing-solutions.com/http://www.testing-solutions.com/http://www.testing-solutions.com/http://www.testing-solutions.com/

7/31/2019 2012 q1 Reid Mette

2/32

Scope

The purpose of standards

Overview of ISO 29119

Applicability

Recent developments

Timeline

Future work

7/31/2019 2012 q1 Reid Mette

3/32

What are standards?

Guidelines documents as they are not compulsoryunless mandated by an individual or anorganization

Agreements because they often reflect a certain

level of consensus

Guidelinedocumentation that reflectsagreementsonproducts,practices, oroperationsbynationally orinternationally

recognizedindustrial, professional, tradeassociations or governmentalbodies

- ISO

7/31/2019 2012 q1 Reid Mette

4/32

Pre-standardization?

7/31/2019 2012 q1 Reid Mette

5/32

Why use standards?

Consumers

Confidence in compliant products

Authors provide expertise in standards

Manufacturers

Conformance and Marketing

Safety from liability

Guidelines on production

But not Best Practice

7/31/2019 2012 q1 Reid Mette

6/32

Quality and Standards

7/31/2019 2012 q1 Reid Mette

7/32

What use are standards?

Basis for:

Communication common terminology

Professional qualifications

Certification/compliance schemes Benchmark of good industry practice

Contracts

Interoperability and consistency

Standards describe a current body ofknowledge that provides the basis for aprofessional discipline

7/31/2019 2012 q1 Reid Mette

8/32

A Case in Point

7/31/2019 2012 q1 Reid Mette

9/32

Standardization Bodies
http://www.brandsoftheworld.com/download/brand/61142.htmlhttp://www.sis.se/Default.aspxhttp://www.bsi-global.com/British_Standards/index.xalterhttp://www.omg.org/http://www.iec.ch/

7/31/2019 2012 q1 Reid Mette

10/32

ISO/IEC Software Standards

0

20

40

60

80

100

120

140

1

987

1

988

1

989

1

990

1

991

1

992

1

993

1

994

1

995

1

996

1

997

1

998

1

999

2

000

2

001

2

002

2

003

2

004

2

005

2

006

2

007

2

008

2

009

2

010

Published

Maintained

7/31/2019 2012 q1 Reid Mette

11/32

Motivation for ISO 29119

Conflicts in definitions, processes & procedures

plethora of standards to be replaced by one

e.g. IEEE 829, IEEE 1008, BS 7925-1/-2, IEEE 1028

practitioners do not know which standard to follow Lacking in current standards:

Organizational Testing not covered

e.g. Test Policy and Organizational Test Strategy

no Project Test Management BS 7925 only covers unit testing

common functional techniques missing

poor coverage of non-functional testing

7/31/2019 2012 q1 Reid Mette

12/32

ISO/IEC SC7 WG26

ISO

TC176

Quality

JTC1

InformationTechnology

SC7

Software &Systems

Engineering

SWG1 ..................

WG26

SoftwareTesting

Representativesof National

Standards Bodies

SC22

Programming

Languages

SC27

IT Security

SC32

DataManagement &

Interchange

IEC

TC56

Dependability

7/31/2019 2012 q1 Reid Mette

13/32

BS 7925-1

BS 7925-2 IEEE 829

Concepts & Vocabulary

Part 1

ISO/IEC 29119 Structure

Process

Assessment

Part 5

TestingTechniques

Part 4

Documentation

Part 3Part 2

Processes

Actually ISO/IEC 33063

7/31/2019 2012 q1 Reid Mette

14/32

SOFTWARE TESTING CONCEPTS

Scope, Conformance, Normative References

TESTING IN DIFFERENT LIFE CYCLE MODELS

ROLES AND RESPONSIBILITIES IN TESTING

ANNEXES Metrics, Examples, Bibliography

DEFINITIONS

Part 1: Concepts & Vocabulary

7/31/2019 2012 q1 Reid Mette

15/32

Part 2: Testing Processes

TEST MANAGEMENT PROCESSES

ORGANIZATIONAL TEST PROCESS

DYNAMIC TEST PROCESSES

7/31/2019 2012 q1 Reid Mette

16/32




Instantiating Testing Processes

7/31/2019 2012 q1 Reid Mette

17/32



TEST

PLANNING

TEST

MONITORING &

CONTROL

TEST

COMPLETION

ORGANIZATIONALTEST

DOCUMENTATION

FEEDBACK ONORGANIZATIONAL TESTDOCUMENTATION

TEST PLAN UPDATES

TESTPLAN

TESTCOMPLETIONREPORT

DYNAMIC TEST

PROCESSES

TEST

MANAGEMENT

PROCESSES

TEST PLAN,TEST COMPLETIONREPORT,TEST MEASURES

TESTMEASURES

TEST PLAN,CONTROLDIRECTIVES

TEST PLAN,CONTROLDIRECTIVES

Test Management Processes

7/31/2019 2012 q1 Reid Mette

18/32

OrganiseTest Plan

Development

Identify &Estimate Risks

Design TestStrategy

DetermineStaffing and

SchedulingDocumentTest Plan

Schedule,Staffing Profile

TestStrategy

AnalyzedRisks

Scope

Identify RiskTreatmentApproaches

GainConsensuson Test Plan

ApprovedTest Plan

DraftTest Plan

TestPlan

PublishTest Plan

UnderstandContext

TreatmentApproaches

Test Planning Process

7/31/2019 2012 q1 Reid Mette

19/32

Test Monitoring &Control Process

Monitor

Report

measures

ControlSet-Up

[testingcomplete]

[testing incomplete]

controldirectives

testprogress

info

test progressinfo

testplan

testmeasures

test status report

testcontrolinfo

...TEST PROCESSES...

Dynamic/Management

7/31/2019 2012 q1 Reid Mette

20/32


TESTSPECIFICATION

[NO ISSUESNOTICED]

TESTENVIRONMENTREADINESS

REPORT

TESTENVT

REQUIREMENTS

INCIDENTREPORT

TEST DESIGN &

IMPLEMENTATION

TEST

ENVIRONMENT

SET-UP

TEST

EXECUTION

TEST

INCIDENT

REPORTING

(LEVEL) TEST MANAGEMENT PROCESS

[ISSUE NOTICED or

RETEST RESULT]

TESTRESULTS

CONTROLDIRECTIVES TEST

MEASURES

(LEVEL)TEST PLAN

Dynamic Test Processes

7/31/2019 2012 q1 Reid Mette

21/32

Test Process Descriptions Example Organizational Test Process

Each testing process is described using a standard template (followingISO 24774), for example:

Purpose

The purpose of the Organizational Test Process is to develop and maintain organizational test specifications,such as the Test Policy and Organizational Test Strategy.

Outcomes

As a result of the successful implementation of the Organizational Test Process:

The organizational test specification is developed based on stakeholder requirements;

The organizational test specification is agreed by stakeholders;

The organizational test specification is published and accessible to stakeholders;

Conformance of stakeholders with the organizational test specification is managed;

The organizational test specification is reviewed on a periodic basis;

Minor updates to the organizational test specification are made as necessary.

Activities and tasks

The tester shall implement the following activities and tasks in accordance with applicable organization policiesand procedures with respect to the Organizational Test Process.

Develop Test Specification

This activity consists of the following tasks:

Analyze any relevant source documents and the current testing practices within the organization toidentify requirements for the organizational test specification.

etc.

Gain Consensus on Test Specification

etc.

Information items

The output of the Organizational Test Process is the organizational test specification. The Organizational TestPolicy and Organizational Test Strategy are typical examples of organizational test specifications.

Note that ISO 24774 (and this format) has changed since we started, sowe have had to rewrite all the process descriptions

7/31/2019 2012 q1 Reid Mette

22/32

Part 3 Test Documentation

TEST DOCUMENTATION

ANNEXES - EXAMPLES

Scope, Conformance,Normative References

7/31/2019 2012 q1 Reid Mette

23/32

Part 3: Test Documentation

Organizational test documentation Test policy

Test strategy

Project test documentation

Project test plan

Test project completion report Test Level documentation

Test plan

Test specification

Test results

Anomaly reports Level test status report

Test environment report

Test level completion report

Appendices

examples of documents at each level of testing

7/31/2019 2012 q1 Reid Mette

24/32

Part 4 Test Techniques

TEST COVERAGE MEASUREMENT

Scope, Conformance, Normative References

ANNEXE

TESTING OF QUALITY CHARACTERISTICS

ANNEXE SELECTION OF TECHNIQUES

ANNEXE TEST TECHNIQUE EFFECTIVENESS

TEST DESIGN TECHNIQUES

BLACK BOX WHITE BOX

7/31/2019 2012 q1 Reid Mette

25/32

Applicability

Intended to be a generic standard support all domains

Some domains are more standards-oriented, e.g.

safety-related

telecoms

Some domains will become more regulated, e.g.

financial banks, stock markets, etc.

Needs trialling in all domains and project types

Applicability to sequential, iterative and agile Applicability to new and maintenance projects

Applicability to IS and embedded

E.g. when wouldnt you be selecting and usingtechniques? When wouldnt you plan?

h

7/31/2019 2012 q1 Reid Mette

26/32

When can you start using itand when can you get involved?

Typical ISO standards take over 7 years

ISO 12207 was conceived in 1988 andpublished in 1995 and represents 17,000

person hours (8.5 person years)

BS 7925-1 & -2 took 8 years to develop

IEEE estimates 2-4 years to develop astandard, at a cost of between $2,000 and$10,000 per page [1998]

7/31/2019 2012 q1 Reid Mette

27/32

May

10

May

11

May

12

May

13

Timelines all going to plan!

Working Draft (WD)Committee Draft (CD)Draft International Standard (DIS)Final Draft International Standard (FDIS)Final International Standard (FIS)

Parts 2 & 3

Parts 1 & 4

WDCD1

DIS FDIS

FIS

WD

CD1

DISFDIS

FIS

CD2CD3

CD2CD3

7/31/2019 2012 q1 Reid Mette

28/32

Do you want to be involved?

Join the working group

representing your national standards body

6 day meetings, every 6 months

contribute between meetings

Join a mirror group

for your national standards body

Contribute materials

Review drafts

7/31/2019 2012 q1 Reid Mette

29/32

Current status

Drafts are out for review Unsurprisingly (for a testing standard) we get

plenty of comments (1000s)

Countries attending meetings include:

Denmark, India, China, UK, South Africa, HongKong, Spain, Germany, USA, Finland,Australia, Korea, Canada, France, Poland,Japan, Russian Federation, Columbia, Sweden,

Brazil, Malaysia, New Zealand, Argentina,India, Peru, Colombia, Turkey

Last meeting Mumbai, Nov 2011

Next meeting Korea, May 2012

7/31/2019 2012 q1 Reid Mette

30/32

Conclusions

International standard will provide practitionerswith guidelines for testing that cover all aspectsof the life cycle

Provides a consistent set of definitions,

processes, procedures & techniques forsoftware testing

Will be adopted by IEEE, BSI, ISO and othernational standards bodies

Currently has representation from 18 nations &is being reviewed by software testingprofessionals world-wide

We still need to ensure widespread applicability

7/31/2019 2012 q1 Reid Mette

31/32

Any Questions?

Any Volunteers?

7/31/2019 2012 q1 Reid Mette

32/32

For more information

[email protected]

if interested in trialling the standard on aproject, reviewing drafts or writing examples

http://softwaretestingstandard.org/

WG26 website

http://www.jtc1-sc7.org/

access to official documents released by WG 26
mailto:[email protected]://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/http://www.jtc1-sc7.org/mailto:[email protected]:[email protected]:[email protected]

Documents

2012 q1 Reid Mette