ISO/IEC/IEEE 29119 勉強会 第3回 - Part 3 テストドキュメント

YAMASAKI Takashi 2014年1月9日

本日のゴール

Organizational Test Process

Test Management Processes

Dynamic Test Processes

Test Policy Organizational

Test Strategy

Test Plan Test Status

Report

Test Completion

Report

Test Environment

Readiness

Report

Test Data

Requirements

Test

Environment

Requirements

Test Data

Readiness

Report

Incident

Report

Test Procedure

Specification

Test Case

Specification

Test Design

Specification

Actual

Result Test

Result

Test Execution

Log

種類

Figure 1 — The hierarchy of test documentation from ISO/IEC/IEEE 29119-2:2013

Dynamic Test Processes Dynamic Test Processes Dynamic Test Processes

Test Management Processes

Test Management Processes

Organizational Test Process

Test Policy

Organizational Test Strategy

Organizational Test Strategy

Test Plan (Project)

Test Plan (Project)

Test Plan (Project)

Test Plan (Sub-process)

Test Plan (Sub-process)

Test Plan (Sub-process)

Test Specification

Test Environment Requirement

Test Data Requirement

Test Environment Readiness Report

Test Data Readiness Report

Perform Dynamic Test

Test Execution Documentation

Incident Report

Test Status Report

Test Completion Report (Sub-proc.)

Test Completion Report (Project)

TEST

PLAN

TEST

DESIGN

SPEC

TEST

CASE

SPEC TEST

PROC

SPEC

ITEM

DOC ITEM

DOC

ITEM

DOC PROJECT

DOC

TEST

LOG

TEST

LOG

TEST

INCIDENT

REPORT

TEST

SUMMARY

REPORT

TEST

ITEM

TEST

EXECUTION

TEST ITEM

TRANSMITTAL

REPORT

TEST

INCIDENT

REPORT

TEST

DESIGN

SPEC

Figure. “Relationship of test documents to testing process” from IEEE Std 829-1998 IEEE Standard for Software Test Documentation).

IEEE Std 829 – 1998 TEST

DESIGN

SPEC

Document specified by this

standard

Document NOT specified by

this standard

Test Item (NOT specified by

the standard)

Process NOT specified by this

standard

Test Management Processes

Dynamic Test Processes

Test

Planning

Process

Test

Monitoring &

Control

Process

Test

Completion

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Organizational

Test

Process

Organizational Test Process

Develop Organizational

Test Specification

(OT1)

Monitor and Control Use of

Test Specification

(OT2)

Update Organizational

Test Specification

(OT3)

Organization Test

Specification

Controlled Organizational

Test Specification

Updated Organization

Test Specification

Figure 4 – Organizational Tet Process, from ISO/IEC/IEEE 29119-2 Test process, P-13

Inputs to activities in this process may include: • Views of primary stakeholders; • Knowledge of the current test

practices within the organization;

• Organization’s Mission Statement;

• IT Policy; • IT Project Management Policy; • Quality Policy;

• Organizational Test Policy; • Organizational Test Strategy; • Feedback on Tet Specification; • Typical test plans from the

organization; and • Industry and/or government

standards.

組織的なテストプロセス

組織的なテスト 仕様を開発する

(OT1)

(組織的な) テスト仕様の

効用をモニターし コントロールする

(OT2)

組織的なテスト 仕様を更新する

(OT3)

組織的な テスト仕様

コントロールされた 組織的なテスト

仕様

更新された 組織的な テスト仕様

Test Management Processes

Dynamic Test Processes

Test Plan Test Status

Report

Test Completion

Report

Test Environment

Readiness

Report

Test Data

Requirements

Test

Environment

Requirements

Test Data

Readiness

Report

Incident

Report

Test Procedure

Specification

Test Case

Specification

Test Design

Specification

Actual

Result Test

Result

Test Execution

Log

Organizational Test Process

Test Policy Organizational

Test Strategy

組織に関するテストについての情報を 提 供 す る ド キ ュ メ ン ト 、 た と え ば プロジェクトに依らない情報など

• 組織的なテスト仕様の例

• 組織的なテストポリシー (organizational test policies)

• 組織的なテスト戦略 (organizational test strategies)

• 経営層レベルのドキュメント

• 組織におけるテストの目的、ゴール、全体的なスコープを記述

※ Part 3では”Organizational”がなく単に”Test Policy”となっている。

Organizationalのありなしで、意味が大きく変わる単語も多いのでちょっと違和感がある。

Document specification information

Introduction

Test policy statement

Objectives of testing (テストの目的)

Test process (テストプロセス)

Test organization structure (テスト組織構造)

Tester training (テスターのトレーニング)

Tester ethics (テスターの行動規範)

Standards (標準)

Other relevant policies (その他の関連のあるポリシー)

Measuring the value of testing（テストの価値の測定）

Test asset archiving and reuse（テスト資産の保管と再利用）

Test process improvement（テストプロセス改善）

• 組織で実行するすべてのプロジェクトにおいて実行されるテストのための包括的(generic)な要求を表現したドキュメントで、どのように テストを実行するかの詳細を提供する

• 注１： 組織的なテストポリシーによって調整される

• 注２：著しく異なるプロジェクトコンテキストをカバー するために、ひとつの組織がひとつ以上の組織的な テスト戦略を持つことができる

Document specific information

Introduction

Project-wide organizational test strategy statements

(プロジェクト横断的な組織的テスト戦略ステートメント…って長いよ！）

Generic risk management（包括的なリスクマネジメント）

Test selection and prioritization (テストの選別と優先順位付け)

Test documentation and reporting (テストの文書化と報告)

Test automation and tools (テストの自動化とツール)

Configuration management of test work products (テスト成果物の構成管理)

Incident management（インシデント管理）

Test subprocesses（テストサブプロセス）

Test sub-process-specific organizational test strategy statements

(テストサブプロセス固有の組織的なテスト戦略ステートメント)

Entry and exit criteria（開始および終了基準）

Test complication criteria （テスト完了基準）

Test documentation and reporting （テストの文書化と報告）

Degree of independence （独立性の度合い）

Test design techniques （テスト設計技法）

Test environment （テスト環境）

Metrics to be collected （収集されるメトリクス）

Retesting and regression testing （再テストと回帰テスト）

Figure 5 – Example test management process relationship, from ISO/IEC/IEEE 29119-2 Test processes P-15

Organizational Test Process

Test Management Processes

Test Planning

Test Monitoring & Control

Test Completion

Test Plan Updates

Test Plan Test

Results

Test Completion

Report

Test Management Processes

Test Management Processes

Dynamic Test Processes

Dynamic Test Processes

Dynamic Test Processes

Test Policy & Organizational Test Strategy

Test Plan, Control

Directives

Test Plan, Test Status Reports, Test Completion Report, Test Measures

Test Plan, Control

Directives

Test Measures

Test Plan, Control

Directives

Test Measures

Feedback on Test Policy & Organizational Test Strategy

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Test

Monitoring &

Control

Process

Test

Completion

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Test

Planning

Process

Figure 6 – Test Planning Process, from ISO/IEC/IEEE 29119-2 Test processes P-16

Test Planning Process

Understand Context (TP1)

Organize Test Plan

Development (TP2)

Design Test Strategy

(TP5)

Identify Risk Mitigation

Approaches (TP4)

Identify & Analyse Risks

(TP3)

Communicate Test Plan And Make Available

(TP9)

Gain Consensus on

Test Plan (TP8)

Determine Staffing and Scheduling

(TP6)

Record Test Plan

(TP7)

Scope

Test Plan Development Schedule

Analyzed Risks

Mitigation Approaches

Test Strategy

Schedule, Staffing Profile

Draft Test Plan

Approved Test Plan

Test Plan The process is shown as purely sequential, but in practice it may be carried out iteratively, with some activities being revisited. See text for detail.

Inputs to activities in this process may include: • Organizational Test Policy; • Organizational Test Strategy; • Regulatory Standards; • Project Test Plan (if planning

testing for a specific phase or type within a project);

• Incident reports; • Project Management Plan; • Applicable product documentation

(e.g. System Requirements, Test Item Specification);

• Software Development Plan; • Project and Product Risks; and • Test Plan Updates.

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Completion

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Test

Monitoring &

Control

Process

Figure 7 – Test monitoring and control process, from ISO/IEC/IEEE 29119-2 Test processes P-22

Test Monitoring & Control Process

Set-Up (TMC1)

<<instantiated>>

Inputs to activities in this process may include: • Test Plan(s); • Applicable product documentation,

e.g. System Requirement, Contract, etc.;

• Organizational Test Policy;

• Organizational Test Strategy • Control Directive (from a higher

level Test Monitoring and Control Process); and

• Measures (from the Test Process being managed).

Monitor (TMC2)

Report (TMC4)

... Test Process ...

Test Management

Processes

Dynamic Test Processes

Control (TMC3)

<<instantiated>>

Test Measures

Control Directives

Test Status Report

Test Measures

Test Plan

Test Progress

Info

Test Control Info [Testing Incomplete]

[Testing Complete]

The process is shown as purely sequential, but in practice it may be carried out iteratively, with some activities being revisited.. See text fir details

Test Progress Info

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Monitoring &

Control

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Test

Completion

Process

Figure 8 – Test Completion Process , from ISO/IEC/IEEE 29119-2 Test processes P-25

Test Completion Process

Inputs to activities in this process may include: • Project Test Plan; • Phase Test Plans; • Incident Reports; • Project Test Status Reports; • Phase/Type Test Completion Reports; and • Organizational Test Strategy (if relevant)

The process is shown as purely sequential, but in practice it may be carried out iteratively, with some activities being revisited.. See text fir details

Archive Test Assets

(TC1)

Archived Test Assets

Clean Up Test

Environment (TC2) Identify

Lessons Learned (TC3) Report

Test Completion

(TC4)

Available Test Environment

Lessons Learned

Test Completion Report

Dynamic Test Processes

Test Environment

Readiness

Report

Test Data

Requirements

Test

Environment

Requirements

Test Data

Readiness

Report

Incident

Report

Test Procedure

Specification

Test Case

Specification

Test Design

Specification

Actual

Result Test

Result

Test Execution

Log

Organizational Test Process

Test Policy Organizational

Test Strategy

Test Management Processes

Test Plan Test Status

Report

Test Completion

Report

Document specific information

Introduction

Context of the testing（テストのコンテキスト）

Project(s).test sub-process(es) (プロジェクト、テストサブプロセス）

Test item(s) (テストアイテム)

Test scope (テストスコープ)

Assumptions and constraints (前提と制約)

Stakeholders (利害関係者)

Testing communication(テストのコミュニケーション)

Risk register（リスク登録簿）

Product risks（プロダクトリスク）

Project risks（プロジェクトリスク）

Test strategy（テスト戦略）

Test sub-processes（テストサブプロセス）

Test deliverables（テスト成果物）

Test design techniques（テスト設計技法）

Test completion criteria（テスト完了基準）

Metrics to be collected（収集されるメトリクス）

Test data requirements（テストデータ要求）

Test environment requirements（テスト環境要求）

Retesting and regression testing（再テストと回帰テスト）

Suspension and resumption criteria（中断と再開基準）

Deviation from the Organizational Test Strategy（組織的なテスト戦略からの逸脱）

Testing activities and estimates（テスト活動と見積もり）

Staffing（人材の配置）

Roles, activities, and responsibilities（役割、活動、責任）

Hiring needs（雇用の必要性）

Training needs（トレーニングの必要性）

Schedule（スケジュール）

Document specific information

Introduction

Test status（テストステータス）

Reporting period（報告時期）

Progress against Test Plan（テストプランと比較した進捗）

Factors blocking progress（進捗を妨げる要因）

Test measures（テストメジャー）

New and changes risks（新規と変更のあったリスク）

Planned testing（計画されたテスト）

Document specific information

Introduction

Testing performed（実施したテスト）

Summary of testing performed（実施したテストの概要）

Deviations from planned testing（計画したテストからの逸脱）

Test completion evaluation（テスト完了評価）

Factors that blocked progress（進捗を妨げた要素）

Test measures（テストメジャー）

Residual risks（残存リスク）

Test deliverables（テスト成果物）

Reusable test assets（再利用可能なテスト資産）

Lesson learned（きづき・教訓）

Test Design & Implementation

Test Environment

Set-up & Maintenance

Test Incident

Reporting

Test Execution

Test Environment

Requirements

Test Specification

Test Environment Readiness Report

Incident Report

Test Results

[Issue Noticed OR Retest Result]

[No Issues Noticed]

Test Management Processes

Figure 9 – Dynamic test processes, from ISO/IEC/IEEE 29119-2 Test processes P-28

Test Plan Test Measures

Control Directives

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Monitoring &

Control

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Test

Completion

Process

Test Design &

Implementation

Process

Test Design & Implementation Process

Figure 10 – Tet Design and Implementation Process , from ISO/IEC/IEEE 29119-2 Test processes P-30

Identify Feature

Sets (TD1)

Derive Test

Conditions (TD2) Drive

Test Coverage

Items (TD3) Derive

Test Cases (TD4)

Assemble Test Sets

(TD5) Drive Test

Procedures （TD6)

Test Design Specification

Test Case Specification

Test Procedure

Specification

Feature Sets

Test Conditions

Test Coverage Items

Test Cases

Test Sets

Test Procedures & Test Scripts

Inputs to activities in this process may include: • Test Basis; • Test plan; • Test strategy; • Test items; and • Test design techniques.

The process is shown as purely sequential, but in practice it may be carried out iteratively, with some activities being revisited.. See text fir details

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Monitoring &

Control

Process

Test Design &

Implementation

Process

Test

Incident

Reporting

Process

Test

Execution

Process

Test

Completion

Process

Test

Environment

Set-up &

Maintenance

Process

Test Environment Set-Up & Maintenance Process

Figure 11 – Test Environment Set-Up & Maintenance process , from ISO/IEC/IEEE 29119-2 Test processes P-34

Establish Test

Environment (ES1)

Maintain Test

Environment (ES2)

Test Environment

Readiness Report

Inputs to activities in this process may include: • Test Plan(s); • Test Environment Requirements; • Intended/operational environment • Test basis • Test procedure(s); and • Test results (where available).

Test Environment

Update(s)

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Monitoring &

Control

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Incident

Reporting

Process

Test

Completion

Process

Test

Execution

Process

Figure 12 – Test Execution process , from ISO/IEC/IEEE 29119-2 Test processes P-36

Test Execution Process

Execute Test

Procedure(s) (TE1)

Record Test

Execution (TE3)

Test Outcome

Inputs to activities in this process may include: • Test Plan(s); • Test Procedure(s); • Test item; • Test basis; • Test environment readiness report (where available); and • Test environment update(s) (where available.

Test Execution

Log Compare

Test Result (TE2)

Test Results

The process is shown as purely sequential, but in practice it may be carried out iteratively, with some activities being revisited.. See text fir details

Organizational

Test

Process

Test Management Processes

Dynamic Test Processes

Figure 2 – The multi-layer model showing all test processes, from ISO/IEC/IEEE 29119-2:2013 Test processes, P-11

Test

Planning

Process

Test

Monitoring &

Control

Process

Test Design &

Implementation

Process

Test

Environment

Set-up &

Maintenance

Process

Test

Execution

Process

Test

Completion

Process

Test

Incident

Reporting

Process

Test Incident Reporting Process

Figure 13 – Test Incident Reporting process , from ISO/IEC/IEEE 29119-2 Test processes P-38

Analyze Test

Result (IR1)

Create/ Update Incident Report (IR2)

Incident Report

Updates

Inputs to activities in this process may include: • Test results; • Test procedures(s) • Test Case(s) • Test item(s) • Test basis; and • Test execution logs (where available).

Incident Report

Organizational Test Process

Test Policy Organizational

Test Strategy

Test Management Processes

Test Plan Test Status

Report

Test Completion

Report

Dynamic Test Processes

Test Environment

Readiness

Report

Test Data

Requirements

Test

Environment

Requirements

Test Data

Readiness

Report

Incident

Report

Test Procedure

Specification

Test Case

Specification

Test Design

Specification

Actual

Result Test

Result

Test Execution

Log

Document specific information

Introduction

Notation convention(s) (表記法)

Feature sets（フィーチャーセット）

Overview（概要）

Unque identifier（UID）

Objective（目的）

Priority（優先度）

Specific strategy（特定の戦略）

Traceability（トレーサビリティ）

Test conditions（テスト条件）

Overview（概要）

Uniqque identifier（UID）

Description（説明）

Priority（優先度）

Traceability（トレーサビリティ）

Test Case Specification（テストケース仕様）

Document specific information

Introduction

Notation convention(s)（表記法）

Test coverage items（テストカバレッジアイテム）

Overview（概要）

Unique identifier（UID）

Priority（優先度）

Traceability（トレーサビリティ）

Test cases（テストケース）

Objective（概要）

Unique identifer（UID)

Traceability（トレーサビリティ）

Inputs（入力）

Expected results（期待結果）

Actual results and test result（実際の結果とテスト結果）

Document specific information

Introduction

Notation convention(s)（表記法）

Test sets（テストセット）

Overview（概要）

Unique identifier（UID)

Objective（目的）

Priority（優先度）

Contents (Traceability) （内容(トレーサビリティ)）

Test procedures（テスト手順）

Overview（概要）

Unique identifier（UID）

Objective（目的）

Priority（優先度）

Start up（開始）

Test cases to be executed (Traceability)（実行されるテストケース）

Relationship to other procedures（その他の手順との関連性）

Stop and wrap up（停止と要約）

Document specific information

Introduction

Detailed test data requirements （詳細化したテストデータ要求）

Overview（概要）

Unique identifier（UID）

Description（説明）

Responsibility（責任）

Period needed（必要な期間）

Resetting needs（リセットの必要性）

Archiving or disposal（保管または破棄）

Document specific information

Introduction

Detailed test environment requirements （詳細化されたテスト環境要求）

Overview（概要）

Unique identifier（UID）

Description（説明）

Responsibility（責任）

Period needed（必要な期間）

Document specific information

Introduction

Test data status（テストデータステータス）

Overview（概要）

Unique identifier（UID）

Description of status（ステータスの説明）

Document specific information

Introduction

Test environment readiness（テスト環境準備）

Overview（概要）

Unique identifer（UID）

Description of status（ステータスの説明）

項目は未定義。

項目は未定義。

Document specific information

Introduction

Events（イベント）

Overview（概要）

Unique identifier（UID）

Time（時間）

Description（説明）

Impact（影響度）

Document specific information

Introduction

Incident details（インシデント詳細）

Timing information（タイミング情報）

Originator（起案者）

Context（コンテキスト）

Description of the incident（インシデントの説明）

Originator's assessment of severity（起案者の深刻度評価）

Originator's assessment of priority（起案者の優先度評価）

Risk（リスク）

Status of incident（インシデントの状態）

IEEE Std 829-1998 IEEE Std829:2008 Te

st P

lan

Test

Des

ign

Spe

cific

atio

n

Test

Cas

e S

peci

ficat

ion

Test

Pro

cedu

re S

peci

ficat

ion

Test

item

tran

smitt

al R

epor

t

Test

Log

Test

Inci

dent

Rep

ort

Test

Sum

mar

y R

epor

t

Mas

ter

Test

Pla

n

Leve

l Tes

t Pla

n

Leve

l Tes

t Des

ign

Leve

l Tes

t Cas

e

Leve

l Tes

t Pro

cedu

re

Leve

l Tes

t Log

Ano

mal

y R

epor

t

Leve

l Int

erim

Tes

t Sta

tus

Rep

ort

Leve

l Tes

t Rep

ort

Mas

ter

Test

Rep

ort

- - - - - - - - Test Policy - - - - - - - - - -

- - - - - - - - Organizational Test Strategy - - - - - - - - - -

x - - - - - - - Test Plan x x - - - - - - - -

- x - - - - - - Test Design Specification - - x - - - - - - -

- - x - - - - - Test Case Specification - - - x - - - - - -

- - - x - - - - Test Procedure Specification - - - - x - - - - -

- - - v - - - - Test Data Requirements - - - - v - - - - -

- - - v - - - - Test Environment Requirements - - - - v - - - - -

- - - - - - - - Test Data Readiness Report - - - - - - - - - -

- - - - - - - - Test Environment Readiness Report - - - - - - - - - -

- - - - - v - - Actual Results - - - - - v - - - -

- - - - - v - - Test Result - - - - - v - - - -

- - - - - x - - Test Execution Log - - - - - x - - - -

- - - - - - x - Incident Report - - - - - - x - - -

- - - - - - - - Test Status Report - - - - - - - x - -

- - - - - - - x Test Completion Report - - - - - - - - x x

ISO/IEC/IEEE 29119-3

Reference ISO/IEC/IEEE 29119関連

• ISO/IEC/IEEE 29119-1:2013 Software testing – Part 1: Concepts and definitions

• ISO/IEC/IEEE 29119-2:2013 Software testing – Part 2: Test processes

• ISO/IEC/IEEE 29119-3:2013 Software testing – Part 3: Test documentation

• ISO/IEC/IEEE DF 29119-4 Software testing – Part 4: Test techniques

• 高信頼な組込みソフトウェアのためのテスト事例と最新動向 (羽田裕氏)

IEEE Std 829関連

• IEEE Std 829:1998 Standard for Software Test Documentation

• IEEE Std 829:2008 Standard for Software and System Test Documentation

ISTQB/JSTQB関連

• テスト技術者資格制度 Foundation Level シラバス 日本語版 Version 2011.J02

• ソフトウェアテスト標準用語集 （日本語版）Version 2.2.J01

ISO/IEC 関連

• ソフトウェアエンジニアリング関連の標準化 ～ ISO/IEC JTC1/SC7による国際標準化とSECの取組み～(高橋光裕氏)

END