Thomas L. Gilchrist tomg@tomgtomg.com Testing Basics Set 4: Strategies & Metrics By Thomas L....

Thomas L. Gilchrist tomg@tomgtomg.com

Testing Basics

Set 4: Strategies & Metrics

Thomas L. Gilchrist, 2009

What are the Tradeoffs

…changes in one require compensation in others...

Tradeoffs

• Schedule• Cost/Resources• Scope• Technology• Quality / Critical Success Factors

How much testing?

…risk!...

Testing and Risk

The degree of risk increases

Where are you now?

Level of testing

Confidence in process

Testing and Risk

Testing Goals

• Testing is about finding errors.• If you don’t find any errors,

testing is a waste of time.

Testing and Risk

Maturity of the development process

Definitions

• Requirement– A condition or capability needed by

a user to solve a problem or achieve an objective

– A condition or capability that must be met or possessed by a system …to satisfy a contract, standard, specification or other formally imposed document.

Test Terms

• Test Categories– Unit Testing– Integration Testing– System Testing– Acceptance Testing– Regression Testing

• Static Testing• Dynamic Testing

The Test Process

The Testing Process

• Step 1 - Set Test Objectives• Step 2 - Develop Test Plan• Step 3 - Execute Tests• Step 4 - Summarize and Report

Results

General Workbench

InputSource

Materials

ProcessProduct

Standards,templates, rulesand checklists

DoWork

Rework

Testing

Testing Workbench

Test Process

Products for Test

Problems

Tested Products

Test Toolbox

Test QC

Test Measures

The Software Test Process

Step 1 - Develop Test Objectives Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control

Step 2 - Develop Test Plan Task 1 - Define Business Functions Task 2 - Define Structural Functions Task 3 - Define Tests Task 4 - Create Function/Test Matrix Task 5 - Admin Test Requirements Task 6 - Formalize System Test Plan Task 7 - System Test Plan QC Task 8 - Develop Unit Test Plan Task 9 - Unit Test Plan QC

Step 3 - Execute Tests Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control

Step 4 - Summarize and Report Results Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings Task 4 - Formalize Report Task 5 - Quality Control

Test Process “V” Diagram

BusinessNeed

DefineRequirements

DesignSystem

BuildSystem

Verify

AcceptanceTest

SystemTest

IntegrationTest

UnitTest

Verify

Validate

Validates

TestPlanning

TestObjectives

TestExecution(Static) Test

Execution(Dynamic) &Test Reporting

The SEICMM

Immature Vs Mature Organization

Immature Mature

Heroic Efforts of Dedicated Team

Repeating Proven Methods of an Organization

Not Managed Managed

People

Management

Technology

Process

Not Fully UtilizedOrganization

al Focus

No "Memory", Improvised

Repeatable, Followed,

Predictable

From: SEI CMM/Bill Curtis, et al.

CMM Architecture

Level Key Process Areas

1Initial

Heroes*

5Optimizing

SystemCQI

Defect PreventionTechnology InnovationProcess Change Management

4Managed

SystemMeasurement

Process Measurement and AnalysisQuality Management

3Defined

EngineeringTechnicalPractices

Organizational Process FocusOrganizational Process DefinitionPeer ReviewsTraining ProgramIntergroup CoordinationSoftware Product EngineeringIntegrated Software Management

2Repeatable

ProjectManagement

Software Project PlanningSoftware Project TrackingSoftware Subcontract ManagementSoftware Quality AssuranceSoftware Configuration ManagementRequirements Management

Process Focus

From: SEI CMM/Bill Curtis, et al.

Taking Action Based

On Data

Measurement

• What’s you least favorite measurement?

• What’s your favorite measurement?

Designing a Metric Program

M* From "A Methodology for Collecting Valid Software Engineering Data"by Victor Basili and David Weiss. IEEE Transactions of Software Engineering,Vol. SE-10, No. 4, Nov. 1984

Question

Metric* From "A Methodology for Collecting Valid Software Engineering Data"by Victor Basili and David Weiss. IEEE Transactions of Software Engineering,Vol. SE-10, No. 4, Nov. 1984

Question

Metric* From "A Methodology for Collecting Valid Software Engineering Data"by Victor Basili and David Weiss. IEEE Transactions of Software Engineering,Vol. SE-10, No. 4, Nov. 1984

Understanding Metric Data

• Data is not information!

DataData InformationInformation

From: “Understanding Variation”, D.J. Wheeler

Understanding Metric Data

• Data is not information!

DataData InformationInformationAnalysisAnalysis

From: “Understanding Variation”, D.J. Wheeler

Key Variation Concepts

• Variation exists everywhere• Two types

– Common causes (noise)– Special case (signal)

• Variation, waste, & predictability.

94% of all problems in an organization are due to common causes and only management can fix them.

(Deming, “Out of Crisis”)

Run Charts

Metric

Specification Targets

Upper Specification Limit

Lower SpecificationLimit

Metric

Voice of the customer

Shewhart’s Control Charts

• Every data set has noise.• “Noise” confuses and clouds

single value comparisons.• Some data sets contain signal

(special causes).• Control charts separate

potential signals from noise.

Control Chart

Metric

Voice of the process

Control Chart

Metric

Upper Control Limit

Lower Control Limit

Signal and Noise

Upper Control Limit

Lower Control Limit

Metric

Voice of the process"Special Cause"

"Common Cause"Region

Data excursions

Serial Correlation

Metric

Upper Control Limit

Lower Control Limit

Recommendations

• Improve measurement systems– Commit to measurement, not

guessing– Devise and use common

measurement tools and definitions– Train data collectors on metric

definitions

• Reduce process variation• Improve process results

Variation In Software

• In software, when one process has wide variation, more than likely, they all have have wide variation.

• High precision in software measurement is unnecessary at this time.

Suboptimization

Suboptimization occurs when one doesn’t think of the total system...it is about dependencies.

Suboptimization

X is big and Y is big = goodnessHow do we allocate resources between x and Y?What are some examples?

X+Y=1X+Y=1

Thomas L. Gilchrist tomg@tomgtomg.com Testing Basics Set 4: Strategies & Metrics By Thomas L....

Documents

Lambaev riw metrics

Marketing Metrics

Sabr metrics

UX metrics

Excel Metrics

Credit Risk Metrics

Gilchrist vs. Cuddy

Agile metrics

Ch4 Performance metrics

Twitter Metrics

Lean Metrics Riga

Amy Gilchrist CV

Agile Metrics - how to use metrics to manage agile teams

Scor 10 Metrics

Rf planning metrics

Egee Sla Metrics

Che Metrics 2009

Mengenal Tuhan Secara Pribadi - By John Gilchrist

Gilchrist Open Computer Lab

952 Sqa Metrics