Generalized Software Reliability Model

(GSRM)Hironori Washizaki

Waseda University

National Institute of InformaticsTwitter: @Hiro_Washi washizaki@waseda.jp

http://www.washi.cs.waseda.ac.jp/

Kiyoshi Honda, Hironori Washizaki, YoshiakiFukazawa, “A Generalized Software Reliability Model Considering Uncertainty and Dynamics in Development,” PROFES 2013

Motivation

• When can we release software?• How many efforts are necessary for further

testing?

2Time

Software Reliability Model (SRM)

3

Counting the defects a day or a week

Approximate actual data to a curve and predict defects

At this time, 95% of all defects will be found

Prediction model

Types of SRM

• Statistic analysis model– From actual data approximate

to a curve.– Gompertz model– Logistic model

• Stochastic process model– The detection of defects

follows stochastic process – Non-homogeneous Poisson

process(NHPP) model [Goel]4[Goel] A.L. Goel and K. Okumoto A non-homogeneous poisson process model for

software reliability and other performance measures, 1979

5

#Defects

Actual

Predicted

Days

Case (Industry)

Further Challenges in SRM

• Uncertainty– Actual projects have many uncertain elements

which cause defects.– E.g. changes of specifications

• Dynamicity– Actual projects have some time dependency.– E.g. changes of developers

6

Idea: Generalized SRM• Conventional Logistic Model

• Assumptions– Number of defects that can be found is variable depending

on time.– Number of defects that can be found contains uncertainty,

which can be simulated with Gaussian white noise.

7

Dynamicity Uncertainty

A generalized software reliability model considering uncertainty and dynamics in development. PROFES ’13

Uncertainty

8

the uncertainty is greater at the start of the project than at the end.

The uncertainty is constant at any given time.

The uncertainty increases near the end.

Dynamicity

9

The number of developers is constant.

The number of developersper unit time changes at a certain time.

The number of developers per unit time increase near the end.

Combination of Uncertainty and Dynamicity

10

Similar to a logistic curve

Constant

Prediction with Probability

11

95% of defects are expected to be foundduring this term.

12

Visualization integrated with Continuous Integration Tool

Case (Industry)

13

abyss_data-api abyss_data-api abyss_data-api

Module Current PredictedTotal

PredictedCurrent

PredictedEnd Day

XYZ 147 144 134 156

Almost all defects seem to be detected.Now more debugging rather than testing.

#Defects #Predicted Total DefectsPredication Error

Case (OSS)

14

Open Research Questions

• How to Predict Uncertainty and Dynamicity?• How to dynamically adapt prediction?• Can we integrate testing and debugging

techniques with (G)SRM?• Any relations among Predicted Reliability and

other measures such as Testing Coverage and Mutation Testing Scores?

Background: Testing and Reliability• Empirical

– Bug Localization in Industry [ICST’14]

– Testing in OSS [PROFES’14]

• Web and Ajax– Gray Box Testing [FASE’13]– Mutation Testing [SEKE’13]– State-Machine Extraction

and Verification [ASE’13][AES’14]

• Software Reliability Model [PROFES’13]

16

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1

The rate of tested

cases

Time

Model 1- 1

Model 2- 1

Model 3- 1

Semi-automatic Incompatibility Localization for Re-engineered Industrial Software, ICST’14Do open source software projects conduct tests enough?, PROFES’14Mutation Analysis for JavaScript Web Applications Testing, SEKE’13Automated Verification of Pattern-based Interaction Invariants in Ajax Applications, ASE’13Validating Ajax Applications Using a Delay-Based Mutation Technique, ASE’14A generalized software reliability model considering uncertainty and dynamics in development. PROFES’13

• Assoc. Prof., Head, Global Software Engineering Laboratory, Waseda University

• Visiting Assoc. Prof., National Institute of Informatics

• Chair, SEMAT Japan Chapter• Chair, ISO/IEC SC7/WG20 Japan• Secretary, IEEE CS Japan• Program Organizations: ASE,

SEKE, SPLC, APSEC, AsianPLoP• Many PCs 17

“Hiro”nori Washizaki

Waseda UniversityGlobal Software Engineering Lab.

• SE researches in global development era

• 6 professors• 17 partner members• 25 students (5 from

China, Norway, Mexico, Indonesia)

18

Recent PublicationsReuse and Development Environments

Unified Code Processing Framework (ICST’13)Evaluation of Separation of Concerns Paradigms (SEKE’13) Traceability Links between Requirements and Code (SPLC’13)Reusable Refactoring Scripts (SEKE’14)

Automated Testing and VerificationBug Localization for Re-engineered Industrial Software (ICST’14)Web Test Code Generator Based on Template (FASE’13)Validating Security Design Pattern Application (ARES’13-14) Mutation Analysis for JavaScript Testing (SEKE’13)Automated Verification of Ajax Applications (ASE’13, ASE’14)

Quality ManagementSoftware Reliability Model Considering Uncertainty (PROFES’13, APSEC’14) Organizational Changes and Quality (APSEC’13)Empirical Study of Testing in OSS (PROFES’14)

Agile, Patterns and EducationEffectiveness of workshops for learning agile (Agile’13)Pattern language for Agile Development and Robotics (PLoP’13-14, Agile’14)Team Characteristics for Educational Effectiveness (CSEET’13-14) 19