Local Bias and its Impacts on the Performance of Parametric

Estimation Models

Accepted by PROMISE2011 (Best paper award)Ye Yang, Lang Xie, Zhimin He (iTechs)Qi Li, Vu Nguyen, Barry Boehm (USC)

Ricardo Valerdi (MIT)

2

Agenda

Background Research questions Measuring local bias Measuring the impacts of local bias Handling Local Bias Conclusions and future work

3

Background

COCOMO II model Proposed by Dr. Barry Boehm; one of the most accurate cost

estimation models; widely adopted by industry. Typical parametric estimation model, need tune parameters

against local data (local calibration)

5

1

170.01

1

ii

B SF

jj

Effort A Size EM

Organization 1

Organization 2

General Model

4

Background (Cont.)

Model usage circle Local calibration relies on local historical data and domain

knowledge, i.e. with local assumptions. In most cases, such local assumptions vary from the general

model assumptions. It is possible that the mismatches between “general assumptions” and “local assumptions” will result in surprising calibration results.

Model Localization

Model Usage

Model Calibration

Model Building

General assumptions

Underlying model

Local data

Calibrationdata

Local assumptions

Model updates

Historical data

E.g., counter-intuitive calibration results: negative values of regression coefficients for level of programmer capability (PCAP), indicating higher PCAP leads to higher effort.

5

Research questions

Research questions: Is there a way to measure the local bias introduced in the

model localization (local calibration) stage? As the historical data accumulates from multiple companies,

how will the associated local bias impact the performance of the general parametric estimation model?

Are there any correlation patterns between local bias and model performance variation after incorporating local dataset into the calibration dataset?

Assumptions: The general parametric model follows a similar structure as the

COCOMO II. In model localization stage, constant A and constant B are

tuned with local data. In model usage stage, locally calibrated A and B are used for

project estimation.

6

Measuring local bias

Definition of local bias:

where A’and B’are model parameters calibrated from local data of each organization, A and B are default constant values of COCOMO II model (A=2.94, B=0.91), and in our study we set Size=100KLOC.

' '| ln( ) | | ln( ) ( ' ) ln( ) |

Effort Alocalbias B B Size

Effort A

7

Measuring local bias (cont.)

Data sets CII 2010 data set; contains two subsets: the CII2000 subset

(161 data points from 16 organizations) and the After2000 subset (92 additional data points newly collected from 10 different organizations since year 2000)

Characteristics CII2000 Subset

CII 2010 Dataset

# data points 161 253# organizations 16 23

Size(KSLOC)min 2.6 1.68max 1292.8 2505.2median 46.92 45.28

Effort(PM)min 6 3.5max 11400 11400median 192.5 170

8

Measuring local bias (cont.)

Analysis procedure Divide After2000 subset into 10 groups according to their

corresponding organization. For each group, we conduct a representative local calibration

using data in that group only and produce its local A’ and B’. Calculate the corresponding local bias value of each group. Compare local bias values among all groups.

CII 2000 SubsetAfter2000 Subset

Subset1

…

A, BA1’, B1’ A2’, B2’ An’, Bn’

local_bias1 local_bias2 local_biasn

CII 2010Dataset

Subset2

Subsetn

Group by Organization_IDDefault Constants: A, B

9

Measuring local bias (cont.)

Parameters of local models: Local bias of each group:

Different local A and B in each group, indicating local bias introduced when adopting local calibration;

Local bias varies in different group, ranging from 0.06 to 2.25; the local bias measures how much relative error the corresponding local model will produce.

10

Measuring the impacts of local bias

Analysis procedure First, for each group ssi in the After2000 subset:

1. combine ssi with CII 2000 data set to produce a new data set dsi ;

2. Assessing model performance on data set dsi , record values of performance indicators;

Then conduct correlation analysis between local bias and model performance

CII 2000 subsetI SS1 Performance Local bias

CII 2000 subsetI SS2 Performance Local bias

…… …… ……

Correlation analysis

11

Measuring the impacts of local bias

Performance assessment Basic performance indicators: MMRE (mean MRE), stdMRE (the

variance of MRE) Assessment procedure:

In our study, we employ Average MMRE, Range of MMRE, Average stdMRE, and Range of stdMRE to assess the performance of an estimation model.

Spliting data set into training set

and test set

Tuning model parameters on

training set

Evaluating model

performance on test set

MMRE, stdMRE

Average MMRERange of MMREAverage stdMRERange of stdMRE

Repeat the above steps for 2000 times

2000 (MMRE, stdMRE) pairs

12

Measuring the impacts of local bias(cont.)

Model performance

13

Measuring the impacts of local bias(cont.) Spearman correlation coefficients between local

bias and model performance:

At the significant level of p-value less than 0.05, the range of stdMRE is significantly positive correlated with local bias and local_bias*num. Both the average stdMRE and the average MMRE are significantly positive correlated with local_bias*num.

Range of stdMRE reflects the uncertainty of model performance. Hence, the bigger the local bias is, the weaker the performance is.

  Range of stdMRE

Average stdMRE

Range of MMRE

Average MMRE

Local bias

Correlation Coefficient 0.7787 0.1677 0.4731 0.1671

p-value 0.0080 0.6435 0.1673 0.6455

Local bias *num

Correlation Coefficient 0.6120 0.8085 0.4731 0.6777

p-value 0.0508 0.0046 0.1673 0.0313

14

Handling Local Bias

Motivation Performance of the general COCOMO II model seriously

decrease on the After2000 subset! Need to calibrate a new version of COCOMO II model on the CII

2010 data set.

　 CII2000 After2010 CII2010

Pred(20) 0.6211 0.2393 0.4822

Pred(25) 0.6957 0.3152 0.5573

Pred(30) 0.7516 0.3696 0.6126Pred(20) Pred(25) Pred(30)

CE/ 通用格式

CE/ 通用格式

CE/ 通用格式

CII2000After2010CII2010

15

Handling Local Bias (cont.)

Local bias handling approach Assumption ： local historical data set with higher local bias

presents more different pattern for cost estimation, and it should be assigned a lower weight when being used for model calibration.

Constraints for weight distribution function Weight=F ( LocalBias )

IF LocalBias =0, THEN Weight =1; IF LocalBias → +∞, THEN Weight → 0; The F should be a decreasing function on interval [0, +∞).

Three functions

11

1F Weight

LocalBias

：

12

1 ln( )F Weight

LocalBias

：

13F Weight LocalBiase

： CE/通用格式

CE/通用格式

CE/通用格式

CE/通用格式

CE/通用格式

CE/通用格式

CE/ 通用格式

CE/ 通用格式

CE/ 通用格式1/(X+1)1/ln(X+1)+11/E^x

16

Handling Local Bias (cont.)

OID LocalBias F1 F2 F3

46 0.03952465 0.96197815 0.962682998 0.961246259

595 0.20628288 0.828992947 0.842074323 0.813602892

179 0.276339409 0.783490655 0.803861012 0.758555426

14 0.302222798 0.767917749 0.791093772 0.73917336

106 0.302948518 0.767490032 0.790745252 0.738637122

99 0.568446596 0.637573509 0.689614414 0.566404611

93 0.726123018 0.579332985 0.646881635 0.483780969

590 1.009427633 0.49765415 0.588980208 0.364427506

599 1.820976691 0.354487154 0.490897974 0.161867579

597 2.190999501 0.313381434 0.462891345 0.1118049441 2 3 4 5 6 7 8 9 10

CE/ 通用格式

CE/ 通用格式

CE/ 通用格式

LocalBias1/(X+1)1/[ln(X+1)+1]1/e^X

Weight assigned to each organization

17

Handling Local Bias (cont.) Model performance on the CII2000 subset

　 COCOMOEqual

Weights F1 F2 F3

Pred(20) 0.6211 0.4907 0.5093 0.5031 0.5342

Pred(25) 0.6957 0.5963 0.6149 0.6149 0.6273

Pred(30) 0.7516 0.677 0.7205 0.7143 0.7081

Model calibrated with equal weights performs worst on the CII2000 subset;

The general COCOMO II model performs best;

Pred(20) Pred(25) Pred(30)CE/ 通用格式

CE/ 通用格式

CE/ 通用格式

COCOMO IIEqual weightsFunction-1Function-2Function-3

18

Handling Local Bias (cont.) Model performance on the After2000 subset

　 COCOMOEqual

Weights F1 F2 F3

Pred(20) 0.2393 0.25 0.2609 0.2717 0.2609

Pred(25) 0.3152 0.3261 0.3261 0.3261 0.3152

Pred(30) 0.3696 0.3804 0.4022 0.4022 0.4022

The general COCOMO II model performans worst on the After 2000 subset

Models calibrated with weights exhibit better performance than models calibrated without weights.

Pred(20) Pred(25) Pred(30)CE/ 通用格式

CE/ 通用格式

COCOMO IIEqual weightsFunction-1Function-2Function-3

19

Handling Local Bias (cont.) Model performance on the whole CII 2010 data set

　 COCOMOEqual

Weights F1 F2 F3

Pred(20) 0.4822 0.4032 0.419 0.419 0.4348

Pred(25) 0.5573 0.498 0.5099 0.5099 0.5138

Pred(30) 0.6126 0.5692 0.6047 0.6008 0.5968

The general COCOMO II model works better on the whole CII 2010 data set than calibrated models;

Models calibrated with weights exhibit better performance than models calibrated without weights.

Pred(20) Pred(25) Pred(30)CE/ 通用格式

CE/ 通用格式

CE/ 通用格式

COCOMO IIEqual weightsFunction-1Function-2Function-3

20

Conclusions

The proposed LocalBias measure can be used to quantitatively measure and analyze potential local bias associated with individual organization data subset in the overall dataset.

As historical data accumulates from multiple companies, the associated local bias will cause the range of stdMRE increase.

The correlation analysis verifies that the model performance is significantly correlated by the degree of local bias and the number of data points associated with each additional group.

Weight calibration helps to reduce impact of local bias and thus improve the usability of cross-company data for model calibration.

21

Future work

More empirical studies on other public dataset to future validate and refine results.

Develop more effective methods for reducing local bias and improving general calibration outcomes.

Thanks!Q&A