phd thesis presentation

Slide 1

Measurement data analysis in quality management systems.

Application to fuel test methods.

PhD Thesis

of

Dimitrios G. Theodorou

October 2015

NATIONAL TECHNICAL UNIVERSITY OF ATHENS

School οf Chemical Engineering

Department οf Synthesis and

Development οf Industrial Processes

Slide 2PhD Thesis – D. Theodorou

Presentation Outline

Introduction and motivation

Statistical and numerical methods overview

Measurement uncertainty arising from sampling

Measurement uncertainty estimation of an analytical procedure

Estimation of the standard uncertainty of a calibration curve

The use of measurement uncertainty and precision data in conformity assessment

Conclusions



Fuels produced and placed on market should comply withstrict requirements introduced by relevant legislation

Directive 98/70/EC

Directive 2003/17/EC

Several laboratory test methods are used for theevaluation and assessment of fuel properties

The social and economic impact of the laboratory gettinga wrong result and the customer consequentlyreaching a false conclusion can be enormous.

The laboratory should provide a high quality service toits customers



Quality = Fitness for purpose (i.e. intended use)

The quality of a result and its fitness for purpose isdirectly related to the estimation of measurementuncertainty

Measurement uncertainty – A key requirement foraccreditation according to international standards:

ISO/IEC 17025, for testing and calibration laboratories

ISO 15189, for medical laboratories

ISO/IEC 17043, for proficiency testing providers

ISO Guide 34, for reference material producers





Measurement Uncertainty (MU) = non-negativeparameter characterizing the dispersion of the quantityvalues being attributed to a measurand, based on theinformation used.

International Vocabulary of Metrology (VIM)



Master document on MU estimation

Guide to the Expression of Uncertainty in Measurement,

GUM

All MU estimation methodologies should give

results consistent with GUM



Modelling approach – GUM uncertainty framework



Modelling approach – GUM uncertainty framework

),...,,( 21 NXXXfY

i

ii xucyu )()( 222

),()()(2 jij

ji

iji xxrxuxucc

Nxxxii

iX

f

x

fc

,...,, 21

)()(

),(),(

ji

ji

jixuxu

xxuxxr

)(ykuU



Modelling approach – Kragten approximation

)(

)())((

i

iii

i xu

xfxuxf

x

f

)())(()()()( iiii

i

iii xfxuxfxux

fxucyu



Modelling approach – Monte Carlo Method



Empirical approaches

Data produced by single laboratory validation approach and analyzed by

ANOVA

Robust ANOVA

Range statistics

Data obtained by proficiency testing schemes

Standard uncertainty estimated as pooled reproducibility limit

Bayesian uncertainty analysis

Type A uncertainty evaluated through a Bayesianapproach



Measurement uncertainty and precision data are used in conformity assessment



SAMPLINGMEASUREMENT

PROCEDURE

REPORTING RESULTS AND

ASSESSING CONFORMITY

Chapter 3 Estimation of sampling uncertainty

Chapter 4Estimation of the uncertainty of a typical measurement procedure

Chapter 5Estimation of the uncertainty of a measurement procedure involving the construction of a calibration curve

Chapter 6Use of measurement uncertainty in conformity assessment



Sampling uncertainty is defined as the part of the total measurement uncertainty attributable to sampling

Empirical approach

Statistical model

analysissamplingtrue Xx2

analysis

2

sampling

2

tmeasuremen

2

analysis

2

sampling

2

tmeasuremen sss

sU 2



Experimental protocol and experimental design

Balanced nested

experimental design

Duplicate diesel samples

were taken from 104

petroleum retail stations

The sampling protocol used was consistent with the standard method ASTM D 4057

The duplicated samples were analyzed in duplicate under repeatability conditions for sulful mass content determination. (ANTEK 9000S sulfur analyzer - ASTM D 5453 /ISO 20846)

Sampling

target

Sample B

Sample A

Analysis A2

Analysis A1

Analysis B2

Analysis B1



Data analysis methods

1. Classical Analysis of Variance (ANOVA)

Variations associated with different sources (analysis and sampling) can be isolated and estimated



Data analysis methods (continued)

2. Robust Analysis of Variance

Particularly appropriate for providing estimated of variances, in cases where the validity of classical ANOVA is doubtful It is insensitive to distributional assumptions (such as normality)

It can tolerate a certain amount of unusual observations (outliers)

It uses robust estimates of the mean and standard deviation calculated by aniterative process (Huber’s method). Extreme values that exceed a certaindistance from the sample mean are downweighted or brought in.



Data analysis methods (continued)3. Range Statistics

The variance of sampling is calculated indirectly as the difference of the variances of measurement and analysis.

2

analysis2

tmeasuremen

2

sampling2

sss

128.1

analysis

analysis

Ds 128.1

tmeasuremen

tmeasuremen

Ds

Sampling

target

Sample B

Sample A

Analysis A2

Analysis A1

Analysis B2

Analysis B1



Results

•Robust ANOVA leads to statistically significantly different results (F-test) compared to the other two methodologies.

•Robust ANOVA, which is not influenced by less than 10% outliers, is considered as the method providing the most reliable estimates



Discussion

Different results is an indication that the assumptions of classical ANOVA and range statistics are not justified

Classical ANOVA and range statistics are strongly affected by the presence of outlying values (9 out of 104 datasets - 8.7 %).



Discussion (continued) The measurement uncertainty of manual sampling of fuels is

dominated by the analytical variance (accounts for the 71 % of the measurement uncertainty)

This leaves “room” for an effective reduction e.g. by making more measurements and calculating their average, instead of making a single measurement.

Then the standard deviation of the mean gets smaller as the number of data increases leading to smaller random error uncertainty contributions.

2

analysiss -20 % Expanded uncertainty of measurement



Test method studied

Gross Heat of Combustion (GHC) (or Higher Calorific Value) determination of a diesel fuel using a bomb calorimeter and following the standard method ASTM D240

Measurement principle

Heat of combustion is determined in this test method by burning a weighed sample in an oxygen bomb calorimeter under controlled conditions. The heat of combustion is computed from temperature observations before, during and after combustion, with proper allowance for thermochemical and heat transfer corrections.



Equipment used – Parr Instruments

Parr 6200 calorimeter

Parr 1108 oxygen bomb

Parr 6510 water handing system

Reference material

Benzoic acid traceable to NIST SRM 39j



Measurement system modeling

repgg

eeeWtQ

321

rept

eegQW '

'

''' 21

gSe 582



Uncertainty estimation methods

GUM uncertainty framework Assumed normal distribution

Assumed t-Student distribution (use of effective degrees of freedom)

Monte Carlo Method (MCM) Fixed number of trials

Adaptive MCM

GUM with Bayesian statistics

Empirical method using interlaboratory study (Proficiency Testing Scheme) data



GUM uncertainty framework vs Monte Carlo Method (MCM)

Formulation- Definition of the output quantity (measurand)

- Determination of the input quantities (sources of

uncertainty)

- Development of a model relating the output quantity

with the input quantities

- Assignment of PDFs to the input quantities on the

basis of available knowledge

Propagation- Propagation of the PDFs of the input quantities through

the model to obtain the PDF for the output quantity

Summarizing- Use of the PDF of the output quantity to obtain the

expectation (measurement result) of the output quantity

- Use of the PDF of the output quantity to obtain the

standard uncertainty associated with expectation

- Use of the PDF of the output quantity to obtain a

coverage interval containing the output quantity with a

specified probability

PDF: Probability Density Function

No difference






uncertainty)















x1, u(x1)

Y = f (X1,X2,X3) y, u(y)x2, u(x2)

x3, u(x3)

Assumed PDF for Y

GUM

MCM

PDF for X1

PDF for X2

PDF for X3

Y = f (X1,X2,X3)

PDF for Y

Propagation of uncertainties

Propagation of distributions






uncertainty)















GUM

MCM

U = k u(y)



GUM uncertainty framework – Uncertainty budget



GUM uncertainty framework – Uncertainty contributions



GUM uncertainty framework – Expanded uncertainty /

Coverage interval estimation

Assumed probability distribution

Standard uncertainty u(Qg) =141.7 J g-1 (33.84 cal g-1)

Normal (Gaussian) distribution

t- distribution

Expanded uncertainty U(Qg)=k . u(Qg)

k=1.96

k=2.08

22 effective degreesof freedom

U(Qg)=277.7 J g-1 (66.3 cal g-1)

U(Qg)=294.6 J g-1 (70.4 cal g-1)

Welch–Satterthwaite formula



Monte Carlo Method (MCM)– Development

The algorithm of MCM was developed in MATLAB®

1

2

…

n

Number of trials

o Fixed (106)

o Increasing number of trials until the results have stabilized in a statistical sense (Adaptive MCM)



Monte Carlo Method (MCM)– MATLAB code flow diagram

Establishment of the process parameters and the

default values of the control variables

Creation of M size row vectors for each of the input

variables

Evaluation of the model. M size file vector

Calculation of the average, standard deviation and

symmetrical interval of the M value sequence

Shortest interval?

Calculation of the shortest coverage interval of the M

value sequence

Value matrices and parameter vectors are formed

First sequence?

Add row to value matrices and parameter vectors

Calculation of the standard deviation of the

parameters

Calculation of the total standard deviation

Calculation of the numerical tolerance related to the

standard deviation

Stabilization?

Calculation of the average and the symmetrical

coverage interval of all the values

Shortest interval?

Calculation of the shortest coverage interval of all the

values

Show results

YES: Interval=1

YES: h=1

YES: Interval=1

YES: comp=1

NO

NO

NO

Lines 3-7*

Lines 8-49*

Lines 50-53*

Lines 54-58*

Lines 59-61*

Lines 62-67*

Lines 68-70*

Lines 71-73*

Lines 74-76*

Lines 77-81*

Lines 82-83*

Lines 84-97*

Lines 98-103*

Lines 104-108*

Lines 109-110*

Lines 111-117*

Lines 118-132*



Monte Carlo Method (MCM)– MATLAB code Flow diagram



Monte Carlo Method (MCM)–– Expanded uncertainty /

Coverage interval estimation

1 using a PC equipped with Intel® Core™ i3 M330, 2.13GHz, 4GB RAM



MCM results vs GUM results (95% coverage intervals)

12% underestimation

7% underestimation



GUM with Bayesian treatment of Type A uncertainties

)(3

1)( iiBayes xs

m

mxu

Standard uncertainty, u(Qg) = 160.7 Jg-1 (38 cal g-1)

95% coverage interval [44.88 – 45.51] MJ kg-1

or [10719 – 10870] cal g-1

Comparable to MCM results !!!



Uncertainty evaluated from proficiency testing data

zl

sl

sz

i

i

z

i

i

Ripooled

R

1

1

2)()1(

• The PTS provider is accredited according to ISO/IEC 17043

• Most of the participants used the standard method ASTM D240 for the measurement

li :number of participating laboratories in round i, z: number of rounds

95% expanded uncertainty 0.30 MJ kg-1 (71 cal g-1)

pooled

Rg sQU 96.1)(

-6,3 % compared to MCM



Calibration often comprises an important uncertainty component of the uncertainty of the whole analytical procedure

The slope and the intercept of a linear calibration model are only estimates based on a finite number of measurements

Therefore their values are associated with uncertainties



2 stage - measurement model

xbbY 10

1

00pred

b

byx

calibration data (pairs of xi yi)

0y

correlated



The standard uncertainty of a calibration curve used for the determination of sulfur mass concentration in fuels has been estimated using 4 methodologies:

GUM uncertainty framework

Kragten numerical method

Monte Carlo method (MCM)

Approximate equation calculating the standard error of prediction

n

i

i xxb

yy

Nnb

SExs

1

22

1

2

0

1

regression

pred

11)(

xbbY 10

1

00

predb

byx



Results

Mean value (ng μL-1)

Standard uncertainty

(ng μL-1)

GUM (correlation included) 8.000 0.175

Kragten method (correlation included) 8.000 0.172

MCM (correlation included) 8.003 0.175

Standard error of prediction equation

(including response uncertainty) 8.000 0.175

Standard error of prediction equation

(no response uncertainty included) 8.000 0.137

GUM (no correlation included) 8.000 0.283

Kragten method (no correlation

included) 8.000 0.279

MCM (no correlation included) 8.005 0.284

n

i

i xxb

yy

Nnb

SExs

1

22

1

2

0

1

regression

pred

11)(

202

2

predpred )()(')( yucxsxu

n

i

i xxb

yy

nb

SExs

1

22

1

2

0

1

regression

pred

1)('

Overestimation of uncertainty by 62%



Bivariate (or joint) Gaussian distribution N(E,V) characterized by the expectation and the covariance (or uncertainty) matrices, E and V

1b

bE

o

)(),(

),()(

1

2

10

100

2

bubbu

bbubuV

A coverage region can be determinedTwo types:•rectangle centered coverage region (separatelydetermined coverage intervals for b1 and b0).•ellipse centered coverage region

specifies a region in 2-dimensional space that contains E with

probability p

Treating calibration curve as a bivariate measurement model



(η – E)T V-1 (η – E) = kp2

2

11

00

1

1

2

10

100

2

1100)(),(

),()(pk

bn

bn

bubbu

bbububnbn

Rectangular and elliptical coverage regions (p=0.95)

p=0.95


Measurement uncertainty and precision data inconformity assessment

Certain approaches should be used to support reliable decisions in conformity assessment of fuels (EN 228, EN 590)

It is necessary to take into account thedispersion of the values that can beattributed to the measurand.



The comparison of the result with the specified requirements should be based on predefined decision rules, which are of key importance when the result is close to the tolerance limit

Use of guard bands to determine acceptance or rejection zones taking into account measurement variability

Guarded acceptance

Guarded rejection (Relaxed acceptance)

No rule



Guard bands minimize the probability of incorrect decisions (risks)

Guarded acceptance decision rules for upperand lower specification limits (TU, TL) andmaximum probability of falseacceptance (Type II error) when guardbands of width w are used

Guarded rejection (relaxed acceptance)decision rules for upper and lowerspecification limits (TU, TL) and maximumprobability of false rejection (Type Ierror) when guard bands of width w areused



Approaches for defining guard bands

The between laboratory precision data approach (ISO 4259 approach)

The intermediate precision (or uncertainty estimate) approach

ISO 4259:2006 Petroleum products -- Determination and application of precision data in relation to methods of test

Widely used in fuel market for resolution of disputes




Sulphur content test method limits

Product Automotive diesel

Specification 10 mg/kg

Dispute Sulphur content off specification

Test method ISO 20846

Reproducibility value R = 0.112*x + 1.12x = sample resultFor x=10, R = 2.24

Upper limit of guard band = TU+0.59*R=10+0.59*2.24=11.3 mg/kg

The product can be considered as FAILING the specification when a single test results falls above 11.3 mg/kg (for 95% confidence level)




krRR

1122

1

For multiple (k) test results

The expressions have been applied for all the parameters related toautomotive fuel quality referred in EN 590:2009 and EN 228:2008 andthe acceptance limits were calculated using the precision data of therelevant test methods



The between laboratory precision data approach (ISO 4259)

unleaded petrol (gasoline) EN228

automotive diesel EN 590




Sulphur content test method limits

Product Automotive diesel

Specification 10 mg/kg

Dispute Sulphur content off specification

Test method ISO 20846

Standard uncertainty value u = 0.31

Upper limit of guard band = TU+1.64*u=10+1.64*0.31=10.5 mg/kg

The product can be considered as FAILING the specification when a single test results falls above 10.5 mg/kg (for 95% confidence level)




2

analysis2

sampling

k

uuu

For multiple (k) test results



Automotive diesel fuel samples were taken from 769 petroleum retail stations and their sulfur mass content was determined in order to assess their compliance with the EU regulatory limit of 10 mg kg-1



The effect of different approaches for defining guard bands, different levelsof confidence or different number of replicate measurements isinvestigated

ISO 4259 approach



Intermediate precision approach



General remarks

The larger the width of the guard band used, the larger the proportion ofsamples that will be judged incorrectly

The differences in the calculations of the two approaches reflect a possiblenumber of samples judged incorrectly when using the ISO 4259 approach(uncertainty estimates represent more precisely the dispersion of the values ofthe measurand)

Minimizing the guard band width by reducing the measurement uncertainty(more replicates, more accurate measurement method) leads to fewer casesof false acceptance or false rejection decisions, reducing as well thecosts associated with these decisions.

At the same time the cost of analysis becomes higher, there is a need that these two costs are balanced against each other in order to find an optimum (target) measurement uncertainty


Conclusions

SAMPLINGMEASUREMENT

PROCEDURE



Statistical and numerical methods have been developed and/or applied concerning the estimation and use of the measurement uncertainty in all parts of the measurement process.


Conclusions

SAMPLINGMEASUREMENT

PROCEDURE



Manual sampling from petroleum retail stations for sulphur determination

Three alternative empirical statistical approaches

Expanded uncertainty of sampling estimated in the range of 0.34 – 0.40 mg kg-1.

The estimation of robust ANOVA (0.40 mg kg-1) is considered more reliable, because of the presence of outliers


Conclusions

SAMPLINGMEASUREMENT

PROCEDURE



Gross Heat of Combustion of diesel fuel

Two alternative modelling approaches (GUM and Monte Carlo Method)

GUM approaches (Gaussian or t-distribution) underestimate measurement uncertainty

Overall Monte Carlo Method is a more reliable tool (subject to fewer assumptions) (0.32 MJ kg-1)

Bayesian treatment of Type A uncertainties “corrects” GUM estimates


Conclusions

SAMPLINGMEASUREMENT

PROCEDURE



Calibration curve construction (Sulphur content determination)

Four alternative approaches (GUM, Monte Carlo Method, Kragten, standard error equation)

All approaches agree well (std uncertainty 0.172 – 0.175 ng μL-1)

Importance of correlation – correct use of standard error equation


Conclusions

SAMPLINGMEASUREMENT

PROCEDURE



Conformity assessment of automotive fuel products (EN 590, EN 228)

Acceptance limits for guarded acceptance and guarded rejection for 95% and 99% confidence levels

Significant differences in the resulting number of non-conforming results when using different approaches for defining guard bands, different levels of confidence or different number of replicate measurements


Applications

The program codes developed in MATLAB in order to apply the Monte Carlo method (adaptive and fixed trials) may be used in any type of measurement

Decision Support System for the Evaluation of Conformity of Fuel Products (key features defined)


Future Work

Bayesian uncertainty analysis – Development of numerical techniques

Conformity assessment – Take into account the variability of both the measuring system and the process (production system)

Sampling - Development of methods for the estimation and the inclusion of sampling bias


List of publications - Conferences

PUBLICATIONS

1. D. Theodorou, Y. Zannikou, G. Anastopoulos, F. Zannikos. Coverage interval estimation of the measurement of Gross Heat of Combustion of fuel by bomb calorimetry: Comparison of ISO GUM and

adaptive Monte Carlo method. Thermochimica Acta (2011) 526: 122– 129

2. D. Theodorou, Y. Zannikou, F. Zannikos. Estimation of the standard uncertainty of a calibration curve:

application to sulfur mass concentration determination in fuels. Accreditation and Quality Assurance (2012) 17: 275–281

3. D. Theodorou, N. Liapis, F. Zannikos. Estimation of measurement uncertainty arising from manual

sampling of fuels. Talanta (2013) 105: 360-365

4. D. Theodorou, F. Zannikos. The use of measurement uncertainty and precision data in conformity

assessment of automotive fuel products. Measurement (2014) 50: 141-151

5. D. Theodorou, Y. Zannikou, F. Zannikos. Components of measurement uncertainty from a measurement

model with two stages involving two output quantities. Chemometrics and Intelligent Laboratory Systems (2015) 146: 305–312

CONFERENCES

5th National Congress on Metrology "Metrologia 2014”

EuroAnalysis 2013 - XVII European Conference on Analytical Chemistry

4th National Congress on Metrology "Metrologia 2012”


Acknowledgments

Advisory Committee

F. Zannikos –Professor – School of Chemical Engineering, NTUA (Supervisor)

K. Tzia - Professor – School of Chemical Engineering, NTUA

D. Karonis – Associate Professor - School of Chemical Engineering, NTUA

Fuels and Lubricants Technology Laboratory

Staff

Partners

Graduate / Post graduate students


ΤΗΑΝΚ YOU FOR YOUR ATTENTION !

ΕΥΧΑΡΙΣΤΩ ΓΙΑ ΤΗΝ ΠΡΟΣΟΧΗ ΣΑΣ !

Documents

phd thesis presentation