Upload
dothuan
View
220
Download
4
Embed Size (px)
Citation preview
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
Chapter 8:Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
In the previous seven chapters, it was assumed that only random, normally distributed errors (with zero mean and known variance) were present in the data.
However, real process data can often include other types of errors, due mainly to instrument malfunction and process leaks. These gross errors are non-random and completely invalidate the statistical basis of data reconciliation techniques.
This is why it is necessary to have methods to check a data set for gross errors. If any data is found to contain gross errors, it must be removed prior to the reconciliation.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
Of course, there are many possible ways to detect gross errors, but the methods discussed in this chapter will be based on how well statistically the data satisfies the balance equations.
The first method discussed, the global test, is fast and useful. As its name implies, the global test can determine whether or not a data set contains gross error(s), but will not indicate precisely where.
Therefore, it is a good preliminary analysis, in order to see iffurther action will be required.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
As shown earlier in the module, the reconciled measurement vector (without gross errors) can be written as:
(8.1)
where y is the vector of redundant raw measurements, and is the vector of random errors. Also, it must be assumed that:
1) Expected value of , E( ) = 02) Covariance matrix, V, is known.
CHAPTER 8
Gross Error Detection
ε+= yy
εε
ε
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
Obviously, gross errors can only be checked for in data that has been measured, so unmeasured data, z, and its corresponding balance equations, Az, need not be considered in the analysis. Therefore, the only balances to be reconciled correspond to:
(8.2)
where G in this case (and for the remainder of Chapter 8) corresponds to the reduced subset of redundant equations(G=Q2
TAy) and y corresponds to the redundant measured variables.
Due to random error, there is bound to be residua, r, leftover in the balances, such that:
(8.3)
CHAPTER 8
Gross Error Detection
0Gy =
εεε GGGy)G(yr =+=+=
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
The global test is based on statistical hypothesis testing, where the null hypothesis, H0 (no gross error is present), is tested against the data set. Under this hypothesis, if the random errors are normally distributed, then Equation 8.3implies that the residua are also normally distributed, and:
(8.4)
Also, the propagation of this error, the covariance matrix of residuum, can be shown by:
(8.5)
CHAPTER 8
Gross Error Detection
0)GE()E(GE(r) === εε
TTTT GVG)GGE()E(rr ===Φ εε
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
The global test function, then, is formulated as:
(8.6)
If G has full row rank equal to m, then has a chi-squared distribution with m (# of equations) degrees of freedom. Therefore, at some specified level of significance, (usually0.05 or 0.10):
(8.7)
For a specific , a critical value of , , can be calculated. If for a data set is larger than this critical , then the data set has a (1- )% chance of containing gross error.
CHAPTER 8
Gross Error Detection rr -1T Φ=τ
τ
τ cττ
α
αcτ
α
αχτ α =≥ − m))(P( 12
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
For example, consider the following redundant measured flow rates, mass balance equations, and covariance matrix:
CHAPTER 8
Gross Error Detection
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
3.88001.22954.79350.1858
y⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−−−−
=2.06.03.01.01.02.01.08.07.02.06.01.0
G
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
0.0400000.00057600000.002500000.000289
V
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
Calculating the covariance matrix of residuum, and using the measured data in the three mass balance equations to find the residuals, results in:
Using Equation 8.6:
CHAPTER 8
Gross Error Detection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
0.0571-0.0059-0.0672-
r⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=Φ
0020.00010.00061.00010.00006.00030.00061.00030.00205.0
8.5347rr -1T =Φ=τ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.1 Global Test
The rank of G is equal to three, so the critical value with 3 degrees of freedom and = 0.05 is:
Since > , there is a 95% chance that the data set contains gross errors in one or more of the measurements.
CHAPTER 8
Gross Error Detection
τ
cτα
cτ
7.815 (3)0.952
c == χτ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Once a gross error has been detected in the data set, the next step is to determine where (which measurements) it has occurred.
Serial elimination is a technique which can test one or more redundant raw measurements and determine if gross errors are present. In this section, it will be shown how to use serial elimination to detect gross errors by going through the data setsequentially, testing one measurement at a time.
Since a different method to test multiple measurements for gross errors will be discussed in the next section, readers interested in using serial elimination for this purpose are encouraged to consult the Suggested Readings.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Assuming that c measurements have gross errors and that g-c do not, the matrix G may be divided as follows:
(8.8)
Where Ag contains the columns of the measurements without suspected gross errors and Ac contains the column(s) of the measurements assumed to have gross errors.
If there is no correlation between the measurements, then the covariance matrix can be divided in the same manner:
(8.9)
CHAPTER 8
Gross Error Detection
[ ]cg AAG =
⎥⎦
⎤⎢⎣
⎡Δ+
=VV0
0VV
c
gn
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
where is the increase in the variance of the c measurements (very large) due to gross error.
The new covariance matrix of residuum, , can be calculated as follows:
(8.10)
The best way to calculate the inverse of this, , is to use the following matrix inversion lemma:
CHAPTER 8
Gross Error Detection
VΔ
nΦ
-1nΦ
Tn n
T Tg g g c c c
Tc c
GV GA V A A (V V)A
A ( V)A
Φ =
= + + Δ
= Φ + Δ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Let Y and Z be nonsingular matrices of order (nxn) and (pxp) respectively, and U a (nxp) matrix.
If (8.11)
Then (8.12)
Applying this lemma to find the inverse of Equation 8.10results in:
(8.13)
CHAPTER 8
Gross Error Detection
TUZUYX ±=
-1T-1-1T-1-1-1-1 YUU]YUU[ZYYX ±= m
}A]AAV)[(A-{I -1Tc
-1c
-1Tc
-1c
-1-1n ΦΦ+ΔΦ=Φ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Since is assumed to be large, its inverse in Equation 8.13can be assumed to be zero, resulting in this modification:
(8.14)
Finally, the value of the least squares objective function, J, can be calculated as shown:
(8.15)
CHAPTER 8
Gross Error Detection
}A]A[AA-{I -1Tc
-1c
-1Tcc
-1-1n ΦΦΦ=Φ
VΔ
T -1nJ r r= Φ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Now, it can be determined which measurements contain gross errors by following these steps:
1) Remove measurement i from the data set, and use its corresponding column from G to make the Ac matrix.
2) Calculate using this Ac matrix as shown in Equation 8.14.
3) Calculate the objective function that results from removing measurement i, Ji.
4) Repeat Steps 1-3 for measurements i+1, i+2, etc.5) If Ji is very small (and smaller than in particular), then
measurement i contains gross errors (obtaining a low value for an objective function indicates less error, thus the measurement removed to obtain a low objective function must contain the error).
CHAPTER 8
Gross Error Detection
-1nΦ
τ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
To illustrate the use of serial elimination, consider the example shown in the last section. Using the global test, it was found that there was a gross error in the data set. Now, by followingthe steps shown on the previous slide, it can be found which measurement contains the gross error.
Step 1:
Removing y1 from the data set results in this Ac matrix:
CHAPTER 8
Gross Error Detection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
0.10.80.1
Ac
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Step 2:
Step 3:
CHAPTER 8
Gross Error Detection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
ΦΦΦ=Φ
5057.6452.1-1440.7-452.1-41.2122.5
1440.7-122.5460.9}A]A[AA-{I -1T
c-1
c-1
cc-1-1
n
7.2953rrJ -1
nT
1
=Φ=
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.2 Serial Elimination
Step 4:
Carrying through Steps 1-3 on the remaining 3 measurements results in these 3 objective functions:
Step 5:
Since J2 is the smallest objective function, we can conclude that it is measurement y2 that contains the gross error. It is also possible that measurement y3 may contain a gross error, as its objective function is smaller than as well.
CHAPTER 8
Gross Error Detection
0.9636J2 = 1.5702J3 = 8.4374J4 =
τ
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
Although serial elimination is a very effective method for finding the location of gross errors, there is a more efficient approach that combines serial elimination (the sequential treatment of measurements) together with the sequential processing of constraints. For this reason, it is called the combined procedure.
First of all, consider the overall problem consisting of m balances and divide it into m sub problems. That is, split the constraint matrix of redundant measured variables, G, into m individual row matrices, A1, A2, A3, etc. This is done so that one equation may be processed at a time.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
For the first equation, A1, the corresponding covariance matrix, V1, is given by:
(8.16)
and the estimate of the measurement errors is:
(8.17)
Also, by replacing the redundant measurement matrix, y, in Equation 8.17 with the covariance matrix, V1, we can obtain the covariance of the estimation error:
(8.18)
CHAPTER 8
Gross Error Detection
VV1 =
yA)AV(AAV 1-1T
111T
111 =ε
11-1T
111T
11 VA)AV(AAVV1
=ε
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
However, when processing the second equation, adjustments must be made by using the estimates of the measurement errors found while processing the first equation. In this way, the estimates for the measurement errors are constantly updated using new information and will more closely approach their true values.
The covariance matrix corresponding to A2 is given by:
(8.19)
or more generally by:
(8.20)
CHAPTER 8
Gross Error Detection
1V-VV 12 ε=
1-iV-VV 1-ii ε=
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
Similarly, the error estimates and their covariance for A2 can be shown by:
(8.21)
(8.22)
Or more generally by:
(8.23)
(8.24)
CHAPTER 8
Gross Error Detection
)-(yA)AV(AAV 12-1T
222T
222 εε =
22-1T
222T
22 VA)AV(AAVV2
=ε
)-(yA)AV(AAV 1-ii-1T
iiiT
iii εε =
ii-1T
iiiT
ii VA)AV(AAVVi
=ε
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
For each equation processed, its least squares objective function can be calculated:
(8.25)
Assuming a normal distribution of the errors, this objective function can be combined with a -test, indicating one or more gross errors in the equation if:
(8.26)
CHAPTER 8
Gross Error Detection
i-1T
iiiT
ii r)AV(ArJ =
2χ
(i)J -12
i αχ>
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.3 The Combined Procedure
Once it has been determined that a constraint contains a gross error, serial elimination can be used among its measurements to find precisely where. The measurement that has a much lower objective function (from Equation 8.15) will contain the gross error.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.4 Estimation of Gross Error Magnitude
Now that a gross error has been detected in a measurement, it is a simple matter to correct it. The measurement can be removed from the data set, the remaining data reconciled, and the faulty measurement estimated from the reconciled data.
However, this is not always desirable or even possible. This section, then, will discuss how to estimate the bias in the faulty measurement and incorporate it into the reconciliation.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.4 Estimation of Gross Error Magnitude
Since it is assumed that there are one or more measurements containing gross errors, a constant bias of magnitude mb is added to the redundant raw measurement vector, y. The measurement equation can now be written in the form:
(8.27)
where B is a (gxs) matrix of ones and zeroes (ones corresponding to faulty measurements and zeroes corresponding to unbiased measurements), s is the number of gross errors in the data, and g is the number of measurements.
CHAPTER 8
Gross Error Detection
bBmyy ++= ε
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.4 Estimation of Gross Error Magnitude
Therefore, the data reconciliation problem becomes:
minimizing(8.28)
subject to
Solving this optimization problem by using Lagrange multipliers results in:
(8.29)
CHAPTER 8
Gross Error Detection
)y-(yV)y-(y)yJ( -1T=
0)Bm--yG( b =ε
]mP-[Gy)(GVGVG-yy bb-1TT=
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.4 Estimation of Gross Error Magnitude
where(8.30)
and(8.31)
While Equation 8.29 can be used to find new estimates for the reconciled values, it is perhaps easier to simply adjust the suspected measurement by its bias and redo the reconciliation.
CHAPTER 8
Gross Error Detection Gy)(GVGP]P)(GVG[Pm -1TT
b-1
b-1TT
bb =
BAP yb =
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.5 Quiz
Question 1:Gross errors have(a) zero mean and a large variance.(b) non-zero mean and a large variance.(c) non-zero mean and known variance.(d) non-zero mean and unknown variance.
Question 2:When checking a data set for gross errors(a) only measured variables are tested.(b) only redundant measured variables are tested.(c) all measured variables and observable unmeasured variables are tested.(d) all variables are tested.
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.5 Quiz
Question 3:The most efficient method to test for the presence of gross errors is(a) the global test.(b) serial elimination.(c) the combined procedure.(d) (b) or (c).
Question 4:Calculation of a low value for an objective function indicates(a) a gross error if using serial elimination.(b) a gross error if sequentially processing constraints.(c) no gross errors if using serial elimination.(d) (b) or (c).
CHAPTER 8
Gross Error Detection
Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation
© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004
8.6 Suggested Readings
Romagnoli, J.A. and Sanchez, M.C. (2000). “Data Processing and Reconciliation for Chemical Process Operations”. Academic Press, San Diego.
CHAPTER 8
Gross Error Detection