NIR
Åsm
un
d R
innan, K
U-LIFE
, Denm
ark, a
ar@
life.ku
.dk
Pre-processing of NIR
Åsmund Rinnan
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
IntroductionNIR
FructoseGlucose255075
755025
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
IntroductionNIR
A “fat” baseline
Baseline with a slope/ curve
NonlinearityFructoseGlucose
255075
755025
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
IntroductionTechniques
• Reference dependent– O-PLS– OSC– OS– SIS
• Reference independent– MSC/ ISC– EMSC/ EISC– SNV– Detrend– Normalization– Savitzky-Golay– Norris-Williams– Finite
difference
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
IntroductionTechniques
• Correction of light scatter– MSC– EMSC– Detrend– SNV
• Derivation– Finite
difference– Savitzky-Golay
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
Effect of pre-processingBefore
Specular effect
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
Effect of pre-processingAfter
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
MSC
Raw Reference
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
MSC
Raw Reference
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
MSC
Reference
Raw
sp
ect
rum
1. P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500
a = Intercept
b = Slope
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
MSC
b
a raw
new
xx
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
...xxλλx known,2known,122
ref
Extended MSC
eb...λλxx1x 22refrefraw
refraw xx 1
Slope & Intercept
eb
=Basic MSC
=Detrend
Wavelengthcorrection
Known spectra
1. H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
2,
2,21,1,
2
4
ref
refrefref
b
bbb λ
new
xx
Extended MSC
bλλ1xx rawλ2
1,refbλ
new
xx
eb...λλxx1x 22refrefraw
Calculated
The correction
Step 1:
Step 2:
1. H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
Extended MSCSummary
ebλλλxx1x 322refrefraw X X
ebλλx1x 2refraw
ebxx refraw 1
ebλλ1x 2raw
1.Parameter setting
2.Parameter setting
3.Paraemter setting
MSC
Detrend
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
SNV
x
orgx
orgnew
xx
1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
SNV vs MSC
b
a org
new
xx
MSC SNV
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
SNV vs. MSCClassificafication of
barley
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
SNV vs. MSC
MSC• Reference
needed• Outlier
sensitive• Only similar
spectra• No
normalization
SNV• Normalization• No reference
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
Derivation
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
DerivationFinite difference
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
Savitzky-Golay vs.Finite difference
Derivativ
e
Method 1 2 3 4
1st
SG0.38
(5)
0.40
(5)
0.47
(6)
0.46
(3)
Finite
difference
0.34
(5)
0.58
(4)
0.73
(4)
0.46
(3)
2nd
SG0.34
(6)
0.51
(4)
0.67
(5)
0.46
(3)
Finite
difference
0.72
(4)
0.88
(5)
0.91
(5)
0.42
(6)
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
DerivationSavitzky-Golay
1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
DerivationSavitzky-Golay
1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639
359711Smoothing
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
DerivationSavitzky-Golay
12345
1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639
7Smoothing Polynomial
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
DerivationSavitzky-Golay
7 point smoothing 15 point smoothing
2nd order derivative
NIR
Åsm
und R
innan, K
U-LIF
E: a
ar@
life.ku
.dk
SummaryPre-processing of NIR
• Run the following for near to optimal results
• MSC with– Only 1st order reference correction or
SNV– 1st order reference correction and 2nd order wavelength
correction– Only 2nd order wavelength correction
• Savitzky-Golay derivation with– 2nd order polynomial smoothing– 7 points smoothing for the 1st derivative– 9 points smoothing for the 2nd derivative