NIR

Åsm

un

d R

innan, K

U-LIFE

, Denm

ark, a

ar@

life.ku

.dk

Pre-processing of NIR

Åsmund Rinnan

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

IntroductionNIR

FructoseGlucose255075

755025

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

IntroductionNIR

A “fat” baseline

Baseline with a slope/ curve

NonlinearityFructoseGlucose

255075

755025

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

IntroductionTechniques

• Reference dependent– O-PLS– OSC– OS– SIS

• Reference independent– MSC/ ISC– EMSC/ EISC– SNV– Detrend– Normalization– Savitzky-Golay– Norris-Williams– Finite

difference

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

IntroductionTechniques

• Correction of light scatter– MSC– EMSC– Detrend– SNV

• Derivation– Finite

difference– Savitzky-Golay

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

Effect of pre-processingBefore

Specular effect

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

Effect of pre-processingAfter

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

MSC

Raw Reference

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

MSC

Raw Reference

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

MSC

Reference

Raw

sp

ect

rum

1. P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500

a = Intercept

b = Slope

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

MSC

b

a raw

new

xx

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

...xxλλx known,2known,122

ref

Extended MSC

eb...λλxx1x 22refrefraw

refraw xx 1

Slope & Intercept

eb

=Basic MSC

=Detrend

Wavelengthcorrection

Known spectra

1. H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

2,

2,21,1,

2

4

ref

refrefref

b

bbb λ

new

xx

Extended MSC

bλλ1xx rawλ2

1,refbλ

new

xx

eb...λλxx1x 22refrefraw

Calculated

The correction

Step 1:

Step 2:

1. H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

Extended MSCSummary

ebλλλxx1x 322refrefraw X X

ebλλx1x 2refraw

ebxx refraw 1

ebλλ1x 2raw

1.Parameter setting

2.Parameter setting

3.Paraemter setting

MSC

Detrend

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

SNV

x

orgx

orgnew

xx

1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

SNV vs MSC

b

a org

new

xx

MSC SNV

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

SNV vs. MSCClassificafication of

barley

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

SNV vs. MSC

MSC• Reference

needed• Outlier

sensitive• Only similar

spectra• No

normalization

SNV• Normalization• No reference

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

Derivation

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

DerivationFinite difference

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

Savitzky-Golay vs.Finite difference

Derivativ

e

Method 1 2 3 4

1st

SG0.38

(5)

0.40

(5)

0.47

(6)

0.46

(3)

Finite

difference

0.34

(5)

0.58

(4)

0.73

(4)

0.46

(3)

2nd

SG0.34

(6)

0.51

(4)

0.67

(5)

0.46

(3)

Finite

difference

0.72

(4)

0.88

(5)

0.91

(5)

0.42

(6)

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

DerivationSavitzky-Golay

1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

DerivationSavitzky-Golay

1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

359711Smoothing

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

DerivationSavitzky-Golay

12345

1. A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

7Smoothing Polynomial

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

DerivationSavitzky-Golay

7 point smoothing 15 point smoothing

2nd order derivative

NIR

Åsm

und R

innan, K

U-LIF

E: a

ar@

life.ku

.dk

SummaryPre-processing of NIR

• Run the following for near to optimal results

• MSC with– Only 1st order reference correction or

SNV– 1st order reference correction and 2nd order wavelength

correction– Only 2nd order wavelength correction

• Savitzky-Golay derivation with– 2nd order polynomial smoothing– 7 points smoothing for the 1st derivative– 9 points smoothing for the 2nd derivative