Seoul National University Neural Network Modeling for Intelligent Novelty Detection 제 2 차 뇌신경정보학 Workshop 일시 : 2002 년 2 월 27 일 ( 수 ) 10:00-18:00 장소

SeoulNationalUniversity

Neural Network Modeling for Intelligent Novelty Detection

제 2 차 뇌신경정보학 Workshop

일시 : 2002 년 2 월 27 일 ( 수 ) 10:00-18:00 장소 : KAIST LG Semicon Hall 1 층 강당

제 2 차 뇌신경 정보학 Workshop

Introduction

Concept of Novelty Detection

– Typical Classification

Class A

Class B

Classifier

A New Instance

Class A or Class B

Training

Classification

Class A

Novelty Detector

A New Instance

Class A or NOT

Training

Classification

– Novelty Detection


Introduction

Applications of Novelty Detection– Authentication of computer system– Detection of counterfeit– Detection of phase transformation in a financial market– Fault detection in a mechanical system

Algorithms for Novelty Detection– Principal Component Analysis (PCA)– Auto-Associative Multi-Layer Perceptron (AAMLP)– Probability Density Estimator : SOM, Mixture of Kernels, etc.


Characteristics of 2-Layer AAMLP

Structure–

–

– Auto-Association Input vectors = Target vectors Normal patterns Smaller errors Novel patterns Larger errors

– Training Algorithms Same as a typical MLP Back-propagation, Gradient-descent, Levenberg-Marquadt, etc.

x1e1

1xf

)( xxf2 )(

Hidden LayerHidden Layer Output LayerOutput Layer

mpm nodesMLP with pm



Properties (Lee, Hwang, Cho; submitted) – A 2-layer AAMLP defines an output-constrained hyperplane– The hyperplane is bounded– The hyperplane lies within the training input vector area– It is trained so that the association error, the sum of distances between the hyperplane and the input vectors, is minimized


Experiments with 2-Layer AAMLP

Comparison with 2-Layer AAMLP (Linear Ftn.) (=PCA)

– AAMLP with Linear Ftn.

– AAMLP with Linear Ftn.



Comparison with 2-Layer AAMLP (Linear Ftn.) (=PCA)

– AAMLP with Linear Ftn. – AAMLP with Linear Ftn.



2-Layer AAMLP with Saturated Linear Ftn.

Equivalent to 2-layer AAMLP with sigmoid ftn.



Limitations of 2-Layer AAMLP

Inability to model a non-linear data



4-Layer AAMLP (Kramer, 1991)

–

– Non-Linear PCA

– Limitations of NLPCA (Malthouse, 1998)

Good at interpolation

But not at extrapolation

A desirable property as a novelty detector

x1e1

1xf

)( xxf2 )(

Mapping LayerMapping Layer

De-Mapping LayerDe-Mapping LayerBottleneck LayerBottleneck Layer

Output LayerOutput Layer



Novelty Detection in A Non-Linear Data

– 4-Layer AAMLP with Sigmoid Ftn.

Ability to model a non-linear data



Novelty Detection in A Non-Linear Data– 4-Layer AAMLP with Saturated Linear Ftn.

Ability to model a non-linear data



Novelty Detection in A Non-Linear Data– Similarity of Sigmoid and Saturated Linear Function

– AAMLP with Sigmoid and Saturated Linear Function Assumption AAMLP-SatLin is similar to 2-layer AAMLP in the output characteristics & is similar to 4-layer AAMLP in the shape of output vectors Hypothesis AAMLP-Sigmoid defines an output-constrained hypersurface

& can minimizes the association error, at least, locally



Novelty Detection in A Multi-Modal Data– 4-Layer AAMLP with Sigmoid Ftn.

Inability to model multi-modal data



Novelty Detection in A Multi-Modal Data– Clustering + 4-Layer AAMLP with Sigmoid Ftn.

Ability to model a multi-modal data



Properties– A 4-layer AAMLP with sigmoid ftn. defines a hypersurface– A one with saturated linear ftn. defines a set of segments– A one with saturated linear ftn. can model a non-linear data in a way similar to that of 2-layer AAMLP– It can be argued that a one with sigmoid ftn. can model a non-linear data & minimizes the association error– A one with sigmoid ftn. cannot model a multi-modal data– A one with sigmoid ftn. can model a multi-modal data if it is preceded by proper clustering algorithms


Conclusion

Conclusion– A 2-layer AAMLP can be a novelty detector for a linear data– A 4-layer one can be a novelty detector for some non-linear data, which a 4-layer one cannot– A 4-layer one can be a novelty detector for a multi-modal data, if the data is pre-clustered


Future Works

Future Works– Probability Density Estimators : SOM, Mixture of Kernels, etc.

Comparison of AAMLP and the density estimators in performances Sophisticated clustering methods using the density estimators

– AAMLP Ensemble The unstableness of AAMLP An AAMLP ensemble may overcome Bagging & Majority voting

Documents

Seoul National University Neural Network Modeling for Intelligent Novelty Detection 제 2 차 뇌신경정보학 Workshop 일시 : 2002 년 2 월 27 일 ( 수 ) 10:00-18:00 장소