41
Computational Intelligence John Sum Institute of Technology Management National Chung Hsing University Taichung, Taiwan ROC

Computational Intelligence

  • Upload
    kizzy

  • View
    64

  • Download
    0

Embed Size (px)

DESCRIPTION

Computational Intelligence. John Sum Institute of Technology Management National Chung Hsing University Taichung, Taiwan ROC. OUTLINE. Historical Background Computational Intelligence Example Problems Methodology Model Structure Model Parameters Parametric Estimation Discussion - PowerPoint PPT Presentation

Citation preview

Page 1: Computational Intelligence

Computational Intelligence

John Sum

Institute of Technology Management

National Chung Hsing University

Taichung, Taiwan ROC

Page 2: Computational Intelligence

John Sum Computational Intelligence 2

OUTLINE

Historical Background Computational Intelligence Example Problems Methodology

Model Structure Model Parameters Parametric Estimation

Discussion Conclusion

Page 3: Computational Intelligence

John Sum Computational Intelligence 3

HISTORY

Page 4: Computational Intelligence

John Sum Computational Intelligence 4

HISTORY

1940 – First computing machine 1957 – Perceptron (First NN model) 1965 – Fuzzy Logic (Rules) 1960s – Genetic Algorithm 1970s – Evolutionary Computing

Page 5: Computational Intelligence

John Sum Computational Intelligence 5

HISTORY

1980s Neural Computing Swarm Intelligence

1990s (Hybrid) Fuzzy Neural Networks NFG, FGN, GNF, etc

Page 6: Computational Intelligence

John Sum Computational Intelligence 6

HISTORY

Beyond 1990s: Research areas converge Computational Intelligence Softcomputing Intelligent Systems

Covering Adaptive Systems Fuzzy Systems Neural Networks Evolutionary Computing Data Mining

Page 7: Computational Intelligence

John Sum Computational Intelligence 7

COMPUTATIONAL INTELLIGENCE Computational Intelligence

Heuristic algorithms (or models) such as in fuzzy systems, neural networks and evolutionary computation.

Techniques that use Simulated annealing, Swarm intelligence, Fractals and Chaos Theory, Artificial immune systems, Wavelets, etc.

Page 8: Computational Intelligence

John Sum Computational Intelligence 8

COMPUTATIONAL INTELLIGENCE Goal: Problem Solving

Financial forecast Customer segmentation (CRM) Supply chain design (SCM) Business process re-engineering System control Pattern recognition Image compression Homeland security

Page 9: Computational Intelligence

John Sum Computational Intelligence 9

COMPUTATIONAL INTELLIGENCE Underlying structure of the model is unknown, or the

model is known but it is too complicated Example: DJI versus HIS (Time Series)

Define system structure NL model (NN, ODE, etc.) Rule-based system

Parametric estimation Deterministic search (Gradient descent or Newton’s

method) Stochastic search (SA or MCMC)

Page 10: Computational Intelligence

John Sum Computational Intelligence 10

COMPUTATIONAL INTELLIGENCE Underlying model structure is known Example: Manufacturing process (SCM)

Define the objective to be maximized Examples: Completion time, Cost, Profit

Optimization Linear programming, ILP, NLP Deterministic search (Gradient descent or Newton’s

method) Stochastic search (SA or MCMC)

Page 11: Computational Intelligence

John Sum Computational Intelligence 11

EG1: Nonlinear Dynamic System

x )(xg

Noise

y

system

Unknown

Page 12: Computational Intelligence

John Sum Computational Intelligence 12

EG2: Nonlinear Function

x )(xg

Noise

y

system

Unknown

Page 13: Computational Intelligence

John Sum Computational Intelligence 13

EG3: Car Price

Predict the price of a car based on Specification of an auto in terms of various

characteristics Assigned insurance risk rating Normalized losses in use as compared to other

cars

Number of attributes: 25 Missing values: Yes!

Page 14: Computational Intelligence

John Sum Computational Intelligence 14

EG3: Car Price

Page 15: Computational Intelligence

John Sum Computational Intelligence 15

EG4: Purchasing Preference

Page 16: Computational Intelligence

John Sum Computational Intelligence 16

EG5: Financial Time Series

7000

8000

9000

10000

11000

12000

13000

14000

1 159 317 475 633 791 949 11071265142315811739

Page 17: Computational Intelligence

John Sum Computational Intelligence 17

EG5: Financial Time Series

What would happen in the next trading day? (Time series prediction problem) Closing value Open value UP or DOWN

Time series prediction + trading rules What should I do tomorrow? HOLD, SELL or BUY When should I BUY and SELL?

Page 18: Computational Intelligence

John Sum Computational Intelligence 18

System Structure Data Types Model

Dynamic System

Unknown Continuous RNN, Fuzzy NN

Nonlinear Function

Unknown Continuous BPN, RBF, Fuzzy NN

Car Price UnknownContinuous

DiscreteBPN, RBF, Fuzzy NN

Purchasing Preference

Known (SEM)

DiscreteSEM

Bayesian Net

Financial Time Series

Unknown Continuous BPN, RBF, Fuzzy NN

Remarks on EG1 ~ EG5

Page 19: Computational Intelligence

John Sum Computational Intelligence 19

COMPUTATIONAL INTELLIGENCE

Statement of Problem Given a set of data collected (or measured) from a system (probably an unknown system), devise a model (by whatever structure, technique, method in CI) that mimics the behavior of that system as ‘good’ as possible.

Making use of the devised model to (1) interpret the behavior of the system, (2) predict the future behavior of the system, (3) control the behavior of the system, (4) make money.

Page 20: Computational Intelligence

John Sum Computational Intelligence 20

METHODOLOGY

Step 1: Data Collection Experiments or measurements Questionnaire Magazine Public data sets

Step 2: Model Structure Assumption IF it is known, SKIP this step. ELSE, DEFINE a model structure

Page 21: Computational Intelligence

John Sum Computational Intelligence 21

METHODOLOGY

Step 3: Parametric Estimation Gradient descent Newton’s method Exhaustive search Genetic algorithms (*) Evolutionary algorithms (*) Swarm intelligence Simulated annealing (*) Markov Chain Monte Carlo (*)

Page 22: Computational Intelligence

John Sum Computational Intelligence 22

METHODOLOGY

Step 4: Model Validation (is it a reasonable good model) Hypothesis test Validation/Testing set Leave one out validation

Step 5: Model Reduction (would there be a simpler model that is also reasonable good) AIC, BIC, MDL Pruning (using testing set)

Page 23: Computational Intelligence

John Sum Computational Intelligence 23

METHODOLOGY

Beyond Model Reduction Any redundant input Any redundant sample (or outlier) Any better structure (alternative) How do we determine a ‘good’ model

Page 24: Computational Intelligence

John Sum Computational Intelligence 24

NN MODEL STRUCTURES

Perceptron Multilayer Perceptron (MLP or BPN) Adaptive Resonance Theory Model (ART) Competitive Learning (CL) Hopfield Network, Associative Network Bidirectional Associative Model (BAM) Recurrent Neural Network (RNN) Boltzmann Machine Brain-State-In-A-Box (BSB) Radial Basis Function Network (RBF Net) Bayesian Networks Self Organizing Map (SOM or Kohonen Map) Learning Vector Quantization (LVQ) Support Vector Machine (SVM) Support Vector Regression (SVR) PCA, ICA, MCA Winner-Take-All Network (WTA) Spike neural networks

Remarks Not all of them is able to learn,

eg BSB, WTA Might need to combine two str

uctures to solve a single problem

Multiple definitions on the ‘neuron’

Page 25: Computational Intelligence

John Sum Computational Intelligence 25

NN MODEL STRUCTURES

Supply Chain Management (Optimization Problem) Hopfield Network

Customer Segmentation (Clustering Problem) CL, SOM, LVQ, ART

Dynamic Systems Modeling RNN, Recurrent RBF

Car Price/NL Function (Function Approximation) MLP, RBF Net, Bayesian Net, SVR, +SOM/LVQ

Financial TS (FA or Time Series Prediction) RNN, SVR, MLP, RBF Net, + SOM/LVQ

Page 26: Computational Intelligence

John Sum Computational Intelligence 26

FUZZY MODEL STRUCTURE

Page 27: Computational Intelligence

John Sum Computational Intelligence 27

FUZZY MODEL STRUCTURE

Page 28: Computational Intelligence

John Sum Computational Intelligence 28

NN MODEL PARAMETERS MLP

Input Weights Output Weights Neuron model

RNN Input Weights Output Weights Recurrent Weights Neuron model

Page 29: Computational Intelligence

John Sum Computational Intelligence 29

NN MODEL PARAMETERS

Page 30: Computational Intelligence

John Sum Computational Intelligence 30

NN MODEL PARAMETERS

Page 31: Computational Intelligence

John Sum Computational Intelligence 31

NN MODEL PARAMETERS

Page 32: Computational Intelligence

John Sum Computational Intelligence 32

FUZZY MODEL PARAMETERS

Page 33: Computational Intelligence

John Sum Computational Intelligence 33

PARAMETRIC ESTIMATION

Page 34: Computational Intelligence

John Sum Computational Intelligence 34

PARAMETRIC ESTIMATION Gradient Descent

Page 35: Computational Intelligence

John Sum Computational Intelligence 35

PARAMERTIC ESTIMATIONGenetic Algorithm

Page 36: Computational Intelligence

John Sum Computational Intelligence 36

PARAMERTIC ESTIMATIONGenetic Algorithm

Page 37: Computational Intelligence

John Sum Computational Intelligence 37

PARAMERTIC ESTIMATIONGenetic Algorithm

Page 38: Computational Intelligence

John Sum Computational Intelligence 38

DISCUSSIONS

CI is not the only method (or structure) to solve a problem.

Even it can solve, its performance might not be better than other methods.

Should compare with other well-known or existing methods

Page 39: Computational Intelligence

John Sum Computational Intelligence 39

DISCUSSIONS

SCM Problem LP, LIP, NLP Lagrangian Relaxation Cutting Plane CPLEX

Function Approximation Polynomial Series Trigonometric Series B-Spline

Page 40: Computational Intelligence

John Sum Computational Intelligence 40

CONCLUSIONS

IF The problem to be solved has been well

formulated The structure has been selected The objective function to evaluation the goodness

of a parametric vector has been defined

THEN Every problem is just an optimization problem

Page 41: Computational Intelligence

John Sum Computational Intelligence 41

JOHN SUM ([email protected]) Taiwan HK-Chinese, PhD (98) and MPhil (95) from CUHK, BEng (92)

from PolyU HK. Taught in HK Baptist University (98-00), OUHK (00) and PolyU HK (0

0-04), Chung Shan Medical University (05-07) Adj. Associate Prof., Institute of Software, CAS Beijing (99-02) Short visit: CityU HK, Griffith University in Australia, FAU, Boca Rato

n FL US, CAS in Beijing, Ching Mai University in Thailand. Assist. Prof., IEC (07-09), Asso. Prof., ITM (09-) NCHU Taiwan 2000 Marquis Who's Who in the World. Senior Member of IEEE, CI Society, SMC Society (05-) GB Member, Asia Pacific Neural Network Assembly (09-) Associate Editor of the IJCA (05-09) Research Interests include NN, FS, SEM, EC, TM