Neural NetworkNeural NetworkMing-Feng Yeh (Ming-Feng Yeh ( 葉明豐葉明豐 ))Department of Electrical Engineering

Lunghwa University of Science and TechnologyE-mail: mfyeh@mail.lhu.edu.tw

Office: F412-III Tel: #5518

Ming-Feng Yeh 2

COURSE OBJECTIVECOURSE OBJECTIVE

This course gives an introduction to basic neural network architectures and learning rules.

Emphasis is placed on the mathematical analysis of these networks, on methods of training them and on their application to practical engineering problems in such areas as pattern recognition, signal processing and control systems.

Ming-Feng Yeh 3

SYLLABUSSYLLABUS

TextbookTextbook: Hagan, Demuth, Beale,

Neural Network DesignNeural Network Design,

PWS Publishing Company

Midterm ExamMidterm Exam: 30%

Final ExamFinal Exam: 30%

Projects:Projects: 40%

Ming-Feng Yeh 4

CONTENTSCONTENTSCh 1. IntroductionCh 2. Neuron Model & Neural ArchitectureCh 3&4. Perceptron (感知機 ) Learning RuleCh 7. Supervised (監督式 ) Hebbian LearningCh 10. Widrow-Hoff LearningCh 11&12. Back-propagation (倒傳遞 ) Ch 13. Associative (關聯 ) LearningCh 14. Competitive (競爭 ) NetworksCh 15. Grossberg NetworksCh 16. Adaptive Resonance (自適應 ) TheoryCh 18. Hopfield Network

Ming-Feng Yeh 5

InformationInformation

Review

Ch 5 – Signal and Weight Vector Spaces

Ch 6 – Linear Transformations for Neural Networks

Ming-Feng Yeh 6

CHAPTER 1CHAPTER 1

IntroductionIntroduction

Ming-Feng Yeh 7

ObjectivesObjectives

As you read these words you are using a complex biological neural network. You have a highly interconnected set of 1011 neurons to facilitate your reading, breathing, motion and thinking.In the artificial neural network, the neurons are not biological. They are extremely simple abstractions of biological neurons, realized as elements in a program or perhaps as circuits made of silicon.

Ming-Feng Yeh 8

History -1History -1

Pre-1940: von Hemholtz, Mach & Pavlov General theories of learning, vision, conditioning No specific mathematical models of neuron operation

1940s: Hebb, McCulloch & Pitts Mechanism for learning in biological neurons (Hebb) Neural-like networks can compute any arithmetic or logical

function (McCulloch & Pitts)

1950s: Rosenblatt, Widrow & Hoff First practical networks and learning rules: the perception n

etwork and associated learning rule (Rosenblatt) & Widrow-Hoff learning rule

Can not successfully modify their learning rules to train the more complex networks.

Ming-Feng Yeh 9

History -2History -2

1960s: Minsky & Papert Demonstrated limitations of existing neural networks Neural network research was largely suspended

1970s: Kohonen, Anderson & Grossberg Kohonen and Anderson independently and separately devel

oped neural networks that could as memories Self-organizing networks (Grossberg)

1980s: Hopfield, Rumelhart & McClelland The use of statistical mechanics to explain the operation of r

ecurrent network: an associative memory (Hopfield) Backpropagation algorithm (Rumelhart & McClelland)

Ming-Feng Yeh 10

ApplicationsApplications

The applications are expanding because neural networks are good at solving problems, not just in engineering, science and mathematics, but in medicine, business, finance and literature as well.

Ming-Feng Yeh 11

Biological InspirationBiological Inspiration

Human brain consists of a large number (about 1011) of highly interconnected elements (about 104 connections per element) called neurons (神經元 ).

Three principle components are the dendrites, the cell body and the axon.

The point of contact is called a synapse.

Ming-Feng Yeh 12

Biological NeuronsBiological NeuronsDendrites(樹突 ): carry electrical into the cell bodyDendrit

es

Cell Body

Cell Body(細胞體 ): sums and thresholds these incoming signalsAxon(軸突 ): carry the signal from the cell body out to other neurons

Axon

Synapse(突觸 ): contact between an axon of one cell and a dendrites of another cell

Synapse

Soma

Ming-Feng Yeh 13

Neural NetworksNeural Networks

Neural Networks: a promising new generation of information processing systems, usually operate in parallel, that demonstrate the ability to learn, recall, and generalize from training patterns or data.

Basic models, learning rules, and distributed representations of neural networks will be discussed.

Ming-Feng Yeh 14

補充資料補充資料Artificial neural network可譯為類神經網路或人工神經網路，是指模仿生物神經網路的一種資訊處理系統。類神經網路是一種計算系統，包括軟體與硬體，它使用大量簡單的相連人工神經元來模仿生物神經網路的能力。人工神經元是生物神經元的簡單模擬，它從外界環境或其它人工神經元取得資訊，並加以簡單的運算，並輸出其結果到外界環境或其它人工神經元。

Ming-Feng Yeh 15

Fuzzy LogicFuzzy Logic

Fuzzy set theory was first proposed by Lotfi Zadeh in 1965.

A mathematical way to represent vagueness in linguistics

A generalization of classical set theory

Ming-Feng Yeh 16

Fuzzy Systems v.s. Fuzzy Systems v.s. Neural Networks Neural Networks

Fuzzy logic is based on the way the brain deals with inexact information.Neural networks are modeled after the physical architecture of the brain.Fuzzy systems and neural networks are both numerical model-free estimator and dynamical systems.They share the common ability to improve the intelligence of systems working in an uncertain, imprecise and noisy environment.

Ming-Feng Yeh 17

Machine IntelligenceMachine Intelligence

Neural networks provide fuzzy systems with learning ability.

Fuzzy systems provide neural networks with a structure framework with high-level fuzzy IF-THEN rule thinking and reasoning.

Ming-Feng Yeh 18

Fuzzy Neural Fuzzy Neural Integrated SystemIntegrated System

Neural fuzzy systems: use of neural networks as tools in fuzzy models.Fuzzy neural networks: fuzzification of conventional neural network models.Fuzzy-neural hybrid systems: incorporation of fuzzy logic technology and neural networks into hybrid systems.

Ming-Feng Yeh 19

Soft Soft / / Hard ComputingHard Computing

Hard computing whose prime desiderata are precision, certainty, and rigor.Soft computing is tolerant of imprecision, uncertainty, and partial truth. (Lotfi Zadeh)

The primary aim of soft computing is to exploit such tolerance to achieve tractability, robustness, a high level of machine intelligence, and a low cost in practical applications.Fuzzy logic, neural networks (including CMAC), probabilistic reasoning (genetic algorithm, evolutionary programming, and chaotic systems)

Ming-Feng Yeh 20

Soft ComputingSoft Computing

Methodology Strength

Neural network Learning and adaptation

Fuzzy set theory Knowledge representation via fuzzy if-then rule

Genetic algorithm and simulated annealing

Systematic random search

Conventional AI Symbolic manipulation

Ming-Feng Yeh 21

Computational IntelligenceComputational Intelligence

Fuzzy logic, neural network, genetic algorithm, and evolutionary programming are also considered the building blocks of computational intelligence. (James Bezdek)

Computational intelligence is low-level cognition in the style of human brain and is contrast to conventional (symbolic) artificial intelligence (AI).