NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS 欢迎大家提出意见建议！ 2003.10.15

NEURAL NETWORK THEORY

NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS

欢迎大家提出意见建议！ 2003.10.15

2


NEURONS AS FUNCTIONS

Neurons behave as functions.Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

3

NEURONS AS FUNCTIONSThe transduction description: a sigmoidal or S-shaped curve the logistic signal function:

cxexS

1

1)(

)()(' 001 cScSdxdSS

The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.


4

NEURONS AS FUNCTIONSS(x)

x

0-∞ - + +∞ Fig.1 s(x) is a bounded monotone-nondecreasing function of x

If c→+∞ ， we get threshold signal function (dash line),Which is piecewise differentiable


5

SIGNAL MONOTONICITY

In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value.The staircase signal function is a piecewise-differentiableMonotone-nondecreasing signal function.


6

SIGNAL MONOTONICITYAn important exception: bell-shaped signal function or Gaussian signal functions

02

cexS cx)(

xScxeS cx ',2'2

The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.


7

SIGNAL MONOTONICITYGeneralized Gaussian signal function define potential or radial basis function :

])(2

1exp[)( 22

n

j

ijj

ii xxS

n

n Rxxx ),,( 1

),,( in

ii 1

input activation vector:

variance:

mean vector:


2i

)(xSi

we shall consider only scalar-input signal functions:

)( ii xS

8

SIGNAL MONOTONICITYneurons are nonlinear but not too much so ---- a property as semilinearityLinear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robustNonlinear signal functions - increases a network’s computational richness - increases a network’s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability


9

SIGNAL MONOTONICITYSignal and activation velocities the signal velocity: =dS/dt

Signal velocities depend explicitly on action velocities

S

xSdtdx

dxdSS '


10

BIOLOGICAL ACTIVATIONS AND SIGNALS

Fig.2 Neuron anatomy神经元 (Neuron) 是由细胞核 (cell nucleus) ，细胞体 (soma) ，轴突 (axon) ，树突 (dendrites) 和突触 (synapse) 所构成的


11

X= （ x1 ， x2 ，…， xn ）W= （ w1 ， w2 ，…， wn ）net=∑xiwinet=XW

x2 w2 ∑ f o=f （ net）

xn wn

…

net=XW

x1 w1

BIOLOGICAL ACTIVATIONS AND SIGNALS


12

BIOLOGICAL ACTIVATIONS AND SIGNALSCompetitive Neuronal Signal

11

21

1

cxcx exS

exS )()(

logical signal function ( Binary Bipolar )

The neuron “wins” at time t if , “loses” ifand otherwise possesses a fuzzy win-loss status between 0 an 1.

a. Binary signal functions : [0,1]b. Bipolar signal functions : [-1,1] McCulloch—Pitts (M—P) neurons


1))(( txS 1))(( txS

13

NEURON FIELDSNeurons within a field are topologically ordered, often by proximity. zeroth-order topology : lack of topological structure Denotation: , , neural system samples the function m times to generate the associated pairs , ... ,The overall neural network behaves as an adaptive filter and sample data changed network parameters.

XF YF ZF },{ YX FF },,{ ZYX FFF

),( mm yx),( 11 yx


pn RRf :YX FF

f

14

NEURONAL DYNAMICAL SYSTEMSDescription:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentialsActivation differential equations: niFFgx YXii ,,),,( 21

piFFhy YXii ,,),,( 21

in vector notation:niFF YXi ,,),,( 21gx

piFF YXi ,,),,( 21hy

(1)(2)

(3)(4)


15

NEURONAL DYNAMICAL SYSTEMSNeuronal State spaces

nn Rtxtxt ))(,),(()( 1X

pp Rtytyt ))(,),(()( 1Y

So the state space of the entire neuronal dynamical system is: pn RR

Augmentation:

ZYX FFF ]|[pn

z RF


},{ YX FF

16

NEURONAL DYNAMICAL SYSTEMSSignal state spaces as hyper-cubesThe signal state of field at time t:

XF

)))((,)),((())(( txStxStXS nXn

X 11

The signal state space: an n-dimensional hypercubeThe unit hypercube : or ,The relationship between hyper-cubes and the fuzzy set : , subsets of correspond to the vertices of


nI n]1,0[

pnyx IIFF , pn

yx IFF | py

nx IFIF ,

nxxxX ,,, 21 n2 X n2nI

17

NEURONAL DYNAMICAL SYSTEMSNeuronal activations as short-term memory

Short-term memory(STM) : activationLong-term memory(LTM) : synapse


18

1 、 Liner Function S(x) = cx + k , c>0

SIGNAL FUNCTION (ACTIVATION FUNCTION)

x

S

o

k


19

2. Ramp Function r if x≥θS(x)= cx if |x|<θ

-r if x≤-θr>0, r is a constant.


r

-r

θ -θ x

S


20

SIGNAL FUNCTION (ACTIVATION FUNCTION) 3 、 threshold linear signal function: a special Ramp Function

Another form:elsecxifcxif

cxxS 0

101

)(

)),max(,min()( cxxS 01

0cS '


21

SIGNAL FUNCTION (ACTIVATION FUNCTION) 4 、 logistic signal function:

xcxc

xc

cx

ee

ee

xS22

2

11)(

Where c>0.01 )(' ScSS

So the logistic signal function is monotone increasing.


22

SIGNAL FUNCTION (ACTIVATION FUNCTION) 5 、 threshold signal function:

Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

TxifTxifTxif

xSxSk

k

k

kk

1

1

1

1

0)(

1)(


23

SIGNAL FUNCTION (ACTIVATION FUNCTION) 6 、 hyperbolic-tangent signal function:

Another form:

)tanh()( cxxS

01 2 )(' ScS

cxcx

cxcx

eeeecx

)tanh(


24

SIGNAL FUNCTION (ACTIVATION FUNCTION) 7 、 threshold exponential signal function:

When ,

),min()( cxexS 1

1cxe

0 cxceS '

02 cxecS ''

0 cxnn ecS )(


25

SIGNAL FUNCTION (ACTIVATION FUNCTION) 8 、 exponential-distribution signal function:

When ,

),max()( cxexS 10

0x

0 cxceS '

0'' 2 cxecS


26

SIGNAL FUNCTION (ACTIVATION FUNCTION) 9 、 the family of ratio-polynomial signal function: An example

For ,

),max()( n

n

xcxxS

0

1n

02

1

)(' n

n

xccnxS


27



28



29

PULSE-CODED SIGNAL FUNCTIONDefinition:

(5)

(6)

t ts

ii dsesxtS )()(

t ts

jj dsesytS )()(

tatpulsenoiftatoccurspulseaif

txi

01

)(

where

(7)


30

PULSE-CODED SIGNAL FUNCTIONPulse-coded signals take values in the unit interval [0,1].Proof: when

0)(txi0

t ts

ii dsesxtS )()(

1)( txi

1

ts

s

tst tsi eedsetS lim)(

when


31

PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals

The first-order linear inhomogenous differential equation: (8)

(9)

)()( tqxtpx The solution to this differential equation:

t tst dsesqextx0

0 )()()(

A simple form for the signal velocity:)()()( tStxtS iii

)()()( tStytS jjj

(10)(11)


t ts

ii dsesxtS )()( (5)

32

PULSE-CODED SIGNAL FUNCTION

The central result of pulse-coded signal functions:The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency. ------------- the velocity-difference property of pulse-coded signal functions


)()()( tStxtS iii (10)

Documents

NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS 欢迎大家提出意见建议！ 2003.10.15