Upload
anonymous-jhqkrr8ia1
View
217
Download
0
Embed Size (px)
Citation preview
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 1/46
OMICS Group
Contact us at: [email protected]
OMICS Group International through its Open Access Initiative iscommitted to make genuine and reliable contributions to thescientifc communit. OMICS Group hosts over !"" leading#edge peer revie$ed Open Access %ournals and organi&es over'"" International Con(erences annuall all over the $orld.OMICS )ublishing Group *ournals have over ' million readers
and the (ame and success o( the same can be attributed to thestrong editorial board $hich contains over '"""" eminentpersonalities that ensure a rapid+ ,ualit and ,uick revie$process. OMICS Group signed an agreement $ith more than-""" International Societies to make healthcare in(ormation
Open Access.
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 2/46
OMICS Group $elcomes submissions that areoriginal and technicall so as to serve both
the developing $orld and developedcountries in the best possible $a.
OMICS %ournals are poised in ecellence bpublishing high ,ualit research. OMICSGroup (ollo$s an /ditorial Manager0 Sstempeer revie$ process and boasts o( a strong
and active editorial board./ditors and revie$ers are eperts in their
feld and provide anonmous+ unbiased anddetailed revie$s o( all submissions.
1he *ournal gives the options o( multiplelanguage translations (or all the articles andall archived articles are available in 21M3+
4M3+ )56 and audio (ormats. Also+ all thepublished articles are archived in repositoriesand indeing services like 5OA%+ CAS+ Google
Scholar+ Scientifc Commons+ IndeCopernicus+ /7SCO+ 2I8A9I and GA3/.
6or more details please visit our $ebsite:
http:omicsonline.orgSubmitmanuscript.php
OMICS %ournals are $elcoming Submissions
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 3/46
8eural 8et$orks
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 4/46
Mario Pavone
)ro(esssor;niversit o( CataniaItal
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 5/46
9ecentl publishedarticles
• Clonal Selection - An Immunological Algorithm for Global Optimization over
Continuous Spaces• Swarm Intelligence Heuristics for Graph Coloring Problem
•O-B-CO!" Optimal Bs for CO!oring Graphs
• scaping !ocal Optima via Parallelization an# $igrationProtein $ultiple
Se%uence Alignment b& H&bri# Bio-Inspire# Algorithms
• ffective Calibration of Artificial Gene 'egulator& (etwor)s• !arge scale agent-base# mo#eling of the humoral an# cellular immune
response
• A $emetic Immunological Algorithm for 'esource Allocation Problem*
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 6/46
7iological inspirations
• Some numbers< – 1he human brain contains about -" billion
nerve cells =neurons> – /ach neuron is connected to the others
through -"""" snapses
• )roperties o( the brain – It can learn+ reorgani&e itsel( (rom eperience
– It adapts to the environment – It is robust and (ault tolerant
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 7/46
7iological neuron
• A neuron has – A branching input =dendrites> – A branching output =the aon>
• 1he in(ormation circulates (rom the dendrites tothe aon via the cell bod
• Aon connects to dendrites via snapses – Snapses var in strength – Snapses ma be ecitator or inhibitor
axon
cell body
synapse
nucleus
dendrites
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 8/46
?hat is an artifcial neuron
• 5efnition : 8on linear+ parameteri&ed(unction $ith restricted output range
- '
$"
+= ∑
−
=
+
+
,
n
i
ii xww f y
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 9/46
Activation (unctions
0 2 4 6 8 10 12 14 16 18 200
2
4
6
8
10
12
14
16
18
20
-10 -8 -6 -4 -2 0 2 4 6 8 10
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-10 -8 -6 -4 -2 0 2 4 6 8 10-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
3inear
3ogistic
2perbolic tangent
x y =
-e.p/+
+
x y
−+=
-e.p/-e.p/
-e.p/-e.p/
x x
x x y
−+−−
=
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 10/46
8eural 8et$orks
• A mathematical model to solve engineeringproblems
– Group o( highl connected neurons to reali&ecompositions o( non linear (unctions
• 1asks – Classifcation
– 5iscrimination
– /stimation
• tpes o( net$orks
– 6eed (or$ard 8eural 8et$orks – 9ecurrent 8eural 8et$orks
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 11/46
6eed 6or$ard 8eural8et$orks
• 1he in(ormation ispropagated (rom theinputs to the outputs
• Computations o( 8o non linear (unctions(rom n input variablesb compositions o( 8c algebraic (unctions
• 1ime has no role =8Occle bet$een outputsand inputs>
- n<..
-st hiddenlaer
nd hiddenlaer
Outputlaer
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 12/46
9ecurrent 8eural 8et$orks
• Can have arbitrartopologies
• Can model sstems $ithinternal states =dnamic
ones>• 5elas are associated to a
specifc $eight
• 1raining is more diBcult
• )er(ormance ma beproblematic – Stable Outputs ma be
more diBcult to evaluate
– ;nepected behavior=oscillation+ chaos+ <>
-
-
"-"
-"
""
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 13/46
3earning
• 1he procedure that consists in estimating the parameters o(neurons so that the $hole net$ork can per(orm a specifctask
• tpes o( learning
– 1he supervised learning – 1he unsupervised learning
• 1he 3earning process =supervised> – )resent the net$ork a number o( inputs and their
corresponding outputs
– See ho$ closel the actual outputs match the desired ones – Modi( the parameters to better approimate the desired
outputs
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 14/46
Supervised learning
• 1he desired response o( the neuralnet$ork in (unction o( particularinputs is $ell kno$n.
• A )ro(essorD ma provide eamplesand teach the neural net$ork ho$ to(ulfll a certain task
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 15/46
;nsupervised learning
• Idea : group tpical input data in (unctiono( resemblance criteria un#kno$n a priori
• 5ata clustering
• 8o need o( a pro(essor – 1he net$ork fnds itsel( the correlations
bet$een the data
– /amples o( such net$orks :• Eohonen (eature maps
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 16/46
)roperties o( 8eural8et$orks
• Supervised net$orks are universal approimators=8on recurrent net$orks>
• 1heorem : An limited (unction can beapproimated b a neural net$ork $ith a fnite
number o( hidden neurons to an arbitrarprecision
• 1pe o( Approimators – 3inear approimators : (or a given precision+ the number
o( parameters gro$s eponentiall $ith the number o(
variables =polnomials> – 8on#linear approimators =88>+ the number o(
parameters gro$s linearl $ith the number o( variables
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 17/46
Other properties
• Adaptivit – Adapt $eights to environment and retrained
easil
• Generali&ation abilit – Ma provide against lack o( data
• 6ault tolerance – Grace(ul degradation o( per(ormances i(
damaged F 1he in(ormation is distributed$ithin the entire net.
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 18/46
• In practice+ it is rare to approimate akno$n (unction b a uni(orm (unction
• black boD modeling : model o( a process• 1he output variable depends on the input
variable $ith kF- to 8
• Goal : /press this dependenc b a(unction+ (or eample a neural net$ork
Static modeling
{ }k pk y x 0
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 19/46
• I( the learning ensemble results (rom measures+the noise intervenes
• 8ot an approimation but a ftting problem
• 9egression (unction• Approimation o( the regression (unction :
/stimate the more probable value o( p (or agiven input
• Cost (unction:
• Goal: Minimi&e the cost (unction b determiningthe right (unction g
[ ]1
+
-0/-/
1
+-/
∑=−=
N
k
k k
p w x g x yw J
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 20/46
/ample
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 21/46
Classifcation=5iscrimination>
• Class ob*ects in defned categories
• 9ough decision O9
• /stimation o( the probabilit (or acertain ob*ect to belong to a specifcclass
/ample : 5ata mining
• Applications : /conom+ speech andpatterns recognition+ sociolog+ etc.
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 22/46
/ample
/amples o( hand$ritten postal codesdra$n (rom a database available (rom the ;S )ostal service
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 23/46
?hat do $e need to use88
• 5etermination o( pertinent inputs
• Collection o( data (or the learning andtesting phase o( the neural net$ork
• 6inding the optimum number o( hiddennodes
• /stimate the parameters =3earning>
• /valuate the per(ormances o( the net$ork
• I6 per(ormances are not satis(actor thenrevie$ all the precedent points
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 24/46
Classical neuralarchitectures
• )erceptron
• Multi#3aer )erceptron
•
9adial 7asis 6unction =976>• Eohonen 6eatures maps
• Other architectures –
An eample : Shared $eights neuralnet$orks
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 25/46
)erceptron
• 9osenblatt =-H>
• 3inear separation
• Inputs :Jector o( real
values
• Outputs :- or #-
KKKK
KK
KK
KK K K
K
K K
K
KKK
KK
K
KK
KK
KKK
K
KK
K
K
K
K
,11++, =++ xc xcc
++= y
+−= y
,c+c 1c
∑
+ x
1 x+
11++, xc xccv ++=
-/v sign y =
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 26/46
3earning =1he perceptronrule>• Minimi&ation o( the cost (unction :
• %=c> is al$as F " =M is the ensemble o( badclassifed eamples>
• is the target value
• )artial cost – I( is not $ell classifed :
– I( is $ell classifed
• )artial cost gradient
• )erceptron algorithm
k x
∑ ∈ −=
M k
k k
p
v yc J /
k
p
y
k k
p
k k
p
k k
p
x yv y
v y
+=<
=>
+-c/) c/)"classifie#not wellis./ ,if
+-c/) c/)"classifie#wellis/. ,if
)
)
k x
k k
p
k v yc J −=-/
,-/ =c J k
k k
p
k
x y
c
c J −=
∂
∂ -/
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 27/46
• 1he perceptron algorithm convergesi( eamples are linearl separable
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 28/46
Multi#3aer )erceptron
• One or morehidden laers
• Sigmoid activations(unctions-st hidden
laer
nd hiddenlaer
Outputlaer
Input data
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 29/46
3earning•
7ack#propagation algorithm
I( the *th node is an output unit
Credit assignment
( )
-/2-/
-/-3/1
+
-/
,
j j j j
j j
j
j j
j
j j
j
j
j
i j
ji
j
j ji
ji
j j j
n
i
i ji j j
net f ot
ot o
E ot E
net f
o
E
net
o
o
E
ow
net
net
E
w
E w
net f o
owwnet
−=
−−=∂
∂=>−=
′
∂
∂−=
∂
∂
∂
∂−=
=∂
∂
∂
∂−=
∂
∂−=∆
=
+= ∑
δ
δ
αδ α α
j
jnet
E
∂∂
−=δ
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 30/46
Momentum term to smooth 1he $eight changes over tim
-/-+/-/
-+/-/-/-/
-/2
t wt wt w
t wt ot t w
wnet f
wo
net
net
E
o
E
ji ji ji
jii j ji
k kjk j j j
k k kjk
j j
∆+−=
−∆+=∆
=
−=∂
∂∂
∂=∂∂
∑
∑ ∑
γ αδ
δ δ
δ
κ
κ κ κ
κ
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 31/46
Structure Types of
Decision Regions
Exclusive-OR
Prole!
"l#sses wit$
Mes$e% regions
Most &ener#l
Region S$#pes
Single-'#yer
Two-'#yer
T$ree-'#yer
(#lf Pl#ne
)oun%e% )y
(yperpl#ne
"onvex Open
Or
"lose% Regions
Abitrar&
/Comple.it&
!imite# b& (o*
of (o#es
A
AB
B
A
AB
B
A
AB
B
B
A
BA
BA
5iLerent non linearl separableproblems
Neural Networks – An Introduction Dr. Andrew Hunter
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 32/46
9adial 7asis 6unctions=976s>
• 6eatures – One hidden laer – 1he activation o( a hidden unit is determined b the distance
bet$een the input vector and a prototpe vector
9adial units
Outputs
Inputs
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 33/46
•
976 hidden laer units have areceptive feld $hich has a centre
• Generall+ the hidden unit (unctionis Gaussian
• 1he output 3aer is linear• 9eali&ed (unction
( )∑
= −Φ=
*
j j j c x+ x s+
/
( )1
e.p
−−=−Φ
j
j
j
c xc x
σ
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 34/46
3earning
• 1he training is per(ormed b deciding on – 2o$ man hidden nodes there should be
– 1he centers and the sharpness o( the
Gaussians• steps
– In the -st stage+ the input data set is used todetermine the parameters o( the basis
(unctions – In the nd stage+ (unctions are kept fed $hile
the second laer $eights are estimated= Simple 7) algorithm like (or M3)s>
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 35/46
M3)s versus 976s
• Classifcation – M3)s separate classes via
hperplanes
– 976s separate classes viahperspheres
• Learning – M3)s use distributed
learning
– 976s use locali&edlearning
– 976s train (aster
• Structure – M3)s have one or more
hidden laers
– 976s have onl one laer
– 976s re,uire more hiddenneurons F curse o(dimensionalit
41
4+
MLP
41
4+
RBF
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 36/46
Sel( organi&ing maps
• 1he purpose o( SOM is to map a multidimensionalinput space onto a topolog preserving map o(neurons – )reserve a topological so that neighboring neurons
respond to similar Ninput patterns
– 1he topological structure is o(ten a or ' dimensionalspace
• /ach neuron is assigned a $eight vector $ith thesame dimensionalit o( the input space
• Input patterns are compared to each $eightvector and the closest $ins =/uclidean 5istance>
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 37/46
• 1he activation o( theneuron is spread in itsdirect neighborhoodFneighbors becomesensitive to the same
input patterns• 7lock distance• 1he si&e o( the
neighborhood isinitiall large but
reduce over time FSpeciali&ation o( thenet$ork
6irst neighborhood
nd neighborhood
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 38/46
Adaptation
• 5uring training+ the$innerD neuron andits neighborhoodadapts to make their$eight vector moresimilar to the inputpattern that causedthe activation
• 1he neurons are
moved closer to theinput pattern
• 1he magnitude o( theadaptation iscontrolled via a
learning parameter$hich deca s over
Shared $eights neural net$orks:
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 39/46
Shared $eights neural net$orks: 1ime 5ela 8eural 8et$orks
=1588s>• Introduced b ?aibel in -HH
• )roperties – 3ocal+ shi(t invariant (eature etraction
–
8otion o( receptive felds combining localin(ormation into more abstract patterns at ahigher level
– ?eight sharing concept =All neurons in a(eature share the same $eights>
• All neurons detect the same (eature but in diLerentposition
• )rincipal Applications – Speech recognition
–
Image analsis
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 40/46
1588s =contPd>
• Ob*ects recognition inan image
• /ach hidden unitreceive inputs onl(rom a small region o(
the input space :receptive feld
• Shared $eights (or allreceptive felds Ftranslation invariancein the response o( thenet$ork
Inputs
2idden3aer -
2idden3aer
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 41/46
• Advantages – 9educed number o( $eights
• 9e,uire (e$er eamples in the training set
• 6aster learning
– Invariance under time or spacetranslation
–
6aster eecution o( the net =incomparison o( (ull connected M3)>
8 l 8 k
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 42/46
8eural 8et$orks=Applications>
• 6ace recognition
• 1ime series prediction
• )rocess identifcation
• )rocess control
• Optical character recognition
• Adaptative fltering
• /tc<
C l i 8 l
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 43/46
Conclusion on 8eural8et$orks
• 8eural net$orks are utili&ed as statistical tools – Ad*ust non linear (unctions to (ulfll a task – 8eed o( multiple and representative eamples but (e$er than
in other methods• 8eural net$orks enable to model comple static
phenomena =66> as $ell as dnamic ones =988>• 88 are good classifers 7;1
– Good representations o( data have to be (ormulated – 1raining vectors must be statisticall representative o( the
entire input space – ;nsupervised techni,ues can help
• 1he use o( 88 needs a good comprehension o( the problem
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 44/46
International 5ournal of Swarm Intelligence an# volutionar&
Computation
International 5ournal of Swarm
Intelligence an# volutionar&
Computation
i l l f S lli #
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 45/46
A Global Collo%uium on Artificial
Intelligence
International 5ournal of Swarm Intelligence an#
volutionar& Computation
OMICS Group Open Access Membership
7/23/2019 Mario Pavone
http://slidepdf.com/reader/full/mario-pavone 46/46
OMICS Group Open Access Membership
OMICS publishing Group Open Access
Membership enables academic and researchinstitutions+ (unders and corporations toactivel encourage open access in scholarlcommunication and the dissemination o(research published b their authors.6or more details and benefts+ click on the
link belo$:http:omicsonline.orgmembership.php