Upload
vutruc
View
233
Download
5
Embed Size (px)
Citation preview
1 © 2011 The MathWorks, Inc.
Global Optimization with MATLAB®
Products
Account Manager
이 장 원 차장
Application Engineer
엄 준 상
2
Agenda
Introduction to Global Optimization
Survey of Solvers with Examples
– MultiStart
– Global Search
– Pattern Search
– Simulated Annealing
– Genetic Algorithm / Multiobjective
Genetic Algorithm
Additional Resources
Question & Answer
-2
0
2
-3-2
-10
12
3
-6
-4
-2
0
2
4
6
8
x
Peaks
y
Local minima
Global minima
3
Optimization – Finding answers to problems automatically
Objectives
Achieved?
NO
Optimal
Design
YES Model or
Prototype
Modify Design
Parameters
Initial
Design
Parameters
OPTIMIZATION PROCESS
Finding better (optimal) designs
Faster design evaluations
Useful for trade-off analysis (N dimensions)
Non-intuitive designs may be found
Optimization benefits include: Design process can be performed:
Antenna Design Using Genetic Algorithm http://ic.arc.nasa.gov/projects/esg/research/antenna.htm
Manually
(trial-and-error or iteratively)
Automatically
(using optimization techniques)
4 0 20 40 60 80 100 1200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
t
c
c=b1e-b
4t+b
2e-b
5t+b
3e-b
6t
Example Global Optimization Problems
Why does fmincon have a hard time finding the
function minimum?
Why didn’t nonlinear regression find a good fit?
Why didn’t fminunc find the maximum volumetric
efficiency?
0 5 10
-10
-5
0
5
10
x
Starting at 10
0 5 10
-10
-5
0
5
10
x
Starting at 8
0 5 10
-10
-5
0
5
10
x
Starting at 6
x s
in(x
) +
x c
os(2
x)
0 5 10
-10
-5
0
5
10
x
Starting at 3
0 5 10
-10
-5
0
5
10
x
Starting at 1
0 5 10
-10
-5
0
5
10
x
Starting at 0
x s
in(x
) +
x c
os(2
x)
Revolutions Per Minute, RPM
Manifold
Pre
ssure
Ratio
Peak VE Value = 0.96144
Start
End
1000 2000 3000 4000 5000 60000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
5
Global Optimization
Goal:
Want to find the lowest/largest value of
the nonlinear function that has many local
minima/maxima
Problem:
Traditional solvers often return one of the
local minima (not the global)
Solution:
A solver that locates globally optimal
solutions Rastrigin’s Function
6
Global Optimization Toolbox – For problems that
contain multiple maxima/minima or are non-smooth
Optimization T
oolbox
Global Optimization
Toolbox
Faster/fewer function eval
uations Larger problems (higher d
imensions) Finds local minima/maxim
a
Finds global minima/maxi
ma (most of the time) Better on
non-smooth
stochastic
discontinuous
undefined gradients
Custom data types
(in GA and SA solvers)
7
MULTISTART
8
What is MultiStart?
Run a local solver from
each set of start points
Option to filter starting
points based feasibility
Supports parallel
computing
11
GLOBAL SEARCH
12
What is GlobalSearch?
Multistart heuristic algorithm
Calls fmincon from multiple
start points to try and find a
global minimum
Filters/removes non-promising
start points
13
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview Schematic Problem
Peaks function
Three minima
Green, z = -0.065
Red, z= -3.05
Blue, z = -6.55
x
y
14
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 0 Run from specified x0
x
y
15
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1 Generate stage 1 start points via Scatter Search
3
6
0
0 0
4
0
-2
x
y
16
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1 Find stage 1 start point with lowest penalty value
3
6
0
0 0
4
0
-2
x
y
17
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1 Run from best stage 1 point
x
y
18
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Generate stage 2 start points using Scatter Search
x
y
19
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn.
x
y
20
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Don’t run points that are in basins of existing
minimum
x
y
21
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn.
x
y
22
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Don’t run points whose penalty value exceeds
threshold
6
Current penalty
threshold value : 4
x
y
23
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn
x
y
24
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Run start point if it satisfies distance & merit criteria
Current penalty
threshold value : 4
-3
x
y
25
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2 Expand basin of attraction if minimum already found
Current penalty threshold value : 2
-0.1
x
y Basins can overlap
28
SIMULATED ANNEALING
29
What is Simulated Annealing?
A probabilistic metaheuristic approach based upon the
physical process of annealing in metallurgy.
Controlled cooling of a metal allows atoms to realign from a
random higher energy state to an ordered crystalline
(globally) lower energy state
By analogy, simulated annealing replaces a current solution
by randomly choosing a nearby solution
A “nearby” solution is determined by the solution
temperature
30
Simulated Annealing Overview – Iteration 1 Run from specified x0
x
y
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
0.9
31
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and
current temperature
3
x
y 0.9
Possible New Points:
Standard Normal N(0,1) * Temperature
Temperature = 1
32
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1 If lower, accept the point, if higher, accept based upon acceptance
probability
3
x
y 0.9
Temperature = 1
11.01
1/)9.03(
Taccepte
P
33
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and
current temperature
3
x
y 0.9
Temperature = 1
0.3
34
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and
current temperature, accept new point if lower value
3
x
y 0.9
Temperature = 1
0.3
35
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 2 Lower temperature according to temperature schedule
3
x
y 0.9
Temperature = 1
0.3
36
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 2 Lower temperature according to temperature schedule and generate new
point
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
37
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N-1 After several iterations, the search radius becomes small and we narrow
in on a local solution
3
x
y 0.9
Temperature = 0.1
0.3
-1.2
-3
38
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reanneling)
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
39
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reannealing)
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
27.01
1/))3(2(
Taccepte
P
40
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reannealing)
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
41
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N+1 Reduce temperature and continue…
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
42
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N+1 Reduce temperature and continue…
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
43
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration … Reduce temperature and continue…
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
-6.5
47
PATTERN SEARCH
(DIRECT SEARCH)
48
What is a Pattern Search?
An approach that uses a pattern of search directions around
the existing points, the mesh
Polls the mesh for a better solution and moves to that point
Expands/contracts the mesh around the current point when a
solution is not found
Does not rely on gradient information
49
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 1 Run from specified x0
x
y
3
50
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 1 Apply pattern vector, poll new points for improvement
x
y
3
Mesh size = 1
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
0_*_ xvectorpatternsizemeshPnew
0]0,1[*1 x1.6
0.4
4.6
2.8
First poll successful
Complete Poll (not default)
51
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 2 Increase mesh size and repeat…
x
y
3
Mesh size = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
Complete Poll
52
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 3 Mesh expansion: increase mesh size and repeat…
x
y
3
Mesh size = 4
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
53
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 4 Refine mesh: decrease mesh size and repeat…
x
y
3
Mesh size = 4*0.5 = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
54
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration N Continue expansion/contraction until convergence…
x
y
3 1.6
0.4
4.6
2.8
-4
0.3 -2.8
-6.5
57
GENETIC ALGORITHM
58
What is a Genetic Algorithm?
Genetic Algorithms use concepts from evolutionary biology to
find exact or approximate solutions to optimization problems
Start with an initial generation of candidate solutions that are
tested against the objective function
Subsequent generations evolve from the 1st
through selection, crossover and mutation
The individual that best minimizes the given
objective is returned as the ideal solution
59
How Evolution Works – Binary Case
Selection
– Retain the best performing bit strings from one generation to the
next. Favor these for reproduction
– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]
Crossover
– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]
– child = [ 1 0 0 0 0 1 1 0 1 0 ]
Mutation
– parent = [ 1 0 1 0 0 1 1 0 0 0 ]
– child = [ 0 1 0 1 0 1 0 0 0 1 ]
60
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 1 Evaluate initial population
x
y
61
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 1 Select a few good solutions for reproduction
x
y
62
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 2 Generate new population and evaluate
x
y
63
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 2 Select a few good solutions for reproduction
x
y
64
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 3 Generate new population and evaluate
x
y
65
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 3 Select a few good solutions for reproduction
x
y
66
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration N Continue process until stopping criteria are met
x
y
Solution found
69
Comparison of Solver (Default) Performance
x
y
rf([x,y])
-6 -4 -2 0 2 4 6-6
-4
-2
0
2
4
6
FuncValue
Min
StartPt
x
y
rf([x,y])
-60 -40 -20 0 20 40 60-60
-40
-20
0
20
40
60
FuncValue
Min
StartPt
x
y
rf([x,y])
-40 -20 0 20 40 60
-40
-20
0
20
40
60FuncValue
Min
StartPt
70
71
Additional Resources
Upcoming Webinars
Speeding Up Optimization with Parallel Computing
(August 2010)
On-demand Webinars
Genetic Algorithm in Financial Applications
Tips & Tricks: Getting Started with Optimization
Introduction to Optimization
72
Contact Information
North America
– Phone: 508-647-7000
– E-mail: [email protected]
Outside North America
– Contact your local MathWorks office or reseller:
www.mathworks.com/contact
73
Questions?
74
MATLAB Provides the Foundation for
Optimization
The leading environment for technical computing
– Customizable
– Numeric computation
– Data analysis and visualization
– The de facto industry-standard, high-level programming language for algorithm development
– Toolboxes for statistics, optimization, symbolic math, signal and image processing, and other areas
– Foundation of the MathWorks product family
75
Optimization Toolbox Solve standard and large-scale optimization problems
Graphical user interface and command line functions for: Linear and nonlinear programming
Quadratic programming
Nonlinear least squares and nonlinear equations
Multi-objective optimization
Binary integer programming
Additional Capabilities:
Parallel computing support in selected solvers
Customizable algorithm options
Choose between standard and large-scale algorithms
Output diagnostics
76
Global Optimization Toolbox Solve multiple maxima, multiple minima, and nonsmooth optimization problems
Graphical user interface and command line functions for: Global Search solver
Multistart solver
Genetic algorithm solver
– Single objective
– Multiobjective with Pareto front
Direct search solver
Simulated annealing solver
Useful for problems not easily addressed with Optimization Toolbox: Discontinuous
Highly nonlinear
Stochastic
Discrete or custom data types
Undefined derivatives
Multiple maxima/minima
77
Anatomy of an Optimization Problem
General Form Accepted by MATLAB Solvers
Objective Function
Subject to Constraints (i.e. such that)
uxl
xcbxA
xcbAx
eqeqeq
0)(,
0)(,
)(min xfx
Typically a linear or nonlinear function
Linear constraints
• inequalities
• equalities
• bounds
Nonlinear constraints
• inequalities
• equalities
Decision variables
78
MATLAB Optimization Products and Example
Applications
MATLAB Statistics Toolbox
Curve Fitting Toolbox
Optimization Toolbox Genetic Algorithm and
Direct Search Toolbox
Solving Equations
f(x) = 0
Root finding
Systems of equations
Real roots finding (1D):
Linear Systems:
F(x) = 0
Ax-b = 0 (i.e. Ax=b)
>> x = A \ b
Real roots finding (N-D):
Nonlinear Systems
F(X) = 0
Noisy, discontinuous root
finding (N-D)
Noisy, discontinuous systems:
F(X) = 0
Curve/Modeling
Fitting
curve fitting
parameter estimation
(model fitting)
Basic (linear) curve fitting Advanced (nonlinear) curve
fitting
Model Fitting (least squares)
Constrained curve fitting
Parameter Estimation
Noisy, Discontinuous
parameter estimation
Trade-Off Studies
Maximization
Minimization
Goal seeking
Multiobjective
Unconstrained nonlinear
minimization
Constrained nonlinear
minimization
Noisy, Discontinuous, ill-
defined mimization
-6 -4 -2 0 2 4 6-100
-50
0
50
100
x
x3-2 x-5
0 0.5 1 1.5 2 2.50.5
0.6
0.7
0.8
0.9