Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
LHS Coordinate Search Nelder-Mead References
Traditional methods
MTH8418
S. Le Digabel, Polytechnique Montreal
Winter 2020(v3)
MTH8418: Traditional methods 1/55
LHS Coordinate Search Nelder-Mead References
IntroductionI We consider the unconstrained version of the problem with
Ω = Rn:
minx∈Rn
f(x)
I Possible to treat constrained problems with meta-strategiessuch as the extreme barrier or the augmented Lagrangianapproaches (discussed in Lecture #9)
I The “traditional” methods were introduced as heuristics. Forsome of them, convergence was studied only in the 90’. Thisopened the era of modern DFO methods
I This lecture describes three of these methods
MTH8418: Traditional methods 2/55
LHS Coordinate Search Nelder-Mead References
Plan
Latin Hypercube Sampling
The Coordinate Search method
The Nelder-Mead method
References
MTH8418: Traditional methods 3/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
The Coordinate Search method
The Nelder-Mead method
References
MTH8418: Traditional methods 4/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube SamplingI Traditional statistical method from [McKay et al., 1979]
I Often used in the design of computer experiments
I Latin square: With n = 2, a grid containing one sample ineach row and each column
I Latin hypercube: Generalization in dimension n
I Bounds for each variable are necessary
I Stochastic method:I Use a good Random Number Generator (RNG)I Use the process identification number (PID) as the random
seedI Change the seed for different resultsI Remember the seed for reproducibility
MTH8418: Traditional methods 5/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
I Optimization:
1. Construct sampling: Generate p sample points2. Evaluate f at the sample points3. Remember the best solution
I Bad algorithm for optimization
I But good strategy to generate a set of starting solutions forother methods
MTH8418: Traditional methods 6/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
One strategy to generate p points in dimension n:
I The range of each variable is divided into p intervals
I The intersections of these intervals define rectangular regions
I Two steps of random draws:
1. Permutations for each variable, to choose one particular region
2. Random location inside the region
MTH8418: Traditional methods 7/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling: Pseudocode
vector<Point> LHS ( n , p , lb , ub )
vector<Point> pts;
for ( k = 0 ; k < p ; ++k )
Point x(n);
for ( i = 0 ; i < n ; ++i )
x[i] = lb[i] + ( ub[i]-lb[i] ) *
( permut(p) + rand(0.0,1.0) ) / p;
pts.insert(x);
return pts;
MTH8418: Traditional methods 8/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling: Example
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100LHS with p=10
x
y
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100LHS with p=100
x
y
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100LHS with p=1000
x
y
n = 2 and p ∈ 10, 100, 1000
MTH8418: Traditional methods 9/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
The Coordinate Search method
The Nelder-Mead method
References
MTH8418: Traditional methods 10/55
LHS Coordinate Search Nelder-Mead References
The Coordinate Search methodI Coordinate Search (CS), or Compass Search
I The CS is the ancestor of the modern direct search methods:I Hooke & Jeeves method: The original Pattern Search method
(also defined “direct search”) [Hooke and Jeeves, 1961]
I Generalized Pattern Search (GPS) [Torczon, 1997]
I Mesh Adaptive Direct Search(MADS) [Audet and Dennis, Jr., 2006]
I Generating Set Search (GSS) [Kolda et al., 2003]
I [Fermi and Metropolis, 1952] (first known reference)
I Tested on one of the first computers, the MANIAC(Mathematical Analyzer, Numerical Integrator, and Computer or Mathematical Analyzer, Numerator,Integrator, and Computer)
MTH8418: Traditional methods 11/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Algorithm
I Initialization:x0 : starting point in Rn
∆0 > 0 : initial step size
I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:
xk+1 ← t∆k+1 ← ∆k
else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2
MTH8418: Traditional methods 12/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Algorithm
I Initialization:x0 : starting point in Rn
∆0 > 0 : initial step size
I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:
xk+1 ← t∆k+1 ← ∆k
else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2
MTH8418: Traditional methods 12/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Algorithm
I Initialization:x0 : starting point in Rn
∆0 > 0 : initial step size
I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:
xk+1 ← t∆k+1 ← ∆k
else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2
MTH8418: Traditional methods 12/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Details
I Evaluation of x0 is counted
I Extension with bounds: If the algorithm generates a trialpoint outside of the boundary, then do not evaluate andconsider as a failure
I If the algorithm generates a trial point x found in the cache,then do not evaluate and consider as a failure
MTH8418: Traditional methods 13/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
•f=4176
MTH8418: Traditional methods 14/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
•f=4176
MTH8418: Traditional methods 14/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
•f=4176
MTH8418: Traditional methods 14/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
•f=4176
MTH8418: Traditional methods 14/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
•f=4176
MTH8418: Traditional methods 14/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x1=(1,2)>,∆1=1
•f=166
•f=4401
•f=262
•f=81
•f=106
MTH8418: Traditional methods 15/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x1=(1,2)>,∆1=1
•f=166
•f=4401
•f=262
•f=81
•f=106
MTH8418: Traditional methods 15/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x2=(0,2)>,∆2=1
•f=81
•f=166
•f=152
•f=2646
•f=36
MTH8418: Traditional methods 16/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x2=(0,2)>,∆2=1
•f=81
•f=166
•f=152
•f=2646
•f=36
MTH8418: Traditional methods 16/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x3=(0,1)>,∆3=1
•f=36
•f=106
•f=81
•f=2466
•f=17
MTH8418: Traditional methods 17/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x3=(0,1)>,∆3=1
•f=36
•f=106
•f=81
•f=2466
•f=17
MTH8418: Traditional methods 17/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x4=(0,0)>,∆4=1
•f=17
•f=82
•f=36
•f=2402
•f=24
•f=17
x4 is a mesh local optimizer
MTH8418: Traditional methods 18/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x4=(0,0)>,∆4=1
•f=17
•f=82
•f=36
•f=2402
•f=24
•f=17
x4 is a mesh local optimizer
MTH8418: Traditional methods 18/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x4=(0,0)>,∆4=1
•f=17
•f=82
•f=36
•f=2402
•f=24
•f=17
x4 is a mesh local optimizer
MTH8418: Traditional methods 18/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x5=(0,0)>,∆5= 12
After 20 eval.:
f(x) = 17
Can we do better?
•f=17
MTH8418: Traditional methods 19/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Basic algorithm
x5=(0,0)>,∆5= 12
After 20 eval.:
f(x) = 17
Can we do better?
•f=17
MTH8418: Traditional methods 19/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 20/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 20/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 20/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 20/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x1=(1,2)>,∆1=1
•f=166
•f=4401
•f=262
•f=81
MTH8418: Traditional methods 21/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x1=(1,2)>,∆1=1
•f=166
•f=4401
•f=262
•f=81
MTH8418: Traditional methods 21/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=166
•f=152
•f=2646
•f=36
MTH8418: Traditional methods 22/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=166
•f=152
•f=2646
•f=36
MTH8418: Traditional methods 22/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x3=(0,1)>,∆3=1
•f=36
•f=106
•f=81
•f=2466
•f=17
MTH8418: Traditional methods 23/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x3=(0,1)>,∆3=1
•f=36
•f=106
•f=81
•f=2466
•f=17
MTH8418: Traditional methods 23/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x4=(0,0)>,∆4=1
•f=17
•f=82
•f=36
•f=2402
•f=24
MTH8418: Traditional methods 24/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x4=(0,0)>,∆4=1
•f=17
•f=82
•f=36
•f=2402
•f=24
MTH8418: Traditional methods 24/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x5(0,0)>,∆5= 12
•f=17
•f=1.0625
MTH8418: Traditional methods 25/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x5(0,0)>,∆5= 12
•f=17
•f=1.0625
MTH8418: Traditional methods 25/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x6=(0.5,0)>,∆6= 12
After 20 eval.:
f(x) = 1.0625
Can we do better?
•f=1.0625
MTH8418: Traditional methods 26/55
LHS Coordinate Search Nelder-Mead References
Opportunistic Coordinate Search
x6=(0.5,0)>,∆6= 12
After 20 eval.:
f(x) = 1.0625
Can we do better?
•f=1.0625
MTH8418: Traditional methods 26/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 27/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 27/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 27/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 27/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x0=(2,2)>,∆0=1
•f=4401
•f=29286
•f=4772
•f=166
MTH8418: Traditional methods 27/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x1=(1,2)>,∆1=1
•f=166
•f=81
MTH8418: Traditional methods 28/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x1=(1,2)>,∆1=1
•f=166
•f=81
MTH8418: Traditional methods 28/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=2646
•f=166
•f=152
•f=36
MTH8418: Traditional methods 29/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=2646
•f=166
•f=152
•f=36
MTH8418: Traditional methods 29/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=2646
•f=166
•f=152
•f=36
MTH8418: Traditional methods 29/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=2646
•f=166
•f=152
•f=36
MTH8418: Traditional methods 29/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x2=(0,2)>,∆2=1
•f=81
•f=2646
•f=166
•f=152
•f=36
MTH8418: Traditional methods 29/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x3=(0,1)>,∆3=1
•f=36
•f=17
MTH8418: Traditional methods 30/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x3=(0,1)>,∆3=1
•f=36
•f=17
MTH8418: Traditional methods 30/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x4=(0,0)>,∆4=1
•f=17
•f=24
•f=2402
•f=82
•f=36
MTH8418: Traditional methods 31/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x4=(0,0)>,∆4=1
•f=17
•f=24
•f=2402
•f=82
•f=36
MTH8418: Traditional methods 31/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x5=(0,0)>,∆5= 12
•f=17
•f=17.25
•f=411
•f=1.0625
MTH8418: Traditional methods 32/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x5=(0,0)>,∆5= 12
•f=17
•f=17.25
•f=411
•f=1.0625
MTH8418: Traditional methods 32/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x6=(0.5,0)>,∆6= 12
•f=1.0625
•f=82
•f=0.375
MTH8418: Traditional methods 33/55
LHS Coordinate Search Nelder-Mead References
Dynamic Coordinate Search
x6=(0.5,0)>,∆6= 12
•f=1.0625
•f=82
•f=0.375
MTH8418: Traditional methods 33/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: 3 strategies
Basic (complete) Opportunistic Dynamic
x> f(x) x> f(x) x> f(x)
After 20 evaluations of f
(0,0) 17 (.5,0) 1.0625 (.5,-.5) .375
After 50 evaluations of f
(.375, -.375) 1.8E-2 (.375, -.312) 5.7E-3 (.375, -.344) 3.1E-3
MTH8418: Traditional methods 34/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Convergence analysis
I Special case of GPS for which convergence was provedby [Torczon, 1997]
I If the sequence of iterates xk is bounded, thenI lim
k∆k = 0
I There exists x ∈ Rn, the limit of a subsequence of mesh localoptimizers, whose size is infinitely small
I If f is continuously differentiable at x, then ∇f(x) = 0
MTH8418: Traditional methods 35/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Failure
f(x) = ‖x‖∞ from x0 = (1, 1)>:
-
6
?
x1
x2Level set:
x ∈ R2 : f(x) = 1 s x0 = (1, 1)>
-
6
?
Algorithm “stuck” at x0
MTH8418: Traditional methods 36/55
LHS Coordinate Search Nelder-Mead References
Coordinate Search: Failure
f(x) = ‖x‖∞ from x0 = (1, 1)>:
-
6
?
x1
x2Level set:
x ∈ R2 : f(x) = 1 s x0 = (1, 1)>
-
6
?
Algorithm “stuck” at x0
MTH8418: Traditional methods 36/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
The Coordinate Search method
The Nelder-Mead method
References
MTH8418: Traditional methods 37/55
LHS Coordinate Search Nelder-Mead References
The Nelder-Mead method
I The Nelder-Mead (NM) algorithm, or the other Simplexmethod [Nelder and Mead, 1965]
I No link with the Simplex method from Dantzig for linearprogramming
I 1965: Good timing with the emergence of blackboxoptimization problems
I Most used and cited method for unconstrained optimization
MTH8418: Traditional methods 38/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: IdeaI Simplex in dimension n: set of n+ 1 points
I At each iteration, such a simplex is defined:
Y = y0, y1, . . . , yn
with f(y0) ≤ f(y1) ≤ . . . ≤ f(yn)
I The centroid yc of the first n points is defined as
yc =
n−1∑i=0
yi/n
I Depending on the success of the iteration, the shape and thesize of the simplex are updated
MTH8418: Traditional methods 39/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Transformations (1/2)
I Five different transformations:
I Reflection
I Expansion
I Outside contraction
I Inside contraction
I Shrink
I Except for the shrink, the worst vertex of the simplex isreplaced by one of the reflection, expansion, or contractionpoints
MTH8418: Traditional methods 40/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Transformations (2/2)
I Reflection point: yr = yc + δr(yc − yn)
I Expansion point: ye = yc + δe(yc − yn)
I Outside contraction point: yoc = yc + δoc(yc − yn)
I Inside contraction point: yic = yc + δic(yc − yn)
I Shrink points: ysi = y0 + γs(yi − y0) for i = 1, 2, . . . , n
I 0 < γs < 1, −1 < δic < 0 < δoc < δr < δe
I Standard choices: δic = −0.5, γs = δoc = 0.5, δr = 1, δe = 2
MTH8418: Traditional methods 41/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Reflection, expansion, contractions
Taken from [Conn et al., 2009]
MTH8418: Traditional methods 42/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Shrink
Taken from [Conn et al., 2009]
MTH8418: Traditional methods 43/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Algorithm (1/4)
[0] Initializationsevaluate f at Y0 = y00 , y10 , . . . , yn0 0 < γs < 1, −1 < δic < 0 < δoc < δr < δe
k ← 0
[1] Iteration k[1.1] Order
order the points in Yk such thatf0 = f(y0k) ≤ f1 = f(y1k) ≤ . . . ≤ fn = f(ynk )
[1.2] Reflectconstruct yc and yr and evaluate fr = f(yr)if (f0 ≤ fr < fn−1)
Yk+1 ← y0k, y1k, . . . , yn−1k , yr
k ← k + 1goto [1]
. . .
MTH8418: Traditional methods 44/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Algorithm (2/4)
[1] Iteration k. . .
[1.3] Expand if (fr < f0)Construct ye and evaluate fe = f(ye)if (fe ≤ fr)
Yk+1 ← y0k, y1k, . . . , yn−1k , ye
otherwiseYk+1 ← y0k, y1k, . . . , y
n−1k , yr
k ← k + 1goto [1]
. . .
MTH8418: Traditional methods 45/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Algorithm (3/4)
[1] Iteration k. . .
[1.4] Contract (fr ≥ fn−1)Outside contraction if (fr < fn)
construct yoc and evaluate foc = f(yoc)if (foc ≤ fr)
Yk+1 ← y0k, y1k, . . . , yn−1k , yoc
k ← k + 1, goto [1]otherwise goto [1.5]
Inside contraction if (fr ≥ fn)construct yic and evaluate f ic = f(yic)if (f ic < fn)
Yk+1 ← y0k, y1k, . . . , yn−1k , yic
k ← k + 1, goto [1]. . .
MTH8418: Traditional methods 46/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Algorithm (4/4)
[1] Iteration k. . .
[1.5] Shrink (fr ≥ fn−1)construct the n points ysi , i = 1, 2, . . . , nevaluate f at these n pointsYk+1 ← y0k, ys1 , ys2 , . . . , ysnk ← k + 1goto [1]
MTH8418: Traditional methods 47/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Other details
I Stopping criteria:
I Minimal diameter size
I Number of evaluations
I Number of shrinks
I etc.
I Number of evaluations at each iteration:
I 1 if the iteration is a reflection
I 2 if the iteration is an expansion or a contraction
I n+ 2 if the iteration is a shrink
MTH8418: Traditional methods 48/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Execution example
See the animation NM animation.pdf taken from Wikipedia, onthe Rosenbrock function f(x, y) = (1− x)2 + 100(y − x2)2
MTH8418: Traditional methods 49/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Convergence (1/2)
I It is a heuristic. No convergence theory. But at the time, themethod proved to be very efficient compared to what wasavailable before
I From [Swann, 1972]: “Although the methods described abovehave been developed heuristically and no proofs ofconvergence have been derived for them, in practice they havegenerally proved to be robust and reliable”
I With time, deficiencies appeared, including the famousMcKinnon family of strictly convex functions, for which NMexecutes an infinite sequence of inside contractions and failsto converge [McKinnon, 1998]
MTH8418: Traditional methods 50/55
LHS Coordinate Search Nelder-Mead References
Nelder-Mead: Convergence (2/2)
I Modified versions of the NM method are in use nowadays.They correct its known deficiencies with:I Use of a sufficient decrease instead of a simple decreaseI Restart when the simplex becomes ill-conditionedI Lexicographic rules to break the ties when ordering
I However NM remains an “old” method and modernalgorithms should be preferred
I But NM is still very popular amongst engineers
I NM is still analyzed to understand why it works so well insome instances
MTH8418: Traditional methods 51/55
LHS Coordinate Search Nelder-Mead References
Latin Hypercube Sampling
The Coordinate Search method
The Nelder-Mead method
References
MTH8418: Traditional methods 52/55
LHS Coordinate Search Nelder-Mead References
References I
Audet, C. and Dennis, Jr., J. (2006).
Mesh adaptive direct search algorithms for constrained optimization.SIAM Journal on Optimization, 17(1):188–217.(MADS).
Conn, A., Scheinberg, K., and Vicente, L. (2009).
Introduction to Derivative-Free Optimization.MOS-SIAM Series on Optimization. SIAM, Philadelphia.(NM description, Section 8.1).
Fermi, E. and Metropolis, N. (1952).
Numerical solution of a minimum problem.Los Alamos Unclassified Report LA–1492, Los Alamos National Laboratory, Los Alamos, USA.(Coordinate Search).
Hooke, R. and Jeeves, T. (1961).
“Direct Search” Solution of Numerical and Statistical Problems.Journal of the Association for Computing Machinery, 8(2):212–229.(H&J).
Kolda, T., Lewis, R., and Torczon, V. (2003).
Optimization by direct search: New perspectives on some classical and modern methods.SIAM Review, 45(3):385–482.(GSS).
MTH8418: Traditional methods 53/55
LHS Coordinate Search Nelder-Mead References
References II
McKay, M., Beckman, R., and Conover, W. (1979).
A comparison of three methods for selecting values of input variables in the analysis of output from acomputer code.Technometrics, 21(2):239–245.(Latin Hypercube Sampling).
McKinnon, K. (1998).
Convergence of the Nelder-Mead simplex method to a nonstationary point.SIAM Journal on Optimization, 9(1):148–158.(NM failure).
Nelder, J. and Mead, R. (1965).
A simplex method for function minimization.The Computer Journal, 7(4):308–313.(NM).
Swann, W. (1972).
Direct Search Methods.In Murray, W., editor, Numerical Methods for Unconstrained Optimization, pages 13–28. Academic Press,London and New York.(Remark on NM).
MTH8418: Traditional methods 54/55
LHS Coordinate Search Nelder-Mead References
References III
Torczon, V. (1997).
On the convergence of pattern search algorithms.SIAM Journal on Optimization, 7(1):1–25.(GPS).
Wright, M. (2012).
Nelder, Mead, and the Other Simplex Method.In Grotschel, M., editor, Documenta Mathematica Extra Volume: Optimization Stories, pages 271–276.Journal der Deutschen Mathematiker-Vereinigung Gegrundet 1996, Berlin.(NM history).
MTH8418: Traditional methods 55/55