Optimization Problems

Optimization Problems

虞台文大同大學資工所智慧型多媒體研究室

ContentIntroductionDefinitionsLocal and Global OptimaConvex Sets and FunctionsConvex Programming

Problems


Introduction

大同大學資工所智慧型多媒體研究室

General Nonlinear Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

Local Minima vs. Global Minima

( )f xminimize


( ) 0 1, ,jh x j p nx R

objective function

constraints

local minimum

global minimum

Convex Programming Problems

( )f xminimize


( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

convex

concave

linear

Local optimality Global optimality

Linear Programming Problems

( )f xminimize


( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear


a special case of convex programming problems


( )f xminimize


( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear


Integer Programming Problems

( )f xminimize


( ) 0 1, ,jh x j p nx Z

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear

The Hierarchy of Optimization Problems

NonlinearPrograms

ConvexPrograms

LinearPrograms

(Polynomial) IntegerPrograms(NP-Hard)

Flowand

Matching





Integer Linear Programming Problems

Optimization Techniques




Integer Linear Programming Problems

ContinuousVariables

DiscreteVariables

ContinuousOptimization

CombinatorialOptimization


Definitions



( )f xminimize


( ) 0 1, ,jh x j p nx R

( )f xminimize



( ) 0 1, ,jh x j p nx R

Define the set of feasible points

F

Minimize cost c: FR1

Definition:Instance of an Optimization Problem

(F, c) F: the domain of feasible points

c: F R1 cost function

Goal: To find f F such that

c( f ) c(g) for all gF.

A global optimum

Definition:Optimization Problem

A set of instances of an optimization problem, e.g.– Traveling Salesman Problem (TSP)– Minimal Spanning Tree (MST)– Shortest Path (SP)– Linear Programming (LP)

Traveling Salesman Problem (TSP)

Traveling Salesman Problem (TSP)

Instance of the TSP – Given n cities and an n n distance matrix [dij], t

he problem is to find a Hamiltonian cycle with minimal total length.

on F n all cyclic permutations objects

( )1

n

j jj

c d

1 2 3 4 5 6 7 8

2 5 3 6 1 8 4 7

e.g.,

Minimal Spanning Tree (MST)

Minimal Spanning Tree (MST)

Instance of the MST – Given an integer n > 0 and an n n symmetric distance m

atrix [dij], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.

( , ) {1,2, , }VF E V n all spanning trees with

( , )

: ( , ) iji j E

c V E d

Linear Programming (LP)

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to



11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1

2

n

c

cc

c

11 12 1

21 22 2

1 2

n

n

m m mn

a a a

a a aA

a a a

1

2

m

b

bb

b

1

2

n

x

xx

x

minimize

Subject to

c x

Ax b0x


, , 0nx x R AF x b x

:c x c x


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize

Subject to

c x

Ax b0x

Example:Linear Programming (LP)


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1 2 34 2 3x x x

1 2 3

1 2 3

2

, , 0

x x x

x x x

4 2 3c

1 1 1A 2b

minimize

Subject to

Example:Linear Programming (LP)


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to


11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1 2 34 2 3x x x

1 2 3

1 2 3

2

, , 0

x x x

x x x

minimize

Subject to

x1

x2

x3

v1

v2

v3

c(v1) = 8

c(v2) = 4

c(v3) = 6

The optimum

The optimal point is at one of the vertices.

Example:Minimal Spanning Tree (3 Nodes)

1 2 34 2 3x x x

1 2 3 2x x x

minimize

Subject to

c1=4

c3=3

c2=2

1 2 3, , {0,1}x x x

x1{0, 1}

x2{0, 1}

x3{0, 1}

Integer Programming

x1

x2

x3

Example:Minimal Spanning Tree (3 Nodes)

1 2 34 2 3x x x

1 2 3 2x x x

minimize

Subject to

c1=4

c3=3

c2=2

x1{0, 1}

x2{0, 1}

x3{0, 1}

Linear Programming

x1

x2

x3

1 2 3, , 0x x x 1 2 3, , 1x x x

Some integer programs can be transformed into linear programs.


Local and Global Optima


Neighborhoods

Given an optimization problem with instance

(F, c),

a neighborhood is a mapping

defined for each instance.

: 2FN F

For combinatorial optimization, the choice of N is critical.

TSP (2-Change)

f F gN2(f )

2 ( ) N f g g F g and can be obtained as above

TSP (k-Change)

( )

.k

g F gN f g

k f

and can be obtained

by changing edges of

MST

f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.

( ) N f g g F g and can be obtained as above

LP

minimize

Subject to

c x

Ax b0x

( ) , 0, N x y Ay b y y x and

Local Optima

Given(F, c)

N

an instance of an optimization problem

neighborhood

f F is called locally optimum with respect to N (or simply

locally optimum whenever N is understood by context) if

c(f ) c(g) for all gN(f ).

0 1 F

c

small

Local Optima

F = [0, 1] R1

( ) , 0, N f x x F y x f and

C

B

A Local minimum

Local minimum

Global minimum

Decent Algorithm

f = initial feasible solution

While Improve(f ) do

f = any element in Improve(f )

return f

Improve( ) ( ) ( ) ( )f s s N f c s c f and

Decent algorithm is usually stuck at a

local minimum unless the neighborhood N

is exact.

Exactness of Neighborhood

Neighborhood N is said to be exact if it makes

Local minimum Global Minimum

Exactness of Neighborhood

0 1 F

c

F = [0, 1] R1

( ) , 0, N f x x F y x f and

C

B

A Local minimum

Local minimum

Global minimum

N is exact if 1.

TSP

N2: not exact

Nn: exact

f F gN2(f )

MST N is exact

f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.

( ) N f g g F g and can be obtained as above


Convex Sets and Functions


Convex Combination

x, y Rn

0 1 z = x +(1)y

A convex combination of x, y.

A strict convex combination of x, y if 0, 1.

Convex Sets

S Rn

z = x +(1)y

is convex if it contains all convex combinations of pairs x, y S.

convex nonconvex

0 1

Convex Sets

S Rn

z = x +(1)y

is convex if it contains all convex combinations of pairs x, y S.

n = 1

S is convex iff S is an interval.

0 1

Convex Sets

Fact: The intersection of any number of convex sets is convex.

c

Convex Functions

x yx +(1)y

c(x)

c(y)c(x) + (1)c(y)

c(x +(1)y)

S Rn a convex set

c:S R a convex function if

c(x +(1)y) c(x) + (1)c(y), 0 1

Every linear function is convex.

LemmaS

c(x)

t

a convex set

a convex function on S

a real number

( ) ,tS c x x Stx

is convex.

Pf) Let x, y St x +(1)y S

c(x +(1)y) c(x) + (1)c(y)

t + (1)t

= t

x +(1)y St

Level Contours

c = 1

c = 2

c = 3

c = 4

c = 5

Concave Functions

S Rn a convex set

c:S R a concave function if

c is a convex

Every linear function is concave as well as convex.




Theorem

(F, c) an instance of optimization problem

a convex set

a convex function

Define ( )N x y y F x y and

( )N x is exact for every > 0.

• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point.

Theorem

(F, c) an instance of optimization probleman instance of optimization problem

a convex set

a convex function

Defi ne ( )N x y y F x y and

( )N x is exact f or every > 0.


a convex set

a convex function



Pf)

xF

( )N x

yNext, we now want to show that c(y) c(x).

• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point. <<1 such that• Since c is convex, we have

• Therefore,

Theorem


a convex set

a convex function




a convex set

a convex function



Pf)

xF

( )N x

yz

(1 ) ( ).x y xz z N and

( ) ( (1 ) )c c x yz

( ) (1 ) ( )c x c y

( ) ( )( )

1

zc c xc y

( ) ( )

1

c x c x

( )c x

( ) ( )zc c x


(F, c)

Defined by ( ) 0, 1, ,ig x i m

: nig R R

Convex function

an instance of optimization problem

Important property:


Concave functions

Convexity of Feasible Set

(F, c)


: nig R R

Convex function


Important property:


Concave functions

( ) : ig x convex

( ) : ig x concave

( ) 0 : ig x convex

( ) 0 : ig x convex

: iF convex

1

: m

ii

F F

convex


(F, c)


: nig R R

Convex function


Important property:


Concave functionsConvex

Theorem

In a convex programming problem, every

point locally optimal with respect to the

Euclidean distance neighborhood N is also

global optimal.

Documents

Optimization Problems