Polyhedral Optimization Lecture 1 – Part 2 M. Pawan Kumar [email protected] Slides available online

Polyhedral OptimizationLecture 1 – Part 2

M. Pawan Kumar

[email protected]

Slides available online http://cvn.ecp.fr/personnel/pawan/

• Computational Complexity

• Convex Sets and Functions

• Linear Programming

Outline

f(n) O(g(n))

There exists k > 0 and n0 such that

f(n) ≤ k g(n)

for all n ≥ n0

f(n) = 5n2 + 3nlog(n)

g(n) = n2

f(n) = 5n3 + 3nlog(n)

g(n) = n2

✔ ✗

f(n) Ω(g(n))

There exists k > 0 and n0 such that

f(n) ≥ k g(n)

for all n ≥ n0

f(n) = 5n2 + 3nlog(n)

g(n) = n2

f(n) = 5n3 + 3nlog(n)

g(n) = n2

✔ ✔

f(n) Θ(g(n))

There exists k1 > 0, k2 > 0 and n0 such that

k1 g(n) ≤ f(n) ≤ k2 g(n)

for all n ≥ n0

f(n) = 5n2 + 3nlog(n)

g(n) = n2

✔

• Computational Complexity– Decision Problems (Answered by ‘yes’ or ‘no’) – P and NP– NP-Complete– NP-Hard– Combinatorial Optimization Problems



Outline

• Finite set Σ called alphabet of size ≥ 2– {0,1}– {a,b,c,d,e,…,x,y,z}

• Set Σ* of all finite length strings (called words)– 0, 1, 00, 01, 10, 11, 000,….– discrete, optimization

• Size of word size(w) = number of letters– size(00) = 2– size(discrete) = 8

Decision Problem

• Problem Π is a subset of Σ*– All words with the answer “yes”

• Informal problem– Given input word x Σ*, does x Π ?

• Polynomial-time solvable problem Π– There exists a polynomial-time algorithm for the

informal problem– Polynomial in size(x)

Decision Problem

Minimum Sum Subset

Given a set of ‘n’ real numbers S

Is there a non-empty subset X S such that ⊆

∑x X∈ x ≤ C

Minimum Sum Subset

Real numbers Σ

{ Σ } Σ , Σ

{1, 2, 3, 4, 5, 6, 7} , -5 Σ*

∉ Π

Minimum Sum Subset

Real numbers Σ

{ Σ } Σ , Σ

{1, 2, 3, 4, 5, 6, 7} , 5 Σ*

Π

Subset Sum


Is there a non-empty subset X S such that ⊆

∑x X∈ x = 0

Subset Sum

Real numbers Σ

{ Σ } Σ , Σ

{1, 2, 3, 4, 5, 6, 7} Σ*

∉ Π

Subset Sum

Real numbers Σ

{ Σ } Σ , Σ

{1, 2, 3, 4, 5, 6, -7} Σ*

Π

Shortest Path

Given graph G and and a path P from u to v

Is the path a shortest u-v path, which minimizes

∑e P∈ length(e)

Shortest Path

v0,v1,v2 … vn Σ

{ Σ

0,1,2, … N Σ

} Σ , Σ

{v0,v1,v2,v3,v4,v5},{{v0,v1},{v0,v3},{v1,v2},{v2,v3},{v2,v4},{v3,v4}},{3,5,5,4,6,1},

v1

v0

v2

v3

v4

3 5

45 1

6

{v0,v3,v4,v2}

Σ*

Shortest Path

v0,v1,v2 … vn Σ

{ Σ

0,1,2, … N Σ

} Σ , Σ


v1

v0

v2

v3

v4

3 5

45 1

6

{v0,v3,v2}

Σ*

Shortest Path

v0,v1,v2 … vn Σ

{ Σ

0,1,2, … N Σ

} Σ , Σ


v1

v0

v2

v3

v4

3 5

45 1

6

{v0,v1,v2}

Σ*

Shortest Path

v0,v1,v2 … vn Σ

{ Σ

0,1,2, … N Σ

} Σ , Σ


v1

v0

v2

v3

v4

3 5

45 1

6

{v0,v1,v2}

Π

• Computational Complexity– Decision Problems– P and NP– NP-Complete– NP-Hard– Combinatorial Optimization Problems



Outline

P

• Polynomial-time solvable problem Π– There exists a polynomial-time algorithm that

decides whether x Σ* belongs to Π or not.– Polynomial in size(x)

• P = {all Π, Π is polynomial time solvable}

• For example– Minimum Sum Subset Problem

NP

• NP is not “not polynomial”

• NP stands for non-deterministic polynomial

• Easy to verify “yes” words

NP

• Π NP– There exists a problem Π’ P– There exists a polynomial p– There exists an x, size(x) ≤ p(size(w)) such that– w Π if and only if wx Π’

• Polynomial-time checkable ‘certificate’

• For example– Subset Sum Problem

NP

• Relationship between P and NP?

• P is a subset of NP

• Is P = NP?– Open Problem– One of Clay Institute’s Millennium Prize problems

If P = NP …

• If P=NP, then the world would be a profoundly different place than we usually assume it to be.

• There would be no special value in "creative leaps," no fundamental gap between solving a problem and recognizing the solution once it's found.

• Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; …

Scott Aaronson




Outline

NP-Complete

• Hardest problems in NP– All problems in NP can be reduced to an NP-

complete problem

• Π is reducible to Λ– There exists a polynomial time algorithm– Given w, returns x– w Π if and only if x Λ

• If Λ P then Π P

NP-Complete


complete problem


• If Λ NP then Π NP

NP-Complete


complete problem


• If Λ NP-complete and Λ P, then P = NP

NP-Complete

• Is there an NP-Complete problem?

• SAT: Is a Boolean expression satisfiable?

• Cook’s Theorem: SAT is NP-Complete

• In more detail …

• Alphabet Σ containing variables x1,x2,…,xn

• And special symbols (, ), ∧,∨,~

• And not containing 0 and 1

• A variable is a Boolean expression

Boolean Expression




• If v and w are Boolean expressions then– (v ∧ w) is a Boolean expression

Boolean Expression




• If v and w are Boolean expressions then– (v ∨ w) is a Boolean expression

Boolean Expression




• If v and w are Boolean expressions then– ~v is a Boolean expression– ~w is a Boolean expression

Boolean Expression




• For example, f(x1,x2,…,xn)

– ((x2 ∧ x3) ∨ ~(x3 ∨ x5) ∧ x2) ∨ ~(x2 ∧ x5)

– ~(~x2 ∨ ~x3) ∧ ~x1

Boolean Expression

• f(x1,x2,…,xn) is satisfiable if

– there exists an assignment x1 = α1, …, xn = αn

– where αi {0,1}

– such that f(x1,x2,…,xn) = 1

• The following identities hold– 0∧0 = 0, 0∧1 = 0, 1∧0 = 0, 1∧1 = 1– 0∨0 = 0, 0∨1 = 1, 1∨0 = 1, 1∨1 = 1– ~0 = 1, ~1 = 0– (0) = 0, (1) = 1

Satisfiability

• Given the alphabet Σ

• Given the identities for 0 and 1

• SAT is a subset of Σ* that is satisfiable

• Informal Problem– Given a Boolean expression w, is w satisfiable

SAT

Karp’s 21 NP-Complete Problems

• If SAT is reducible to Π, Π is NP-complete

• SAT, 0-1 integer programming, Clique, Set Packing, Vertex Cover, Set Covering, Feedback node set, Feedback arc set, Directed Hamilton circuit, Undirected Hamilton circuit, 3-SAT, Chromatic number, Clique cover, Exact cover, Hitting set, Steiner tree, 3-dimensional matching, Knapsack, Job sequencing, Partition, Max cut

• A special case of SAT

• Given variables x1,x2,…,xn Σ– B1 consists of x1,~x1,…,xn,~xn

– B2 consists of (w1∨…∨wk), wi B1, 1≤k≤3

– B3 consists of w1∧w2∧…∧wm, wi B2

– e.g., (x1∨~x2∨x3) ∧ (x2∨~x3∨x4) ∧ (~x1∨~x2)

• 3-SAT is the subset of B3 that is satisfiable

3-SAT

• x1 = x2 ∨ x3

– (x1∨~x2) ∧ (x1∨~x3) ∧ (~x1∨x2∨x3)

• x1 = x2 ∧ x3

– (~x1∨x2) ∧ (~x1∨x3) ∧ (x1∨~x2∨~x3)

• x1 = ~x2

– (x1∨x2) ∧ (~x1∨~x2)

SAT is reducible to 3-SAT

NP-Complete Problems

• Today, we know several such problems

• http://en.wikipedia.org/wiki/List_of_NP-complete_problems

• No proof if they are in P

• No proof if they are not in P

http://en.wikipedia.org/wiki/List_of_NP-complete_problems

http://en.wikipedia.org/wiki/List_of_NP-complete_problems




Outline

NP-hard

• Need not be in NP– No polynomial-time checkable certificate– Not even a decision problem (e.g. optimization)

• At least as hard as NP-complete problems

• Λ NP-hard– There exists Π NP-complete– Π is reducible to Λ

NP-hard

Image Courtesy of Wikipedia




Outline

Combinatorial Optimization Problem

• Given a discrete set of feasible solutions

• And a measure of the quality of each solution

• Find the best quality solution

Combinatorial Optimization Problem

• I = set of instances of a problem

• Given an instance i ∈I

• Discrete set of feasible solutions Fi

• Measure of quality mi: Fi → Real

• maxx mi(x), x F∈ i

Minimum Sum Subset


Find a non-empty subset X S that minimizes⊆

∑x X∈ x

Minimum Sum Subset


Find a non-empty subset X S that maximizes⊆

-∑x X∈ x

Minimum Sum Subset Instance


Find a non-empty subset X S that maximizes⊆

-∑x X∈ x

{-28, 53, -58, -99, 13, 27, -55, -31, -91, 12, -87, -68}

Feasible solutions Fi? Measure mi?

Subset Sum



| ∑x X∈ x |

Subset Sum Instance



| ∑x X∈ x |

{-28, 53, -58, -99, 13, 27, -55, -31, -91, 12, -87, -68}


Shortest Path

Given graph G and vertices u and v

Find the path from u to v that minimizes

∑e P∈ length(e)

Shortest Path Instance

v1

v0

v2

v3

v4

3 5

45 1

6


Complexity

• Easy problems have polynomial-time solution

• P optimization problems

• Hard problems P optimization problems∉

• NP-hard problems


• Convex Sets and Functions– Convex Set– Convex Hull and Convex Cone– Polyhedron and Polytope– Convex Function


Outline

Convex Set

x1 x2

λ x1 + (1 - λ) x2

λ [0,1]

Line Segment

Endpoints

Convex Set

x1 x2

All points on the line segment lie within the set

For all line segments with endpoints in the set

Non-Convex Set

x1 x2

Examples of Convex Sets

x1 x2

Line Segment


x1 x2

Line


Hyperplane aTx - b = 0


Halfspace aTx - b ≤ 0


Second-order Cone ||x|| ≤ t

t

x2

x1


Semidefinite Matrices {X | X 0}

aTXa ≥ 0, for all a Rn

All eigenvalues of X are non-negative

aTX1a ≥ 0 aTX2a ≥ 0

aT(λX1 + (1-λ)X2)a ≥ 0

Operations that Preserve ConvexityIntersection

Operations that Preserve ConvexityAffine Transformation x Ax + b

Separating Hyperplane

Closed Convex Set C

z C∉

Property

For any closed convex set C, and z C ∉

There exists a separating hyperplane

Proof?

Proof Sketch

Let y = argminx C∈ ||z-x||

Define a = z - y b = (||z||2 - ||y||2)/2

We can show that aTz > b

We can show that aTx < b, for all x C ∈

Proof Sketch

aTz = (z-y)Tz

> (z-y)Tz - ||z-y||2/2

= b

Proof Sketch

Let aTx ≥ b, for all x C ∈

aTy < aTy + ||a||2/2 = b

Therefore, there exists 0 < λ ≤ 1

λ < 2aT(x-y)/||x-y||2

Let w = λx + (1-λ)y

We can show ||w-z||2 < ||y-z||2 Contradiction

Closed Convex Set

C is a closed convex set

C is an intersection of halfspaces

The two statements are equivalent

Proof?




Outline

Convex Hull

Intersection preserves convexity

Given a set X

Smallest convex set containing X exists

Convex Hull

Convex Hull of a Convex Set

Convex Hull of Vectors

Convex Hull

x belongs to the convex hull of X

There exists a natural number t

x = λ1x1 + λ2x2 + … + λtxt xi ∈ X

λ1 + λ2 + … + λt = 1 λi > 0

Proof?

Convex Cone

Convex Set C

If x1 and x2 belong to C

λ1x1 + λ2x2 ∈ C For all λ1, λ2 ≥ 0

Cone

x belongs to the cone(X)

There exists a natural number t

x = λ1x1 + λ2x2 + … + λtxt xi ∈ X λi > 0

Given a set X

cone(X): Smallest convex cone containing X




Outline

Polyhedron

a1Tx ≤ b1

a2Tx ≤ b2

amTx ≤ bm

.

.

.

Polyhedron

Ax ≤ b

A : m x n matrix

b: n x 1 vector

Bounded Polyhedron

Vertex

Formal definition?

Vertex

z is a vertex of P = {x, Ax ≤ b}

z is not a convex combination of two points in P

There does not exist x, y P and 0 < λ < 1∈

x ≠ z and y ≠ z

such that z = λ x + (1-λ) y

Vertex

z is a vertex of P = {x, Ax ≤ b}

Az is a submatrix of A

Contains all rows of A such that aiTz = bi

Recall A is an m x n matrix

Vertex

z is a vertex of P

Rank of Az = n

⟺

Proof?

Proof Sketch: Necessity

Let z be a vertex of P

Suppose rank(Az) < n Azc = 0 for some c ≠ 0

Then there exists a d > 0 such that

z - dc ∈ P z + dc ∈ P Contradiction

Proof Sketch: Sufficiency

Suppose rank(Az) = n but z is not a vertex of P

z = (x+y)/2 for some x, y ∈ P, x ≠ y ≠ z

For each a in Az

Implies Az(x-z) = 0 Contradiction

aTx ≤ b = aTz aTy ≤ b = aTz

Polytope

Convex hull of a finite number of vectors

Polytope and Polyhedron

P is a polytope

P is a bounded polyhedron

⟺

Skipping the proof (available on webpage)




Outline

Convex Function

x

g(x)

Blue point always lies above red point

x1

x2

Convex Function

x

g(x)

g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2)

x1

x2

Domain of g(.) has to be convex

Convex Function

x

g(x)

x1

x2

-g(.) is concave

g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2)

Convex Function

Once-differentiable functions

g(y) + g(y)T (x - y) ≤ g(x)

x

g(x)

(y,g(y))

g(y) + g(y)T (x - y)

Twice-differentiable functions

2g(x) 0

Convex Function and Convex Sets

x

g(x)

Epigraph of a convex function is a convex set

Examples of Convex Functions

Linear function aTx

p-Norm functions (x1p + x2

p + xnp)1/p, p ≥ 1

Quadratic functions xT Q x

Q 0

Operations that Preserve ConvexityNon-negative weighted sum

x

g1(x)

w1

x

g2(x)

+ w2 + ….

xT Q x + aTx + b

Q 0

Operations that Preserve ConvexityPointwise maximum

x

g1(x)

max

x

g2(x)

,

Pointwise minimum of concave functions is concave




Outline

Examples taken from the slides of Ziv Bar-Yossef (Technion)

Linear Program

Maximize a linear function

Over a polyhedral feasible region

s.t. A x ≤ b

maxx cTx

A: m x n matrix

b: m x 1 vector

c: n x 1 vector

Objective function

Constraints

x: n x 1 vector

Example

4x1 – x2 ≤ 8

maxx x1 + x2

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

s.t.

What is c? A? b?

Example

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 = 8

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2x1 + x2 = 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2

x1 + x2 = 8 Optimal solution



• Linear Programming– Duality– Complementary Slackness– Solving the LP

Outline

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

3 x

2 x 7 x

Scale the constraints, add them up

3x1 + x2 + 2x3 ≤ 90 Upper bound on solution

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

1 x

1 x


3x1 + x2 + 2x3 ≤ 36 Upper bound on solution

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

1 x

1 x


3x1 + x2 + 2x3 ≤ 36 Tightest upper bound?

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

y1, y2, y3, y4, y5, y6 ≥ 0

We should be able to add up the inequalities

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y1 + y4 + 2y5 + 4y6 = 3

Coefficient of x1 should be 3

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y2 + y4 + 2y5 + y6 = 1


Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y3 + 3y4 + 5y5 + 2y6 = 2


Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

miny 30y4 + 24y5 + 36y6

Upper bound should be tightest

Dual

miny 30y4 + 24y5 + 36y6

s.t. y1, y2, y3, y4, y5, y6 ≥ 0

-y1 + y4 + 2y5 + 4y6 = 3

-y2 + y4 + 2y5 + y6 = 1

-y3 + 3y4 + 5y5 + 2y6 = 2

Original problem is called primal

Dual of dual is primal

Dual

s.t. A x ≤ b

maxx cTx

Dual

- yT(A x – b)maxx cTxminy≥0

KKT Condition? ATy = c

miny≥0 bTy

s.t. ATy = c

miny≥0 bTy

s.t. ATy = c

s.t. A x ≤ b

maxx cTxPrimal

Dual

Strong Duality

miny≥0 bTy

s.t. ATy = c

s.t. A x ≤ b

maxx cTxPrimal

Dual

p =

d =

If p ≠ ∞ or d ≠ ∞, then p = d.

Skipping the proof (available on webpage)

Think back to the intuition of dual

Question

s.t. A1 x ≤ b1

maxx cTx

A2 x ≥ b2

A3 x = b3

Dual?




Outline

Weak Duality

Let x be a feasible solution of the primal

cTx ≤ bTy

Objective of any primal ≤ Objective of any dual

Think back to the intuition of dual

Proof?

Let y be a feasible solution of the dual

Proof Sketch

cTx

= (ATy)Tx

When does equality hold?

Optimal primal and dual solutions: x* and y*

= yTAx

≤ yTb

= bTy

Equality

(y*)TAx* - (y*)Tb = 0

Recall that y* ≥ 0

Recall that Ax* ≤ b

If y*j > 0 ajTx* = bj

Complementary slackness

Necessary and sufficient for optimality




Outline

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2

x1 + x2 = 8

Optimal solution at a vertex

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1

x1 = 3


Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x2

x2 = 6


Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

An optimal solution always at a vertex

Proof? maxx cTx

• Dantzig (1951): Simplex Method– Search over vertices of the polyhedra– Worst-case complexity is exponential– Smoothed complexity is polynomial

• Khachiyan (1979, 1980): Ellipsoid Method– Polynomial time complexity– LP is a P optimization problem

• Karmarkar (1984): Interior-point Method– Polynomial time complexity– Competitive with Simplex Method

Solving the LP

Optimization vs. Feasibility

s.t. A x ≤ b

maxx cTxOptimization

Feasibility asks if there exists an x such that

cTx ≥ K

A x ≤ b

Optimization via binary search on K

Feasible solution

For a given K

Feasibility via Ellipsoid Method

Feasible region of LP


Ellipsoid containing feasible region of LP


Centroid of ellipsoid


Separating hyperplane for centroid


Smallest ellipsoid containing “truncated” ellipsoid




Separating hyperplane for centroid


Smallest ellipsoid containing “truncated” ellipsoid




Terminate when feasible solution is found

• Separating hyperplane in polynomial time– Check each of the ‘m’ LP constraints in O(n) time

• New ellipsoid in polynomial time– Shor (1971), Nemirovsky and Yudin (1972)

• Polynomial iterations (Khachiyan)– Volume of ellipsoid reduces exponentially

• Only requires a separation oracle– Constraint matrix A can be very large

Ellipsoid Method

UsefulFact !!

Questions?

Documents

Polyhedral Optimization Lecture 1 – Part 2 M. Pawan Kumar [email protected] Slides available online