Discrete Optimization Lecture 2 – Part 2

M. Pawan Kumar

pawan.kumar@ecp.fr

Slides available online http://cvn.ecp.fr/personnel/pawan/

• Integer Programming Formulation

• LP Relaxation

Outline

Integer Programming Formulation

Va Vb

Label l0

Label l12

5

4

2

0

1 1

0

2Unary Potentials

a;0 = 5

a;1 = 2

b;0 = 2

b;1 = 4

Labeling

f(a) = 1

f(b) = 0

ya;0 = 0 ya;1 = 1

yb;0 = 1 yb;1 = 0

Any f(.) has equivalent boolean variables ya;i

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Unary Potentials

a;0 = 5

a;1 = 2

b;0 = 2

b;1 = 4

Labeling

f(a) = 1

f(b) = 0

ya;0 = 0 ya;1 = 1

yb;0 = 1 yb;1 = 0

Find the optimal variables ya;i

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Unary Potentials

a;0 = 5

a;1 = 2

b;0 = 2

b;1 = 4

Sum of Unary Potentials

∑a ∑i a;i ya;i

ya;i {0,1}, for all Va, li∑i ya;i = 1, for all Va

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Pairwise Potentials

ab;00 = 0

ab;10 = 1

ab;01 = 1

ab;11 = 0

Sum of Pairwise Potentials

∑(a,b) ∑ik ab;ik ya;iyb;k

ya;i {0,1}

∑i ya;i = 1

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Pairwise Potentials

ab;00 = 0

ab;10 = 1

ab;01 = 1

ab;11 = 0

Sum of Pairwise Potentials

∑(a,b) ∑ik ab;ik yab;ik

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Label l0

Label l1

Integer Programming Formulation

min ∑a ∑i a;i ya;i + ∑(a,b) ∑ik ab;ik yab;ik

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Integer Programming Formulation

min Ty

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

= [ … a;i …. ; … ab;ik ….]

y = [ … ya;i …. ; … yab;ik ….]

One variable, two labels

ya;0

ya;1

ya;0 {0,1} ya;1 {0,1} ya;0 + ya;1 = 1

y = [ ya;0 ya;1] = [ a;0 a;1]

Two variables, two labels

q = [ a;0 a;1 b;0 b;1

ab;00 ab;01 ab;10 ab;11]y = [ ya;0 ya;1 yb;0 yb;1

yab;00 yab;01 yab;10 yab;11]

ya;0 {0,1} ya;1 {0,1} ya;0 + ya;1 = 1

yb;0 {0,1} yb;1 {0,1} yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

In General

Marginal Polytope

In General

q R(|V||L| + |E||L|2)

y {0,1}(|V||L| + |E||L|2)

Number of constraints

|V||L| + |V| + |E||L|2

ya;i {0,1} ∑i ya;i = 1 yab;ik = ya;i yb;k

Integer Programming Formulation

min Ty

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

= [ … a;i …. ; … ab;ik ….]

y = [ … ya;i …. ; … yab;ik ….]

Integer Programming Formulation

min Ty

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Solve to obtain MAP labelling y*

Integer Programming Formulation

min Ty

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

But we can’t solve it in general

• Integer Programming Formulation

• LP Relaxation

Outline

Linear Programming Relaxation

min Ty

ya;i {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Two reasons why we can’t solve this

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

yab;ik = ya;i yb;k

One reason why we can’t solve this

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

∑k yab;ik = ∑kya;i yb;k

One reason why we can’t solve this

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

One reason why we can’t solve this

= 1∑k yab;ik = ya;i∑k yb;k

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

One reason why we can’t solve this

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

No reason why we can’t solve this *

*memory requirements, time complexity

One variable, two labels

ya;0

ya;1

ya;0 {0,1} ya;1 {0,1} ya;0 + ya;1 = 1

y = [ ya;0 ya;1] = [ a;0 a;1]

One variable, two labels

ya;0

ya;1

ya;0 [0,1] ya;1 [0,1] ya;0 + ya;1 = 1

y = [ ya;0 ya;1] = [ a;0 a;1]

Two variables, two labels

q = [ a;0 a;1 b;0 b;1

ab;00 ab;01 ab;10 ab;11]y = [ ya;0 ya;1 yb;0 yb;1

yab;00 yab;01 yab;10 yab;11]

ya;0 {0,1} ya;1 {0,1} ya;0 + ya;1 = 1

yb;0 {0,1} yb;1 {0,1} yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

q = [ a;0 a;1 b;0 b;1

ab;00 ab;01 ab;10 ab;11]y = [ ya;0 ya;1 yb;0 yb;1

yab;00 yab;01 yab;10 yab;11]

ya;0 [0,1] ya;1 [0,1] ya;0 + ya;1 = 1

yb;0 [0,1] yb;1 [0,1] yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

q = [ a;0 a;1 b;0 b;1

ab;00 ab;01 ab;10 ab;11]y = [ ya;0 ya;1 yb;0 yb;1

yab;00 yab;01 yab;10 yab;11]

ya;0 [0,1] ya;1 [0,1] ya;0 + ya;1 = 1

yb;0 [0,1] yb;1 [0,1] yb;0 + yb;1 = 1

yab;00 + yab;01 = ya;0

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

q = [ a;0 a;1 b;0 b;1

ab;00 ab;01 ab;10 ab;11]y = [ ya;0 ya;1 yb;0 yb;1

yab;00 yab;01 yab;10 yab;11]

ya;0 [0,1] ya;1 [0,1] ya;0 + ya;1 = 1

yb;0 [0,1] yb;1 [0,1] yb;0 + yb;1 = 1

yab;00 + yab;01 = ya;0

yab;10 + yab;11 = ya;1

In General

Marginal Polytope

LocalPolytope

In General

q R(|V||L| + |E||L|2)

y [0,1](|V||L| + |E||L|2)

Number of constraints

|V||L| + |V| + |E||L|

Linear Programming Relaxation

min Ty

ya;i [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

No reason why we can’t solve this

Linear Programming Relaxation

Extensively studied

Optimization

Schlesinger, 1976

Koster, van Hoesel and Kolen, 1998

Theory

Chekuri et al, 2001 Archer et al, 2004

Machine Learning

Wainwright et al., 2001

Linear Programming Relaxation

Many interesting properties

• Global optimal MAP for trees

Wainwright et al., 2001

But we are interested in NP-hard cases

• Preserves solution for reparameterization

• Global optimal MAP for submodular energy

Chekuri et al., 2001

Linear Programming Relaxation

• Large class of problems

• Metric Labeling• Semi-metric Labeling

Many interesting properties - Integrality Gap

Manokaran et al., 2008

• Most likely, provides best possible integrality gap