Algorithms, Graph Theory, and the Soluon of …mcgrew/media/Events/distinguished/...the Soluon of Laplacian Linear Equaons Daniel A. Spielman Yale University Rutgers, Dec 6, 2011 Outline

Algorithms,GraphTheory,andtheSolu7onofLaplacianLinear

Equa7ons

DanielA.SpielmanYaleUniversity

Rutgers, Dec 6, 2011

Outline

LinearSystemsinLaplacianMatricesWhat?Why?Classicwaystosolvethesesystems.

Approxima7ngGraphsbyTrees

SparseApproxima7onsofGraphs

LocalGraphClustering

LaplacianLinearSystems

Solvein7mewhere=numberofnon‐zerosentriesofA

7mesfor‐approximatesolu7on.

Enablessolu7onofallsymmetric,diagonally‐dominantsystems,includingsub‐matricesofLaplacians.

O(m logc m)m

log(1/!) !!!x!A!1b

!!A" !

!!A!1b!!A

Ax = b

LaplacianQuadra7cFormof

For x : V ! IR

xTLGx =!

(u,v)!E

(x (u)! x (v))2

G = (V,E)


For x : V ! IR

!1!3 01

3

x :

xTLGx =!

(u,v)!E

(x (u)! x (v))2

G = (V,E)


For x : V ! IR

!1!3 0x :

xTLGx = 15

22 1212

32

xTLGx =!

(u,v)!E

(x (u)! x (v))2

1

3

G = (V,E)


For x : V ! IR

0x :

12

xTLGx =!

(u,v)!E

(x (u)! x (v))2

1

G = (V,E)

0 1

10

0

0

xTLGx = 1

Laplacian Quadratic Form, examples

When x is the characteristic vector of a set S, countstheedgesontheboundaryofS

00

0

1

1

1

S 0xTLGx = |bdry(S)|

Laplacian Quadratic Form, examples

When x is the characteristic vector of a set S, countstheedgesontheboundaryofS

00

0

1

1

1

S 0xTLGx = |bdry(S)|

xTLGx

xTx=

|bdry(S)||S|

=edge‐expansionofS

LearningonGraphs[Zhu‐Ghahramani‐Lafferty’03]

Infervaluesofafunc7onatallver7cesfromknownvaluesatafewver7ces.

Minimize xTLGx =!

(u,v)!E

w(u,v) (x (u)! x (v))2

Subjecttoknownvalues

0

1

0

10.5

0.5

0.6250.375

Taking deriva,ves, minimize by solving Laplacian

Infervaluesofafunc7onatallver7cesfromknownvaluesatafewver7ces.

Minimize xTLGx =!

(u,v)!E

w(u,v) (x (u)! x (v))2

Subjecttoknownvalues

LearningonGraphs[Zhu‐Ghahramani‐Lafferty’03]

OtherApplica7ons

Solveforcurrentwhenfixvoltages

1V

0V

Compu7ngeffec7veresistancesinresistornetworks:

OtherApplica7ons

Solveforcurrentwhenfixvoltages

1V

0V

Compu7ngeffec7veresistancesinresistornetworks:

0.5V

0.5V

0.625V0.375V

LaplacianQuadra7cFormforWeightedGraphs

xTLGx =!

(u,v)!E

w(u,v) (x (u)! x (v))2

G = (V,E,w)

w : E ! IR+ assignsaposi7veweighttoeveryedge

MatrixLGisposi7vesemi‐definitenullspacespannedbyconstvector,ifconnected

LaplacianMatrixofaWeightedGraph

LG(u, v) =

!"#

"$

!w(u, v) if (u, v) " E

d(u) if u = v

0 otherwise

4 -1 0 -1 -2 -1 4 -3 0 0 0 -3 4 -1 0 -1 0 -1 2 0 -2 0 0 0 2

1 2

34

51

1

2

1

3

d(u) =!

(v,u)!E w(u, v)

the weighted degree of u

isadiagonallydominantmatrix

ClassicApplica7ons

Compu7ngeffec7veresistances.

SolvingEllip7cPDEs.

Compu7ngEigenvectorsandEigenvaluesofLaplaciansofgraphs.

SolvingMaximumFlowbyInteriorPointMethods

SolvingLaplacianLinearEqua7onsQuickly

Fastwhengraphissimple,byelimina7on.

Fastapproxima7onwhengraphiscomplicated*,byConjugateGradient

*=randomgraphorhighexpansion

CholeskyFactoriza7onofLaplacians

AlsoknownasY‐Δ

Wheneliminateavertex,connectitsneighbors.

3 -1 0 -1 -1 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2 0 -1 0 0 0 1

1 2

34

51

1

1

1

1


AlsoknownasY‐Δ


3 -1 0 -1 -1 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2 0 -1 0 0 0 1

1 2

34

51

1

1

1

1 .33

.33

.33

3 0 0 0 0 0 1.67 -1.00 -0.33 -0.33 0 -1.00 2.00 -1.00 0 0 -0.33 -1.00 1.67 -0.33 0 -0.33 0 -0.33 0.67


AlsoknownasY‐Δ


3 -1 0 -1 -1 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2 0 -1 0 0 0 1

1 2

34

5

1

1 .33

.33

.33

3 0 0 0 0 0 1.67 -1.00 -0.33 -0.33 0 -1.00 2.00 -1.00 0 0 -0.33 -1.00 1.67 -0.33 0 -0.33 0 -0.33 0.67

3 0 0 0 0 0 1.67 -1.00 -0.33 -0.33 0 -1.00 2.00 -1.00 0 0 -0.33 -1.00 1.67 -0.33 0 -0.33 0 -0.33 0.67

3 -1 0 -1 -1 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2 0 -1 0 0 0 1

3 0 0 0 0 0 1.67 0 0 0 0 0 1.4 -1.2 -0.2 0 0 -1.2 1.6 -0.4 0 0 -0.2 -0.4 0.6

1 2

34

51

1

1

1

1

1 2

34

5.33

.33

1

1 .33

1 2

34

5 .2

1.2

.4

1 0 0 0 0 0 2 -1 0 -1 0 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2

1 -1 0 0 0 -1 3 -1 0 -1 0 -1 2 -1 0 0 0 -1 2 -1 0 -1 0 -1 2

1 0 0 0 0 0 2 0 0 0 0 0 1.5 -1 -0.5 0 0 -1.0 2 -1.0 0 0 -0.5 -1 1.5

2 3

45

11

1

1

1

1

2 3

45

11

1

1

1

2 3

45

1

1

1

0.5

Theordermaeers

ComplexityofCholeskyFactoriza7on

#ops~Σv(degreeofvwheneliminate)2

Tree #ops~O(|V|)



#ops~O(|V|)Tree



Tree #ops~O(|V|)



Tree #ops~O(|V|)



Tree #ops~O(|V|)



Tree #ops~O(|V|)



Tree #ops~O(|V|)

Planar #ops~O(|V|3/2)Lipton‐Rose‐Tarjan‘79



Tree #ops~O(|V|)

Planar #ops~O(|V|3/2)Lipton‐Rose‐Tarjan‘79

Expander likerandom,butO(|V|)edges

#ops≳Ω(|V|3)Lipton‐Rose‐Tarjan‘79

For S ! V

!G = minS!V !(S)

S

ExpansionandCholeskyFactoriza7on

!(S) =|bdry(S)|

min (|S| , |V ! S|)

For S ! V

!G = minS!V !(S)

S

ExpansionandCholeskyFactoriza7on

!(S) =|bdry(S)|

min (|S| , |V ! S|)

CholeskyslowwhenexpansionhighCholeskyfastwhenlowforGandallsubgraphs

Cheeger’sInequalityandtheConjugateGradient

Cheeger’sinequality(degree‐dunwtedcase)

=second‐smallesteigenvalueofLG ~d/mixing7meofrandomwalk

!2

neardforexpandersandrandomgraphs

1

2

!2

d! !G

d!

!2!2

d

Cheeger’sInequalityandtheConjugateGradient

Cheeger’sinequality(degree‐dunwtedcase)

=second‐smallesteigenvalueofLG ~d/mixing7meofrandomwalk

!2

ConjugateGradientfinds∊ ‐approxsolu7ontoLG x = b

inmultsbyLGO(!d/!2 log "!1)

isops

1

2

!2

d! !G

d!

!2!2

d

O(dm!!1G log !!1)

Fastsolu7onoflinearequa7ons

ConjugateGradientfastwhenexpansionhigh.

Elimina7onfastwhenlowforGandallsubgraphs.



Planargraphs

Wantspeedofextremesinthemiddle




Planargraphs

Wantspeedofextremesinthemiddle

Notallgraphsfitintothesecategories!


Precondi7onedConjugateGradient

SolveLG x = bby

Approxima7ngLGbyLH (theprecondi7oner)

Ineachitera7onsolveasysteminLHmul7plyavectorbyLG

∊ ‐approxsolu7onaserO(

!!(LG, LH) log "!1) itera7ons

condi,on number/approx quality

Inequali7esandApproxima7on

if for all x, xTLHx ! xTLGxLH ! LG

Example:ifHisasubgraphofG

xTLGx =!

(u,v)!E

w(u,v) (x (u)! x (v))2


!(LG, LH) ! t LH ! LG ! tLHif


CallsuchanHat‐approxofG


!(LG, LH) ! t iff


CallsuchanHat‐approxofG

!c : cLH ! LG ! ctLH

Vaidya’sSubgraphPrecondi7oners

Precondi7onGbyasubgraphH

LH ! LG Justneedtoknowts.t. LG ! tLH

EasytoboundtifHisaspanningtree

And,easytosolveequa7onsinLH byelimina7on

H

ApproximateLaplacianSolvers

ConjugateGradient[Hestenes‘51,S7efel’52]

Vaidya‘90:AugmentedMST

Boman‐Hendrickson’01:UsingLow‐StretchSpanningTrees

S‐Teng’04:Spectralsparsifica7on

Kou7s‐Miller‐Peng‘11:Elegance

O(m logc n)

O(m log n)

TheStretchofSpanningTrees

Where

Boman‐Hendrickson‘01:

stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT


path‐len3

Where


stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT


path‐len5

Where


stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT


path‐len1

Where


stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT


Inweightedcase,measureresistancesofpaths

Where


stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT

49

FundamentalGraphicInequality

1 8

1 2 3

8 7

4 5

6

edge k times path of length k

With weights, corresponds to resistors in serial (Poincaré inequality)

1 2 3

8 7

4 5

6

2 3

7

4 5

6

50

WhenTisaSpanningTree

G T

EveryedgeofGnotinThasuniquepathinT

51

WhenTisaSpanningTree


Where


stT (G) =!

(u,v)!E

path-lengthT (u, v)

LG ! stT (G)LT

Low‐StretchSpanningTrees

(Alon‐Karp‐Peleg‐West’91)

(Elkin‐Emek‐S‐Teng’04,Abraham‐Bartal‐Neiman’08)

ForeveryGthereisaTwith

where m = |E|

Solvelinearsystemsin7me O(m3/2 logm)

stT (G) ! m1+o(1)

stT (G) ! O(m logm log2 logm)

SpectralSparsifica7on[S‐Teng‘04]

ApproximateGbyasparseHwith

!(LG, LH) ! 1 + "

CutSparsifica7on[Benczur‐Karger‘96]

S S

ApproximateGbyasparseH,approximatelypreservingallcuts

Sparsifica7on

Goal:findsparseapproxima7onforeveryG

S‐Teng‘04:ForeveryGisanHwithO(n log7 n/!2) edgesand!(LG, LH) ! 1 + "

Sparsifica7on



S‐Srivastava‘08:withedgesbyrandomsamplingbyeffec7veresistances

O(n log n/!2)

0V

0.53V

0.27V

0.33V0.2V

1V

0V

u

v1/(currentflowatonevolt)

Sparsifica7on



S‐Srivastava‘08:withedges

Batson‐S‐Srivastava‘09

determinis7c,poly7me,andedges

O(n log n/!2)

O(n/!2)

Ultra‐Sparsifiers[S‐Teng]

ApproximateG byatreeplusedges

Sparsifiers Low‐StretchTrees

n/ log2 n

LH ! LG ! c log2 n LH

CholeskyfactortosmallersystemEliminatedegree1and2nodes









Getsystemofsize,solverecursively[Joshi‘97,Reif‘98,S‐Teng’04‘09]

O(n/ log2 n)

Ultra‐Sparsifiers

SolvesystemsinH by:1.Choleskyelimina7ngdegree1and2nodes

2.recursivelysolvingreducedsystem

Time

O(m logc m)

Kou7s‐Miller‐Peng‘11

Solvein7me O(m log n log2 log n log(1/!))

BuildUltra‐Sparsifierby:1.Construc7nglow‐stretchspanningtree2.Addingotheredgeswithprobability

pu,v ! path-lengthT (u, v)

CodebyYiannisKou7s

GivenvertexofinterestfindnearbyclusterSwithsmallexpansion*in7meO(|S|)

LocalGraphClustering[S‐Teng‘04]

*Actually,useconductance.Countver7cesbydegree.

Prove:GivenasetSofsmallexpansionandarandomvertexvofSprobablyfindasetTofsmallexpansion mostofTinsideS

in7meO(|T|)


S


in7meO(|T|)


S


in7meO(|T|)


S


in7meO(|T|)


S


in7meO(|T|)


Sv


in7meO(|T|)


Sv

T

UsingApproximatePersonalPageRankVectors

Jeh‐Widom‘03,Berkhin‘06,Andersen‐Chung‐Lang’06

Spillingpaintinagraph:startatonenodeateachstep,αfrac7ondriesofwetpaint,halfstaysput,halftoneighbors




1 0 0

dry

wet




.66 0 0

dry

wet

(α=1/3)

.33




.66 0 0

dry

wet

(α=1/3)

.33




.33 .33 0

dry

wet

(α=1/3)

.33




.22 .22 0

dry

wet

(α=1/3)

.44 .11




dry

wet

(α=1/3)

.44 .11

.22 .22 0




.17 .22 .06

dry

wet

(α=1/3)

.44 .11



Spillingpaintinagraph:

1 0 0

dry

wet

Timedoesn’tmaeer,canpushasynchronously

Approximate:onlypushwhenalotofpaint




1 0 0

dry

wet








.33 .33 0

dry

wet

.33




.33 .33 0

dry

wet

.33






.11 .06

dry

wet

.33 .11



.39




.11 .06

dry

wet

.33 .11



.39




.24 .06

dry

wet

.46 .11



.13

Volume‐BiasedEvolvingSetMarkovChain

[Andersen‐Peres‘09]

Walkonsetsofver7cesstartsatonevertex,endsatV

Dualtorandomwalkongraph

Whenstartinsidesetofconductancefindsetofconductance!1/2 log1/2 n

withwork |S| logc n/!1/2

Volume‐BiasedEvolvingSetMarkovChain

[Andersen‐Peres‘09]

Walkonsetsofver7cesstartsatonevertex,endsatV

Dualtorandomwalkongraph

Whenstartinsidesetofconductancefindsetofconductance!1/2 log1/2 n

withwork |S| logc n/!1/2

can we eliminate this?

OpenProblems

FasterandbeeerLow‐StretchSpanningTrees.

Fasterhigh‐qualitysparsifica7on.

Fasterlocalclusteringandgraphdecomposi7on.

Otherfamiliesoflinearsystems.

Conclusions

LaplacianSolversareapowerfulprimi7ve!FasterMaxflow:Chris7ano‐Kelner‐Madry‐S‐Teng

FasterRandomSpanningTrees:Kelner‐Madry‐Propp

AllEffec7veResistances:S‐Srivastava

Maybewecansolveallwell‐condi7onedgraphproblemsinnearly‐linear7me.

Don’tfearlargeconstants

Documents

Algorithms, Graph Theory, and the Soluon of …mcgrew/media/Events/distinguished/...the Soluon of Laplacian Linear Equaons Daniel A. Spielman Yale University Rutgers, Dec 6, 2011 Outline