View
18
Download
2
Category
Preview:
Citation preview
ELE 538B: Sparsity, Structure and Inference
Gaussian Graphical Models and Graphical Lasso
Yuxin Chen
Princeton University, Spring 2017
Multivariate Gaussians
Consider a random vector x ∼ N (0,Σ) with pdf
f(x) = 1(2π)p/2 det (Σ)1/2 exp
{−1
2x>Σ−1x
}∝ det (Θ)1/2 exp
{−1
2x>Θx
}where Σ = E[xx>] � 0 is covariance matrix, and Θ = Σ−1 is inversecovariance matrix / precision matrixGraphical lasso 5-2
Undirected graphical models
x1 ⊥⊥ x4 | {x2, x3, x5, x6, x7, x8}
• Represent a collection of variables x = [x1, · · · , xp]> by a vertexset V = {1, · · · , p}• Encode conditional independence by a set E of edges
◦ For any pair of vertices u and v,
(u, v) /∈ E ⇐⇒ xu ⊥⊥ xv | xV\{u,v}
Graphical lasso 5-3
Gaussian graphical models
Fact 5.1(Homework) Consider a Gaussian vector x ∼ N (0,Σ). For any u andv,
xu ⊥⊥ xv | xV\{u,v}
iff Θu,v = 0, where Θ = Σ−1.
conditional independence ⇐⇒ sparsity
Graphical lasso 5-4
Gaussian graphical models
∗ ∗ 0 0 ∗ 0 0 0∗ ∗ 0 0 0 ∗ ∗ 00 0 ∗ 0 ∗ 0 0 ∗0 0 0 ∗ 0 0 ∗ 0∗ 0 ∗ 0 ∗ 0 0 ∗0 ∗ 0 0 0 ∗ 0 00 ∗ 0 ∗ 0 0 ∗ 00 0 ∗ 0 ∗ 0 0 ∗
︸ ︷︷ ︸
Θ
Graphical lasso 5-5
Likelihoods for Gaussian models
Draw n i.i.d. samples x(1), · · · ,x(n) ∼ N (0,Σ), then log-likelihood(up to additive constant) is
` (Θ) = 1n
n∑i=1
log f(x(i)) = 12 log det (Θ)− 1
2n
n∑i=1x(i)>Θx(i)
= 12 log det (Θ)− 1
2 〈S,Θ〉 ,
where S := 1n
∑ni=1 x
(i)x(i)> is sample covariance; 〈S,Θ〉 = tr(SΘ)
Maximum likelihood estimation
maximizeΘ�0 log det (Θ)− 〈S,Θ〉
Graphical lasso 5-6
Challenge in high-dimensional regime
Classical theory says MLE coverges to the truth as sample size n→∞
Practically, we are often in the regime where sample size n is small(n < p)• In this regime, S is rank-deficient, and MLE does not even exist
Graphical lasso 5-7
Graphical lasso (Friedman, Hastie, &Tibshirani ’08)
Many pairs of variables are conditionally independent⇐⇒ many missing links in the graphical model (sparsity)
Key idea: apply lasso by treating each node as a response variable
maximizeΘ�0 log det (Θ)− 〈S,Θ〉 − λ‖Θ‖1︸ ︷︷ ︸lasso penalty
• It is a convex program! (homework)• First-order optimality condition
Θ−1 − S − λ ∂‖Θ‖1︸ ︷︷ ︸subgradient
= 0 (5.1)
=⇒ (Θ−1)i,i = Si,i + λ, 1 ≤ i ≤ p
Graphical lasso 5-8
Blockwise coordinate descentIdea: repeatedly cycle through all columns/rows and, in each step,optimize only a single column/row
Notation: use W to denote working version of Θ−1. Partition allmatrices into 1 column/row vs. the rest
Θ =[
Θ11 θ12θ>12 θ22
]S =
[S11 s12s>12 s22
]W =
[W11 w12w>12 w22
]Graphical lasso 5-9
Blockwise coordinate descent
Blockwise step: suppose we fix all but the last row / column. Itfollows from (5.1) that
0 ∈W11β − s12 − λ∂‖θ12‖1 = W11β − s12 + λ∂‖β‖1 (5.2)
where β = −θ12/θ̃22 (since[
Θ11 θ12θ>12 θ22
]−1=[
∗ − 1θ̃22
Θ−111 θ12
∗ ∗
]︸ ︷︷ ︸
matrix inverse formula
) with
θ̃22 = θ22 − θ>12Θ−111 θ12 > 0
This coincides with the optimality condition for
minimizeβ12∥∥W 1/2
11 β −W−1/211 s12
∥∥22 + λ‖β‖1 (5.3)
Graphical lasso 5-10
Blockwise coordinate descent
Algorithm 5.1 Block coordinate descent for graphical lassoInitialize W = S + λI and fix its diagonals {wi,i}.Repeat until covergence:
for t = 1, · · · p:(i) Partition W (resp. S) into 4 parts, where the upper-left part
consists of all but the jth row / column(ii) Solve
minimizeβ12‖W
1/211 β −W
−1/211 s12‖2 + λ‖β‖1
(iii) Update w12 = W11β
Set θ̂12 = −θ̂22β with θ̂22 = 1/(w22 −w>12β)
Graphical lasso 5-11
Blockwise coordinate descent
The only remaining thing is to ensure W � 0. This is automaticallysatisfied:
Lemma 5.2 (Mazumder & Hastie, ’12)
If we start with W � 0 satisfying ‖W − S‖∞ ≤ λ, then everyrow/column update maintains positive definiteness of W .
• If we start with W (0) = S + λI, then W (t) will always bepositive definite
Graphical lasso 5-12
Proof of Lemma 5.2
A key observation for the proof of Lemma 5.2
Fact 5.3 (Lemma 2, Mazumder & Hastie, ’12)
Solving (5.3) is equivalent to solving
minimizeγ (s12 + γ)>W−111 (s12 + γ) s.t. ‖γ‖∞ ≤ λ (5.4)
where solutions to 2 problems are related by β̂ = W−111 (s12 + γ̂)
• Check that optimality condition of (5.3) and that of (5.4) match
Graphical lasso 5-13
Proof of Lemma 5.2Suppose in tth iteration one has ‖W (t) − S‖∞ ≤ λ and
W (t) � 0
⇐⇒ W(t)11 � 0; w22−w(t)>
12
(W
(t)11
)−1w
(t)12 > 0 (Schur complement)
We only update w12, so it suffices to show
w22 −w(t+1)>12
(W
(t)11
)−1w
(t+1)12 > 0 (5.5)
Recall that w(t+1)12 = W t
11βt+1. It follows from Fact 5.3 that and
‖w(t+1)12 − s12‖∞ ≤ λ;
w(t+1)>12
(W
(t)11)−1w
(t+1)12 ≤ w(t)>
12(W
(t)11)−1w
(t)12 .
Since w22 = s22 + λ remains unchanged, we establish (5.5).Graphical lasso 5-14
Reference
[1] ”Sparse inverse covariance estimation with the graphical lasso,”J. Friedman, T. Hastie, and R. Tibshirani, Biostatistics, 2008.
[2] ”The graphical lasso: new insights and alternatives,” R. Mazumder andT. Hastie, Electronic journal of statistics, 2012.
[3] ”Statistical learning with sparsity: the Lasso and generalizations,”T. Hastie, R. Tibshirani, and M. Wainwright, 2015.
Graphical lasso 5-15
Recommended