Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
The Computational Complexity ofSpark, RIP, and NSP
Andreas M. TillmannResearch Group Optimization, TU Darmstadt, Germany
joint work with Marc E. Pfetsch
SPARS 201307/08 – 07/11/2013, Lausanne, Switzerland
07/08/2013 | TU Darmstadt | A. M. Tillmann | 1
Sparse Recovery Conditions
. min{ ‖x‖0 : Ax = b } is NP-hard (also with constraint ‖Ax − b‖2 ≤ ε)
. various conditions for k -sparse solution uniqueness and recoverability byheuristics such as OMP or `1-minimization
Complexity Rumors ...
Spark, RIP, and NSP are very often mentioned to be intractable / NP-hard,but apparently no proof or reference anywhere in CS literature!
. In particular, hardness often “explained” solely by “combinatorial nature”
(this reasoning is false – many combinatorial problems are in P)
07/08/2013 | TU Darmstadt | A. M. Tillmann | 2
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 3
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 4
P, NP, and coNP – Hardness and completeness(informally...)
. P : deterministic-polynomial-time solvable (decision) problems
. NP : nondeterministic-polynomial-time solvable (decision) problemsI polynomial certificate for “yes” answers, but no poly.-time solution algorithm
(unless P=NP)
. coNP : complementary class of NPI polynomial certificate for “no” answers, but no poly.-time solution algorithm
(unless P=NP)
. coNP-hard ≡ NP-hard : (decision or optimization) problems for whichexistence of a polynomial solution algorithm would imply P=NP
. (co)NP-complete : NP-hard (decision) problems contained in (co)NP
07/08/2013 | TU Darmstadt | A. M. Tillmann | 5
Examples
. LINEAR PROGRAMMING ∈ P.
. A classical NP-complete problem: the k -CLIQUE problemGiven a graph G and a positive integer k , does G contain a clique of size k?
Example: 4-clique {3, 4, 5, 7}.
1
8
6
2
7
3
4
5
7
3
4
5
07/08/2013 | TU Darmstadt | A. M. Tillmann | 6
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 7
The Spark of a Matrix
Definitionspark(A) := min ‖x‖0 s.t. Ax = 0, x 6= 0
. why care?I unique k -sparse `0-solution if and only if k < spark(A)/2
. a.k.a. girth of the vector matroidM(A) on A:
spark(A) = min{ |C| : C circuit ofM(A) },
circuit: inclusion-wise minimal collection of linearly dependent columns
. for graphic matroids: polynomial time; for transversal matroids: NP-hard
07/08/2013 | TU Darmstadt | A. M. Tillmann | 8
Spark Complexity – An overlooked early result
. Khachiyan, 1995:
Given A ∈ Qm×n, it is NP-complete to decide whether Ahas an (m ×m)-submatrix with zero determinant.
mIt is NP-complete to decide whether spark(A) ≤ m.
. Observation: “Is A full-spark?” (“spark(A) = m + 1?”) is coNP-complete.
(previously only known to be “hard for NP under randomized reductions”,based on probabilistic matrix representation of transversal matroids [Alexeevet al.])
07/08/2013 | TU Darmstadt | A. M. Tillmann | 9
Spark Complexity – New Result
Theorem 1 (T. & Pfetsch)
Given a matrix A ∈ Qm×n (with rank(A) = m < n) and a positive integer k < m,it is NP-complete to decide whether spark(A) ≤ k (or spark(A) = k ).
. Difference to Khachiyan’s result: k < m with full (row)-rank A(Khachiyan’s proof extends to k < m only by appending zero-rows)
Corollary
Given a matrix A, computing spark(A) is NP-hard.
(Polyn. algo. to compute spark(A) could decide “spark(A) ≤ k?” in poly-time. �)
07/08/2013 | TU Darmstadt | A. M. Tillmann | 10
Proof Sketch for Theorem 1
Reduction from k -CLIQUE:
. given instance: G = (V , E) and k ∈ N (wlog k > 4), with n := |V | and m := |E |
. construct a matrix A of size (n +(k
2
)− k − 1)×m
I first n rows: set aie = 1 iff i ∈ e, and 0 else (incidence matrix of G)
I remaining rows (n + i for i = 1, ... ,(k
2
)− k − 1): set a(n+i)e = (U + i + 1)e−1
(sub-Vandermonde matrix)
. G has a k -clique if and only if spark(A) ≤(k
2
)(in fact, spark(A) =
(k2
)).
I a specific choice of U [cf. Chistov et al.] and some technical auxiliary results ongraphs and incidence matrices yield the desired linear (in)dependency properties.
. containment in NP: “guess” x with Ax = 0 (⇒ can assume x ∈ Qn);can verify Ax = 0, ‖x‖0 =
(k2
), and that supp(x) is a circuit in poly-time.
07/08/2013 | TU Darmstadt | A. M. Tillmann | 11
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 12
The Restricted Isometry Property (RIP)
DefinitionA matrix A ∈ Rm×n satisfies the RIP of order k with constant δk if
(1− δk )‖x‖22 ≤ ‖Ax‖2
2 ≤ (1 + δk )‖x‖22 ∀ x : ‖x‖0 ≤ k . (k , δk )-RIP
Restricted Isometry Constant (RIC): δk := min{ δk : A satisfies (k , δk )-RIP }
why care?
. `o-`1-equivalence for k -sparse solutions if δ2k <√
2− 1 [Candès, 2008],or if δk < 0.307 [Cai, Wang & Xu, 2010], ...
. certain random matrices have desirable RIP with high probability
07/08/2013 | TU Darmstadt | A. M. Tillmann | 13
Central RIP-related Complexity Issues
. RIC computation: Is it hard to compute the RIC δk (given A and k )?
. RIP certification: Is it hard to decide whether δk < δ (given A, k , δ)?
07/08/2013 | TU Darmstadt | A. M. Tillmann | 14
Complexity of RIP Certification
Theorem 2 (RIP Certification I) (T. & Pfetsch)
Given a matrix A ∈ Qm×n and a positive integer k , deciding whether there existssome constant δk < 1 such that A satisfies the (k , δk )-RIP is coNP-complete.
Theorem 3 (RIP Certification II) (T. & Pfetsch)∗
Given a matrix A ∈ Qm×n, a positive integer k , and some constant δk ∈ (0, 1),deciding whether A satisfies the (k , δk )-RIP is (co)NP-hard.
∗ independently obtained by [Bandeira et al.], using Khachiyan’s spark result (i.e., k = m).
07/08/2013 | TU Darmstadt | A. M. Tillmann | 15
Complexity of RIC Computation
Corollary
Computation of the RIC δk is NP-hard.
Proof: A polynomial algorithm to compute the RIC could be used to decideRIP CERTIFICATION (I or II) in polynomial time. �
07/08/2013 | TU Darmstadt | A. M. Tillmann | 16
A Useful Lemma
Lemma 1Let A = (aij ) ∈ Qm×n and define α := max |aij |, C := 2dlog2(α
√mn)e, and A := 1
C A.Then
‖A x‖22 ≤ (1 + δ)‖x‖2
2 for all x ∈ Rn and δ ≥ 0.
Why useful?
(k , δk )-RIP for A reduces to “(1− δk )‖x‖22 ≤ ‖Ax‖2
2 ∀ k -sparse x”, i.e.,only the lower RIP inequality is relevant!
07/08/2013 | TU Darmstadt | A. M. Tillmann | 17
Proof of Theorem 2 (RIP CERTIFICATION I)
(“(k, δk)-RIP for some δk < 1”?)
Reduction from SPARK (“spark(A) ≤ k?”):
. instance for RIP-problem: A = 1C A, k (noteM(A) =M(A))
. If spark(A) ≤ k , there exists k -sparse x 6= 0 with Ax = 0. Then
(1− δk )‖x‖22 ≤ ‖Ax‖2
2 = 0 ⇒ δk ≥ 1.
. Conversely, suppose there is no δk < 1 s.t. A is (k , δk )-RIP. Then
∃ x with 1 ≤ ‖x‖0 ≤ k s.t. 0 ≥ (1− δk )‖x‖22 = ‖Ax‖2
2 ≥ 0,
hence Ax = 0. ⇒ ∃ circuit (⊆supp(x)) of size at most k , thus spark(A) ≤ k .
. RIP CERTIFICATION I ∈ coNP: certificate is x with 1 ≤ ‖x‖0 ≤ k whichtigthly satisfies the (k , 1)-RIP; implies Ax = 0 (so can assume x ∈ Qn). �
07/08/2013 | TU Darmstadt | A. M. Tillmann | 18
Another Useful Lemma
Lemma 2Given a matrix A ∈ Qm×n and a positive integer k ≤ n, if spark(A) > k , thereexists a rational constant ε > 0 such that
‖Ax‖22 ≥ ε ‖x‖2
2 for all x with 1 ≤ ‖x‖0 ≤ k .
Why useful?
reveals a “rationality gap”: δk < 1 ⇔ δk ≤ 1− ε
07/08/2013 | TU Darmstadt | A. M. Tillmann | 19
Complexity of RIP CERTIFICATION II
(“(k, δk)-RIP with δk ≤ δ for given δ ∈ (0, 1)”?)
Proof Sketch:
. essentially extend the proof of Theorem 2 by means of previous Lemma:
spark(A) ≤ k ⇔ A not (k , δk )-RIP with some δk < 1 (Theorem 2)
⇔ A not (k , 1− ε)-RIP (Theorem 3)
Remark: Containment in coNP not known.
(rationality of the certificate x is not obvious, since no longer Ax = 0)
07/08/2013 | TU Darmstadt | A. M. Tillmann | 20
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 21
The Nullspace Property (NSP)
DefinitionA matrix A ∈ Rm×n satisfies the NSP of order k with constant αk if
‖x‖k ,1 := maxS:|S|=k
∑i∈S
|xi | ≤ αk‖x‖1 ∀ x : Ax = 0. (k ,αk )-NSP
Nullspace Constant (NSC): αk := min{ αk : A satisfies (k ,αk )-NSP }
why care?
. `0-`1-equivalence if and only if αk < 1/2
07/08/2013 | TU Darmstadt | A. M. Tillmann | 22
Complexity of the NSP
Theorem 4 (T. & Pfetsch)
Given a matrix A ∈ Qm×n and a positive integer k , deciding whether there existssome constant αk < 1 such that A satisfies the (k ,αk )-NSP is coNP-complete.
Corollary
Computation of the NSC αk is NP-hard.
07/08/2013 | TU Darmstadt | A. M. Tillmann | 23
Proof of Theorem 4
Reduction from SPARK (“spark(A) ≤ k?”):
. instance for NSP-decision problem: A, k
. If spark(A) ≤ k , there exists x with Ax = 0 and 1 ≤ ‖x‖0 ≤ k . Then,‖x‖k ,1 = ‖x‖1, and therefore αk ≥ 1 (in fact, αk = 1).
. Conversely, suppose there is no αk < 1 s.t. A satisfies the (k ,αk )-NSP.Then there is some x with Ax = 0 and 1 ≤ ‖x‖0 ≤ k such that ‖x‖k ,1 = ‖x‖1
(otherwise αk < 1 was possible).⇒ ∃ circuit (⊆supp(x)) of size at most k , whence spark(A) ≤ k .
. ∈coNP: αk ≤ 1 (trivially)⇒ “no”-certificate is a k -sparse x ∈ Qn s.t. Ax = 0and ‖x‖k ,1 ≤ αk‖x‖1 with αk = 1 is tightly satisfied, i.e., ‖x‖k ,1 = ‖x‖1 �
07/08/2013 | TU Darmstadt | A. M. Tillmann | 24
Outline
1 Computational Complexity Basics
2 Confirming the Intractability Rumors ...SparkRestricted Isometry Property (RIP)Nullspace Property (NSP)
3 Conclusions
07/08/2013 | TU Darmstadt | A. M. Tillmann | 25
Concluding Remarks
. Suspicions confirmed: Spark, RIP, and NSP are all NP-hard indeed
CLIQUE SPARK
Spark computation
RIP-CERT. I
RIP-CERT. II
RIC computation
NSP NSC computation
07/08/2013 | TU Darmstadt | A. M. Tillmann | 26
Concluding Remarks
. Suspicions confirmed: Spark, RIP, and NSP are all NP-hard indeed
. Existing approximation/relaxation algorithms well justified
. More work on exact algorithms desirableI NP-hardness means not all instances can be solved efficiently
– existence of practically efficient methods not necessarily excluded!
. Still open: Complexity of verifying (e.g.) δk < 0.307, αk < 1/2, ... ?Complexity of approximating δk or αk ?
. Details, and more results, in our paper
arXiv: 1205.2081 (new version v4!)
07/08/2013 | TU Darmstadt | A. M. Tillmann | 27