SVD and Applications
Olga Sorkine-Hornung
December 23, 2016
Dimensionality reduction
December 23, 2016
n
= m
σ1 σ2
σn
UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n
data vectors aj (reduced) SVD of A
Olga Sorkine-Hornung # 2
Dimensionality reduction
December 23, 2016
n
m
σ1 σ2
σn
UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n
data vectors aj (reduced) SVD of A
nullify the last (small) singular values!
=
Olga Sorkine-Hornung # 3
Dimensionality reduction
December 23, 2016
n
m
σ1 σk
σn
Uk A VkT Σk
m⇥ n
data vectors aj
m⇥ k k ⇥ k k ⇥ n
≈
Olga Sorkine-Hornung # 4
≈
Dimensionality reduction
December 23, 2016
n
m
σ1 σk
σn
Uk A VkT Σk
m⇥ n
data vectors aj
m⇥ k k ⇥ k k ⇥ n
basis for the reduced subspace
Olga Sorkine-Hornung # 5
≈
Dimensionality reduction
December 23, 2016
n
m
σ1 σk
σn
Uk A VkT Σk
m⇥ n
data vectors aj
m⇥ k k ⇥ k k ⇥ n
basis for the reduced subspace
coordinates of the projections
of the data vectors onto the
reduced subspace
Olga Sorkine-Hornung # 6
≈
Also called PCA, Principal Component Analysis
December 23, 2016
n
m
σ1 σk
σn
Uk A VkT Σk
m⇥ n
data vectors aj
m⇥ k k ⇥ k k ⇥ n
basis for the reduced subspace
coordinates of the projections
of the data vectors onto the
reduced subspace
Olga Sorkine-Hornung # 7
The space of human faces
December 23, 2016
x1 x2 x3
xm
……
..
Concatenate grayscale pixel values into data point a 2 Rm
Olga Sorkine-Hornung # 8
Database of faces ● Many 100x100 pixel face images ● Different people ● Different poses ● Different illumination ● … ● Each image is labeled with the
person’s name/identification
● Hypothesis: Faces live in a low-dimensional subspace of
December 23, 2016
R10,000
Olga Sorkine-Hornung # 9
PCA on face database: Eigenfaces
December 23, 2016
average face
aavg =1
n
nX
j=1
aj
Olga Sorkine-Hornung # 10
PCA on face database: Eigenfaces
December 23, 2016
average face
aavg =1
n
nX
j=1
aj
data vectors aj := aj � aavg
Olga Sorkine-Hornung # 11
PCA on face database: Eigenfaces
December 23, 2016
average face
n
m
Am⇥ n
aavg =1
n
nX
j=1
aj
data vectors aj := aj � aavg
Olga Sorkine-Hornung # 12
PCA on face database: Eigenfaces
December 23, 2016
average face
n
≈ m
σ1 σk
σn
Uk A VkT Σk
m⇥ n m⇥ k k ⇥ k k ⇥ naavg =
1
n
nX
j=1
aj
data vectors aj := aj � aavg
Olga Sorkine-Hornung # 13
PCA on face database: Eigenfaces
December 23, 2016
average face
n
≈ m
σ1 σk
σn
Uk A VkT Σk
m⇥ n m⇥ k k ⇥ k k ⇥ naavg =
1
n
nX
j=1
aj
data vectors aj := aj � aavg
Olga Sorkine-Hornung # 14
PCA on face database: Eigenfaces
December 23, 2016
average face
aavg =1
n
nX
j=1
aj
u1 u2 u3 u4
u5 u6 u7 u8
k=8 first principal components (columns of U)
Olga Sorkine-Hornung # 15
PCA on face database: Eigenfaces
December 23, 2016
u1 u2 u3 u4
u5 u6 u7 u8
k=8 first principal components (columns of U)
Coordinates of each projection of each aj onto the subspace of eigenfaces:
VkT Σk
k ⇥ n
Each face is now represented by k numbers!
Olga Sorkine-Hornung # 16
Face recognition
December 23, 2016
downsample to 100x100 = m pixels,
subtract aavg aproject onto subspace,
i.e. calculate coordinates w.r.t. uj
⇠1 = aTu1
⇠2 = aTu2
⇠8 = aTu8
...
m⇥ 1
Olga Sorkine-Hornung # 17
input image
Face recognition
December 23, 2016
≈ + ⇠2 + ⇠3 + ⇠4
+ ⇠5 + ⇠6 + ⇠7 + ⇠8
+ ⇠1
u1 u2 u3 u4
u5 u6 u7 u8
a aavg u1
Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of
˜aj in the database
Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest
Olga Sorkine-Hornung # 18
Face recognition
December 23, 2016
≈ + ⇠2 + ⇠3 + ⇠4
+ ⇠5 + ⇠6 + ⇠7 + ⇠8
+ ⇠1
u1 u2 u3 u4
u5 u6 u7 u8
a aavg u1
Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of
˜aj in the database
Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest
Olga Sorkine-Hornung # 19 most likely this face in the database is labeled as “Marc Pollefeys” J
Face recognition ● Searching for the nearest neighbor in a k-dimensional space is very
fast if k is small! And gives quite accurate results
● Searching for the most similar face in a m=10,000-dimensional space is practically infeasible
● Most 10,000-dimensional vectors do not represent a human face!
● This kind of dimensionality reduction is the basis of many various machine learning methods.
December 23, 2016 Olga Sorkine-Hornung # 20
Recommended