Upload
alexa-mclean
View
215
Download
1
Embed Size (px)
Citation preview
Giansalvo EXIN CirrincioneGiansalvo EXIN Cirrincione
unité #5unité #5
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
021 n valeurs singulièresvaleurs singulières
aam1m1
aamnmn
aa1111 aa1n1n
UU
00
00
00 nn
11 00
VV
=
Full SVDFull SVDHA U V
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
aam1m1
aamnmn
aa1111 aa1n1n
UU
00
00
00 nn
11 00
VV
=
Reduced SVDReduced SVDˆ ˆ HA U V aam1m1
aamnmn
aa1111 aa1n1n
UU 00 nn
11 00
VV
= ^̂
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
Full SVDFull SVD
unitaire
unitaire
diagona le
m m
n n
m n
H
U
A U V
V
Reduced SVDReduced SVD
colonnes orthonormales
un
ˆ
ˆ
itaire
diagon
ˆ
al
ˆ
e
m n
n n
m
H
n
A U
V
V
U
ˆ ˆˆˆ ˆˆ ˆ ˆ ˆH
HH H 2H HH HA U V U U U V VVA V V
ˆ ˆˆ ˆ ˆ
ˆ ˆ ˆˆ ˆ ˆˆ
H11H H H 2 H H
H H H2 1 H
A A A A
V V
U V V V U
V
P
UU
V
UVU
ˆ ˆ HP UU
Approximation au sens des moindres carrées
x
2
Given , , ,
find such that is minimized.
m n m
n
A m n b
x b Ax
Example: polynomial data fittingExample: polynomial data fitting
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8-4
-3
-2
-1
0
1
2
f(x)
xi
yi
1,données : ,i i i m
x y
1
1
2
1
1
1
approximation
mi
( )
équations et inconnues ( ) approximation au s
interpolation
( )
ens
n (
) m
i ii
nj
nj
j
i i j i i
j
j
k
n m
f x y
f x x
n n
n m
f x y x y
des moindres carrés
Approximation au sens des moindres carrées
discrete square wavediscrete square wave
interpolationinterpolation
m m = = n n = = 1111
least squaresleast squares
m m = = 11 11 , , nn = = 88
Approximation au sens des moindres carrées
Posons le problème matriciellement
(1) ( 1) ( 1)1 2 1 1 1 1
(1) ( 1) ( 1)1 2 2 2 2 2
(1) ( 1) ( 1)1 2
1 2
... ...
... ...
... ...
j nj n
j nj n
j ni j i n i i
x x x f x
x x x f x
x x x f x
x
(1) ( 1) ( 1)... ...j nm j m n m mx x f x
1
1
1,
( )
pour ,
kj
jj
i i i m
f x x
x y
(1) ( 1) ( 1)11 1 1
(1) ( 1) ( 1)22 2 2
(1) ( 1) ( 1)
(1) ( 1) ( 1)
(1)
1
1) )
2
( ( 1
1 ... ...
1 ... ...
1 ... ...
1 ... ...
1
j n
j n
j nii i i
j nn n n
j nm m m
i
n
f xx x x
f xx x x
f xx x x
f xx x x
x x x
a
n
mf x
X f
Approximation au sens des moindres carrées
2
1
1
2
1
1 1
1
1 1
1 1 1
* argmin ( )
min ( ) avec ( )
principe : 1,...,
2 0
( *)
n
0
mi ( )m
i ii
mj
i ii
m nj
j i ii j
m
j
nj
i i iij
J J x yf x y
x xx y x
JnJ j
J
1
1 1
n mj
i ii
y x
système linéaire de n équations et n inconnues
11 1 1
1
1
0
1
1
1
( )
1
1
kj
i i i j i ij
n
ni i i
nn n n
nm m m
e x x
e x x
e x x
e x x
e f x y x y
e X y
1
1
2
2 2
( ) 2
( )
( ) 0
i
nn
T
m
T T
y
y
y
X X X
y
J e X y
J X X y yJ
Approximation au sens des moindres carrées
erreur erreur d’approximationd’approximation
Matrice de Vandermonde (1735-1796)
Approximation au sens des moindres carrées
2 2
1min
( )
2
min ( )
'( ) 2 '( ) 0
22
T
T T T T T
T
T T
J e X y
X y X y
X X X y y y
J
J X X y J
G h
G h
T TX X X y
Équations normalesÉquations normales
forme quadratiqueforme quadratique
x
2
Given , , ,
find such that is minimi
zed.
m n m
n
A m n
Ax b
b
x b Ax
find the closest point in range to so that
the norm of the residual (residu) is minimized
mAx A b
r b Ax
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
range(range(AA)) range nA
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
y = A xy = A x
r = b - A xb
b m
ranger A
1
1 1
H H
H H H H
P A A A A
Pb A A A A b A A A A Ax Ax
Pb Ax
0
normal equations
H
H H
A r
A Ax A b
The system is nonsingular iff A has full rank.
= Pb= Pb
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
1
1 1
H H
H H H H
P A A A A
Pb A A A A b A A A A Ax Ax
Pb Ax
0
normal equations
H
H H
A r
A Ax A b
The system is nonsingular iff A has full rank.
1
x1
pseudoinverse of
H H
H H
n m
x A bA A A b
A A A A
A
Solution par les équations normales
factorisation de Cholesky
AHA est une matrice n x n hermitienne strictement définie positive
1. Form the matrix AAHHAA and the vector AAH H bb2. Compute the Cholesky factorization AAHHAA = = RRHHRR3. Solve the lower-triangular system RRHH w w = = AAH H bb for w4. Solve the upper-triangular system R xR x = = ww for x
3
2
3
2
computation of flops
computation
WORK
o
f
: fl
f
o
lop3
s
s
p3
H
H
A A mn
nR R
nmn
Solution par la factorisation QR (Householder)
2 32WORK : flops
32 nmn
x xˆ ˆˆ ˆ , , m n n nA QR Q R reduced reduced QR factorizationQR factorization
1
ˆ ˆ
ˆ ˆ
ˆˆˆ ˆ
ˆ ˆ
H
H
H H
P QQ
y Pb QQ b Ax Q
A
x
x Q
R
Q RR b
1. Compute the reduced QR factorization ˆ ˆA QR
2. Compute the vector ˆ HQ b
3. Solve the upper-triangular system for xˆˆ HRx Q b
Solution par la SVD
2 3WORK : 1 f o s2 1 l pnmn
1
ˆ ˆ
ˆ ˆ ˆ
ˆ ˆˆ
ˆ
ˆ
H
H H
H HH
P UU
y Pb UU b Ax U V x
V x U b A V U
1. Compute the reduced SVD ˆ ˆ HA U V
2. Compute the vector ˆ HU b
3. Solve the diagonal system for wˆˆ Hw U b
4. Set x Vw
For this cost is approximately the same as for QR factorization,
but for the SVD is more expensive
m n
m n
Comparison of algorithms
� speed : normal equations� standard : QR factorization� A close to singular : SVD
DrawbacksDrawbacks• normal equationsnormal equations : not always stable in the presence of rounding errorsnot always stable in the presence of rounding errors• QR factoriz.QR factoriz.: less-than-ideal stability properties if less-than-ideal stability properties if AA is close to singular is close to singular• SVDSVD : expensive for expensive for mm nn
Conditionnement et précision
rela
su
tive condition number
px
f x
xf x
p
su supx x
A x x Ax x
xA
A
A A
A x Ax
Ax x
x
x
x x
1
1
1
1
1
1
x b Ax
b x A b
A x A b
xA A A
b
bA A A
x
A A A
accuracy : relative erro
r
f x f x
f x
backward stability
accuracy of
a B
accurac
S algorithm
y
machine
f x f x
fxO
x
Conditionnement du problème des moindres carrées
range(range(AA))y = A xy = A x
r = b - A xb
= Pb= Pb
x A b y Pb ˆ ˆ ˆ ˆH HP QQ UU AA
Données : A , b Solutions : x , y
12 2 2
2
A A A
arccos 2
2
y
b
closeness of the fit
2 2 2 2
2 2
A x A x
y Ax 2 2 22 2 2 2
2 2 2
A AA AA xA
Ax Ax Ax
b Ax
0 11 A2A
Conditionnement du problème des moindres carrées
Données : A , b Solutions : x , y
12 2 2
2
A A A
arccos 2
2
y
b
2 2 2 2
2 2
A x A x
y Ax
0 11 A2A
2-norm relative condition numbers
exact for certain exact for certain b b
upper boundsupper bounds
Stabilité des méthodes des moindres carrées
exemple
Least squares fitting of the function exp(sin(4)) on the interval [0,1] by a polynomial of degree 14
x15 = 1
highly ill-conditioned basis
very close fit
16machine 10
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
The rounding errors have been amplified by a factor of order 10 9. This inaccuracy is explained by ill-conditioning, not instability.
reducedreduced
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
implicit calculation of the product QH b
explicit formingQ
same accuracy
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
implicit calculation of the product QH b
explicit formingQ
same accuracy\ QR factorization
with column pivoting
ˆ ˆAP
b
R
A
Q
an order of magn. more acc.
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
backward stable backward stable
This is true whether is computed via
ˆformation of or . It also holds for
House
expli
holde
mi
cit
implicitly
column pivot
r triangulari
n
za
in
tion with arbitr
.g
a y
ˆ
r
machine
H
AA A x
Q b
A
Q
b O
Stabilité des méthodes des moindres carrées
exemple
SVD
16machine 10
backward stable backward stable
min machine
AA A x b O
A
It beats Householder triangularization with column It beats Householder triangularization with column pivoting ( pivoting ( MATLAB's \MATLAB's \ ) by a factor of about ) by a factor of about 33
Stabilité des méthodes des moindres carrées
exemple
équations normales
16machine 10
unstable unstable
not even a single digit of accuracynot even a single digit of accuracy
factorisation de Cholesky
Stabilité des méthodes des moindres carrées
BS least squares algorithmBS least squares algorithm
2 tanmachine
x xO
x
suppose ill-conditioned, i.e. 1, and is bounded away from 2A
and
ta 1
n
or
tan
0
2machineO machineO
The condition The condition number of the LS number of the LS problem may lie problem may lie anywhere in the anywhere in the range range to to 22 . .
Stabilité des méthodes des moindres carrées
BS least squares algorithmBS least squares algorithm Cholesky factorization (BS)Cholesky factorization (BS)
2 tanmachine
x xO
x
2machine
x xO
x
suppose ill-conditioned, i.e. 1, and is bounded away from 2A
and
ta 1
n
or
tan
0
2machineO machineO 2
machineO 2machineO
2cond HA A
The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.
Stabilité des méthodes des moindres carrées
The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.
The solution of the full-rank least squares problem via the normal equations is unstable. Stability can be achieved, however, by restriction to a class of problems in which (A) is uniformly bounded above or (tan)/ is uniformly bounded below.
FINEFINE