Vous êtes sur la page 1sur 20

MATH 3795

Lecture 9. Linear Least Squares. Using SVD

Decomposition.
Dmitriy Leykekhman

Fall 2008

Goals
I

SVD-decomposition.

Linear Least Squares

SVD Decomposition.
For any matrix A Rmn there exist orthogonal matrices U Rmm ,
V Rnn and a diagonal matrix Rmn , i.e.,

1
0 ... 0

..

r
for m n
=

..

.
0

...

with diagonal entries

1 r > r+1 = = min {m,n} = 0
such that A = U V T
D. Leykekhman - MATH 3795 Introduction to Computational Mathematics

Linear Least Squares

SVD Decomposition.
For any matrix A Rmn there exist orthogonal matrices U Rmm ,
V Rnn and a diagonal matrix Rmn , i.e.,

..

=
for m n
..

.
..
..
0
0
with diagonal entries
1 r > r+1 = = min {m,n} = 0
such that A = U V T
D. Leykekhman - MATH 3795 Introduction to Computational Mathematics

Linear Least Squares

SVD Decomposition.
I

The decomposition
A = U V T
is called Singular Value Decomposition (SVD). It is very
important decomposition of a matrix and tells us a lot about its
structure.

The diagonal entries i of are called the singular values of A. The

columns of U are called left singular vectors and the columns of V
are called right singular vectors.

Linear Least Squares

SVD Decomposition.
I

The decomposition
A = U V T
is called Singular Value Decomposition (SVD). It is very
important decomposition of a matrix and tells us a lot about its
structure.

The diagonal entries i of are called the singular values of A. The

columns of U are called left singular vectors and the columns of V
are called right singular vectors.

Using the orthogonality of V we can write it in the form

AV = U
We can interpret it as follows: there exists a special orthonormal set
of vectors (i.e. the columns of V ), that is mapped by the matrix A
into an orthonormal set of vectors (i.e. the columns of U ).

Applications of SVD Decomposition.

Given the SVD-Decomposition of A,
A = U V T
with
1 r > r+1 = = min {m,n} = 0
one may conclude the following:
I

rank(A) = r,

Applications of SVD Decomposition.

Moreover if we denote
Ur = [u1 , . . . , ur ],

r = diag(1 , . . . , r ),

then we have
A = Ur r VrT =

r
X

Vr = [v1 , . . . , vr ],

i ui viT

i=1

This is called the dyadic decomposition of A, decomposes the matrix A

of rank r into sum of r matrices of rank 1.

I

The 2-norm and the Frobenius norm of A can be easily computed

from the SVD decomposition
kAxk2
= 1
x6=0 kxk2
m X
n
q
X
kAkF =
a2ij = 12 + + p2 ,
kAk2 = sup

i=1 j=1
I

From the SVD decomposition of A it also follows that

AT A = V T V T

and AAT = U T U T .

Thus, i2 , i = 1, . . . , p are the eigenvalues of symmetric matrices

AT A and AAT and vi and ui are the corresponding eigenvectors.

Applications of SVD Decomposition.

Theorem
Let the SVD of A Rmn be given by
A = Ur r VrT =

r
X

i ui viT

i=1

with r = rank(A). If k < r

Ak =

k
X

i ui viT ,

i=1

then
min
rank(D)=k

kA Dk2 = kA Ak k2 = k+1 ,

and
min
rank(D)=k

v
u p
uX
kA DkF = kA Ak kF = t
i2 ,

p = min {m, n}.

k+1
Linear Least Squares

min kAx bk22
x

Using the orthogonality of U and V we have

T
2
T
kAx bk22 = kU T (AV V T x b)k22 = k V
| {zx} U b)k2
=z

r
X
i=1

m
X

(i zi uTi b)2 +

(uTi b)2 .

i=r+1

I

Thus,
min kAx bk22 =
x

r
X
i=1

m
X

(i zi uTi b)2 +

(uTi b)2 .

i=r+1

I

Thus,
min kAx bk22 =
x

r
X

m
X

(i zi uTi b)2 +

i=1

i=r+1

The solution is given

uTi b
, i = 1, . . . , r,
i
zi = arbitrary, i = r + 1, . . . , n.
zi =

(uTi b)2 .

I

Thus,
min kAx bk22 =
x

r
X

m
X

(i zi uTi b)2 +

i=1

i=r+1

The solution is given

uTi b
, i = 1, . . . , r,
i
zi = arbitrary, i = r + 1, . . . , n.
zi =

As a result
min kAx bk22 =
x

(uTi b)2 .

m
X

(uTi b)2 .

i=r+1

Recall that z = V T x. Since V is orthogonal, we find that

kxk2 = kV V T xk2 = kV T xk2 = kzk2 .
All solutions of the linear least squares problem are given by z = V T x
with
uT b
zi = i , i = 1, . . . , r,
i
zi = arbitrary, i = r + 1, . . . , n.

10

Solving LLS with SVD Decomposition. Minimum norm

solution
The minimum norm solution of the linear least squares problem is given
by
x = V z ,
where z Rn is the vector with entries
zi =

uTi b
,
i

zi = 0,

i = 1, . . . , r,

i = r + 1, . . . , n.

x =

r
X
uT b
i

i=1

vi

11

% compute the SVD:

[U,S,V] = svd(A);
s
= diag(S);
% determine the effective rank r of A using singular values
r = 1;
while( r < size(A,2) & s(r+1) >= max(size(A))*eps*s(1) )
r = r+1;
end
d = U*b;
x = V* ( [d(1:r)./s(1:r); zeros(n-r,1) ] );

12

I

Suppose that the data b are

b = bex + b,
where b represents the measurement error.

x =

r
X
uT b
i

i=1

vi =

r  T
X
u b
i

i=1

uTi b
i

13


vi .

I

Suppose that the data b are

b = bex + b,
where b represents the measurement error.

x =

r
X
uT b
i

i=1
I

vi =

r  T
X
u b
i

i=1

uTi b
i

uT (b)


vi .

If a singular value i is small, then i i could be large, even if

uTi (b) is small. This shows that errors b in the data can be
magnified by small singular values i .

13

Conditioning of a Linear Least Squares Problem.

% Compute A
t = 10.^(0:-1:-10);
A = [ ones(size(t)) t t.^2 t.^3 t.^4 t.^5];
% compute SVD of A
[U,S,V] = svd(A); sigma = diag(S);
% compute exact data
xex = ones(6,1); bex = A*xex;
for i = 1:10
% data perturbation
deltab = 10^(-i)*(0.5-rand(size(bex))).*bex;
b = bex+deltab;
% solution of perturbed linear least squares problem
w = U*b;
x = V * (w(1:6) ./ sigma);
errx(i+1) = norm(x - xex); errb(i+1) = norm(deltab);
end
loglog(errb,errx,*);
ylabel(||x^{ex} - x||_2); xlabel(||\delta b||_2)
D. Leykekhman - MATH 3795 Introduction to Computational Mathematics

14

I

1 3.4
2 2.1
3 8.2 102

4 7.2 104
5 6.6 107
6 5.5 1011

We see that small perturbations b in the measurements can lead to

large errors in the solution x of the linear least squares problem if
the singular values of A are small.

15