Matrix Computation Norms for Vectors and Matrices

Lecture Notes 3: Norms for Vectors and Matrices

Professor: Zhihua Zhang Scribe:Qi Yu, Yizhi Liu
1 Vector Norms
Denition 1.1 A vector norm on R
n
is a function f : R
m
R that satises the following properties
1. f(x) 0, x R
m
2. f(x) iff x =0
3. f(x+y) f(x) +f(y), x, y R
m
4. f(x) =||f(x), R, x R
m
We denote vector norm as: f(x) =||x||
Remarks: f(x+(1)y) f (x) +f((1)y) =f(x) +(1)f(y)
Thus vector norm is a convex function.
1.1 p-norm
Denition 1.2 ||x||
p
=(
m

i =1
|x
i
|
p
)
1
p
), p 1
example:
||x||
1
=
m

i =1
|x
i
|
||x||
2
=(
m

i =1
|x
i
|
2
)
1
2
||x||

= max
1i m
|x
i
| = lim
p
||x||
p
{||x||
1
1| |x
1
| +|x
2
| 1} {||x||
2
1| |x
1
|
2
+|x
2
|
2
1} {||x||

1| max|x
1
|, |x
2
| 1}
1.2 weighted norm
Denition 1.3 W=di ag(w
1
, , w
m
), w
i
=0
x
w
=||wx|| =(
n

i =1
|w
i
x
i
|
p
)
1
p
3 - 1
1.3 zero-norm
||x||
0
= number of nonzero of x
However, ||x||
0
is not a vector norm. Consider x = (1, 1)
T
, = 5, it is obvious that zero-norm
contradicts the forth property of vector norm.
1.4 Cauchy-Schantz inequality
<x, y >=
n

i =1
x
i
y
i
| <x, y >|

<x, x ><y, y >=||x||||y||

2
proof: <x+t y ><x+t y >0
<y, y >t
2
+2 <x, y >t +<x, x >0
=4 <x, y >
2
4 <y, y ><x, x >0
| <x, y >|

<x, x ><y, y >

1.5 Holder inequality
<x, y > ||x||
p
||y||
q
(
1
p
+
1
q
=1)
when p =1, q =, the inequality still holds.
Proposition 1.1 p-norm propositions
||x||
2
||x||
1

n||x||
2
||x||

||x||
2

n||x||

||x||

||x||
1
n||x||

proof:
||x||
2
||x||
1
is self-evident. For ||x||
1

n||x||
2
, let
z
i
=
_
1 i f x
i
0
1 i f x
i
<0
, then ||x||
1
=
m

i =1
z
i
x
i
. Using Cauchy-Schantz inequality, we have | <x, z >| ||x||
2
||z||
2
=

n||x||
2
Theorem1.1 there are constants C
1
, C
2
<0, such that for any x, C
1
||x||

||x||

C
2
||x||

2 convergence
Denition 2.1 We say that a sequence {x
(k)
} of m-vectors converges to x if lim
k
||x
(k)
x|| = 0,if

(k)
=x
k
x, that is lim
k
||
(k)
|| =0.
Corollary 2.1 x
(k)
x iff x
(k)
i
x
i
Denition 2.2 (Cauchy sequence) A sequence {x
(k)
} in a vector space is a Cauchy sequence with
respect to the vector norm ||.|| if for each > 0, there is a positive integer N()such that ||x
(k
1
)

x
(k
2
)
|| whenever k
1
, k
2
N()
3 - 2
3 Dual Norm
Denition 3.1 Let ||.|| be a norm all of R
m
. The function ||y||
D
max
||x||=1
|y
T
x|. ||y||
D
is called the
dual norm of y w.r.t ||.||
Proposition 3.1 Dual norm is a vector norm
proof:
1. Since |.| 0, max
||x||=1
|y
T
x| 0
2. y =0 ||y||
D
=0
y =0 ||y||
D
=max
||x||=1
|y
T
x| |
yy
T
||y||
| =||y||
2
>0 max
||x||=1
|y
T
x| =0 iff y =0
3. ||y
1
+y
2
||
D
=max
||x||=1
|(y
T
1
+y
T
2
)x| =max
||x||=1
|y
T
1
x+y
T
2
x| max
||x||=1
|y
T
1
x| +max
||x||=1
|y
T
2
x| =||y
1
||
D
+||y
2
||
D
4. The forth proposition of vector norm is self-evident
Lemma 3.1 Let ||.|| be a norm on R
m
. Then |y
T
x| ||x||||y||
D
, |y
T
x| ||x||
D
||y|| for x, y R
m
proof: x= 0, the conclusion holds. When x =0, ||y||
D
=max
||x||=1
|y
T
x|
||x||

|y
T
x|
||x||
Proposition 3.2 Dual Norm proposition
1. ||y||
D
1
=||y||

2. ||y||
D

=||y||
1
3. ||y||
D
2
=||y||
2
proof:
1. x, y R
m
|y
T
x| =|
m

i =1
y
i
x
i
|
m

i =1
|y
i
x
i
| max
1i m
|y
i
|
m

i =1
|x
i
| =||y||

||x||
1
||y||
D
1
= max
||x||
1
=1
|y
T
x|
||x||
1
||y||

let
x
i
=
_
_
_
1 i f i =ar gmax
j
|y
j
|
0 ot her wi se
||y||
D
1
= max
||x||
1
=1
|y
T
x|
||x||
1
||y||

thus ||y||
D
1
=||y||

2. |y
T
x| =|
m

i =1
y
i
x
i
|
m

i =1
|y
i
x
i
| max
1i m
|x
i
|
m

i =1
|y
i
| =||x||

||y||
1
||y||
D

= max
||x||

=1
|y
T
x|
||x||

||y||
1
let
x
i
=
_
1 i f y
i
0
1 i f y
i
<0
3 - 3
||y||
D

= max
||x||

=1
|y
T
x|
||x||

||y||
1
thus ||y||
D

=||y||
1
3. Using Cauchy-Schantz inequality, we have |y
T
x| ||y||
2
||x||
2
||y||
D
2
= max
||x||
2
=1
|y
T
x|
||x||
2

||y||
2
||x||
2
||x||
2
=||y||
2
Let x =y/||y||
2
||y||
D
2
= max
||x||
2
=1
|y
T
x|
||x||
2
||y||
2
thus ||y||
D
2
=||y||
2
Theorem3.1 Let ||.|| be a vector norm on R
m
. Let ||.||
D
be its dual norm. Let C >0 be given. Then
||x|| =C||x||
D
for all x R
m
iff ||.|| =

C||.||
2
, in particular, ||.|| =||.||
D
iff ||.|| =||.||
2
proof:
: ||.|| =

C||.||
2
, then||x||
D
=max
||y||=1
|y
T
x| = max
||y||
2
=
1

C
|y
T
x| = max
||y||
2
=1
y
T
x

C
=
1

C
max
||y||
2
=1
|y
T
x| =
1

C
||x||
D
2
=
1

C
1

C
||x|| =
1
C
||x||
: ||.|| =C||.||
D
for some C >0. Then ||x||
2
2
=x
T
x ||x||||x||
D
=
1
C
||x||
2
which means||x||

C||x||
2
1
C
||x|| =||x||
D
=max
||y||=1
|x
T
y| =max(
|x
T
y|
||y||
2
||y||
||y||
)
1

C
max
y=0
|x
T
y|
||y||
2
=
1

C
||x||
D
2
which means||x||

C||x||
D
2
So ||x|| =

C||x||
D
2
4 Matrix Norms
Denition 4.1 f : R
mm
R is a matrix norm if the following properties hold:
1. f(A) 0, A R
mm
2. f(A) =0 iff A0
3. f(A+B) f(A) +f(B), A, B R
mm
4. f(A) =||f(A), R, A R
mm
example:
||A||

=max
i , j
|a
i j
|
||A||
1
=

i , j
|a
i j
|
||A||
F
=(

i , j
|a
i , j
|
2
)
1
2
=tr(AA
T
)
1
2
(Frobenius Norm)
4.1 Consistency
Denition 4.2 A R
mn
, B R
np
if ||AB|| ||A||||B||, then matrix norm||.|| is consistent.
3 - 4
example:
Frobenius-norm is consistent.||vec(AB)||
2
=|vec(A)
T
vec(B)| ||vec(A)||
2
||vec(B)||
2
||.||

is not consistent. For example, A=

_
1 1
1 1
_
4.2 Operation Norm
Denition 4.3 Let Abe mn matrix. ||.||be a vector norm on R
m
. Then |||A||| = sup
x=0
||Ax||
||x||
=
max
||x||=1
||Ax|| is called an operation norm or induced matrix norm.
Lemma 4.1 |||.||| is a consistent matrix norm
proof:
1. Since vector norm||.|| 0 |||.||| 0
2. when A=0, it is obvious that |||A||| =0
when A=0, Ax =0, from the properties of vector norm, we have |||A||| =0
3. |||A+B||| =max
||x||=0
||(A+B)x||
||x||
max
||x||=0
(
||Ax||
||x||
+
||Bx||
||x||
) max
||x||=0
||A||
||x||
+max
||x||=0
||B||
||x||
=|||A||| +|||B|||
4. |||A||| =|||A||| is self-evident.
5. ||ABx|| |||A||| ||Bx|| |||A|||| |||B||| ||x||
|||AB||| |||A||| |||B|||
Lemma 4.2 Ax Ax for a vector normand its corresponding operator norm, or Ax
2

A
F
x
2
, where x R
n
and A R
mn
Proof: It is easy to prove Ax Ax from the denition of operator norm. We just focus
on the second conclusion:
Ax=
_
_
_
_
_
_
a
1
x
a
2
x
.
.
.
a
m
x
_
_
_
_
_
_
where a
i
is the i th row of A. Then
Ax
2
2
=
m

i =1
(a
i
x)
2

i =1
(a
j

2
2
x
2
2
) =x
2
2
m

i =1
a
j

2
2
=A
2
F
x
2
2
4.3 Propositions for Matrix Norms
Proposition 4.1 Some propositions of matrix norms.
(1) The max norm

and Frobenius norm

F
are NOT operator norms.
3 - 5
(2) QAZ
F
=A
F
and QAZ
2
=A
2
, where Q
T
(mm)
Q
(mm)
=I
m
, Z
T
(nn)
Z
(nn)
=I
n
.
(3) A

=max
x=0
Ax

=max
i

j
|a
i j
|
(4) A
1
=max
x=0
Ax
1
x
1
=max
j

i
|a
i j
|
(5) A
2
=max
x=0
Ax
2
x
2
=
_

max
(A
T
A), where
max
(A
T
A) denotes the largest eigenvalue of A
T
A
(6) A
2
=A
T

2
.
(7) If Ais an mn matrix, then
n

1
2
A

A
2
m
1
2
A

1
2
A
1
A
2
n
1
2
A
1
Proof:
(1) We have already known that

is not consistent, thus not operator norm. For Frobenius

norm, consider I =
_
1 0
0 1
_
, we have I
F
=

2, while
I =max
x=0
Ix
x
=max
x=0
x
x
=1 2
(2) First consider the Frobenius norm:
t r (QAZZ
T
A
T
Q
T
) =t r (QAA
T
Q
T
)
=t r (A
T
Q
T
QA) =t r (AA
T
)
=A
2
F
Similarly,
QAZ
2
=max
x=0
QAZx
2
x
2
=max
x=0
_
x
T
Z
T
A
T
Q
T
QAZx

x
T
x
y=Zx
======max
y=0
_
y
T
A
T
Ay
_
y
T
y
=max
y=0
Ay
2
y
2
=A
2
2
(3) Since
|

j
a
i j
x
j
|

j
|a
i j
x
j
| (max
j
|x
j
|)

j
|a
i j
|
we have
Ax

=
max
i
|

j
a
i j
x
j
|
max
j
|x
j
|

max
i
(max
j
|x
j
|)

j
|a
i j
|
max
j
|x
j
|
=max
i

j
|a
i j
|
(1)
3 - 6
Now let p =argmax
i

j
|a
i j
|, and
y
j
=
_
1 if a
p j
0
1 otherwise
Then,
Ay

= max
i

j
|a
i j
|, which implies the equation in (1) can be achieved. Thus comes
to the conclusion. 2
(4) Let A=(a
1
, a
2
, , a
n
), then Ax=(x
1
a
1
+x
2
a
2
+ +x
n
a
n
) =
n

j =1
x
j
a
j
Since
Ax
1
=
n

j =1
x
j
a
j

j =1
(|x
j
|a
j

1
) (max
j
a
j
)x
1
We have
Ax
1
x
1
max
j
a
j

1
(2)
Dene e
j
=
_
_
_
_
_
_
_
_
_
0
.
.
.
1
.
.
.
0
_
_
_
_
_
_
_
_
_
j , then Ae
j
=
_
_
_
_
_
_
a
1j
a
2j
.
.
.
a
mj
_
_
_
_
_
_
.
Let p = argmax
j
a
j

1
, y = e
p
. We have
Ay
1
y
1
=
max
j

i
|a
i j
|
1
= max
j
a
j

1
, which implies the
equation in (2) can be achieved. Thus comes to the conclusion. 2
(5) Consider the following problem:
maximize
s.t.x
2
2
=1
f (x) =Ax
2
2
We use the method of Lagrangian multiplier to solve this problem:
L(x) =x
T
A
T
Ax(x
T
x1)
L
x
=2A
T
Ax2x
Let
L
x
=0, we have
A
T
Ax=x
which implies that is the eigenvalue of A
T
A. Thus,
A
2
2
= max
x
2
2
=1
Ax
2
2
= max
x
2
=1
(x
T
A
T
Ax) = max
x
2
=1
(x
T
x)
=max() =
max
(A
T
A) 2
3 - 7
Denition 4.4 The spectral radius (A) of a matrix A R
mm
is
(A) max{|| : is an eigenvalue of A}
Theorem4.1 If is any consistent matrix norm, then (A) A for any A R
mm
.
Proof: Let X=(x
1
, x
2
, , x
m
)
(nm)
, where Ax
i
=x
i
, x
i
=0. Then,
AX=(Ax
1
, Ax
2
, , Ax
m
) =X
||X =AX AX 2
Theorem4.2 Let A R
mm
, and >0 be given. There is a consistent matrix norm such that
(A) A (A) +.
Proof: We rst give the following theorem:
Theorem4.3 (Jordan Decomposition) If A C
nn
, then there exists a nonsingular X C
nn
such that X
1
AX=diag(J
1
, , J
t
) where
J
i
=
_
_
_
_
_
_
_
_
_
_

i
1 0
0
i
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 1
0 0
i
_
_
_
_
_
_
_
_
_
_
is m
i
-by-m
i
and m
1
+ +m
t
=n.
Now apply Jordan Decomposition to A, we have X
1
AX=, where

=1+|
i
|

=max(1+|
i
|)
Let
D=
_
_
_
_
_
_
1

1
.
.
.

(m1)
_
_
_
_
_
_
D
1
=
_
_
_
_
_
_
1

.
.
.

m1
_
_
_
_
_
_

1
=
_
_
_
_
_
_
_
_
_
_

1
1 0
0
1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 1
0 0
1
_
_
_
_
_
_
_
_
_
_
(r r )
D
1
=
_
_
_
_
_
_
_
_
1

1
.
.
.

(r 2)

(r 1)
_
_
_
_
_
_
_
_
3 - 8
Then,
D
1

1
D
1
1
=
_
_
_
_
_
_
_
_
_

1
1 0 0
0
1

1

1
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
(r 2)
0 0
(r 1)

1
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
1

.
.
.

r 2

r 1
_
_
_
_
_
_
_
_
=
_
_
_
_
_
_
_
_
_

1
0 0
0
1

.
.
.
.
.
.
.
.
.
.
.
.
1

0 0
1
_
_
_
_
_
_
_
_
_
Thus, DD
1
=diag(
1
, ,
p
), where
i
=
_
_
_
_
_
_
_
_
_
_

i
0
0
i
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0
i
_
_
_
_
_
_
_
_
_
_
Since for any vector norm and nonsingular matrix A, we have Ax is also a vector norm,
we can dene an operator normA =max
y=0
DX
1
Ay

DX
1
y

. Then,
A =max
y=0
DX
1
AXD
1
DX
1
y

DX
1
y

z=DX
1
y
=========max
z=0
DX
1
AXD
1
z

=DX
1
AXD
1

=max(+|
i
|) =+(A) 2
5 Matrix Convergence
Denition 5.1 A sequence {A
(k)
|A
(k)
R
mn
} converges to A R
mn
if lim
k
A
(k)
A =0
Lemma 5.1 Let A R
mm
be a given matrix. If there is a consistent matrix norm such that
A 1, then lim
k
A
k
=0.
Proof:
A
k
A
k1
A =A
k
Since A <1, lim
k
A
k
=0, which implies lim
k
A
k
=0. Thus, lim
k
A
k
=0 2
Theorem5.1 Let A R
mm
, then lim
k
A
k
=0 iff (A) <1.
3 - 9
Proof:
1. Sufciency: Consider Ax=x and A
k
x=
k
x,
lim
k
A
k
x=0 lim
k

k
x=0
which implies || <1. Thus (A) <1.
2. Necessity:
(A) <1 1(A) >0
Let =
1(x)
2
, use Theorem 4.2 we can nd a norm such that
A =(A) +
1A
2
=
1+(A)
2
<1
Use Lemma 5.1 we have lim
k
A
k
=0 2
3 - 10