Académique Documents
Professionnel Documents
Culture Documents
Chia-Ping Chen
Professor
Department of Computer Science and Engineering
National Sun Yat-sen University
Linear Algebra
1/33
Positive Definite, Negative Definite, Indefinite
2/33
Prof. Chen Positive Definite Matrix
Multi-Variate Pure Quadratic Function
3/33
Prof. Chen Positive Definite Matrix
Minimization of Pure Quadratic Function
The function
ax 2 + 2bxy + cy 2
is minimized at x = y = 0 if
a ≥ 0, ac − b 2 ≥ 0
b2
a ≥ 0, ac − b 2 ≥ 0 ⇒ a ≥ 0, c − ≥0
!a
2b
⇒ ax 2 + 2bxy + cy 2 = a x 2 + xy + cy 2
a
!2
b2
!
b
=a x+ y + c− y2 ≥ 0
a a
4/33
Prof. Chen Positive Definite Matrix
Maximization
A function
ax 2 + 2bxy + cy 2
is maximized at x = y = 0 if
a ≤ 0, ac − b 2 ≥ 0
(−b)2
a ≤ 0, ac − b 2 ≥ 0 ⇒ − a ≥ 0, (−c) − ≥0
(−a)
n X
X n
f (x) = cij xi xj
i=1 j=1
Let
1
aij = (cij + cji ), A = {aij }
2
Then
A = AT
and
f (x) = xT Ax
6/33
Prof. Chen Positive Definite Matrix
Positive Definite Matrix
xT Ax > 0, ∀ x 6= 0
A is positive semi-definite if
xT Ax ≥ 0, ∀ x
7/33
Prof. Chen Positive Definite Matrix
Conditions for Positive Definite Matrix
A is positive definite if
A: (definition) xT Ax > 0, ∀ x 6= 0
B: it has only positive eigenvalues
C: it has only principal sub-matrices of positive determinant
D: it has only positive pivots
8/33
Prof. Chen Positive Definite Matrix
A⇒B
Ax = λx
⇒ xT Ax = λ(xT x)
xT Ax
⇒ λ=
xT x
⇒ λ>0
9/33
Prof. Chen Positive Definite Matrix
B⇒A
A = QΛQT
yT y
z }| { z }| {
⇒ xT Ax = xT Q Λ QT x
= yT Λy
λi yi2
X
=
i
> 0, ∀ x 6= 0
10/33
Prof. Chen Positive Definite Matrix
A⇒C
⇒ Ak is positive definite
⇒ all eigenvalues of Ak are positive
⇒ |Ak | = λk,1 × · · · × λk,k > 0
11/33
Prof. Chen Positive Definite Matrix
C⇒D
A matrix with only principal sub-matrices of positive
determinant has only positive pivots.
A = LDU
⇒ Ak = Lk Dk Uk
⇒ |Ak | = |Dk | = d1 . . . dk
|Ak |
⇒ dk = > 0, k = 1, . . . , n
|Ak−1 |
12/33
Prof. Chen Positive Definite Matrix
D⇒A
A = LDLT
13/33
Prof. Chen Positive Definite Matrix
Example
2 −1 0
A = −1 2 −1
0 −1 2
xT Ax < 0, ∀ x 6= 0
A is negative semi-definite if
xT Ax ≤ 0, ∀ x
15/33
Prof. Chen Positive Definite Matrix
Conditions for Negative Definite Matrix
A is negative definite if
16/33
Prof. Chen Positive Definite Matrix
Local Approximation and Optimization
17/33
Prof. Chen Positive Definite Matrix
First-order Approximation
where ∂f
∂x1
..
∇f =
.
∂f
∂xn
18/33
Prof. Chen Positive Definite Matrix
Second-order Approximation
19/33
Prof. Chen Positive Definite Matrix
Approximation around a Stationary Point
x0 is a stationary point if
∇f (x0 ) = 0
f (x) ≈ f (x0 )
20/33
Prof. Chen Positive Definite Matrix
Minimum, Maximum, and Saddle Point
21/33
Prof. Chen Positive Definite Matrix
Example
Approximate
f (x) = 2x 2 + 4xy + y 2
near x = 0.
" #
4 4
∇f (0) = 0, H(0) =
4 2
1
⇒ f (x) ≈ f (0) + xT H(0)x = 2x 2 + 4xy + y 2
2
22/33
Prof. Chen Positive Definite Matrix
Example
Approximate
near x = 0.
" #
4 4
∇f (0) = 0, H(0) =
4 2
1
⇒ F (x) ≈ F (0) + xT H(0)x = 7 + 2x 2 + 4xy + y 2
2
23/33
Prof. Chen Positive Definite Matrix
Singular Value Decomposition
24/33
Prof. Chen Positive Definite Matrix
Singular Values
25/33
Prof. Chen Positive Definite Matrix
Positivity
Singular values are always positive.
AT A is positive semi-definite, so its eigenvalues are
non-negative.
By definition, the square root of a positive number is
positive, so A has only positive singular values
σi > 0
26/33
Prof. Chen Positive Definite Matrix
Number of Singular Values
A matrix of rank r has r singular values.
AT A and A have the same nullspace
AT Ax = 0 ⇔ xT AT Ax = 0 ⇔ Ax = 0
Let A be of order m × n. Then AT A has n non-negative
eigenvalues, n − r of which are 0s, since
dim N (A) = n − r = dim N AT A
n − (n − r ) = r
27/33
Prof. Chen Positive Definite Matrix
Singular Value Decomposition
A = UΣVT
U is an orthogonal eigenvector matrix of AAT
V is an orthogonal eigenvector matrix of AT A
Σ is a rectangular matrix with singular values on the
leading diagonal positions, and 0s elsewhere.
28/33
Prof. Chen Positive Definite Matrix
Proof
Find orthonormal eigenvectors of AT A . Construct
" #
V = v1 . . . vr
vr +1 . . . vn
Let u1 . . . ur be
29/33
Prof. Chen Positive Definite Matrix
Consider UT AV with
For j = 1 . . . r
For j = r + 1 . . . n
AT A vj ⇒ Avj = 0 ⇒ uTi Avj = 0
Thus
UT AV = Σ
That is
A = UΣVT
30/33
Prof. Chen Positive Definite Matrix
Example
The eigenvalues of
1 −1 0
T
A A = −1 2 −1
0 −1 1
31/33
Prof. Chen Positive Definite Matrix
Orthonormal eigenvectors of AT A are
1 −1 1
1 1 1
v1 = √ −2 , v2 = √ 0 , v3 = √ 1
6 1 2 1 3 1
Orthonormal eigenvectors of AAT are
" # " #
Av1 1 −1 Av2 1 1
u1 = =√ , u2 = =√
σ1 2 1 σ2 2 1
1
− √26 √1
# "√ √
− √12 √1
" #
3 0 0 61 6
⇒ A = UΣVT = 2 − √ 0 √1
√1 √1 0 1 0 2 2
2 2 √1 √1 √1
3 3 3
32/33
Prof. Chen Positive Definite Matrix
Application to Image Processing