Num LA

Numerical Linear Algebra
Introduction
Ng Tin Yau (PhD)
Department of Mechanical Engineering
The Hong Kong Polytechnic University
Jan 2015
By Ng Tin Yau (PhD) 1/65
Table of Contents
Matrices & Determinants
Linear Systems
Applications to Differential Equations
Direct Methods
Iterative Methods
Eigenproblems
Diagonalization Problem
A Transformation Method Jacobi
Exercises
By Ng Tin Yau (PhD) 2/65
Matrix
Denote K as either the set of real numbers R or complex numbers C.
An m n matrix A is an array of numbers that belong to K, that is,
a11 a12 a1n

a21 a22 a2n
A= .
..
.. aij K, 1 i m, 1 j n
..
..
.
.
.
am1 am2
amn
In the case where m = n, we call it a square matrix. A matrix with all

entries equal to zero is called the zero matrix.
0 0 0
0 0 0
O = . . .
. . ...
.. ..
0 0
Two matrices A and A0 are said to be equal if and only if aij = a0ij .
Matrices & Determinants By Ng Tin Yau (PhD) 3/65
Some important types of matrix

The tranpose of an m n matrix A, denoted by AT , is defined to be
the n m matrix that results from interchanging the rows and columns
of A. A matrix A is said to be symmetric if A = AT , that is
aij = aji .
A square matrix A is said to be invertible if there exists another
square matrix B such that AB = BA = I where
1 0 0
0 1 0
I = . . .
..
.
.
.
. .
. .
0 0
is the identity matrix. If A is invertible, then we denote B = A1

and is called the inverse of A. A square matrix Q is said to be an
orthogonal matrix if Q1 = QT . Finally, a matrix of dimension
n 1 is called a column matrix or sometimes called it an
n-dimensional vector in geometry.
Matrix Addition
Let A and B be two matrices of same dimension. Then the addition
of A and B is defined as
A + B aij + bij
(1)
Suppose that k K, then the scalar multiplication of k and A is

defined as
kA kaij
(2)
In the case where k = 1, we write (1)A = A.
Example
Compute 2A B with

1 0
A=
2 3
0 4
and B =
1 1
Solution:

1 0
0 4
2 4
2A B = 2
=
2 3
1 1
3 5
Matrix Multiplication
Let A and B be matrices of dimensions n p and p m, respectively.
Then the matrix multiplication of A and B is defined as
AB
p
X
aik bkj
(3)
k=1
Example
Given

A=
1 0
2 3

and B =

4
2
Compute AB.
Solution:

4
AB =
14
Note that BA is undefined. Therefore, in general, AB 6= BA.
Determinant: Cofactor Expansion Approach

Given an n n real matrix A,
a11
a21
A= .
..
a12
a22
..
.
..
.
an1 an2
a1n
a2n
..
.
ann
ij to be the (n 1) (n 1) matrix obtained

For n 2, we define A
from A by deleting row i and column j.
If n = 1, so that A = a11 , we define det(A) = a11 . For n 2, we define
the determinant of A recursively as
det(A) =
n
X
1j )
(1)1+j a1j det(A
(4)
j=1
This formula is called cofactor expansion along the first row of A. The
ij ) is called the cofactor of the entry of A in row
scalar (1)i+j det(A
i, column j.
A 3 3 Example
For a 2 2 matrix, we have

a11 a12
a11 a12

= a11 a22 a12 a21
det
=
a21 a22
a21 a22
(5)
Given a 3 3 matrix A,
a11 a12 a13

A = a21 a22 a23
a31 a32 a33
Then
det A =
3
X
1j )
(1)1+j a1j det(A
j=1
11 ) a12 det(A
12 ) + a13 det(A
13 )
= a11 det(A

a22 a23
a21 a23
a21 a22

a11
a12
+ a13
a32 a33
a31 a33
a31 a32
Properties of Determinants
1. Suppose that A is a n n matrix and k is any scalar, then
det(kA) = k n det(A)
(6)
2. Suppose that A and B are n n matrices, then

det(AB) = det(A) + det(B)
(7)
3. If A is invertible, then
1
det(A)
(8)
det(A) = det(AT )
(9)
det(A1 ) =
4. If A is a square matrix, then
Inverse of a 2 2 Matrix
Given a matrix

A=
a b
c d
Then
det(A) = ad bc
and
1

1
d b
=
det(A) c a
(10)
Properties of Matrix Operations

1. If the sizes of the matrices are such that the stated operations can
be performed, then
(AT )T = A
(11)
and
(AB)T = BT AT
(12)
2. Suppose that A is invertible, then

(A1 )1 = A
(13)
3. If A and B are invertible matrices, then AB is invertible and

(AB)1 = B1 A1
(14)
4. If A is an invertible matrix, then then AT is also invertible and

(A1 )T = (AT )1
(15)
System of Linear Equations

Many practical applications of engineering and science lead to a system
of linear algebraic equations. A set of simultaneous linear algebraic
equation can be expressed in matrix form:

a11 a12 a1n
x1

b1
x2
a21 a22 a2n

b2
=
(16)
..
..
.
.
.
..
..
..

..
.
.
.

xn
bn
an1 an2 ann
In symbolic notation, we write Ax = b where A is called the
coefficient matrix of the system. Also, the augmented matrix of
is the matrix
A, denoted A
a11 a1n b1
a21 a2n b2
[A | b] = .
(17)
..
..
..
..
.
.
.
an1
ann bn
Linear Systems By Ng Tin Yau (PhD) 12/65
Conditions for Invertibility of a Square Matrix

Given a linear system Ax = b of n equations.
Theorem
Let A be a square matrix with real entries. Then the following
statements are equivalent:
1
A is invertible
det(A) 6= 0
Given any vector b Rn , there is exactly one vector x such that

Ax = b.
Ax = 0 has only the trivial solution, namely, x = 0.
The rank (the number of linearly independent column vectors of

A) of A is equal to n.
Common Methods
Several methods can be used for solving systems of linear equations.

These methods can be divided into two types: direct and iterative.
Common direct methods include elimination method (Gauss,
LU-decomposition) and Choleskys method (for symmetric, positive
definite matrix). Iterative methods include Jacobi method,
Gauss-Seidel method, relaxation methods, to name a few.
BVP1 - ODE
We consider the discretization of the boundary value problem for the

ordinary differential equation
u00 (x) = f (x, u(x)) x (0, 1)
(18)
with boundary conditions u(0) = u(1) = 0. Here, f : (0, 1) R R is a

given continuous function, and we are looking for a solution
u C 2 (0, 1). Boundary value problem of this type occur, for example,
in elastic bar problem and heat conduction problem.
BVP1 - discretization
For the approximate solution we choose an equidistant subdivision of
the interval [0, 1] by setting
xj = jh
j = 0, 1, . . . , n + 1
(19)
where the step size is given by h = 1/(n + 1) with n N.

At the interval grid points xj with j = 1, . . . , n, we replace u00 (xj ) by
the difference quotient
u00 (xj )
1
[u(xj+1 ) 2u(xj ) + u(xj1 )]
h2
(20)
By using the notation u(xj ) uj , we arrive the system of equations
1
[uj1 2uj + uj+1 ] = f (xj , uj )
h2
(21)
for j = 1, . . . , n.
BVP1 - 3 3 case
In the case where n = 3
2
1
1
h2
0
and notice that u0 = u4 = 0, then we arrive

1 0 u1 f (x1 , u1 )
2 1 u2 = f (x2 , u2 )
(22)

f (x3 , u3 )
u3
1 2
In symbolic form, we have Au = f (x, u) where
2 1 0
f (x1 , u1 )
1
A = 2 1 2 1 and f = f (x2 , u2 )
h
0 1 2
f (x3 , u3 )
(23)
In this type of problems, the matrix A is tridiagonal. Moreover, if f

depends linearly on the second variable u, then the tridiagonal system
of equations also is linear.
BVP2 - PDE
Let x = (x1 , x2 ) and = (0, 1) (0, 1) and consider the following

elliptic partial differential equation:

2
u 2u
+
= f (x, u(x)) x
(24)
x21 x22
with Dirichlet boundary condition u(x) = 0 for all x . Here,
f : R R is a given continuous function, and we are looking for a
solution u C 2 (). Boundary value problems of this type arise, for
example, in torsional bar problem and steady-state heat conduction
problem.
BVP2 - discretization
For the approximate solution we choose an equidistant subdivision of
the region [0, 1] [0, 1] by setting
xi,j = (ih, jh) j = 0, 1, . . . , n + 1
(25)
where the step size is given by h = 1/(n + 1) with n N.

Analogously to the previous example, At the internal grid points xi,j
with i, j = 1, . . . , n, we replace the Laplacian by the difference quotient
1
2u 2u
+ 2 2 [u(xi,j+1 ) + u(xi+1,j ) 4u(xi,j ) + u(xi,j1 ) + u(xi1,j )]
2
h
x1 x2
(26)
By using the notation u(xi,j ) ui,j , we arrive the system of equations
1
[4ui,j ui,j+1 ui+1,j ui,j1 ui1,j ] = f (xi,j , ui,j )
h2
(27)
for i, j = 1, . . . , n.
Elementary Row Operations
Intuitively, the following operations for matrices

1
Interchange of two rows
Addition of a constant multiple of one row to another row
Multiplication of a row by a nonzero constant c
should not change the solution of any system of equations.

We now call a linear system S1 row-equivalent to a linear system S2 if
S1 can be obtained from S2 by finitely many row operations. In fact,
we have the following theorem:
Theorem
Row-equivalent linear systems have the same set of solutions
An Example
Consider the following linear system

2 1 1 x1 4
4 3 1 x2 = 6

3 2
2
x3
15
(28)
Put it in the form of augmented matrix and perform row operations on

this matrix.
2 1 1
4
2 1 1
4
2E1 +E2
4 3 1 6
0 5 3 2
3/2E1 +E3
1
9
3 2
2 15
0 27
2
2 1 1
4
7/10E2 +E3
0 5 3 2
52
13
0 0
5
5
To this end, we finished the so-called forward elimination. Next
procedure called the back substitution. That is, we have x3 = 4 and
the solve for x2 = 2 and finally x1 = 1.
Idea of the Gauss Elimination

This standard method for solving linear systems is a systematic process
of elimination that reduces the system to triangular form because
the system can then be easily solved by back substitution. In the
case of a 3 3 system, a triangular system takes the form
(1)
(2)
a11 a12 a13 b1
a11 a12 a13 b1
a11 a12 a13 b1
a21 a22 a23 b2 0 a22 a23 b2 0 a22 a23 b2
a31 a32 a33 b3
0 a32 a33 b3
0
0 a33 b3
(2)
(2)
(2)
Now one can solve for x3 = b3 /a33 provided a33 6= 0.

(0)
In the first step of the elimination process, the first equation E1 of

the system is called the pivot equation and a11 is called the pivot.
(0)
(0)
We use this pivot equation to eliminate x1 from E2 to E3 . In the
(1)
second step we take the new second equation E2 as the pivot
(1)
equation and use it to eliminate x2 of equation E3 .
Gauss Elimination Method - General Procedures

In general, for an n n system after n 1 steps the system is
transformed to a triangular system that can be solved by back
substitution.

b1
a11 a12 a13 a1n
x1
0 a22 a23 a2n

b2
x
..
..
.
.
.
.
.
0
a
a
=
33
3n
..

..
..
..
..
..
..
.
.
.
.
.
.
.

bn
xn
0
0
0 ann
Now if we set aj,n+1 = bj where 1 j n, then
an,n+1
xn =
ann
and for i = n 1, . . . , 1 we have
n
X
1
xi =
ai,n+1
aij xj
aii
(29)
(30)
(31)
j=i+1
How about if akk = 0?

In general, the pivot akk (in step k) must be different from zero and
should be large in absolute value so as to avoid roundoff manification by
the multiplication in the elimination. For this we choose as our pivot
equation one that has the absolutely largest ajk in column k on or
below the main diagonal (actually, the upper most if there are several
such equations). This popular method is called partial pivoting.
Consider the system as follows:

0 8 2 x1 7
3 5 2 x2 =
8

26
x3
6 2 8
In this case, we have a11 = 0, therefore, pivoting is necessary. The
greatest coefficient in column 1 is |a31 | = 6, in this case we interchange
E1 and E3 to give the system as in problem 1.
Guildlines for Pivoting

In step k, if akk = 0, we must pivot. If |akk | is small, we should pivot.
Difficulty with Small Pivots

The solution of the system

0.0004 1.402
x1
1.406
=
0.4003 1.502 x2
2.501
is x1 = 10 and x2 = 1. Solve this system by Gauss elimination using
4-digit floating-point arithmetric.
Picking the first of the given equations as the pivot equation, we have
to multiply this equation by 0.4003/0.0004 = 1000.75 1001 and
subtract the result from the second equation, obtaining
1405
x2 = 1404 (round-off error here!). Hence,
x
2 = (1404)/(1405) = 0.9993 (round-off error again!) which gives
x
1 = 12.4535. This failure occurs because |a11 | is small compared with
|a12 | so that a small round-off error in x2 leads to a large error in x1 .
That is, using the first equation to obtain
0.0004e1 + 1.402e2 = 0
(32)
where ei = xi x
i . Notice that e2 = 0.0007 which gives e1 = 2.4535.
Swaping the rows

The solution of the system

0.4003 1.502 x1
2.501
=
0.0004 1.402
x2
1.406
In this case we use the factor 0.0004/0.4003 0.0010 to make a22 = 0
which yields the system

0.4003 1.502 x1
2.501
=
0.0000 1.404 x2
1.403
which gives x
2 = 0.9993 and
0.4003
x1 1.502(0.9993) = 2.501 x
1 = 9.9974
Now using the first equation we have
0.4003e1 1.502e2 = 0
(33)
In this case e2 = 0.0007 which is the same as without pivioting,

however, e1 = 0.0026 which is definitely better than before.
Iteration Methods
The linear systems Ax = b that occur in many applications can have a

very large order. For such systems, the Gaussian elimination method is
often too expensive in either computation time or computer memory
requirements, or possibly both.
In an iterative method, a sequence of progressively accurate iterates is
produced to approximate the solution. Thus, in general, we do not
expect to get the exact solution in a finite number of iteration steps,
even if the round-off error effect is not taken into account. In the study
of iteration methods, the most important issue is the convergence
property.
Vectors in Rn
Denote x = (x1 , x2 , . . . , xn )T Rn where xi R. Then Rn becomes a
vector space if for all elements x, y of Rn and scalars R we have
1
x + y = (x1 + y1 , x2 + y2 , . . . , xn + yn )T
x = (x1 , x2 , . . . , xn )T
To introduce the notion of length to Rn , we have the following

so-called normed linear space axioms:
Normed Linear Space Axioms

Let x, y be elements of Rn . A vector norm k k on Rn is a
nonnegative real-valued function such that it satisfies the following
axioms:
1
kxk = 0 if and only if x = 0
kxk = ||kxk for all R
kx + yk kxk + kyk for all x, y Rn (Triangle Inequality)
Some Vector Norms

Let x = (x1 , x2 , . . . , xn )T Rn where xi R. Common norms
employed in numerical analysis include:
1 p-norm:
!1/p
n
X
p
kxkp =
|xi |
for p N
(34)
i=1
2
Infinity norm:
kxk = max {|xi |}
1in
(35)
In particular if p = 2 we call it the Euclidean norm.
Example
Let x = (1, 1, 2)T R3 . Calculate kxk2 and kxk .
Ans:
kxk2 =
(1)2 + 12 + (2)2 = 6
and
kxk = max{| 1|, |1|, | 2|} = 2
What norm we should use?

Theorem
For each x Rn , we have
kxk kxk2
nkxk
(36)
Proof: Let xj be a coordinate of x such that

kxk = max |xi | = |xj |
1in
Then
kxk2 = |xj |2 = x2j
n
X
x2i = kxk22
(37)
(38)
i=1
so kxk kxk2 . On the other hand

kxk22 =
Hence, kxk2
n
X
i=1
x2i
n
X
x2j = nx2j = nkxk2
(39)
i=1
nkxk .
Convergent Sequences
n
A sequence {x(k) }
k=0 of vectors in R is said to converge to x with
respect to the norm k k if, given any > 0, there exists an integer
N () such that
kx(k) xk < k N ()
(40)
The following theorem provides a useful stopping criterion for many

numerical algorithms.
Theorem
The sequence of vectors {x(k) } converges to x in Rn with respect to
(k)
k k if and only if limk xi = xi for each i = 1, 2, . . . , n.
(k)
Proof: () Notice that kx(k) xk = max1in |xi xi |, in other

words, there exists j {1, . . . , n} such that
(k)
kx(k) xk = |xj xj | < and we have the desired result.
() Given > 0 then for each i {1, . . . , n} there exists Ni such that
(k)
|xi xi | < whenever k(i) Ni . Let N = max1in {N1 , . . . , Nn }
(k)
and whenever k N , kx(k) xk = max1in |xi xi | < .
Example 1
Let x(k) R4 be defined by
T

1 3 k
(k)
x = 1, 2 + , 2 , e sin k
k k
Since limk 1 = 1, limk 2 + 1/k = 2, limk 3/k 2 = 0 and
limk ek sin k = 0, therefore, x(k) converges to x = (1, 2, 0, 0)T with
respect to k k . In other words, given > 0, there exists an integer
N (/2) with property that

kx(k) xk <
2
kx(k) xk < /2 whenever k N (/2). Since the Euclidean norm
and the infinity norm are equivalent, this implies that

kx(k) xk2 < 4kx(k) xk < 2

=
2
whenever k N (/2). So {x(k) } also converges to x with respect to
k k2 .
The Jacobi Method
The Jacobi iterative method is to obtained by solving the i-th

equation in Ax = b for xi to obtain (provided aii 6= 0)
aii xi =
n
X
(aij xj ) + bi
for i = 1, 2, . . . , n
(41)
j=1,j6=i
(k)
For each k 1, generate the components xi of x(k) from the

components x(k1) by
n
X
1
(k)
(k1)
xi =
aij xj
+ bi for i = 1, 2, . . . , n
aii
(42)
j=1,j6=i
Example 2
Given a linear system Ax = b as follows:
10x1 x2 + 2x3 + x4 = 6
x1 + 11x2 x3 + 3x4 = 25
2x1 x2 + 10x3 x4 = 11
3x2 x3 + 8x4 = 15
Use Jacobis iterative technique to find approximations x(k) to x
starting with x(0) = (0, 0, 0, 0)T until
kx(k) x(k1) k
< 0.0002
kx(k) k
Solution: First Iteration

Rewrite the system to yield
10x1 = x2 2x3 x4 + 6
11x2 = x1 + x3 3x4 + 25
10x3 = 2x1 + x2 + x4 11
8x4 = 3x2 + x3 + 15
From the initial approximation x(0) = (0, 0, 0, 0)T , we have x(1) given
by
(1)
x1 =
(1)
x2 =
(1)
x3 =
(1)
x4 =
Since
(0)
(0)
(0)
1
10 (x2 2x3 x4 + 6) = 0.6000
(0)
(0)
(0)
1
11 (x1 + x3 3x4 + 25) = 2.2727
(0)
(0)
(0)
1
10 (2x1 + x2 + x4 11) = 1.1000
(0)
(0)
1
8 (3x2 + x3 + 15) = 1.8750
kx(1) x(0) k
= 1 > 0.0002
kx(1) k
Solution: Second Iteration

Let us try one more iteration. Using x(1) calculated in the first
iteration, we have
(2)
x1 =
(2)
x2 =
(2)
x3 =
(2)
x4 =
Now
(1)
(1)
(1)
1
10 (x2 2x3 x4 + 6) = 0.8598
(1)
(1)
(1)
1
11 (x1 + x3 3x4 + 25) = 1.7159
(1)
(1)
(1)
1
10 (2x1 + x2 + x4 11) = 0.8052
(1)
(1)
1
8 (3x2 + x3 + 15) = 0.8852
kx(2) x(1) k
= 0.5768 > 0.0002
kx(2) k
Therefore, we have to continue.
The Approximation Solution

Continue with the computational work we have the following results:
k
1
2
3
4
5
6
7
8
9
10
11
(k)
x1
0.6000
0.8598
0.8441
0.8929
0.8868
0.8944
0.8932
0.8943
0.8941
0.8943
0.8943
(k)
x2
2.2727
1.7159
2.0363
1.9491
1.9987
1.9842
1.9920
1.9896
1.9909
1.9905
1.9907
(k)
x3
1.1000
0.8052
1.0118
0.9521
0.9852
0.9750
0.9802
0.9785
0.9794
0.9791
0.9792
(k)
x4
1.8750
0.8852
1.1309
0.9849
1.0251
1.0023
1.0090
1.0055
1.0066
1.0060
1.0062
ek
1.0000
0.5768
0.1573
0.0582
0.0249
0.0147
0.0039
0.0018
0.0006
0.0003
0.0001
Since e11 = 0.0001 < 0.0002, therefore the approximate solution is x(11)
and Matlab gives (0.8943, 1.9906, 0.9792, 1.0061)T .
A Sufficient Condition for Convergence
For a n n linear system Ax = b, a sufficient condition for convergence

of the Jacobis method is that in each row of the matrix of cofficients
aij , the absolute value of the diagonal element is greater than the sum
of the absolute values of the off-diagonal elements. That is
|aii | >
j=n
X
|aij |
(43)
j=1,j6=i
When this condition is satisfied, the matrix A is classified as diagonally

dominant and the iteration process converges toward the solution. The
solution, however, might converge even when the above condition is not
satisfied.
What is an eigenproblems?
We shall denote Kn be the collection of all n-tuples such that each
component belongs to K. Unless otherwise stated, we shall denote A to
be any n n matrix over K.
Given a square matrix A. Suppose that there exist K and a
nonzero vector v Kn such that
Av = v
(44)
Then we say is an eignvalue of A and v is an accompanying

eigenvector. Finding such a pair (, v), also called an eigenpair that
satisfies equation (1) is called an eigenproblem.
Eigenproblem plays a significant role in engineering applications and is
frequently encountered in numerical methods.
Eigenproblems By Ng Tin Yau (PhD) 39/65
How to determine the eigenvalues of a square matrix?

Rewrite equation (1) as
(I A)v = 0
or
(A I)v = 0
(45)
where I is a the identity matrix. If is the eigenvalue of A and v 6= 0,

then we must have
det(I A) = 0
(46)
Define the characteristic polynomial of A by
p() , det(I A)
(47)
Thus, the zeros of p() are eigenvalues of the matrix A. Hence, we

have the following theorem:
Theorem
Let A be a complex square matrix. Then C is an eigenvalue of A if
and only if det(I A) = 0.
Facts about eigenvalues
Theorem
Let A be a square matrix over Kn . Then
1
Every A has an eigenvalue.
The eigenvalue associated with an eigenvector is uniquely

determined.
If v(1) , v(2) Kn are eigenvectors with the same eigenvalue , then

for every scalar c1 , c2 , the vector c1 v(1) + c2 v(2) , if it is nonzero, is
also an eigenvector with eigenvalue .
A warm up example
Example
Given

1 4 3
A=
10 6 7
Verfy that

1
1,
2

and

1
1
,
10 1
are the eigenpairs of matrix A.

Solution:
and

4/10 3/10 1
1
=1
6/10 7/10 2
2

1
1
4/10 3/10
1
=
6/10 7/10 1
10 1
Bookkeeping of cars of a car rental company

Example
A car rental company has offices in Edmonton and Calgary. Relying on
its records, the company knows that on a monthly basis 40% of rentals
from the Edmonton office are returned there and 60% are one-way
rentals that are dropped off in the Calgary office. Similarly, 70% of
rentals from the Calgary office are returned there, whereas 30% car
dropped off in Edmonton.
(1) Obtain a matrix equation that describes the number of cars at the
depots in Edmonton and Calgary at the beginning of month.
Solution: Let xk and yk denote the number of cars at the depots in
Edmonton and Calgary, respectively, at the beginning of month k
(k = 0, 1, 2, . . .). We can express this information in terms of difference
equations:
xk+1 = 0.4xk + 0.3yk
yk+1 = 0.6xk + 0.7yk
Cont
(2) Estimate the number of cars at the depots in Edmonton and
Calgary in the long run.
Analysis: Denote zk = (xk , yk )T and A as the coefficient matrix of
the system of difference equations for the model. Then, zk+1 = Azk
and notice that zk = Ak z0 . Recall that the eigenvalues of A are 1 = 1
T
and 2 = 0.1 and their corresponding eigenvectors
P2 v1 = (1, 2) and
T
v2 = (1, 1) , respectively. By writing z0 = i=1 ci vi and using the
fact that Avi = i vi , then we have
k
zk = A z0 =
2
X
i=1
ci A vi =
2
X
i=1
ci ki vi

=
c1 + c2 (0.1)k
2c1 c2 (0.1)k
Now if k , we have limk zk = (c1 , 2c1 )T . Thus, in the long run,

the number of cars in Edmonton depot tends to a value that is half the
number of cars at the Calgary office.
Example 3
Example
Compute the eigenvalues and their corresponding eigenvectors of
matrix

3 4
A=
2 6
First, we need to compute
det(I A) = det

3
4
=0
2 + 6
This leads to the charisteristic equation

( 3)( + 6) + 8 = 2 + 3 10 = ( 2)( + 5) = 0
Thus, the eigenvalues are 1 = 2 and 2 = 5.
Example 3 cont
Next we need to use the equation (I A)v = 0 to determine the
corresponding eigenvectors.
For = 2, the equation becomes

( (1) )
1
4
v1
0
(1 I A)v(1) =
=
(1)
2 8
0
v2
(1)
(1)
(2)
(2)
Set v2 = 1, then v1 = 4, hence v(1) = (4, 1)T .

For = 5, the equation becomes
( (2) )

8
4
v1
0
=
(2 I A)v(2) =
(2)
2 1
0
v2
Set v1 = 1, then v2 = 2, hence v(2) = (1, 2)T .
Example 4
Example
Compute the eigenvalues of matrix

2 6
A=
3
4
The characteristic polynomial is

+2
6
det(I A) = det
= 2 2 + 10 = 0
3 4
Notice that if we restrict that R, then we have no eigenvalue that
satisfy the characteristic polynomial! However, if we allow C, then
we have
1 = 1 + 3i and 2 = 1 3i
where i = 1.
Example 5
Example
Determine the eigenvalues for the following matrix.
7
13 16
A = 13 10 13
16 13
7
First, we need to compute
( + 7)
13
16
( + 10)
13 = 0
det(A I) = det 13
16
13
( + 7)
This leads to the charisteristic equation
p() = 3 + 242 405 + 972 = 0
Solving this equation gives
1 = 36 2 = 9 3 = 3
A Mechanical Vibration Problem

Consider the vibration of a 3-spring 2-mass problem:
Using Newtons second law to obtain

m1 0
x
1
k1 + k2
k2
x1
0
+
=
0 m2 x
2
k2
k2 + k3 x2
0
or in matrix form
M
x + Kx = 0
Mechanical Vibration cont

Setting A = M1 K, that is,

(k1 + k2 )/m1
k2 /m1
A=
k2 /m2
(k2 + k3 )/m2
= Ax. By assuming that the solution is purely
then we have x
oscillatory, we have x = veit . Substitute this solution into the
equations of motion to yield
2 veit = Aveit
Since eit 6= 0 which gives
Av = v
where = 2 . Physically, i represents the natural frequency of mi
and the eigenvectors v(1) and v(2) are the mode shapes of the two
masses.
Diagonalization Problem
A square matrix A is called a diagonal matrix if Aij = 0 when i 6= j.
It is easy to see that working with a diagonal matrix is much more
convenient than working with a nondiagonal matrix. A matrix A is
said to be similar to a matrix B if there exists an nonsingular matrix
P such that P1 AP = B. In particular, if A is similar to a diagonal
matrix D, then it is said to be diagonalizable.
Now the two key questions are:
1
How do we know that a given square matrix is diagonalizable?
Suppose that the square matrix is diagonalizable, then how to

obtain a matrix P?
In the sequel, we shall see that the diagonalization problem is directly

related to an eigenproblem.
Idea of constructing P
Suppose that A is diagonalizable, then P1 AP = D. Denote Dii = i
where i is the diagonal entry at row i. Since P is invertible, then the
column space forms a linearly independent subset of Cn . Denote
column i of P as v(i) and thus,
P = [v(1)
v(2)
v(n) ]
(48)
From the relation AP = PD, we must have

Av(i) = i v(i)
(49)
or equivalently (i I A)v(i) = 0. Since v(i) cannot be zero, therefore,

det(i I A) = 0. This suggest that we can solve for all i and then
obtain the corresponding v(i) and if all the v(i) s are linearly
independent, then we are succeeded! Hence, all we need are some
theorems to guarantee that the eigenvectors forms a linearly
independent set. At least we must have n eigenvectors at the outset.
A matrix with n distinct eigenvalues

Theorem
Let 1 , 2 , . . . , n be distinct eigenvalues of an n n matrix A. If
v(1) , v(2) , . . . , v(n) are the eigenvectors of A such that i corresponds to
v(i) , then the set {v(1) , v(2) , . . . , v(n) } is linearly independent.
Corollary
Let Abe an n n matrix. If A has n distinct eigenvalues, then A is
diagonalizable.
Example

7
13 16
3 4
The matrices
and 13 10 13 are diagonalizable.
2 6
16 13
7
Brief Survey of Numerical Methods

Many methods can be used to determine the eigenvalues and
eigenvectors of a square matrix. For example, the power method,
deflation methods, QR method, Jacobi method, to name a few.
The power method can be used to find the dominant eigenvalue and
an associated eigenvector for an arbitrary matrix. The inverse power
method will find the eigenvalue closest to a given value and associated
eigenvector. This method is often used to refine an approximate
eigenvalue and to compute an eigenvector once an eigenvalue has been
found by some other technique.
Methods based on similarity transformations, such as Householders
method, are used to convert a symmetric matrix into a similar matrix
that is tridiagonal (or upper Hessenberg if the matrix is not
symmetric). Techniques such as the QR method can then be applied to
the upper Hessenberg matrix to obtain approximations to all the
eigenvalues. The associated eigenvectors can be found by using an
iterative method, such as the Inverse Power Method, or by modifying
the QR method to include the approximation of eigenvectors.
Symmetric Matrices
A matrix A is said to be symmetric if A = AT . An n n matrix Q
is said to be an orthogonal matrix if Q1 = QT .
Theorem
If A is a real symmetric square matrix and D is a diagonal matrix
whose diagonal entries are the eigenvalues of A, then there exists an
orthogonal matrix Q such that D = QT AQ.
The following corollary to the above theorem demonstrate some of the
interesting properties of symmetric matrices.
Corollary
If A is a real symmetric n n matrix, then there exist n eigenvectors
of A that form an orthonormal set and the eigenvalues of A are real
numbers.
The Jacobis Method

Suppose that we have a real symmetric square matrix A. The Jacobis
method is an iterative method that produce all the eigenvalues and
eigenvectors of a real symmetric matrix simultaneously. This method is
based on a theorem in linear algebra stated previously, that is we need
to determine an orthogonal matrix Q such that D = QT AQ.
However, from a practical viewpoint, it may not be possible to obtain a
truely diagonal matrix D. Instead we seek a sequence of matrics
{Dk }kN and hoping that
lim Dk = D
(50)
where
Dk = QT
k Dk1 Qk
kN
(51)
with D0 A. Then the eigenvalues are given by the diagonal entries

(k)
Dii of matrix Dk . The corresponding eigenvectors {v(i) }ni=1 are given
by the columns of the matrix V(k) where it is given by

V(k) = Q1 Q2 Qk = v(1) v(2) v(n)
(52)
How to convert a 2 2 matrix to a diagonal one?

Using the idea of rotating a vector in the plane, we have the rotation
matrix given by

cos sin
R=
(53)
sin cos
Then we can obtain an orthogonal transformation A0 = RT AR. Carry
out the matrix multiplication, we have
A011 = A11 cos2 + 2A12 sin cos + A22 sin2
A012
A021
= (A22 A11 ) sin cos + A12 (cos sin )
A022
= A11 sin 2A12 sin k cos + A22 cos
(54)
(55)
(56)
To obtain a diagonal matrix, we need to kill the off-diagonal terms. In

other words, we require A012 = A021 = 0 and using the identities
cos 2 = cos2 sin2 and sin 2 = 2 sin cos to yield
tan 2 =
2A12
A11 A22
(57)
How to determine matrix Qk ?

Extending this idea to the n n case and using our notation in
previous analysis, we have
I
0
0
0
0
0 cos k 0 sin k 0
0
0
I
0
0
Qk =
0 sin k 0 cos k 0
0
0
0
0
I nn
(58)
for all k N. Here the sine and cosine entries appear in the position
(i, i), (i, j), (j, i) and (j, j). In this case, we require
(k+1)
(k+1)
Dij
= Dji
= 0 which gives
(k)
tan 2k+1 =
2Dij
(k)
(k)
(59)
Dii Djj
Thus, each step of Jacobis method reduces a pair of off-diagonal

elements to zero.
Example 6
Example
Find the eigenvalues and eigenvectors of the matrix
1 1 1
A = 1 2 2
1 2 3
Ans: The largest off-diagonal term is A23 = 2. In this case, we have
i = 2 and j = 3. Thus

4
1
2A23
1
1 = tan
tan 21 =
= 37.981878
A22 A33
2
23
and
1
0
0
1.0
0
0
0.7882054 0.6154122
Q1 = 0 cos 1 sin 1 = 0
0 sin 1 cos 1
0 0.6154122 0.7882054
Example 6 - First Iteration

With D0 = A and using the calculated Q1 , we have
1.0
0.1727932 1.4036176
0.0
D1 = QT
1 D0 Q1 = 0.1727932 0.4384472
1.4036176
0.0
4.5615525
Now we try to reduce the largest off-diagonal term of D1 , namely,
(1)
D13 = 1.4036176 to zero. In this case, we have i = 1 and j = 3.
(1)
tan 22 =
2D13
(1)
D11
(1)
D33
1
2 = tan1
2
2.8072352
1.0 4.5615525
= 19.122686
and
cos 2 0 sin 2
0.9448193
0 0.3275920
1
0 =
0
1.0
0
Q2 = 0
sin 2 0 cos 2
0.3275920 0 0.9448193
Example 6 - Second Iteration

Using the calculated Q2 , we have
0.5133313 0.1632584
0.0
D2 = Q T
2 D1 Q2 = 0.1632584 0.4384472 0.0566057
0.0
0.0566057 5.0482211
2 = 0.1632584 to zero.
The largest off-diagonal term of D2 , namely, D12
In this case, we have i = 1 and j = 2. Now tan 23 =

gives
1
3 = tan1
2
0.3265167
0.5133313 0.4384472
(2)
2D12
(2)
(2)
D11 D22
which
= 38.541515
and
cos 3 sin 3 0
0.7821569 0.6230815 0.0
Q3 = sin 3 cos 3 0 = 0.6230815 0.7821569 0.0
0
0
1
0.0
0.0
1.0
Example 6 - Third Iteration
Using the calculated Q3 , we have
0.6433861
0.0
0.0352699
0.0
0.3083924 0.0442745
D3 = Q T
3 D2 Q3 =
0.0352699 0.0442745 5.0482211
Suppose that you want to stop the process, then the three eigenvalues
are
1 = 0.6433861 2 = 0.3083924 3 = 5.0482211
In fact the eigenvalues obtained by Matlab are
1 = 0.6431 2 = 0.3080
3 = 5.0489
Example 6 - Eigenvectors
To obtain the corresponding eigenvectors we compute
0.7389969 0.5886994 0.3275920

0.7421160 0.5814533
V(3) = Q1 Q2 Q3 = 0.3334301
0.5854125 0.3204631 0.7447116
Then the eigenvectors are given by the columns of the matrix Q(3) .
That is,
0.7389969
0.5886994
0.3275920
0.3334301
0.7421160
v(1) =
v(2) =
v(3) = 0.5814533
0.5854125
0.3204631
0.7447116
Using Matlab, we have the corresponding eigenvectors
0.7370
0.591
0.3280
0.3280
0.7370
v(1) =
v(2) =
v(3) = 0.5910
0.5910
0.3280
0.7370
Set
(1) Solve the following system using Gauss elimination method with
partial pivoting.
x1 x2 + 2x3 x4 = 8
2x1 2x2 + 3x3 3x4 = 20
x1 + x2 + x3 = 2
x1 x2 + 4x3 + 3x4 = 4
(2) Suppose that z = x y. Compute kzk5 and kzk if = 3 and
= 2, and x = (5, 3, 8)T and y = (0, 2, 9)T .
(3) Perform five iterations to the following linear system using the
Jacobi method. (Using x(0) = 0 as the initial approximation.)
4x1 + x2 x3 = 5
x1 + 3x2 + x3 = 4
2x1 + 2x2 + 5x3 = 1
Exercises By Ng Tin Yau (PhD) 64/65
Set
(4) Given a matrix
3
1 2
5
C = 1 0
1 1 4
Determine the eigenvalues and eigenvectors of C
method.
(5) Given matrices
6 7 2
3
A = 4 5 2 and B = 1
1 1 1
0
by conventional
1 0
4 2
2 3
Determine the eigenvalues and eigenvectors of matrix A by the

conventional method and matrix B by the Jacobis method with 3
iterations ONLY.
Exercises By Ng Tin Yau (PhD) 65/65

Num LA

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Num LA

Transféré par

Droits d'auteur :

Formats disponibles

Numerical Linear Algebra

By Ng Tin Yau (PhD) 1/65

Matrices & Determinants

By Ng Tin Yau (PhD) 2/65

a11 a12 a1n

In the case where m = n, we call it a square matrix. A matrix with all

Matrices & Determinants By Ng Tin Yau (PhD) 3/65

Some important types of matrix

is the identity matrix. If A is invertible, then we denote B = A1

Matrices & Determinants By Ng Tin Yau (PhD) 4/65

Suppose that k K, then the scalar multiplication of k and A is

Matrices & Determinants By Ng Tin Yau (PhD) 5/65

Note that BA is undefined. Therefore, in general, AB 6= BA.

Matrices & Determinants By Ng Tin Yau (PhD) 6/65

Determinant: Cofactor Expansion Approach

ij to be the (n 1) (n 1) matrix obtained

Matrices & Determinants By Ng Tin Yau (PhD) 7/65

a11 a12 a13

Matrices & Determinants By Ng Tin Yau (PhD) 8/65

2. Suppose that A and B are n n matrices, then

Matrices & Determinants By Ng Tin Yau (PhD) 9/65

Matrices & Determinants By Ng Tin Yau (PhD) 10/65

Properties of Matrix Operations

2. Suppose that A is invertible, then

3. If A and B are invertible matrices, then AB is invertible and

4. If A is an invertible matrix, then then AT is also invertible and

Matrices & Determinants By Ng Tin Yau (PhD) 11/65

System of Linear Equations

Linear Systems By Ng Tin Yau (PhD) 12/65

Conditions for Invertibility of a Square Matrix

Given any vector b Rn , there is exactly one vector x such that

Ax = 0 has only the trivial solution, namely, x = 0.

The rank (the number of linearly independent column vectors of

Linear Systems By Ng Tin Yau (PhD) 13/65

Several methods can be used for solving systems of linear equations.

Linear Systems By Ng Tin Yau (PhD) 14/65

We consider the discretization of the boundary value problem for the

with boundary conditions u(0) = u(1) = 0. Here, f : (0, 1) R R is a

Linear Systems By Ng Tin Yau (PhD) 15/65

where the step size is given by h = 1/(n + 1) with n N.

By using the notation u(xj ) uj , we arrive the system of equations

Linear Systems By Ng Tin Yau (PhD) 16/65

and notice that u0 = u4 = 0, then we arrive

In symbolic form, we have Au = f (x, u) where

In this type of problems, the matrix A is tridiagonal. Moreover, if f

Linear Systems By Ng Tin Yau (PhD) 17/65

Let x = (x1 , x2 ) and = (0, 1) (0, 1) and consider the following

Linear Systems By Ng Tin Yau (PhD) 18/65

where the step size is given by h = 1/(n + 1) with n N.

Linear Systems By Ng Tin Yau (PhD) 19/65

Elementary Row Operations

Intuitively, the following operations for matrices

Interchange of two rows

Addition of a constant multiple of one row to another row

Multiplication of a row by a nonzero constant c

should not change the solution of any system of equations.

Linear Systems By Ng Tin Yau (PhD) 20/65

Put it in the form of augmented matrix and perform row operations on

Linear Systems By Ng Tin Yau (PhD) 21/65

Idea of the Gauss Elimination

Now one can solve for x3 = b3 /a33 provided a33 6= 0.

In the first step of the elimination process, the first equation E1 of

Linear Systems By Ng Tin Yau (PhD) 22/65