System of Linear Equations

Direct & Iterative Methods

Ng Tin Yau (PhD)
Department of Mechanical Engineering
The Hong Kong Polytechnic University

Jan 2016

By Ng Tin Yau (PhD) 1/36

Introduction
Many practical applications of engineering and science lead to a system
of linear algebraic equations. A set of simultaneous linear algebraic
equation can be expressed in matrix form:

a11 a12 a1n
x1

b1

x2
a21 a22 a2n

b2

=
(1)
..

..
.
.
.
..
..
..

..

.
.
.

xn
bn
an1 an2 ann
In symbolic notation, we write Ax = b where A is called the
coefficient matrix of the system. Also, the augmented matrix of
is the matrix
A, denoted A

a11 a1n b1
a21 a2n b2

[A | b] = .
(2)
..
..
..
..
.
.
.
an1

ann bn
By Ng Tin Yau (PhD) 2/36

Conditions for Invertibility of a Square Matrix

Given a linear system Ax = b of n equations. The matrix A is said to
be invertible if there exists a matrix B such that AB = BA = I
where I is the n n identity matrix. If A is invertible, then we denote
B = A1 and is called the inverse of A.

Theorem
Let A be a square matrix with real entries. Then the following
statements are equivalent:
1

A is invertible

det(A) 6= 0

Given any vector b Rn , there is exactly one vector x such that

Ax = b.

Ax = 0 has only the trivial solution, namely, x = 0.

The rank (the number of linearly independent column vectors of

A) of A is equal to n.
By Ng Tin Yau (PhD) 3/36

Linear Circuit
A simple electrical network contains a number of resistances and a
single DC source as shown in the figure.

Using Kirchhoffs laws and Ohms law, we can write a system of linear
equations that govern the circuit.
(7 + 2 + 6)i1 2i2 6i3 = 300
2i1 + (2 + 5 + 1 + 4)i2 4i3 i4 = 0
6i1 4i2 + (4 + 9 + 6)i3 9i4 = 0
i2 9i3 + (1 + 11 + 9)i4 = 0
By Ng Tin Yau (PhD) 4/36

Linear Circuit
Simplify the equations yield the system
15i1 2i2 6i3 = 300
2i1 + 12i2 4i3 i4 = 0
6i1 4i2 + 19i3 9i4 = 0
i2 9i3 + 21i4 = 0
In matrix form, we have

300
15 2 6 0
i1

2 12 4 1 i2 0

6 4 19 9 i3 = 0

i4
0 1 9 21
0
The solution to the system is
i1 = 26.5

i2 = 9.35 i3 = 13.3

i4 = 6.13
By Ng Tin Yau (PhD) 5/36

Elementary Row Operations

Intuitively, the following operations for matrices

1

Interchange of two rows

Addition of a constant multiple of one row to another row

Multiplication of a row by a nonzero constant c

should not change the solution of any system of equations.

We now call a linear system S1 row-equivalent to a linear system S2 if
S1 can be obtained from S2 by finitely many row operations. In fact,
we have the following theorem:

Theorem
Row-equivalent linear systems have the same set of solutions

By Ng Tin Yau (PhD) 6/36

Row Operations
Try to perform row operations on the following system:
5x1 2x2 = 1

()

x1 + 8x2 = 15

()

Clearly, interchanging Eq. and Eq. will not changing the solution
of the system. Now five times Eq. gives
5x1 2x2 = 1
5x1 + 40x2 = 75
Add Eq. to the new Eq. yields to another new Eq. , namely
38x2 = 76 x2 = 2
To solve for x1 , we use Eq. .
x1 =

1 + 2x2
=1
5
By Ng Tin Yau (PhD) 7/36

Numerical Methods for Solving Linear Systems

Several methods can be used for solving systems of linear equations.

These methods can be divided into two types: direct and iterative.
Common direct methods include
elimination method (Gauss, LU-decomposition)
Choleskys method (for symmetric, positive definite matrix)
Iterative methods include
Jacobi method, Gauss-Seidel method
Relaxation methods
to name a few.

By Ng Tin Yau (PhD) 8/36

Idea of the Gauss Elimination

This standard method for solving linear systems is a systematic process
of elimination that reduces the system to triangular form because
the system can then be easily solved by back substitution. We
describe the Gauss Elimination method by taking a 3 3 system of
equations. Let us put the system in augmented matrix form, that is,

a11 a12 a13 b1

a21 a22 a23 b2
a31 a32 a33 b3
To zero out a21 and a31 , we take the multipliers
m21 =

a21
a11

and m31 =

a31
a11

Then we perform the following calculations.

E2 E2 + m21 E1

and E3 E3 + m31 E1
By Ng Tin Yau (PhD) 9/36

Forward Elimination
Now

a11 a12 a13 b1

a11 a12 a13 b1
(1)
(1)
(1)
a21 a22 a23 b2
0 a22 a23 b2
(1)
(1)
(1)
a31 a32 a33 b3
0 a32 a33 b3
(1)

To make a32 becomes zero, we take the multiplier m32

and then do the following E3 E3 + m32 E2 . That is,

a11 a12 a13 b1

a11 a12 a13

(1)
(1)
(1)
(1)
(1)
0 a22 a23 b2 0 a22 a23
(1)
(1)
(1)
(2)
0 a32 a33 b3
0
0 a33

(1)

(1)

= a32 /a22

b1
(1)
b2
(2)
b3

In the first step of the elimination process, the first equation E1 of the
system is called the pivot equation and a11 is called the pivot. We
(0)
(0)
use this pivot equation to eliminate x1 from E2 to E3 . In the second
(1)
step we take the new second equation E2 as the pivot equation and
(1)
use it to eliminate x2 of equation E3 .
By Ng Tin Yau (PhD) 10/36

Backward Substitution
In short, we perform row operations on the original system to make it
becomes an equivalent triangular form.

a11 a12 a13 b1

a11 a12 a13 b1
(1)
(1)
(1)
a21 a22 a23 b2
0 a22 a23 b2
(2)
(2)
a31 a32 a33 b3
0
0 a33 b3
Now we are ready to do the backward substitution.
(2)

(2)

x3 = b3 /a33
(1)

(1)

(1)

x2 = (b2 a23 x3 )/a22

x1 = (b1 a12 x2 a13 x3 )/a11

By Ng Tin Yau (PhD) 11/36

Gauss Elimination Method - General Procedures

In general, for an n n system after n 1 steps the system is
transformed to a triangular system that can be solved by back
substitution.

b1
a11 a12 a13 a1n

x1

0 a22 a23 a2n

b2
x

..
..
.
.

.
.
.
0
a

a
=
33
3n

..

..
..
..
..
..
..

.
.
.
.
.
.
.

bn
xn
0
0
0 ann
Now if we set aj,n+1 = bj where 1 j n, then
an,n+1
xn =
ann
and for i = n 1, . . . , 1 we have

n
X
1
xi =
ai,n+1
aij xj
aii

(3)

(4)

(5)

j=i+1

By Ng Tin Yau (PhD) 12/36

Example 1
Consider the following linear system

2 1 1 x1 4
4 3 1 x2 = 6

3 2
2
x3
15
Put it in the form of augmented matrix and perform row operations on
this matrix.

2 1 1
4
2 1 1
4
2E1 +E2
4 3 1 6
0 5 3 2
3/2E1 +E3
1
9
3 2
2 15
0 27
2

2 1 1
4
7/10E2 +E3
0 5 3 2
52
13
0 0
5
5
To this end, we finished the so-called forward elimination. Next
procedure called the back substitution. That is, we have x3 = 4 and
the solve for x2 = 2 and finally x1 = 1.
By Ng Tin Yau (PhD) 13/36

Example 2
Example
Solve the following system using Gauss elimination method (3 decimal
places).
3x1 + 2x2 + x3 4x4 = 5
x1 5x2 + 2x3 + x4 = 18
5x1 + x2 3x3 + 2x4 = 4
2x1 + 3x2 + x3 + 5x4 = 11
Put the system in augmented matrix form:

3 2
1 4 5
1 5 2
1 18

5 1 3 2 4
2 3
1
5 11
By Ng Tin Yau (PhD) 14/36

Example 2 cont

With

Thus,

3
1

5
2

m21 = 1/3 = 0.333

E2 E2 + m21 E1

m31 = 5/3 = 1.667

E3 E3 + m31 E1

m41 = 2/3 = 0.667

E4 E4 + m41 E1

2
1 4 5
3
2
1
4
5

5 2
1 18
16.35
0 5.666 1.667 2.332

0 2.334 4.667 8.668 12.335

1 3 2 4
3
1
5 11
0 1.666 0.333 7.668
7.665

By Ng Tin Yau (PhD) 15/36

Example 2 cont
With
m32 = 2.334/5.666 = 0.412
m42 = 1.666/5.666 = 0.294

E3 E3 + m32 E2
E4 E4 + m42 E2

Thus,

3
2
1
4
5
0 5.666 1.667 2.332
16.35

0 2.334 4.667 8.668 12.335

0 1.666 0.333 7.668
7.665

3
2
1
4
5
0 5.666 1.667 2.332
16.35

0
0
5.354 7.707 19.065
0
0
0.823 8.354 12.467

By Ng Tin Yau (PhD) 16/36

Example 2 cont
With
m43 = 0.823/5.354 = 0.154

E4 E4 + m43 E3

Thus,

3
2
1
4
5
0 5.666 1.667 2.332
16.35

0
0
5.354 7.707 19.065
0
0
0.823 8.354 12.467

3
2
1
4
5
0 5.666 1.667 2.332
16.35

0
0
5.354 7.707 19.065
0
0
0
9.541
9.531

By Ng Tin Yau (PhD) 17/36

Example 2 cont
In short, the forward

3 2
1 4
1 5 2
1

5 1 3 2
2 3
1
5

elimination yields

3
2
1
4
5
5

16.35
18

0 5.666 1.667 2.332

0
0
5.354 7.707 19.065
4
0
0
0
9.541
9.531
11

Back Substitution:
x4 = 9.531/93531 = 0.999
x3 = [19.065 (7.707)(0.999)]/(5.354) = 4.999
x2 = [16.335 (2.332)(0.999) (1.667)(4.999)]/(5.666) = 1.001
x1 = [5 + (4)(0.999) (1)(4.999) (2)(1.001)]/3 = 2.000
The exact answer is x1 = 2, x2 = 1, x3 = 5 and x4 = 1.
By Ng Tin Yau (PhD) 18/36

How about if akk = 0?

In general, the pivot akk (in step k) must be different from zero and
should be large in absolute value so as to avoid roundoff manification by
the multiplication in the elimination. For this we choose as our pivot
equation one that has the absolutely largest ajk in column k on or
below the main diagonal (actually, the upper most if there are several
such equations). This popular method is called partial pivoting.
Consider the system as follows:

0 8 2 x1 7
3 5 2 x2 =
8

26
x3
6 2 8
In this case, we have a11 = 0, therefore, pivoting is necessary. The
greatest coefficient in column 1 is |a31 | = 6, in this case we interchange
E1 and E3 to give the system as in problem 1.

Guildlines for Pivoting

In step k, if akk = 0, we must pivot. If |akk | is small, we should pivot.
By Ng Tin Yau (PhD) 19/36

Difficulty with Small Pivots

The solution of the system

  

0.0004 1.402
x1
1.406
=
0.4003 1.502 x2
2.501
is x1 = 10 and x2 = 1. Solve this system by Gauss elimination using
4-digit floating-point arithmetric.
Picking the first of the given equations as the pivot equation, we have
to multiply this equation by 0.4003/0.0004 = 1000.75 1001 and
subtract the result from the second equation, obtaining
1405
x2 = 1404 (round-off error here!). Hence,
x
2 = (1404)/(1405) = 0.9993 (round-off error again!) which gives
x
1 = 12.4535. This failure occurs because |a11 | is small compared with
|a12 | so that a small round-off error in x2 leads to a large error in x1 .
That is, using the first equation to obtain
0.0004e1 + 1.402e2 = 0

(6)

where ei = xi x
i . Notice that e2 = 0.0007 which gives e1 = 2.4535.
By Ng Tin Yau (PhD) 20/36

Swaping the rows

The solution of the system

  

0.4003 1.502 x1
2.501
=
0.0004 1.402
x2
1.406
In this case we use the factor 0.0004/0.4003 0.0010 to make a22 = 0
which yields the system

  

0.4003 1.502 x1
2.501
=
0.0000 1.404 x2
1.403
which gives x
2 = 0.9993 and
0.4003
x1 1.502(0.9993) = 2.501 x
1 = 9.9974
Now using the first equation we have
0.4003e1 1.502e2 = 0

(7)

In this case e2 = 0.0007 which is the same as without pivioting,

however, e1 = 0.0026 which is definitely better than before.
By Ng Tin Yau (PhD) 21/36

Iteration Methods

The linear systems Ax = b that occur in many applications can have a

very large order. For such systems, the Gaussian elimination method is
often too expensive in either computation time or computer memory
requirements, or possibly both.
In an iterative method, a sequence of progressively accurate iterates is
produced to approximate the solution. Thus, in general, we do not
expect to get the exact solution in a finite number of iteration steps,
even if the round-off error effect is not taken into account. In the study
of iteration methods, the most important issue is the convergence
property.

By Ng Tin Yau (PhD) 22/36

Vectors in Rn
Denote x = (x1 , x2 , . . . , xn )T Rn where xi R. Then Rn becomes a
vector space if for all elements x, y of Rn and scalars R we have
1

x + y = (x1 + y1 , x2 + y2 , . . . , xn + yn )T

x = (x1 , x2 , . . . , xn )T

To introduce the notion of length to Rn , we have the following

so-called normed linear space axioms:

Normed Linear Space Axioms

Let x, y be elements of Rn . A vector norm k k on Rn is a
nonnegative real-valued function such that it satisfies the following
axioms:
1

kxk = 0 if and only if x = 0

kxk = ||kxk for all R

kx + yk kxk + kyk for all x, y Rn (Triangle Inequality)

By Ng Tin Yau (PhD) 23/36

Some Vector Norms

Let x = (x1 , x2 , . . . , xn )T Rn where xi R. Common norms
employed in numerical analysis include:
1 p-norm:
!1/p
n
X
p
kxkp =
|xi |
for p N

(8)

i=1
2

Infinity norm:
kxk = max {|xi |}
1in

(9)

In particular if p = 2 we call it the Euclidean norm.

Example
Let x = (1, 1, 2)T R3 . Calculate kxk2 and kxk .
Ans:
kxk2 =

(1)2 + 12 + (2)2 = 6

and
kxk = max{| 1|, |1|, | 2|} = 2
By Ng Tin Yau (PhD) 24/36

What norm we should use?

Theorem
For each x Rn , we have
kxk kxk2

nkxk

(10)

Proof: Let xj be a coordinate of x such that

kxk = max |xi | = |xj |
1in

Then
kxk2 = |xj |2 = x2j

n
X

x2i = kxk22

(11)

(12)

i=1

so kxk kxk2 . On the other hand

kxk22 =
Hence, kxk2

n
X
i=1

x2i

n
X

x2j = nx2j = nkxk2

(13)

i=1

nkxk . 
By Ng Tin Yau (PhD) 25/36

Convergent Sequences
n
A sequence {x(k) }
k=0 of vectors in R is said to converge to x with
respect to the norm k k if, given any  > 0, there exists an integer
N () such that
kx(k) xk <  k N ()
(14)

The following theorem provides a useful stopping criterion for many

numerical algorithms.

Theorem
The sequence of vectors {x(k) } converges to x in Rn with respect to
(k)
k k if and only if limk xi = xi for each i = 1, 2, . . . , n.
(k)

Proof: () Notice that kx(k) xk = max1in |xi xi |, in other

words, there exists j {1, . . . , n} such that
(k)
kx(k) xk = |xj xj | <  and we have the desired result.
() Given  > 0 then for each i {1, . . . , n} there exists Ni such that
(k)
|xi xi | <  whenever k(i) Ni . Let N = max1in {N1 , . . . , Nn }
(k)
and whenever k N , kx(k) xk = max1in |xi xi | < . 
By Ng Tin Yau (PhD) 26/36

Example 3
Let x(k) R4 be defined by
T

1 3 k
(k)
x = 1, 2 + , 2 , e sin k
k k

(15)

Since limk 1 = 1, limk 2 + 1/k = 2, limk 3/k 2 = 0 and

limk ek sin k = 0, therefore, x(k) converges to x = (1, 2, 0, 0)T with
respect to k k . In other words, given  > 0, there exists an integer
N (/2) with property that

kx(k) xk <
2
kx(k) xk < /2 whenever k N (/2). Since the Euclidean norm
and the infinity norm are equivalent, this implies that


kx(k) xk2 < 4kx(k) xk < 2

=
2
whenever k N (/2). So {x(k) } also converges to x with respect to
k k2 .
By Ng Tin Yau (PhD) 27/36

The Jacobi Method

The Jacobi iterative method is to obtained by solving the i-th

equation in Ax = b for xi to obtain (provided aii 6= 0)
aii xi =

n
X

(aij xj ) + bi

for i = 1, 2, . . . , n

(16)

j=1,j6=i
(k)

For each k 1, generate the components xi of x(k) from the

components x(k1) by

n
X
1
(k)
(k1)
xi =
aij xj
+ bi for i = 1, 2, . . . , n
aii

(17)

j=1,j6=i

By Ng Tin Yau (PhD) 28/36

Example 4
Given a linear system Ax = b as follows:
10x1 x2 + 2x3 + x4 = 6
x1 + 11x2 x3 + 3x4 = 25
2x1 x2 + 10x3 x4 = 11
3x2 x3 + 8x4 = 15
Use Jacobis iterative technique to find approximations x(k) to x
starting with x(0) = (0, 0, 0, 0)T until
kx(k) x(k1) k
< 0.0002
kx(k) k

By Ng Tin Yau (PhD) 29/36

Solution: First Iteration

Rewrite the system to yield
10x1 = x2 2x3 x4 + 6
11x2 = x1 + x3 3x4 + 25
10x3 = 2x1 + x2 + x4 11
8x4 = 3x2 + x3 + 15
From the initial approximation x(0) = (0, 0, 0, 0)T , we have x(1) given
by
(1)

x1 =
(1)

x2 =
(1)

x3 =
(1)

x4 =
Since

(0)
(0)
(0)
1
10 (x2 2x3 x4 + 6) = 0.6000
(0)
(0)
(0)
1
11 (x1 + x3 3x4 + 25) = 2.2727
(0)
(0)
(0)
1
10 (2x1 + x2 + x4 11) = 1.1000
(0)
(0)
1
8 (3x2 + x3 + 15) = 1.8750

kx(1) x(0) k
= 1 > 0.0002
kx(1) k
By Ng Tin Yau (PhD) 30/36

Solution: Second Iteration

Let us try one more iteration. Using x(1) calculated in the first
iteration, we have
(2)

x1 =
(2)

x2 =
(2)

x3 =
(2)

x4 =
Now

(1)
(1)
(1)
1
10 (x2 2x3 x4 + 6) = 0.8598
(1)
(1)
(1)
1
11 (x1 + x3 3x4 + 25) = 1.7159
(1)
(1)
(1)
1
10 (2x1 + x2 + x4 11) = 0.8052
(1)
(1)
1
8 (3x2 + x3 + 15) = 0.8852

kx(2) x(1) k
= 0.5768 > 0.0002
kx(2) k

Therefore, we have to continue.

By Ng Tin Yau (PhD) 31/36

The Approximation Solution

Continue with the computational work we have the following results:
k
1
2
3
4
5
6
7
8
9
10
11

(k)

x1
0.6000
0.8598
0.8441
0.8929
0.8868
0.8944
0.8932
0.8943
0.8941
0.8943
0.8943

(k)

x2
2.2727
1.7159
2.0363
1.9491
1.9987
1.9842
1.9920
1.9896
1.9909
1.9905
1.9907

(k)

x3
1.1000
0.8052
1.0118
0.9521
0.9852
0.9750
0.9802
0.9785
0.9794
0.9791
0.9792

(k)

x4
1.8750
0.8852
1.1309
0.9849
1.0251
1.0023
1.0090
1.0055
1.0066
1.0060
1.0062

ek
1.0000
0.5768
0.1573
0.0582
0.0249
0.0147
0.0039
0.0018
0.0006
0.0003
0.0001

Since e11 = 0.0001 < 0.0002, therefore the approximate solution is x(11)
and Matlab gives (0.8943, 1.9906, 0.9792, 1.0061)T .
By Ng Tin Yau (PhD) 32/36

A Sufficient Condition for Convergence

For a n n linear system Ax = b, a sufficient condition for convergence

of the Jacobis method is that in each row of the matrix of cofficients
aij , the absolute value of the diagonal element is greater than the sum
of the absolute values of the off-diagonal elements. That is
|aii | >

j=n
X

|aij |

(18)

j=1,j6=i

When this condition is satisfied, the matrix A is classified as diagonally

dominant and the iteration process converges toward the solution. The
solution, however, might converge even when the above condition is not
satisfied.

By Ng Tin Yau (PhD) 33/36

Inverse of a 2 2 Matrix

Given a matrix


A=

a b
c d

Then
det(A) = ad bc
and
1



1
d b
=
det(A) c a

(19)

By Ng Tin Yau (PhD) 34/36

Determinant of an n n Matrix
ij to be the
Given A Mnn (K), for n 2, we define A
(n 1) (n 1) matrix obtained from A by deleting row i and column
j.
If n = 1, so that A = A11 , we define det(A) = A11 . For n 2, we
define the determinant of A recursively as
det(A) =

n
X

1j )
(1)1+j A1j det(A

(20)

j=1

This formula is called cofactor expansion along the first row of A. The
ij ) is called the cofactor of the entry of A in row
scalar (1)i+j det(A
i, column j.

By Ng Tin Yau (PhD) 35/36

Exercises
(1) Solve the following system using Gauss elimination method with
partial pivoting.
x1 x2 + 2x3 x4 = 8
2x1 2x2 + 3x3 3x4 = 20
x1 + x2 + x3 = 2
x1 x2 + 4x3 + 3x4 = 4
(2) Suppose that z = x y. Compute kzk5 and kzk if = 3 and
= 2, and x = (5, 3, 8)T and y = (0, 2, 9)T .
(3) Perform five iterations to the following linear system using the
Jacobi method. (Using x(0) = 0 as the initial approximation.)
4x1 + x2 x3 = 5
x1 + 3x2 + x3 = 4
2x1 + 2x2 + 5x3 = 1
By Ng Tin Yau (PhD) 36/36