SOLVING LINEAR

SYSTEMS
Dr. Indah Soesanti, S.T., M.T.
Universitas Gadjah Mada

 Solution of sets of linear

equations is required in many
applications.
Many methods exist for this
purpose: direct methods and
iterative methods.

Gaussian Elimination
A direct method frequently used in
many applications is the Gaussian
elimination method
Gaussian Elimination transforms the
problem into a triangular system

 First, lets review the concept of simultaneous

linear equations. A set of linear simultaneous
equations may be written as:

a11x1 a12 x2 a13 x3 ... a1N xN b1

a21x1 a22 x2 a23 x3 ... a2 N xN b2
a31x1 a32 x2 a33 x3 ... a3N xN b3
.......

aM 1 x1 aM 2 x2 aM 3 x3 ... aMN xN bM

 Here we have a total of N unknowns (xj, j=1,

2 N), related through M equations. The
coefficients in the left sides of the equations
(aij, i=1, 2 N ; j=1, 2 M ) are known
parameters, and so are also the coefficients on
the right side (bi, i=1, 2 M).

The equations may alternatively be written in a

compact form:

A x b

where A is the coefficient matrix, and b is the

right hand side vector
a11 a12 ... a1N
a

a
...
a
21
22
2N
A

...

aM 1 aM 2 ... aMN

b1
x1
b
x
2
2

b
x
...
...

bM

x M

Gaussian Elimination
The method consists of two steps:
Forward Elimination: the system is reduced to
upper triangular form. A sequence of
elementary operations is used.
Backward Substitution: Solve the system
starting from the last variable.

a11 a12 a13 x1 b1

a
x b
a
a
21 22 23 2 2
a31 a32 a33 x3 b3

a11 a12 a13 x1 b1

0 a ' a ' x b '
22
23 2

2
0
0 a33 ' x3 b3 '

Gaussian elimination method

For simplicity, lets use the following set of 3 equations and 3 unknowns, ie.
N=3 and M=3, in order to illustrate the Gaussian elimination method:

a11x1 a12 x2 a13 x3 b1

a21x1 a22 x2 a23 x3 b2
a31x1 a32 x2 a33 x3 b3

(4)
(5)
(6)

The method starts by multiplying Eq. (4) by a21/a11 and then

add it to Eq. (5). The resulting equation becomes:

x2 a23
x3 b' 2
a22
We then multiply Eq.(4) by a31/a11 and add it to Eq.(6), resulting in:
'
3
32 2
33 3

a x a x b

The set of equation has now become

a11x1 a12 x2 a13 x3 b1
x2 a23
x3 b' 2
a22
x2 a33
x3 b'3
a32

(7)
(8)
(9)

Next, we multiply Eq. (8) by a32/a22 and add it to Eq.

(9), so that the set of equations become:
a11x1 a12 x2 a13 x3 b1

(10)

x2 a23
x3 b'2
a22

(11)

x3 b"3
a33

(12)

This completes the first part of the Gaussian elimination

method, called the forward elimination process.

Eq. (12) may now be used to solve directly for x3:

x3 b"3 / a33

(13)

After completion of the forward elimination process, determined the last

unknown of the vector (x3) by Eq. (13), we will perform a back substitution
process. This simply means that as the unknowns are calculated, in our simple
example from x3 and downwards, they are substituted into the equations above,
and the next unknown may be computed. For Eqs. (12) and (11) this process is
carried out as follows:

x3 ) / a22

x2 (b' 2 a23

(14)

x1 (b1 a12 x2 a13 x3 ) / a11

(15)

Based on the example above, we may formulate a general procedure for the forward
elimination, solution for the last unknown, and back substitution to get the rest
of the unknowns

Gaussian Elimination in 3D
2x 4 y 2z 2
4 x 9 y 3z 8
2 x 3 y 7 z 10
Using the first equation to eliminate x from
the next two equations

Gaussian Elimination in 3D
2x 4 y 2z 2
y z 4
y 5z 12
Using the second equation to eliminate y
from the third equation

Gaussian Elimination in 3D
2x 4 y 2z 2
y z 4
4z 8
Using the second equation to eliminate y
from the third equation

Solving Triangular Systems

We now have a triangular system which is
easily solved using a technique called
Backward-Substitution.

2x 4 y 2z 2
y z 4
4z 8

Solving Triangular Systems

If A is upper triangular, we can solve
Ax = b by:

xn bn / Ann
n

xi bi Aij x j / Aii , i n 1,
j i 1

,1

Backward Substitution
From the previous work, we have

2x 4 y 2z 2
y z 4
z 2

And substitute z in the first two equations

Backward Substitution
2x 4 y 4 2
y 2 4
z 2
We can solve y

Backward Substitution
2x 4 y 4 2
y
2
z 2
Substitute to the first equation

Backward Substitution
2x 8 4 2
y
2
z 2
We can solve the first equation

Backward Substitution
x
y

1
2
z 2

Gauss-Jordan Method
The method reduces the general system of equations
to
where I is an identity matrix.

AX=B IX=B

Only Forward elimination is done and no backward

substitution is needed.
It has the same problems as Naive Gaussian elimination and
can be modified to do partial scaled pivoting.

It takes 50% more time than Naive Gaussian method.

GAUSS-JORDAN
a11x1 + a12x2 + a13x3 + a14x4 = b1
a21x1 + a22x2 + a23x3 + a24x4 = b2
a31x1 + a32x2 + a33x3 + a34x4 = b3
a41x1 + a42x2 + a43x3 + a44x4 = b4

 a11
a
21
a31

a41

a12 a13
a22 a23
a32 a33
a42 a43

a14 x1 b1

a24 x2 b2

a34 x3 b3

a44 x4 b4

 1 a'12 a'13 a'14 x1 b'1

a

a22 a23 a24 x2

b2
21

a31 a32 a33 a34 x3 b3

a41 a42 a43 a44 x4 b4

1 a'12 a'13 a'14 x1 b'1

0 1 a' '

a
'
'
x
b
'
'
23
24
2

2
0 a'32 a'33 a'34 x3 b'3

0 a'42 a'43 a'44 x4 b'4

1
0

0
1
0
0

a' '13
a' '23
a' '33
a' '43

a' '14 x1 b' '1

a' '24 x2 b' '2

a' '34 x3 b' '3

a' '44 x4 b' '4

1
0

0
1
0
0

0 x1 b

0 x2 b

0 x3 b

1 x4 b

IV
1
IV
2
IV
3
IV
4

0
0
1
0

x1 b

IV
1

x2 b

IV
2

x3 b

IV
3

x4 b

IV
4

Gauss-Jordan Method
Example
2 2 2 x1 0
1 1 1
4 2 1 x 7
4 2 1

2 2 4 x3 2
2 2 4
Step 1 Eleminate x1 from equations 2and 3

x1 0
x 7
2
x3 2

eq1 eq1 / 2
1 1 1 x1 0
4
eq2 eq2 eq1 0 6 5 x2 7
1
0 0 2 x3 2
2

eq3 eq3 eq1
1
CISE301_Topic3

27

Gauss-Jordan Method
Example

1
1 1 1 x1 0
1 1
0 6 5 x 7
0 1 0.833

0 0 2 x3 2
2
0 0
Step 2 Eleminate x 2 from equations1 and 3

eq2 eq2 / 6
1 0 0.1667
1
eq1 eq1 eq2 0 1 0.8333
1
0 0
2
0

eq3 eq3 eq2

1

CISE301_Topic3

x1 0
x 1.1667
2

x3 2

x1 1.1667
x 1.1667
2

x3 2

28

Gauss-Jordan Method
Example
1 0 0.1667
0 1 0.8333

0 0
2

x1 1.1667
1 0 0.1667
x 1.1667
0 1 0.8333
2

x3 2
1
0 0
Step 3 Eleminate x 3 from equations1 and 2

eq3 eq3 / 2
1 0 0 x1 1

0.1667

eq1 eq1
eq
3

0 1 0 x2 2
1
0 0 1 x 1
3
0.8333
eq2 eq2
eq3
1

x1 1.1667
x 1.1667
2

x3 1

Gauss-Jordan Method
Example

2 2 2 x1 0
4 2 1 x 7

2
2 2 4 x3 2
is transformed to
1 0 0 x1 1
x1 1
0 1 0 x 2 solutionis x 2

2
2
0 0 1 x3 1
x3 1
CISE301_Topic3

30

LU Decomposition
This motivates a factorization approach to
solving arbitrary systems:
Find a way of writing A as LU, where L and U are
both triangular
Ax=b LUx=b Ly=b Ux=y
Time for factoring matrix dominates computation

LU Decomposition
ii 1

LU = A
Thus A x = (LU) x = L (U x) = b

Let y = U x, then solve y in L y = b by

forward substitution
Solve x in U x = y by backward substitution

Crouts Method
a11

a21
a
31

a12
a22
a32

a13 1

a23 l21
a33 l31

u11 a11
l21u11 a21
l31u11 a31

0
1
l32

u12 a12

0 u11

0 0
1 0

u12
u22
0

u13 a13
a21
l21
u11
a31
l31
u11

u13

u23
u33

Crouts Method
a11

a21
a
31

a12
a22
a32

a13 1

a23 l21
a33 l31

0
1
l32

0 u11

0 0
1 0

u12
u22
0

u11 a11 u12 a12 u13 a13

a21
l21u11 a21 l21
u11
a31
l31u11 a31 l31
u11
l21u12 u22 a22 u22 a22 l21u12
a32 l31u12
l31u12 l32u22 a32 l32
u22

u13

u23
u33

Crouts Method
a11

a21
a
31

a12
a22
a32

a13 1

a23 l21
a33 l31

0
1
l32

0 u11

0 0
1 0

For i = 1..n
For j = 1..i
j 1

For j = i+1..n

u ji a ji l jk uki
k 1

i 1

l ji

a ji l jk uki
k 1

uii

u12
u22
0

u13

u23
u33

Cholesky Decomposition
For symmetric matrices, choose U=LT
Perform decomposition
a11

a12
a
13

a12
a22
a23

a13 l11

a23 l21
a33 l31

0
l22
l32

0 l11

0 0
l33 0

l21
l22
0

l31

l32
l33

Ax=b LLTx=b Ly=b LTx=y

Cholesky Decomposition
a11

a12
a
13

a12
a22
a23

a13 l11

a23 l21
a33 l31

0
l22
l32

0 l11

0 0
l33 0

l21
l22
0

l112 a11 l11 a11

a12
l11
a
l31 13
l11

l11l21 a12 l21

l11l31 a13

l212 l22 2 a22 l22 a22 l212

l21l31 l22l32 a23

a23 l21l31
l32
l22

l31

l32
l33

Cholesky Decomposition
a11

a12
a
13

a12
a22
a23

a13 l11

a23 l21
a33 l31

0
l22
l32

0 l11

0 0
l33 0

i 1

lii aii lik2

k 1

i 1

l ji

aij lik l jk
k 1

lii

l21
l22
0

l31

l32
l33

Thanks