Linear Algebra

What we will do during the next 3 weeks What we will do during the next 3 weeks
Review of matrices
S t f li l b i ti AX B Systems of linear algebraic equations AX=B
Finding solution(s) by row operations
Inverse of a square matrix A q
Finding inverse by row operations
Determinant of a square matrix A
C l l i d i b i Calculating determinant by row operations
Bigger pictures: how are these 3 topics connected together? Bigger pictures: how are these 3 topics connected together?
M t i i bl
In between, we will look at vector space and linearly independent
vectors
Matrix eigenproblem
Diagonalisation problem
Review on matrices
What is a matrix? What is a matrix?
An MN matrix is an array of MN numbers enclosed within a
pair of brackets and arranged in M rows and N columns.
1
2 1 3 6 9
1
(3) 2
t
| |
|
| | | |
|
Examples:
1
(3) 2
1 2 2 3 9 2
5 6
| | | |
|
| |
|
\ . \ .
|
\ .
The numbers making up a matrix are referred to as elements
of the matrix.
In a square matrix the number of rows equals the number of In a square matrix, the number of rows equals the number of
columns.
We use bold capital letters such as A, B, P and Q to denote
matrices E g we can write:
| |
matrices. E.g. we can write:
1 2 3
4 5 6
| |
=
|
\ .
A
Notation otat o
Let A be an MN matrix.
D t th l t i th i th d j th Denote the element in the i-th row and j-th
column of A by a
ij
. Then we can write:
| |
( ) a = A
11 12 1
21 22 2
N
N
a a a
a a a
| |
|
|
=

( )
ij
a = A
1 2 M M MN
a a a
|
=
|
|
\ .

1 2 M M MN \ .
Let A = (a ) and B = (b ) be both MN (same order)
Equality of matrices
Let A = (a
ij
) and B = (b
ij
) be both MN (same order).
A and B are said to be equal that is A = B if a = b A and B are said to be equal, that is, A = B, if a
ij
= b
ij
.
E.g.
3 a b c c
c a b d b
+
| | | |
=
| |

\ . \ .
c a b d b
\ . \ .
, 3, , a b c c c d a b b + = = = =
Addition of matrices Addition of matrices
If A+B = C then C=(c
ij
) is MN and c
ij
= a
ij
+ b
ij
.
1 2 3 7 8 9 1 7 2 8 3 9
4 5 6 10 11 12 4 10 5 11 6 12
+ + +
| | | | | |
+ =
| | |
\ . \ . \ .
E.g.
4 5 6 10 11 12 4 10 5 11 6 12
| | |
+ + +
\ . \ . \ .
Multiplication of a number to a matrix
If A = (a
ij
) and c is a number then cA = (ca
ij
) and
cA has the same order as A.
E.g.
1 2 2 4
2
3 4 6 8

| | | |
=
| |

\ . \ .
We write (1)A as A. So BA = B + (A).
E.g.
2 3 1 2 2 1 3 2
4 4 3 2 4 3 4 2

| | | | | |
=
| | |

\ . \ . \ . \ . \ . \ .
The transpose of a matrix A is denoted by A
T
. p y
1 2 3 1 4 7
4 5 6 2 5 8
| | | |
| |
T
A A 4 5 6 2 5 8
7 8 9 3 6 9
| |
= =
| |
| |
\ . \ .
T
A A
If A = A
T
then A is said to be symmetric. If A A then A is said to be symmetric.
1 3 5 1 3 5
| | | |
1 3 5 1 3 5
3 0 6 3 0 6
| | | |
| |
= =
| |
| |
T
A A
5 6 9 5 6 9
| |
\ . \ .
Product of matrices
Let A=(a
ij
) and B=(b
ij
) be MN and PQ matrices
respectively.
If N P f th d t t i AB If N=P, we can form the product matrix AB.
If AB is denoted by C= (c
ij
) then C is MQ and the y (
ij
)
element c
kp
is calculated using the k-th row of A and p-th
column of B as follows.
b
| |
kp
c =
( )
1
2
1 2
p
p
k k kN
b
b
a a a
| |
|
|
|
|

1 1 2 2 k p k p kN Np
a b a b a b = + + +
p
A typo here in the 238
page document (p56)
Np
b
|
|
|
\ .

1
N
kn n j
n
a b
=
=

1 n
What is the condition for forming the product BA?
What is the order of BA if it can be formed?
Example:
1 2
| |
2 4
1 2
5 1 2 2
3 4
3 3 1 2
5 6

| |
| |
|
= =
|
|
\ .
|
\ .
P Q
3 2
5 6

\ .
We can form PQ but not QP.
1 2 11 7 4 6
5 1 2 2
3 4 27 15 10 14
| | | |
| |
| |
= =
|
| |
PQ
2 4
3 2 3 4
3 4 27 15 10 14
3 3 1 2
5 6 43 23 16 22

= =
|
| |
\ .
| |
\ . \ .
PQ
Multiplication of matrices is not commutative that is even if AB and Multiplication of matrices is not commutative, that is, even if AB and
BA can be formed, AB may or may not be equal to BA.
Identity matrices de t ty at ces
An NN matrix (c
ij
) such that c
11
= c
22
= c
33
== c
NN
=1 and c
ij
= 0
for i not equal to j is called an identity matrix.
1 0 0 0
1 0 0
1 0 0 1 0 0
| |
| |
|
| |
|
|
Examples:
1 0 0 1 0 0
0 1 0
0 1 0 0 1 0
0 0 1
0 0 0 1
| |
|
|
|
|
|
\ .
|
|
\ .
\ . \ .
Whats special about identity matrices?
Let I be an identity matrix and A be any matrix Let I be an identity matrix and A be any matrix.
If the product IA can be formed then IA = A.
Similarly if the product AI can be formed then AI = A Similarly, if the product AI can be formed then AI = A.
Lets start on linear algebra now
A good starting point is to look at a
system of linear algebraic equations.
Whats a linear algebraic equation?
Many simultaneous linear algebraic equations
form a system.
How can we solve a system of linear
algebraic equations?
Whats a linear algebraic equation?
An example of a linear algebraic equation in
one unknown x is:
2 1 0 x + =
The solution of the above linear algebraic
equation is:
1
A linear algebraic equation in two unknowns x and y
1
2
x =
A linear algebraic equation in two unknowns x and y
is an equation of the form:
3 9 x y = 3 9 x y =
If we let y =0 then x = 3. So (x,y)=(3,0) is a solution of the above equation.
Whats a linear algebraic equation?
A li l b i ti i N k A linear algebraic equation in N unknowns x
1
, x
2
,
, x
N
and x
N
is an equation of the form:
1 1 2 2
...
N N N
c x c x c x d + + + =
Why bother? y
Many problems in engineering and
h i l i f l t d i physical sciences are formulated in
terms of a system of linear algebraic
ti equations.
This temperature p
profile in the human
eye was obtained from
the boundary element the boundary element
method by solving a
system of 470 linear
algebraic equations in
470 unknowns.
Heres a system of N linear algebraic
equations in N unknowns.
a x a x a x a x b + + + + =
11 1 12 2 13 3 1 1
21 1 22 2 23 3 2 2
...
...
N N
N N
a x a x a x a x b
a x a x a x a x b
+ + + + =
+ + + + =
b + + + +

1 1 2 2 3 3
...
N N N NN N N
a x a x a x a x b + + + + =
a
ij
a is the (constant) coefficient of the unknown x
j
in the i-th equation.
b
is the constant term in the i-th equation
i
b
is the constant term in the i-th equation.
The system can be written in matrix y
form AX =B, where:
11 12 1
...
N
a a a
| |
|
Example:
12 22 2
1 2
...
N
N N NN
a a a
a a a
|
|
=
|
|
\ .
A

2 3 10
0
x y
x y
+ =
+ =
1 2 N N NN
a a a
\ .
1
2
x
x
| |
|
|
X
0 x y +
2 3 10 x
| | | | | |
=
| | |
N
x
|
=
|
|
\ .
X

1 1 0 y
| | |

\ . \ . \ .
1
2
b
b
| |
|
|
=
|
B

2 3 10
0
x y
x y
+
| | | |
=
| |
+
\ . \ .
N
b
|
|
\ .

0 x y +
\ . \ .
Solutions of a linear algebraic equation g q
Consider a single linear algebraic equation
c x c x c x d + + + =
1 1 2 2
...
N N N
c x c x c x d + + + =
1 1
x
x
o
o
| | | |
| |
| |
is said to be a solution of the above linear
l b i ti if th LHS f th ti
2 2
N N
x
x
o
o
| |
=
| |
| |
\ . \ .

algebraic equation if the LHS of the equation
equals its RHS when we replace x
1
, x
2
, , x
N
by o
1
, o
2
, , o
N
respectively.
N N \ . \ .
Example:
We can find many other solutions easily.
2 0 x y z + =
1
2
x
| | | |
| |
is a sol tion b t is not
2
1
x
y
| | | |
| |
=
| |
2
3
y
z
| |
=
| |
| |
\ . \ .
is a solution but is not.
1
3
y
z
=
| |
| |
\ . \ .
Solution of a system of linear
algebraic equations
1 1
x o
| | | |
| |
2 2
x o
| |
| |
=
| |
| |

If
is a solution of
N N
x o
| |
\ . \ .
each and every linear algebraic y g
equation in the system then it is
said to be a solution of the system said to be a solution of the system.
It is possible that a system of linear
algebraic equations has no solution.
Example:
1 0 1 0
5
x y
x y
+ =
+ =
Such a system is said to be inconsistent
y
Such a system is said to be inconsistent.
It is possible that a consistent system It is possible that a consistent system
has more than one solutions.
E l Example:
1 0 x y + =
2 2 2 0 x y + =
10 x y + =
The system really contains only one linear algebraic
equation in 2 unknowns!
So we can find infinitely many solutions for the system So we can find infinitely many solutions for the system.
If a consistent system of linear algebraic
equations has only one solution, we say q y , y
the system has a unique solution.
To summarise, a system of linear algebraic
equations can either be consistent or inconsistent.
If it is consistent, it can have either a unique
solution or infinitely many solutions.
Given a system of linear algebraic
equations AX = B, how do we know
whether it is consistent or not?
If it is consistent how do we find all If it is consistent, how do we find all
its solutions?
Reduce the system AX = B to a
simpler but equivalent system
UX = C.
AX = B and UX = C are equivalent if they have
tl th l ti ( ) exactly the same solution(s).
If we can work out the solution(s) of UX = C,
we would have solved AX B we would have solved AX = B.
If the square matrix U is an upper triangular
matrix the system UX C would be simpler matrix, the system UX = C would be simpler
enough for us to work out its solution(s) (if any).
Wh t i t i l t i ? What is an upper triangular matrix?
1 4 5 6
| |
|
E.g.
0 1 6 8
|
|
|
0 0 2 0
0 0 0 3
|
|
\ .
0 0 0 3
\ .
To solve AX = B, reduce it to an
i l t t UX C h equivalent system UX = C, where
U is an upper triangular matrix.
How can this be done?
Write AX = B in tableau form A | B.
2 3 4 6 x y z + + =
2 3 4 6
3 5 2 7
E.g. E.g.
3 5 2 7
10 5 9
x y z
x y z
+ =
+ + =
3 5 2 7
1 10 5 9

Use legitimate row operations in a systematic

manner to reduce the tableau to become U | C. manner to reduce the tableau to become U | C.
There are 2 types of legitimate row
operations.
i j
R R
Interchange i-th and j-th rows.
i j
i i j
R R R o | +
Use row j to change row i to
become o R
i
+ | R
j
.
Important. The constant o is not allowed to be zero.
Why? Why R
1
R
3
(say) is not allowed?
A simple rule to observe.
In changing a tableau by the second type of row
operation, keep one row fixed. Use the fixed row
to change other row(s). to change other row(s).
Example:
Solve the system (if it is consistent) Solve the system (if it is consistent)
2 3 2
5
x y z w
x y z w
+ + =
+ + 5
2 4 1
x y z w
x y z
+ + =
+ =
8 y z w + + =
1 1 2 3 2
1 1 2 3 2
1 1 1 1 5

1 1 2 3 2
0
2 2 1
R R R
0 1 -2 3
2 1 4 0 1
0 1 1 1 8

0 1 1 1 8
0
3 3 1
2 R R R
-3 8 -6 -3
0 1 1 1 8
0 1 1 1 8
1 1 2 3 2
1 1 2 3 2
0 0 1 2 3
0 3 8 6 3

1 1 2 3 2
0 1 1 1 8
0 3 8 6 3
2 4
R R
0 3 8 6 3
0 1 1 1 8

0 3 8 6 3
0 0 1 2 3

1 1 2 3 2
1 1 2 3 2
0 1 1 1 8

1 1 2 3 2
0 1 1 1 8

0 0 11 3 21
0 0

0
0 0 1 2 3
0 3 3 2
3 R R R +
11 -3 21
0
-19 12
4 4 3
11 R R R
x y z w
1 1 2 3 2
0 1 1 1 8

8 y z w + + =
2 3 2 x y z w + + =
0 0 11 3 21
0 0 0 19 12

19 12 12 / 19
11 3 21 (21 3 )/11 33/19 z w z w = = + =
y
0 0 0 19 12
19 12 12 / 19 w w = =
The onl sol tion of the s stem is The only solution of the system is
9/19 x
| | | |
| |
131/19
33/19
y
z
| |
| |
=
| |
| |
12/19 w
| |

\ . \ .
Linear Algebra
What we will do during the next 3 4 weeks What we will do during the next 3-4 weeks
Review of matrices
S t f li l b i ti AX B

Systems of linear algebraic equations AX=B

Finding solution(s) by row operations
Inverse of a square matrix A q
Finding inverse by row operations
Determinant of a square matrix A
C l l i d i b i Calculating determinant by row operations
Bigger pictures: how are these 3 topics connected together? Bigger pictures: how are these 3 topics connected together?
M t i i bl Matrix eigenproblem
Diagonalisation problem
Review
We were looking at solving a system
of linear algebraic equations. of linear algebraic equations.
a system of linear
algebraic equations
legitimate row
operations
algebraic equations
i j
R R
operations
a simpler but
( 0)
i i j
R R R o | o + =
a simpler but
equivalent system
E l 67
upper triangular matrix
Example on p67
Example
Solve the system (if it is consistent):
2 3 6
5 6 7 8
x y z + + =
1 2 3 6
5 6 7 8
Solve the system (if it is consistent):
5 6 7 8
8 9 10 0
x y z
x y z
+ + =
+ + =
5 6 7 8
8 9 1 0 0
1 2 3 6
0
5 R R R
4 8 22
0
0
2 2 1
5 R R R
4 8 22
3 3 1
8 R R R
7 14 48
1 2 3 6
0 4 8 22
0 7 14 48


1 2 3 6
0 4 8 2 2
0 0
3 3 2
4 7 R R R
0 38
The last row gives a nonsensical statement!
The system is inconsistent The system is inconsistent.
Example
x y z
2 3 8
5 6 7 24
x y z
x y z
+ + =
+ + =
1 2 3 8
0 4 8 16
5 6 7 24
8 9 10 36
x y z
x y z
+ +
+ + =
0 0 0 0
There is nothing wrong with the last row. It tells that
there are only 2 independent equations. The system
has infinitely many solutions has infinitely many solutions.
Let z = s (s is any arbitrary real number).
4 8 16 4 2
x s
| | | |
| |
4 8 16 4 2 y s y s = =
2(4 2 ) 3 8 x s s x s + + = =
4 2 y s
z s
| |
=
| |
| |
\ . \ .
Read note at the bottom of p70.
\ . \ .
Example
x y z w
1 2 3 2 1
0 1 0 0 2
2 3 2 1
2 5 6 4 0
x y z w
x y z w
+ + + =
+ + + =
0 0 0 0 0
0 0 0 0 0
3 7 9 6 1
3 3 2 1
x y z w
x y z w
+ + + =
+ + + =
There are only two independent equations here.
We can let two of the unknowns be any values
but second row in the final tableau tells us that
y = 2.
L t t d t
1 2
2( 2) 3 2 1 x t t + + + =
Let z=t
1
and w=t
2
.
The Big Picture e g ctu e
AX B =
original system
AX B =
legitimate row eg t ate o
operations
UX C =
simpler but equivalent
system
A homogeneous system of linear
algebraic equation is one of the form algebraic equation is one of the form
a
11
x
1
+ a
12
x
2
+ + a
1(N1)
x
N

1
+ a
1N
x
N
= 0
a
21
x
1
+ a
22
x
2
+ + a
2(N

1)
x
N

1
+ a
2N
x
N
= 0
a
31
x
1
+ a
32
x
2
+ + a
3(N 1)
x
N 1
+ a
3N
x
N
= 0 a
31
x
1
+ a
32
x
2
+ + a
3(N

1)
x
N

1
+ a
3N
x
N
0
a
N1
x
1
+ a
N2
x
2
+ + a
(N

1)
x
N

1
+ a
NN
x
N
= 0
In matrix form, it can be written as AX = 0
0
| |
where 0 = .
0
0
| |
|
|
|

0
|
|
\ .

A homogeneous system AX=0 is

always consistent.
0
| | | |
x
1
2
0
0
| | | |
| |
| |
=
| |

x
x
is a solution, no matter what A is.
0
N
| |
| |
\ . \ .

x
trivial solution
A | 0 U | 0
Depending on what A is, the homogeneous system
AX=0 has either only one (unique) solution given by
X=0 or infinitely many solutions (one of which is X=0).
Fi d ll l ti f th h t Find all solutions of the homogeneous system
2 3 0 x y z + + = 1 2 3 0
5 6 7 0
5 6 7 0
8 9 10 0
x y z
z z z
+ + =
+ + =
5 6 7 0
8 9 1 0 0
1 2 3 0
0
2 2 1
5 R R R
-4 -8 0
z t =
0
0
2 2 1
4 8 0
3 3 1
8 R R R
-7 -14 0
1 2 3 0
2
z t
y t
=
=
1 2 3 0
0 4 8 0
0

0
4 7 R R R
0 0
x t =
infinitely many
sol tions
0
0
3 3 2
4 7 R R R
0 0
solutions
A | B U | C | |
We can make the following general observations:
If ll th di l l t f U t If all the diagonal elements of U are not zero,
the system AX=B has only one solution.
(For a homogeneous system, this means X=0 ( g y ,
is the only solution.)
If at least one diagonal element of U is zero If at least one diagonal element of U is zero,
the system AX=B has either no solution or
infinitely many solutions.
(F h t thi th t (For a homogeneous system, this means that
there are infinitely many solutions, one of
which is X=0.) )
Review
Solving a system of linear algebraic
equations: equations:
a system of linear
algebraic equations
legitimate row
operations
algebraic equations
i j
R R
operations
a simpler but
( 0)
i i j
R R R o | o = +
a simpler but
equivalent system
upper triangular matrix
Linear combination of vectors
Vectors: a matrix with only one row or one Vectors: a matrix with only one row or one
column
Let u and v
1
, v
2
, , v
K1
and v
K
be vectors, each
with N elements.
We say that u is a linear combination of v
1
, v
2
,
, v
K1
and v
K
if we can find real numbers
d h th t a
1
, a
2
, , a
K1
and a
K
such that
u = a v + a v + +a v + a v u = a
1
v
1
+ a
2
v
2
+ +a
K1
v
K1
+ a
K
v
K
Example:
If possible express as a linear combination of
5
3
| |
|

|
1 3
1 0
| | | |
| |
| |
If possible, express as a linear combination of
3
4
|
|

\ .
and .
0
1
| |
|
|
1 , 0
0 1
| |
| |
\ . \ .
3
|
|
\ .
If possible, find a, b and c such that
| |
|
5
| | | | | |
| | |
1 3 0
| |
|
+ 3b a
|
|
|
\ .
-3 =
-4
| | |
| | |
| | |
\ . \ . \ .
1 0 1
0 1 3
+ b + c a
|
|
|
\ .
+
+ 3
c
b c
= a
If possible, solve:
a + 3b +0c = 5
a + 0b + c = 3
0a + b +3c = 4
Write the equations in tableau form:
1 3 0 5
a b c
1 3 0 5
0
R R R
3 1 8
1 0 1 3
0 1 3 4

0 1 3
0
4
2 2 1
R R R
3 1 8
10c = 20 c = 2
1 3 0 5
10c = 20 c = 2
3b+(2) = 8 b = 2
0 3 1
0 0
8
3 3 2
3 R R R +
10 20
a+3(2) = 5 a = 1
3 3 2
Yes we can express as a linear combination of
5
3
| |
|
|
1 3
1 0
| | | |
| |
| |
Yes, we can express as a linear combination of
3
4

|
|

\ .
and !
0
1
| |
|
|
1 , 0
0 1
| |
| |
\ . \ .
3
|
|
\ .
5 1 3 0
| | | | | | | |
Specifically, we find that:
5 1 3 0
3 ( 1) 1 2 0 ( 2) 1
| | | | | | | |
| | | |
= + +
| | | |
| | | |
4 0 1 3
| | | |
| | | |

\ . \ . \ . \ .
More examples: Problems 5 and 6 on page 86
Let w w w and w be vectors each with
Linearly independent vectors
Let w
1
, w
2
, , w
P1
and w
P
be vectors, each with
N elements.
We say that w
1
, w
2
, , w
P 1
and w
P
are linearly We say that w
1
, w
2
, , w
P1
and w
P
are linearly
independent vectors if we cannot find any one
of these vectors to be a linear combination of
the other vectors.
How do we check if w
1
, w
2
, , w
P1
and w
P
are
li l i d d ? linearly independent?
Form the homogeneous system
c
1
w
1
+ c
2
w
2
+ + c
P1
w
P1
+ c
P
w
P
= 0
If c
1
= c
2
= = c
P1
= c
P
= 0 is the only solution
of the system then the vectors are linearly
independent.
Why does this work?
How do we check if w
1
, w
2
, , w
P1
and w
P
are
linearly independent? linearly independent?
Form the homogeneous system
c
1
w
1
+ c
2
w
2
+ + c w
P 1
+ c
P
w
P
= 0 c
1
w
1
+ c
2
w
2
+ + c
P1
w
P1
+ c
P
w
P
= 0
If c
1
= c
2
= = c
P1
= c
P
= 0 is the only solution
of the system then the vectors are linearly of the system then the vectors are linearly
independent. Why does this work?
For example if we can find c which is not zero then For example, if we can find c
1
which is not zero then
we can write:
3 2 P
c c c
= w w w w
1 2 3
1 1 1
P
c c c
... = w w w w
That is, we can express w
1
as a linear combination of w
2
, w
2
, ,
and hence the ectors are not linearl independent w
P1
and w
P
, hence the vectors are not linearly independent.
p77
Are
1 2 1
1 , 1 , 0
1 1 1
| | | | | |
| | |
| | |
| | |
\ . \ . \ .
linearly independent?
1 1 1
| | |

\ . \ . \ .
1 1 2 0
| | | | | | | |
| | | |
c
1
c
2
c
3
1 2 3
0 1 1 0
1 1 1 0
c c c
| | | | | | | |
| | | |
+ + =
| | | |
| | | |

\ . \ . \ . \ .
1 1 2 0
0 1 1 0
-1 -1 -1 0
Is c
1
= c
2
= c
3
= 0 the only
solution?
1 1 2 0 Yes! 1 1 2 0
0 1 1 0
0 0 1 0
Yes!
Vectors are linearly independent.
0 0 1 0
1 2 1
1 , 2 , 2
| | | | | |
| | |
| | |
| | |
What about ?
2 4 3
| | |
| | |
\ . \ . \ .
1 2 1 0
| | | | | | | |
1 2 1 0
1 2 3
1 2 1 0
1 2 2 0
2 4 3 0
c c c
| | | | | | | |
| | | |
+ + =
| | | |
| | | |
\ . \ . \ . \ .
1 2 1 0
1 2 2 0
2 4 3 0
\ . \ . \ . \ .
1 2 1 0
0 0 1 0
0 0 0 0
c
1
=c
2
=c
3
=0 is not the only solution.
A possible non-trivial solution is:
c = 2 c = 1 and c =0
0 0 0 0
c
1
= 2, c
2
= 1 and c
3
=0.
1 2 1 0 2 1
2 1 2 0 2 0 2 2 1
| | | | | | | | | | | |
| | | | | |
+ = =
| | | | | |
Vectors are not linearly
independent.
2 1 2 0 2 0 2 2 1
2 4 3 0 4 2
+ = =
| | | | | |
| | | | | |
\ . \ . \ . \ . \ . \ .
p