Primer on Linear Algebra

Linear Algebra
Basic Concepts on Vectors and Matrices
Reading:
Many primers (check internet)

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Matrices & Vectors

Vector

1
.
= . = (1 ) = [ ]
.

1 1

, outer =
Dot product =
1
=1

Matrix

11
=
1

Stefanos Zafeiriou

1
.
= . = [ ]
.

Adv. Statistical Machine Learning(495)

Matrix Multiplication
Matrix multiplication

=1

= 1

= B=

1
1
=

1
=

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Matrix Multiplication
1
1
=[ ]

Matrix/vector multiplication = =


Identity matrix = =

1 0 0
= 0 1 0
0 0 1

3x3 Identity matrix

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Matrix Transpose
Matrix transpose
11
=
1

= 1

1
.
= .
.

Property () =
Symmetric Matrix

Example

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Trace

() =

properties

=
+ = + ()
=

in general

)(

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Determinants
=
||=

||=
7

+
(1)
| |
=1

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Determinants
Properties

= ||

+ + ||

then

=1

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Matrix Inverse
Matrix inverse
1

exists iff

Solving systems

Properties

1 = 1 =
0

= 1

()1 = 1 1
1
1
| | =
||

Stefanos Zafeiriou

1 =

Adv. Statistical Machine Learning(495)

Matrix Inverse
Inverse is only for square matrices. What happens for
non-square (pseudo-inverse)?
Solving the system

=
( )

= ( )

Pseudo-inverse + = ( )
10

Stefanos Zafeiriou

+ =

Adv. Statistical Machine Learning(495)

Rank of matrix (Definition 1)

Equal to the dimension of the largest square sub-matrix of a
matrix that has a non-zero determinant.

has rank 3

=0

11

but

4 5 2
| 3 9 6 | = 63
8 10 7

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Rank of matrix (Definition 2)

A set of vectors 1 , 2 , 3 , , are linearly independent
then 1 1 +2 2 +3 3 + + =0
iff 1 , 2 , 3 , , = 0

Alternative definition of rank: the maximum number of linearly

independent columns (or rows) of
Therefore,
rank is not 4 !

Example:

12

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Properties

If in , = iff is nonsingular (i.e.,

invertible)
If in , = iff 0 (full rank).
If in , < iff is singular.

13

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Orthogonal/Orthonormal matrices
1
=

=0 for every

= [1 ]
=I

14

= 0 for every

=L

=L

=I

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Eigenvalues & Eigenvectors

Vector x is an eigenvector of matrix and is an eigenvalue
of if:
= x

x0

Interpretation: the linear transformation implied by

cannot change the direction of the eigenvectors x, only
their magnitude.

15

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Eigenvalues & Eigenvectors

To find the eigenvalues of a matrix A, find the roots of the
characteristic polynomial:

= 0
Example:

16

5
6

2
|=0
2

2 3 + 2 = 0

Stefanos Zafeiriou

1 = 1, 2 = 2

Adv. Statistical Machine Learning(495)

Eigenvalues & Eigenvectors

Find the eigenvectors by solving the two homogenous
systems:

17

( 1 ) = 0

1/2
=
1

( 2 ) = 0

1/3
=
1

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Eigenvalues & Eigenvectors

Eigenvalues and eigenvectors are only defined for
square matrices (i.e., = )
Eigenvectors are not unique: If is an eigenvector, so is
(solution of a homogenous system)
Suppose 1 , 2 , , are the eigenvalues of , then:

= ()
=

18

= ||
=

Stefanos Zafeiriou

= 0
is singular

Adv. Statistical Machine Learning(495)

Diagonalisation
Given matrix , find P such that 1 is
diagonal (i.e., diagonalises )
Take = 1 , 2 , , , where 1 , 2 , ,
are the eigenvectors of

19

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Diagonalisation
Are all matrices diagonalizable with
1 , 2 , , > 0?
Only if 1 exists (i.e., must have linearly independent
eigenvectors, that is, = )
If has n distinct eigenvalues 1 , 2 , , , then the
corresponding eigen-vectors 1 , 2 , , form a basis:
(1) linearly independent
(2) span

20

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Diagonalisation
Symmetric Matrices
The eigenvalues of symmetric matrices are all real.
The eigenvectors corresponding to distinct eigenvalues
are orthogonal. 1 =

= =

( )
=1

21

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)

Properties in the white board

22

Stefanos Zafeiriou

Adv. Statistical Machine Learning(495)