ARMAX for System

Identification
Dr. Robert Oates
(Based on notes written by Prof. William Harwin)
1
Summary
Systems Identification
Least Squares Linear Regression
Example How many Harrys are
Pottering?
ARMAX Models and Least Squares Linear
Regression
Common ARMAX Variants
2
WARNING MATHS!
I tend to avoid using equations in
presentations
It is totally unavoidable here
3
Systems Identification and Modelling
Noise
Real System
System
Model
Adaptive
Component
+
+
+
-
Input
(u)
Output
(y)
Modelling Error
(e)
Model
Reparameterisation
4
Systems Identification and State
Estimation
There is an obvious connection between
Identifying unknown parameters that relate
input to output (Systems ID)
ARMAX
ARMA
ARX
Identifying unknown parameters that
represent a systems state (State Estimation)
Particle Filters
Kalman Filters
5
Systems Identification and State
Estimation
In fact both sets of algorithms use Gauss
Least Squares Linear Regression
This is an algorithm designed to minimise
the errors for a given function
6
Least Squares Linear Regression
Notation
The matrix of observed outputs of a
system
The inputs to the system
The system model parameters
The estimate of y based on a system
model and the inputs to the system
y

U
7
Least Squares Linear Regression
Calculating the estimate of the output
U y =

8
Least Squares Linear Regression
Calculating the model error
Calculating the sum-of-squares error
(SSE)
y y e

=
e e
T
=
=

=
SSE
n
i
i SSE
e
e e
1
2
9
Least Squares Linear Regression
Substituting in the definition of e
Substituting in the definition of and
transposing the 1
st
bracket
) y (y ) y (y
T

=
SSE
e
y

U) )(y U (y
T T T
=
SSE
e
10
Least Squares Linear Regression
Expanding and rearranging
( )
( ) ( ) ( ) ( ) y U U U U U y U U U
y U U U U y y y
T
1
T T
T
T
1
T
T
1
T T T

+
=
SSE
e
11
Least Squares Linear Regression
Expanding and rearranging
Ux U x K
T T
+ =
SSE
e
y U U) U(U y y y K
T 1 T T T
=
y U U) (U x
T 1 T
=
12
Least Squares Linear Regression
But wait!
If
Is the only thing we can change, the
smallest possible value of error is when
x = 0 i.e.
y U U) (U x
T 1 T
=
y U U) (U
T 1 T
=
13
LSE Example
Mr Gauss, meet Mr Potter
The National Office of Statistics keeps
track of all the baby names for certain
years (1998, 1999, 2007,2008, 2009)
In 1997 Harry Potter and the Philosophers
Stone was released
Assuming that this causes an exponential
growth in the number of Harrys can we
use the data to predict 2009s result?
14
Building the Model
Lets assume that the number is growing
exponentially and use the following model
Lets also use two model parameters,
making
U y
y
U
=
=
) log(
e
| |
0 1
b b =

15
Building the Model
As the dimensionality of U and has to
match, we pad U with 1s

(
(
(
(

=
1 2008
1 2007
1 1999
1 1998
U
16
Building the Model
To use the standard formulation of the
LSE well linearise the data by taking logs
(
(
(
(

=
|
|
|
|
|
.
|

\
|
(
(
(
(

= =
8.7008477
8.674368
8.499844
8.468213
6008
5851
4914
4761
log ) log(
'
y y
17
Calculating the Parameters
Using the equation
We can calculate the values of b
0
and b
1
that minimise the error
y U U) (U
T 1 T
=
(

=
6 36.8790619 -
0.02269839

18
Example R Code
19
Predicting the Future
Using the optimal model parameters we
can estimate how many Harrys will be
born in 2009
| |
6136

1 2009
= =

y e
Actual number of Harrys from 2009 : 6143
(from National Office for Statistics)
20
ARMAX and Recursive Least
Squares
For modelling generic systems ARMAX is
the standard
AR AutoRegressive
The current output has a relationship to the previous
values of the output
MA Moving average
The noise model used
X eXogeneous inputs
The system relies not only on the current value of the
input, but the history of inputs
21
ARMAX and Recursive Least
Squares
An ARMAX Model
i sc i sc i i
sb i sb i i
sa i sa i i
i
e e c e c e c
u b u b u b
y a y a y a
y
+ + + +
+ + +

=



...
...
...
2 2 1 1
2 2 1 1
2 2 1 1
22
ARMAX and Recursive Least
Squares

i
=
i
y
| |
T
i i i i i i
e e u u y y ... ... ...
2 1 2 1 2 1
=
i

| |
T
c c b b a a ... ... ...
2 1 2 1 2 1
=
Collecting all of these instances together gives:
e + = y
23
ARMAX and Recursive Least
Squares
Using the normal LSE for an ARMAX
model
But there are problems
Every time a new observation is added we
have to recalculate the entire thing
Were performing a matrix inversion on a big
matrix- VERY costly
y ) (
T 1 T
=
24
ARMAX and Recursive Least
Squares
Recursion to the rescue!
Can we rephrase the update into a
recursive form?
Correction
becomes
1
+ =
=

n n

y ) (
T 1 T
25
ARMAX and Recursive Least
Squares
Kalman of Kalman Filters fame provides
us with
Where:
n n n n
K c + =
1

Gain Kalman the is
n
K
n n n
y y

= c
26
ARMAX and Recursive Least
Squares
But theres still a problem!
New equations
Were still performing a massive matrix
inversion
( )
1
1
1
1
1

+ =
=
+ =
=
T
n n n n
n n n
n n n n
n
T
n n n
K
K
y
P P
P


c
c
27
ARMAX and Recursive Least
Squares
However, there is The Matrix Inversion
Lemma that allows us to convert this
inversion into a more manageable form
( )
n n
T
n
n
T
n n n
n
T
n n n n
P
P P
P
P P
1
1 1
1
1
1
1
1

+
=
+ =
28
Noise
The previous model assumes we know
what the noise terms are
This is plainly ridiculous!
However, we already compute an
estimate of the noise
We can simply substitute this in for e
n
n
c
29
Variants of ARMAX
There are two important variants of
ARMAX
ARMAX with Forgetting
Instrumental ARMAX
30
ARMAX With Forgetting
For when the systems properties are slowly
shifting
Allows less emphasis to be placed on older
observations
All equations stay the same except for P
Introduce a forgetting factor () between 0-1

1
1
1 1
1
|
|
.
|

\
|
+
=

n n
T
n
n
T
n n n
n n
P
P P
P P

31
Instrumental ARMAX
ARMAX assumes that the noise is not
changing in proportion to the input
Not true for many systems
Introduce an instrumental variable
that allows us to avoid the results
becoming skewed by assuming
independence
32
Instrumental Variables - Example
Smoking is correlated with poor health
But can we be sure that is a causal
relationship?
What if there is something which causes
both?
What if poor health causes smoking!?
33
Instrument Variables - Example
Smoking Health
Watching
Television
Adverts
Changing
The
Taxes on
Smoking
34
Instrumental Variables and
Estimators
Input
Output
F(Input,Noise)
Noise
Instrumental
Variable
Output
F(Noise(Input),
Input)
35
Instrumental ARMAX
Two of the standard ARMAX Equations
change
36
Instrumental ARMAX
Two equations change
n n
T
n
n
T
n n n
n n
n n n
K
z P
P z P
P P
z P
1
1 1
1
1

+
=
=
37
Instrumental ARMAX
All that remains is to find values for z that
are not correlated with the noise, but are
correlated with the input
A popular choice is an estimate of the
current output which uses old model
parameters so unaffected by the latest
noise
38
Summary
LSE is an effective tool for estimating
parameters that you cant directly
measure
ARMAX is a generalised model for
discrete, time-varying systems
Many different variants and techniques
available to address a variety of problems
39