System Identification

&
Parameter Estimation
Wb2301: SIPE lecture 2
Correlation functions in time & frequency domain
Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE
2/11/2010

Delft
University of
Technology

Contents
Correlation functions in time domain
Autocorrelations (Lecture 1)
Cross-correlations

General properties of estimators

Bias / variance / consistency

Estimators for correlation functions

Fourier transform
Correlation functions in frequency domain
Auto-spectral density
Cross-spectral density
Estimators for spectral densities
SIPE, lecture 2

2 | xx

Lecture 1: (auto)correlation functions

Autocorrelation function

xx ( ) = E [ x (t )x(t )]
Autocovariance function

Cxx ( ) = E ( x (t ) x )( x (t ) x ) = xx ( ) x2

Autocorrelation coefficient

x (t ) x x (t ) x xx ( ) x2 Cxx ( )
=
rxx ( ) = E

=
2
x
x
Cxx (0)
x

SIPE, lecture 2

3 | xx

Lecture 1: autocovariance function

Autocovariance function:

Cxx ( ) = E ( x(t ) x )( x (t ) x )
= E [ x(t )x (t )] E [ x (t ) x ] E [ x x (t )] + E x2
= xx ( ) x2 x2 + x2
= xx ( ) x2
If = 0, then autocovariance and autocorrelation functions are identical
At zero lag:

Cxx (0) = E ( x (t ) x ) = x2

SIPE, lecture 2

4 | xx

Example autocovariances
signal

white

5
2
Matlab
demo: Lec2_example_Cuu.m
0
-5

0
0

low-pass

1
0
-1

sinusoid
sin. + noise

-2
-2
0.05

-0.05
-2
0.5

-1

-1

-1

0
tau [s]

-0.5
-2
2

-5

-1

-1

Cuu

2
3
time [s]

-2
-2

SIPE, lecture 2

5 | xx

2D Probability density function

fxy ( x, y; ) dxdy = Pr{x < x (t ; ) x + dx y < y (t + ; ) y + dy}

SIPE, lecture 2

6 | xx

2D Probability density function

Probability for certain values of y(t) given certain values of x(t)
Co-variance of y(t) with x(t): y(t) is related with x(t)
No co-variance between y(t) and x(t): 2D probability density function
is circular
Covariance between y(t) and x(t): 2D probability density function is
ellipsoidal

No co-variance between y(t) and x(t):

y(t) and x(t) are independent
no relation exist
transfer function is zero !

SIPE, lecture 2

7 | xx

Cross-correlation functions
Measure for common structure in two signals:
Cross-correlation

xy ( ) = E [ x (t )y (t )]

Cross-covariance

Cxy ( ) = E ( x (t ) x ) ( y (t ) y ) = xy ( ) x y
Cross-correlation coefficient

x (t )
x
rxy ( ) = E
x

y (t ) y

y

Cxy ( )
=
Cxx (0)Cyy (0)

SIPE, lecture 2

8 | xx

Example: Estimation of a Time Delay

Consider the system:

y (t ) = x (t 0 ) + v (t )
Additive noise v(t) is stochastic and uncorrelated to x(t),

Cross-covariance of the measured input and output in case:

input x(t) is white noise (stochastic)

SIPE, lecture 2

9 | xx

Example: Estimation of a Time Delay

u
5

u(t)

Matlab demo: Lec2_example_Cuy_delay.m

0
-5

10

15

20

25
time [s]
u

30

35

40

45

50

10

15

20

25
time [s]

30

35

40

45

50

y(t)

5
0
-5

Cuy ()

1
0
-1
-2

-1.5

-1

-0.5

0
[s]

0.5

1.5

SIPE, lecture 2

10 | xx

Effects of Noise on Autocorrelations

Often, correlation functions must be estimated from measurements.
Consider x (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.:
w ( t ) = x (t ) + n (t )
z(t ) = y (t ) + v (t )

Noises have zero mean and are independent of the signals and of each
other.
Autocorrelation: zz ( ) = E ( y (t ) + v (t ))( y (t ) + v (t ) )
= yy ( ) + yv ( ) + vy ( ) + vv ( )
= yy ( ) + vv ( )

Additive noise will bias autocorrelation functions!

SIPE, lecture 2

11 | xx

Effects of Noise on Cross-correlations

Cross-correlation: wz ( ) = E ( x (t ) + n(t ) )( y (t ) + v (t ) )

= xy ( ) + xv ( ) + ny ( ) + nv ( )
= xy ( )
Additive noise will not bias cross-correlation functions!

SIPE, lecture 2

12 | xx

Noise Reduction
Longer recordings
More information, same amount of noise

Repeat the experiment

Noise cancels out by averaging from exactly the same inputs

Improve Signal-to-Noise Ratio

Concentrate power at specified frequencies, assuming the noise
power remains the same

SIPE, lecture 2

13 | xx

Properties of Estimators
Formula given (autocovariances / spectral densities) are
estimators for the true relations.
What are the properties of these estimators?
bias / variance / consistency

Bias: structural error

Variance: random error
Consistent: A consistent estimator is an estimator that converges,
in probability, to the quantity being estimated as the sample size
grows.

SIPE, lecture 2

14 | xx

Example: estimator for mean value

and variance of a signal
Signal xk , with k=1N
Estimator for the signal mean:

1 N
x = xk
N k =1

1 N
2
Estimator for the signal variance: = ( xk x )
N k =1
2
x

What are the expected values of both estimators?

Expectation operator: E{.}

SIPE, lecture 2

15 | xx

Estimator for the mean value

1 N
x = xk
N k =1
1 N 1 N
E{ x } = E xk = E{xk }
N k =1 N k =1
E{xk } = E{x} = x
1 N
E{ x } = x = x
N k =1
SIPE, lecture 2

16 | xx

Estimator for the variance

2

1 N

( xk x )

xk N xl
k =1
k =1
l =1

1 N 2 2 N
1 N N

= xk xk xl + 2 xl xk
N k =1
N l =1
N l =1 k =1

1
x2 =
N

1
=
N

2 N
1 N N

+
E
x
E
x
x
E
x
x
{
}
{
}

k l
l k
{ k } N
2
N
k =1
l =1
l =1 k =1

1 N 2
2 2 N
1 2 N N

2
2
2
E { x } = x + x x K xk xl 2 x + 2 x K xk xl + x2
N k =1
N
N
l =1
l =1 k =1

1
E { } =
N

2
x

1
1 N 1 2
2

E { x2 } = x2 1 1 + 2 N = x2 1 =
x
N
N

N N
Estimator is biased and consistent
SIPE, lecture 2

17 | xx

Estimator for the variance

1

(x )

=
biased estimator:

N
N

2
x

k =1

1 N 1 2

E x2 = x2 1 =
x
N N

{ }

unbiased estimator:
(default!)

1 N
2

(
)
=
x

k x
N 1 k =1
2
x

{ }

E x2 = x2

SIPE, lecture 2

18 | xx

Estimates of Correlation Functions

Cross-correlation:

xy ( ) = E [ x(t )y (t )]

Let x(t ) and y(t ) are finite time realizations of ergodic

T
processes. Then:
1

xy ( ) = lim

2T

x(t )y (t )dt

Practically, signals are finite time and sampled every t.

Signal x(t) is sampled at t=0, t, , (N-1)t
giving x(i) with i=1, 2, , N.
Then:
xy ( ) = 1
N

x(i )y (i )
i=

with i =1, 2, , N
SIPE, lecture 2

19 | xx

Estimates of Correlation Functions

Unbiased estimator:

xy ( ) = 1

x(i )y (i )
i=

Variance of the estimator increases with lag!

To avoid this, divide by N (biased estimator):
N
1
xy ( ) = x(i )y (i )

N i =

Use large N to minimize bias! I.e.

N
1
N

Similar estimators can be derived for the covariance and

correlation coefficient functions
SIPE, lecture 2

20 | xx

Additional demos
Calculation of cross-covariance: Lec2_calculation_Cuy.m
N
1
xy ( ) = x (i )y (i )

N i =

SIPE, lecture 2

21 | xx

Cross-covariance of some basic systems

Example systems (Matlab Demo: Lec2_example_systems.m):
Time delay
1st order
2nd order

Input signals:
White noise (WN)
Low pass filtered noise (LP: 1/(0.5s+1) )

SIPE, lecture 2

22 | xx

1st order system

Cuy (WN)

time delay
1

0.02

0.5

0.01

0
0

0.1
Cuy (LP)

0.05

-0.5
-5

-0.01
-5

0.04

-0.05
-5

0
time [s]

0.2

0.02
0

0
0

-0.1
-5

1
impulse

2nd order system

-0.02
-5

-0.2
-5
10
5

0.5

1
0

0
-5

0
time [s]

0
-5

0
time [s]

-5
-5

SIPE, lecture 2

23 | xx

Intermezzo: Fourier Transform

Fourier Transform:

X ( f ) = ( x (t )) =

x ( t ) e j 2 ft dt

Mapping between time-domain and frequency-domain

One-to-one mapping
Unique: inverse Fourier Transform exists
Linear technique

Inverse Fourier Transform:

x(t ) = 1 ( X ( f ) ) =

X ( f ) e j 2 tf df

SIPE, lecture 2

25 | xx

Fourier transformation
y(t) is an arbitrary signal

Y ( f ) = y (t )e j 2 ft dt
Euler formula: e

jz

= cos ( z ) + j sin ( z )

symmetric part: cos(z)

anti-symmetric part: sin(z)

e j 2 ft = re(e j 2 ft ) + im(e j 2 ft ) = cos(2 ft ) j *sin(2 ft )

SIPE, lecture 2

26 | xx

Example Fourier Transformation

Y ( f ) = y (t )e j 2 ft dt
y (t ) = sin (0.5 Hz ) + sin (5 Hz )

Y(0.5Hz): 5 Hz signal will be averaged out

1.5

0.5

-0.5

-1

-1.5
0

0.5

1.5

2.5

3.5

4.5

SIPE, lecture 2

27 | xx

Example Fourier Transformation

Y() = (-0.5)

y(t) = sin(0.5t)
1.5

5
4.5

1
4
3.5

0.5

2.5
2

-0.5
1.5
1

-1

0.5

-1.5
0

0.5

1.5

2.5

3.5

4.5

0
-0.5

0.5

1.5

SIPE, lecture 2

28 | xx

Frequency Domain Expressions

Discrete Fourier Transform:
N

U (f ) = ( u(t ) ) = u(t )e

j 2

ft
N

t =1

where f takes values 0, 1, , N-1 multiples of

f =

1
N t

Inverse Fourier Transform:

ft
N
j 2
1
u(t ) = 1 (U (f ) ) = U (f )e N
N f =1

SIPE, lecture 2

29 | xx

Fourier transform of signals

Discrete Fourier Transform (DFT)

u(k );

k [0,1,, N 1]
N 1

U (r ) = DFT{u(k )} = u(k )e j 2rk / N

k =0

U (r );

r [0,1,, N 1]

DFT maps N real values in time domain to N complex values in

frequency domain
Double information?
DFT is symmetric U(-r)=U(r)*
Information for N/2 complex values

r 0,1, ,
2

SIPE, lecture 2

30 | xx

Power spectrum or
auto-spectral density
auto-spectral density Sxx is Fourier Transform of autocorrelation

S xx ( ) = ( xx ( ) ) =

)
e
d
xx

S xx ( f ) = ( xx ( ) ) =

2 jf

(
)
e
d

xx

properties:
Real values (no imaginary part)
Symmetry: Sxx(f)=Sxx(-f) => only positive frequencies are analyzed
Cxx(0) = Sxx(f) df = x2
Area under Sxx(f) is equal to variance of signal (Parsevals theorem)
SIPE, lecture 2

31 | xx

Cross-spectrum or
Cross-spectral density
cross-spectral density Sxy:

S xy ( f ) = ( xy ( ) ) =

j 2 f

)
e
d
xy

properties:
Complex values
Sxy*(f)=Sxy(-f) => only positive frequencies are analyzed
Sxy(f) describes the interdependency of signals x(t) and y(t) in
frequency domain (gain and phase)
if xy() = 0, then Sxy(f) = 0 for all frequencies

SIPE, lecture 2

32 | xx

Estimators for the power spectrum

Fourier transform of the autocorrelation function, the power spectrum:
N 1

uu ( )e
Suu (f ) =

j 2

f
N

=0

1 N

using the time average estimate: uu ( ) = u(i )u(i )

N i =
and multiplication with e

j 2

( i i )f
N

= e0 = 1

( i )f N
if
N 1
j 2
j 2
1
N
gives: S (f ) =
u(i )e
u(i )e N

uu
N =0
i =

1 *
U (nf )U (nf )
N

Indirect approach
and direct approach
give equal result!
* : complex conjugate:

U * (nf ) = U ( nf )

SIPE, lecture 2

33 | xx

Estimator for the cross-spectrum

Fourier transform of the cross-correlation function is called the cross-spectrum:
N 1

uy ( )e
Suy (f ) =

j 2

f
N

=0

1 *

Suy (f ) = U (nf )Y (nf )

N

SIPE, lecture 2

34 | xx

Time-domain vs. Frequency-domain

Time Domain

input, output

Fourier
Transformation

x(t), y(t)

cross-correlation
function

xy()

cross-covariance
function

Cxy()

correlation
coefficient

rxy()

Frequency Domain

X(f), Y(f)

Sxy(f)

xy(f)

input, output
cross-spectral
density

coherence

SIPE, lecture 2

35 | xx

Discrete and Continuous Signals

Reconstruction
DA conversion
Distortion: Hold Circuits

Sampling
AD conversion
Distortion: "Dirac Comb"

Continuous system: Plant

Digital Controller: Plant performance enhancement
SIPE, lecture 2

36 | xx

Sampling, Diracs Comb

SIPE, lecture 2

37 | xx

Models of Linear Systems

System:

u(t)

y (t ) = N (u(t ))
N

y(t)

Linear systems obey both scaling and superposition property!

k1y1 (t ) = N (k1u1 (t ))
k 2 y 2 (t ) = N (k 2u2 (t ))
k1y1 (t ) + k 2 y 2 (t ) = N (k1u1 (t ) + k 2u2 (t ))

SIPE, lecture 2

38 | xx

Time domain models

Assume input is a pulse with:
1
t
for
t
0
<

d ( t , t ) = t
2
0
otherwise

Response of linear system N to

pulse:

N ( d ( t , t ) ) = h ( t , t )

Assume u(t) is weighted sum of

pulses:

u (t ) =

u d ( t k t , t )

k =

SIPE, lecture 2

39 | xx

Time domain models

The output is:

y (t ) = N (u(t ))
=

u N ( d ( t k t, t ) )

k =

u h ( t k t, t )

k =

Limit case: unit-area pulse d(t,t) is an impulse: lim d ( t , t ) = ( t )

t 0

Then h(t) is the impulse response function (IRF) and the output
the convolution integral

y (t ) =

h ( ) u ( t )d

SIPE, lecture 2

40 | xx

Time domain models

Causal system: h(t)=0 for t<0
Finite memory: h(t)=0 for t>T
T

y (t ) = h ( ) u ( t )d
o

Discrete

T 1

y (t ) = h ( ) u ( t )
=0

SIPE, lecture 2

41 | xx

Frequency domain models

Time-domain: convolution integral

Fourier transform:

y (t ) =

h ( ) u ( t )d

Y (f ) = ( y (t ) ) = h ( ) u ( t )d

j 2 ft
dt
= h ( ) u ( t )d e

2 ft
h

u
t

e
dtd

(
)
(
)

Convolution in time =domain

U (f ) ish (multiplication
) e 2 f d in
frequency domain (and vice-versa)

Y (f ) = H (f )U (f )
SIPE, lecture 2

42 | xx

Estimators in time domain

Cross-covariance
When analyzing the dynamics of a dynamical system interested in
variations of the signals around it mean (and not average value).

Assuming signals with zero-mean

y (t ) = (h u )(t ) =

h(t ')u (t t ')dt '

Cuy ( ) = E {u ( t ) y ( t )} = E h(t ')u (t )u (t t ')dt '

SIPE, lecture 2

43 | xx

Basic identification with crosscovariance

n(t)
?

y(t)

u(t)
y ( t ) = n ( t ) + h ( t ') u ( t - t' ) dt'

multiply with u (t ): u ( t ) y ( t ) = u ( t ) n ( t ) + h ( t ') u (t )u ( t - t' ) dt'

Cuy ( ) = Cun ( ) + h ( t' )Cuu ( t ') dt '
white noise:

Cuu ( ) = 0 for 0;

Cuu ( 0 ) = 1

Cuy ( ) = Cun ( ) + h ( )
Other 'tricks' needed when u(t) is not white
SIPE, lecture 2

44 | xx

Basic identification with spectral

densities
n(t)
?

y(t)

u(t)
y ( t ) = n ( t ) + h ( t ') u ( t - t' ) dt'

multiply with u (t ): u ( t ) y ( t ) = u ( t ) n ( t ) + h ( t ') u (t )u ( t - t' ) dt'

Cuy ( ) = Cun ( ) + h ( t' )Cuu ( t ' ) dt '
Fourier transform:

Suy ( f ) = Sun ( f ) + H ( f ) Suu ( f )

if Sun ( f ) = 0 :

H(f )=

Suy ( f )

Suu ( f )

SIPE, lecture 2

45 | xx

Identification:
time-domain vs. frequency-domain
n(t)
?
u(t)

y(t)

Time domain
Unknown system: impulse response function (IRF) of h(t) over
limited time
Mostly: direct model parameterization (fit of physics model)

Frequency domain
Unknown system: frequency response function (FRF) H(f) for number
of frequencies

SIPE, lecture 2

46 | xx

Example IRF & FRF of typical

systems

Time delay
First order system
Second order system
Unstable system

(try Matlab!)

SIPE, lecture 2

47 | xx

Book: Westwick & Kearney

Lecture 1: signals
Chapter 1, all
Chapter 2, sec. 2.1 2.3.1

Lecture 2: Correlation functions in time & frequency domain

Chapter 2, sec. 2.3.2 2.3.4
Chapter 3, sec. 3.1 3.2

Lecture 3: Estimators for impulse & frequency response functions

Chapter 5, sec. 5.1 5.3

SIPE, lecture 2

48 | xx

Next week: lecture 3

Lecture 3:
Estimation of IRF and FRF
Estimator for coherence
Open loop and closed loop system identification

SIPE, lecture 2

49 | xx