Estimation Theory

Estimation Theory
(Introduction)
Estimation Theory
Course Objective
Extract information from noisy signals
Parameter Estimation Problem : Given a set of measured data
{x[0], x[1], . . . , x[N 1]}
which depends on an unknown parameter vector , determine an estimator
= g (x[0], x[1], . . . , x[N 1])
where g is some function.
Applications : Image processing, communications, biomedicine, system

identication, state estimation in control, etc.
(Introduction)
Estimation Theory
Some Examples
Range estimation : We transmit a pulse that is reected by the aircraft.
An echo is received after second. Range is estimated from the equation
= c/2 where c is the lights speed.
System identication : The input of a plant is excited with u and the
output signal y is measured. If we have :
y [k] = G (q 1 , )u[k] + n[k]
where n[k] is the measurement noise. The model parameters are estimated
using u[k] and y [k] for k = 1, . . . , N.
DC level in noise : Consider a set of data {x[0], x[1], . . . , x[N 1]} that
can be modeled as :
x[n] = A + w [n]
where w [n] is some zero mean noise process. A can be estimated using the
measured data set.
(Introduction)
Estimation Theory
Outline
Classical estimation ( deterministic)
Minimum Variance Unbiased Estimator (MVU)
Cramer-Rao Lower Bound (CRLB)
Best Linear Unbiased Estimator (BLUE)
Maximum Likelihood Estimator (MLE)
Least Squares Estimator (LSE)
Bayesian estimation ( stochastic)
Minimum Mean Square Error Estimator (MMSE)
Maximum A Posteriori Estimator (MAP)
Linear MMSE Estimator
Kalman Filter
(Introduction)
Estimation Theory
References
Main reference :
Fundamentals of Statistical Signal Processing

Estimation Theory
by Steven M. KAY, Prentice-Hall, 1993 (available in Library de La
Fontaine, RLC). We cover Chapters 1 to 14, skipping Chapter 5 and
Chapter 9.
Other references :
Lessons in Estimation Theory for Signal Processing, Communications
and Control. By Jerry M. Mendel, Prentice-Hall, 1995.
Probability, Random Processes and Estimation Theory for Engineers.
By Henry Stark and John W. Woods, Prentice-Hall, 1986.
(Introduction)
Estimation Theory
Review of Probability and Random Variables
(Probability and Random Variables)
Estimation Theory
Random Variables
Random Variable : A rule X () that assigns to every element of a sample
space a real value is called a RV. So X is not really a variable that varies
randomly but a function whose domain is and whose range is some
subset of the real line.
Example : Consider the experiment of ipping a coin twice. The sample
space (the possible outcomes) is :
= {HH, HT , TH, TT }
We can dene a random variable X such that
X (HH) = 1, X (HT ) = 1.1, X (TH) = 1.6, X (TT ) = 1.8
Random variable X assigns to each event (e.g. E = {HT , TH} ) a
subset of the real line (in this case B = {1.1, 1.6}).
Estimation Theory
Probability Distribution Function

For any element in , the event {|X () x} is an important event.
The probability of this event
Pr [{|X () x}] = PX (x)
is called the probability distribution function of X .
Example : For the random variable dened earlier, we have :
PX (1.5) = Pr [{|X () 1.5}] = Pr [{HH, HT }] = 0.5
PX (x) can be computed for all x R. It is clear that 0 PX (x) 1.
Remark :
For the same experiment (throwing a coin twice) we could dene
another random variable that would lead to a dierent PX (x).
In most of engineering problems the sample space is a subset of the
real line so X () = and PX (x) is a continuous function of x.
Estimation Theory
Probability Density Function (PDF)

The Probability Density Function, if it exists, is given by :
dPX (x)
dx
When we deal with a single random variable the subscripts are removed :
pX (x) =
p(x) =
Properties :
(i )
(ii )
(iii )
dP(x)
dx
p(x)dx = P() P() = 1

Pr [{|X () x}] = Pr [X x] = P(x) =

x2
Pr [x1 < X x2 ] =
p(x)dx
x1
Estimation Theory
p()d
Gaussian Probability Density Function

A random variable is distributed according to a Gaussian or normal
distribution if the PDF is given by :
p(x) =
1
2 2
(x)2
22
The PDF has two parameters : , the mean and 2 the variance.
We note X N (, 2 ) when the random variable X has a normal
(Gaussian) distribution with the mean and the standard deviation .
Small means small variability (uncertainty) and large means large
variability.
Remark : Gaussian distribution is important because according to the
Central Limit Theorem the sum of N independent RVs has a PDF that
converges to a Gaussian distribution when N goes to innity.
Estimation Theory
Some other common PDF

Uniform (b > a) :
p(x) =
Exponential ( > 0) :
p(x) =
Rayleigh ( > 0) :
p(x) =
Chi-square 2 :
p(x) =
where (z) =
1
ba
a<x <b
otherwise
1
exp(x/)u(x)
x
x 2
exp(
)u(x)
2
2 2

1
x n/21 exp( x2 ) for x > 0
2n (n/2)
0
for x < 0
t z1 e t dt and u(x) is the unit step function.
Estimation Theory
Joint, Marginal and Conditional PDF

Joint PDF : Consider two random variables X and Y then :
x2 y2
Pr [x1 < X x2 and y1 < Y y2 ] =
p(x, y )dxdy
x1
Marginal PDF :

p(x) =
y1
p(x, y )dy
and
p(y ) =
p(x, y )dx
Conditional PDF : p(x|y ) is dened as the PDF of X conditioned on

knowing the value of Y .
Bayes Formula : Consider two RVs dened on the same probability space
then we have :
p(x, y )
p(x, y ) = p(x|y )p(y ) = p(y |x)p(x)
or
p(x|y ) =
p(y )
Estimation Theory
Independent Random Variables

Two RVs X and Y are independent if and only if :
p(x, y ) = p(x)p(y )
A direct conclusion is that :

p(x|y ) =
p(x, y )
p(x)p(y )
=
= p(x)
p(y )
p(y )
and
p(y |x) = p(y )
which means conditioning does not change the PDF.

Remark : For a joint Gaussian pdf the contours of constant density is an
ellipse centered at (x , y ). For independent X and Y the major (or
minor) axis is parallel to x or y axis.
Estimation Theory
Expected Value of a Random Variable

The expected value, if it exists, of a random variable X with PDF p(x) is
dened by :

E (X ) =
xp(x)dx
Some properties of expected value :

E {X + Y } = E {X } + E {Y }
E {aX } = aE {X }
The expected value of Y = g (X ) can be computed by :

E (Y ) =
g (x)p(x)dx
Conditional expectation : The conditional expectation of X given that a

specic value of Y has occurred is :

xp(x|y )dx
E (X |Y ) =
Estimation Theory
Moments of a Random Variable

The r th moment of X is dened as :

E (X r ) =
x r p(x)dx
The rst moment of X is its expected value or the mean ( = E (X )).

Moments of Gaussian RVs : A Gaussian RV with N (, 2 ) has
moments of all orders in closed form
E (X )
E (X 2 )
E (X 3 )
E (X 4 )
E (X 5 )
E (X 6 )
=
=
=
=
=
=
2 + 2
3 + 3 2
4 + 62 2 + 3 4
5 + 103 2 + 15 4
6 + 154 2 + 452 4 + 15 6
Estimation Theory
Central Moments
The r th central moment of X is dened as :

r
(x )r p(x)dx
E [(X ) ] =
The second central moment (variance) is denoted by 2 or var(X ).

Central Moments of Gaussian RVs :

0
if r is odd
r
E [(X ) ] =
r (r 1)!! if r is even
where n!! denotes the double factorial that is the product of every odd
number from n to 1.
Estimation Theory
Some properties of Gaussian RVs

If X N (x , x2 ) then Z = (X x )/x N (0, 1).
If Z N (0, 1) then X = x Z + x N (x , x2 ).
If X N (x , x2 ) then Z = aX + b N (ax + b, a2 x2 ).
If X N (x , x2 ) and Y N (y , y2 ) are two independent RVs, then
aX + bY N (ax + by , ax2 + by2 )
The sum of square of n independent RV with standard normal
distribution N (0, 1) has a 2n distribution with n degree of freedom.
For large value of n, 2n converges to N (n, 2n).
The Euclidian norm X 2 + Y 2 of two independent RVs with

standard normal distribution has the Rayleigh distribution.
Estimation Theory
Covariance
For two RVs X and Y , the covariance is dened as

xy =
xy = E [(X x )(Y y )]

(x x )(y y )p(x, y )dxdy
If X and Y are zero mean then xy = E {XY }.

var(X + Y ) = x2 + y2 + 2xy
var(aX ) = a2 x2
Important formula : The relation between the variance and the mean of
X is given by
2 = E [(X )2 ] = E (X 2 ) 2E (X ) + 2
= E (X 2 ) 2
The variance is the mean of the square minus the square of the mean.
Estimation Theory
Independence, Uncorrelatedness and Orthogonality

If xy = 0, then X and Y are uncorrelated and
E {XY } = E {X }E {Y }
X and Y called orthogonal if E {XY } = 0.
If X and Y are independent then they are uncorrelated.
p(x, y ) = p(x)p(y ) E {XY } = E {X }E {Y }
Uncorrelatedness does not imply the independence. For example, if X
is a normal RV with zero mean and Y = X 2 we have p(y |x) = p(y )
but
xy = E {XY } E {X }E {Y } = E {X 3 } 0 = 0
The correlation only shows the linear dependence between two RV so
is weaker than independence.
For Jointly Gaussian RVs, independence is equivalent to being
uncorrelated.
Estimation Theory
Random Vectors
Random Vector : is a vector of random variables 1 :
x = [x1 , x2 , . . . , xn ]T
Expectation Vector : x = E (x) = [E (x1 ), E (x2 ), . . . , E (xn )]T
Covariance Matrix : Cx = E [(x x )(x x )T ]
Cx is an n n symmetric matrix which is assumed to be positive
denite and so invertible.
The elements of this matrix are : [Cx ]ij = E {[xi E (xi )][xj E (xj )]}.
If the random variables are uncorrelated then Cx is a diagonal matrix.
Multivariate Gaussian PDF :

1
1
T 1
exp (x x ) Cx (x x )
p(x) =
2
(2)n det(Cx )
1. In some books (including our main reference) there is no distinction between random
variable X and its specific value x. From now on we adopt the notation of our reference.
Estimation Theory
Random Processes
Discrete Random Process : x[n] is a sequence of random variables
dened for every integer n.
Mean value : is dened as E (x[n]) = x [n].
Autocorrelation Function (ACF) : is dened as
rxx [k, n] = E (x[n]x[n + k])
Wide Sense Stationary (WSS) : x[n] is WSS if its mean and its
autocorrelation function (ACF) do not depend on n.
Autocovariance function : is dened as
cxx [k] = E [(x[n] x )(x[n + k] x )] = rxx [k] 2x
Cross-correlation Function (CCF) : is dened as
rxy [k] = E (x[n]y [n + k])
Cross-covariance function : is dened as
cxy [k] = E [(x[n] x )(y [n + k] y )] = rxy [k] x y
Estimation Theory
Discrete White Noise

Some properties of ACF and CCF :
rxx [0] |rxx [k]|
rxx [k] = rxx [k]
rxy [k] = ryx [k]
Power Spectral Density : The Fourier transform of ACF and CCF gives
the Auto-PSD and Cross-PSD :
Pxx (f ) =
Pxy (f ) =

k=
rxx [k] exp(j2fk)

rxy [k] exp(j2fk)
k=
Discrete White Noise : is a discrete random process with zero mean and
rxx [k] = 2 [k] where [k] is the Kronecker impulse function. The PSD of
white noise becomes Pxx (f ) = 2 and is completely at with frequency.
Estimation Theory
Introduction and Minimum Variance Unbiased

Estimation
(Minimum Variance Unbiased Estimation)
Estimation Theory
The Mathematical Estimation Problem

Parameter Estimation Problem : Given a set of measured data
x = {x[0], x[1], . . . , x[N 1]}
which depends on an unknown parameter vector , determine an estimator
= g (x[0], x[1], . . . , x[N 1])
where g is some function.
The rst step is to nd the PDF of data as a

function of : p(x; )
Example : Consider the problem of DC level in white Gaussian noise with
one observed data x[0] = + w [0] where w [0] has the PDF N (0, 2 ).
Then the PDF of x[0] is :

1
1
2
exp 2 (x[0] )
p(x[0]; ) =
2
2 2
Estimation Theory

Example : Consider a data sequence that can be modeled with a linear
trend in white Gaussian noise
x[n] = A + Bn + w [n]
n = 0, 1, . . . , N 1
Suppose that w [n] N (0, 2 ) and is uncorrelated with all the other
samples. Letting = [A B] and x = [x[0], x[1], . . . , x[N 1]] the PDF is :

N1
N1

1
1
2
p(x; ) =
p(x[n]; ) =
exp 2
(x[n] A Bn)
2
( 2 2 )N
n=0
n=0
The quality of any estimator for this problem is related to the assumptions
on the data model. In this example, linear trend and WGN PDF
assumption.
Estimation Theory

Classical versus Bayesian estimation
If we assume is deterministic we will have a classical estimation
problem. The following method will be studied : MVU, MLE, BLUE,
LSE.
If we assume is a random variable with a known PDF, then we will
have a Bayesian estimation problem. In this case the data are
described as the joint PDF
p(x, ) = p(x|)p()
where p() summarizes our knowledge about before any data is
observed and p(x|) summarizes our knowledge provided by data x
conditioned on knowing . The following methods will be studied :
MMSE, MAP, Kalman Filter.
Estimation Theory
Assessing Estimator Performance

Consider the problem of estimating a DC level A in uncorrelated noise :
x[n] = A + w [n]
n = 0, 1, . . . , N 1
Consider the following estimators :

N1

1 = 1
A
x[n]
N
n=0
2 = x[0]
A
1 = 0.95 and A
2 = 0.98. Which estimator is better ?
Suppose that A = 1, A
An estimator is a random variable, so its

performance can only be described by its PDF or
statistically (e.g. by Monte-Carlo simulation).
Estimation Theory
Unbiased Estimators
An estimator that on the average yield the true value is unbiased.
Mathematically
=0
E ( )
for a < < b
1 and A
2 :
Lets compute the expectation of the two estimators A
N1
N1
N1
1
1
1
E (x[n]) =
E (A + w [n]) =
(A + 0) = A
E (A1 ) =
N
N
N
n=0
n=0
n=0
2 ) = E (x[0]) = E (A + w [0]) = A + 0 = A
E (A
Both estimators are unbiased. Which one is better ?
Now, lets compute the variance of the two estimators :
N1
N1

1
1
1
2
1 ) = var
var(A
x[n] = 2
var(x[n]) = 2 N 2 =
N
N
N
N
n=0
n=0
2 ) = var(x[0]) = 2 > var (A

1)
var(A
Estimation Theory
Unbiased Estimators
Remark : When several unbiased estimators of the same parameters from
independent set of data are available, i.e., 1 , 2 , . . . , n , a better estimator
can be obtained by averaging :
1
i
=
n
n
=
E ()
i =1
Assuming that the estimators have the same variance, we have :

n
1
1
var(i )
var() = 2
var(i ) = 2 n var(i ) =
n
n
n
i =1
By increasing n, the variance will decrease (if n , ).

It is not the case for biased estimators, no matter how many estimators are
averaged.
Estimation Theory
Minimum Variance Criterion

The most logical criterion for estimation is the Mean Square Error (MSE) :
= E [( )2 ]
mse()
Unfortunately this type of estimators leads to unrealizable estimators (the
estimator will depend on ).
= E {[ E ()
+ E ()
]2 } = E {[ E ()
+ b()]2 }
mse()
is dened as the bias of the estimator. Therefore :
where b() = E ()
= E {[ E ()]
2 } + 2b()E [ E ()]
+ b 2 () = var()
+ b 2 ()
mse()
Instead of minimizing MSE we can minimize the variance of

the unbiased estimators :
Minimum Variance Unbiased Estimator
Estimation Theory
Minimum Variance Unbiased Estimator
Existence of MVU Estimator : In general MVU estimator does not

always exist. There may be no unbiased estimator or non of unbiased
estimators has a uniformly minimum variance.
Finding the MVU Estimator : There is no known procedure which
always leads to the MVU estimator. Three existing approaches are :
1
Determine the Cramer-Rao lower bound (CRLB) and check to see if

some estimator satises it.
Apply the Rao-Blackwell-Lehmann-Schee theorem (we will skip it).
Restrict to linear unbiased estimators.
Estimation Theory
Spring 2013
31 / 152
Cramer-Rao Lower Bound
(Cramer-Rao Lower Bound)
Estimation Theory
CRLB is a lower bound on the variance of any unbiased estimator.

CRLB()
var()
Note that the CRLB is a function of .
It tells us what is the best performance that can be achieved
(useful in feasibility study and comparison with other estimators).
It may lead us to compute the MVU estimator.
Estimation Theory

Theorem (scalar case)
Assume that the PDF p(x; ) satises the regularity condition

ln p(x; )
E
=0
for all
Then the variance of any unbiased estimator satises

2
1
ln p(x; )
var() E
2
An unbiased estimator that attains the CRLB can be found i :
ln p(x; )
= I ()(g (x) )
for some functions g (x) and I (). The estimator is = g (x) and the
minimum variance is 1/I ().
Estimation Theory

Example : Consider x[0] = A + w [0] with w [0] N (0, 2 ).

1
1
2
p(x[0]; A) =
exp 2 (x[0] A)
2
2 2
1
ln p(x[0], A) = ln 2 2 2 (x[0] A)2
2
Then
ln p(x[0]; A)
1
= 2 (x[0] A)
A
2 ln p(x[0]; A)
1
= 2
2
A
According to Theorem :
2
var (A)
and
I (A) =
1
2
= g (x[0]) = x[0]
and A
Estimation Theory

Example : Consider multiple observations for DC level in WGN :
x[n] = A + w [n]
n = 0, 1, . . . , N 1
with w [n] N (0, 2 )
N1
1
exp 2
(x[n] A)2
p(x; A) =
2
( 2 2 )N
n=0
Then
ln p(x; A)
A
N1

1
(x[n] A)2
ln[(2 2 )N/2 ] 2
A
2
n=0
N1

N1
1
N
1
(x[n] A) = 2
x[n] A
2
N
n=0
n=0
According to Theorem :
2

var (A)
N
N
and I (A) = 2
N1
1
and A = g (x[0]) =
x[n]
N
Estimation Theory
n=0
Transformation of Parameters
If it is desired to estimate = g (), then the CRLB is :

2
1 2
ln p(x; )
g
var() E
2
Example : Compute the CRLB for estimation of the power (A2 ) of a DC

level in noise :
2
4A2 2
var(A2 )
(2A)2 =
N
N
Denition
Ecient estimator : An unbiased estimator that attains the CRLB is said
to be ecient.
Example : Knowing that x =
N1
1
x[n] is an ecient estimator for A, is
N
n=0
x2 an ecient estimator for A2 ?

Estimation Theory
Solution : Knowing that x N (A, 2 /N), we have :
E (
x 2 ) = E 2 (
x ) + var(
x ) = A2 +
2
= A2
N
2 = x2 is not even unbiased.

So the estimator A
Lets look at the variance of this estimator :
x 4 ) E 2 (
x 2)
var(
x 2 ) = E (
but we have from the moments of Gaussian RVs (slide 15) :
E (
x 4 ) = A4 + 6A2
2
2
+ 3( )2
N
N
Therefore :

2
3 4
2
4A2 2 2 4
2
var(
x ) = A + 6A
+ 2 A +
=
+ 2
N
N
N
N
N
2
Estimation Theory
Remarks :
2 = x2 is biased and not ecient.
The estimator A
As N the bias goes to zero and the variance of the estimator
approaches the CRLB. This type of estimators are called
asymptotically ecient.
General Remarks :
is an
If g () = a + b is an ane function of , then g
() = g ()
ecient estimator. First, it is unbiased : E (a + b) = a + b = g (),

moreover :
2
g

= a2 var()
var(g ())
var()
so that the CRLB is

but var(g
()) = var(a + b) = a2 var(),
achieved.
If g () is a nonlinear function of and is an ecient estimator,
is an asymptotically ecient estimator.
then g ()
Estimation Theory

Theorem (Vector Parameter)
Assume that the PDF p(x; ) satises the regularity condition

ln p(x; )
E
=0
for all
Then the variance of any unbiased estimator satises C I1 () 0

where 0 means that the matrix is positive semidenite. I() is called the
Fisher information matrix and is given by :

2
ln p(x; )
Iij () = E
i j
An unbiased estimator that attains the CRLB can be found i :
ln p(x; )
= I()(g(x) )
Estimation Theory
CRLB Extension to Vector Parameter

Example : Consider a DC level in WGN with A and 2 unknown.
Compute the CRLB for estimation of = [A 2 ]T .
N1
N
1
N
2
(x[n] A)2
ln p(x; ) = ln 2 ln 2
2
2
2
n=0
The Fisher information matrix is :

2
I() = E
ln p(x;)
A2
2 ln p(x;)
A2
2 ln p(x;)
A2
2 ln p(x;)
(2 )2

=
N
2
0
N
24
The matrix is diagonal (just for this example) and can be easily inverted to
yield :

var()
2
N
2 )
var(
2 4
N
Is there any unbiased estimator that achieves these bounds ?

Estimation Theory
Spring 2013
41 / 152
If it is desired to estimate = g(), and the CRLB for the covariance of
is I1 (), then :
2

1 T
ln p(x; )
g
g
E
C
Example : Consider a DC level in WGN with A and 2 unknown.

Compute the CRLB for estimation of signal to noise ratio = A2 / 2 .
We have = [A 2 ]T and = g () = 12 /2 , then the Jacobian is :

g () g ()
A2
2A
g ()
=
4
=
1
2
2
So the CRLB is :

2A
var(
)
2
A2
4

N
2
1
N
24
Estimation Theory
2A
2
A2
4

=
4 + 22
N
Spring 2013
42 / 152
Linear Models with WGN

If N point samples of data observed can be modeled as
x = H + w
where
x
H
=N 1
=N p
=p1
=N 1
observation vector
observation matrix (known, rank p)
vector of parameters to be estimated
noise vector with PDF N (0, 2 I)
Compute the CRLB and the MVU estimator that achieves this bound.
Step 1 : Compute ln p(x; ).

2 ln p(x; )
and the covariance
Step 2 : Compute I() = E
2
matrix of : C = I1 ().
Step 3 : Find the MVU estimator g(x) by factoring
ln p(x; )
= I()[g(x) ]
(Linear Models)
Estimation Theory
Spring 2013
43 / 152
Step 1 : ln p(x; ) = ln( 2 2 )N 21 2 (x H)T (x H).

Step 2 :
1 T
ln p(x; )
= 2
[x x 2xT H + T HT H]
2
1 T
[H x HT H]
=
2

2
1
ln p(x; )
= 2 HT H
Then
I() = E
2
Step 3 : Find the MVU estimator g(x) by factoring

HT H
ln p(x; )
= I()[g(x) ] =
[(HT H)1 HT x ]
2
Therefore :
C = I1 () = 2 (HT H)1
= g(x) = (HT H)1 HT x
(Linear Models)
Estimation Theory
Spring 2013
44 / 152

For a linear model with WGN represented by x = H + w the MVU
estimator is :
= (HT H)1 HT x
This estimator is ecient and attains the CRLB.
That the estimator is unbiased can be seen easily by :
= (HT H)1 HT E (H + w) =
E ()
The statistical performance of is completely specied because is a
linear transformation of a Gaussian vector x and hence has a Gaussian
distribution :
N (, 2 (HT H)1 )
(Linear Models)
Estimation Theory
Spring 2013
45 / 152
Example (Curve Fitting)

Consider tting the data x[n] by a p-th order polynomial function of n :
x[n] = 0 + 1 n + 2 n2 + + p np + w [n]
We have N data samples, then :
x = [x[0], x[1], . . . , x[N 1]]T
w = [w [0], w [1], . . . , w [N 1]]T
= [0 , 1 , . . . , p ]T
so x = H + w, where H is :
1
0
0
1
1
1
1
2
4
..
..
..
.
.
.
1 N 1 (N 1)2
..
.
0
1
2p
..
.
(N 1)p
N(p+1)
Hence the MVU estimator is : = (HT H)1 HT x

(Linear Models)
Estimation Theory
Spring 2013
46 / 152
Example (Fourier Analysis)

Consider the Fourier analysis of the data x[n] :
x[n] =
M

2kn
2kn
)+
) + w [n]
bk sin(
N
N
M
ak cos(
k=1
k=1
so we have = [a1 , a2 , . . . , aM , b1 , b2 , . . . , bM ]T and x = H + w where :

H = [ha1 , ha2 , . . . , haM , hb1 , hb2 , . . . , hbM ]
with
hak
1
cos( 2k
N )
2k2
cos( N )
..
.
hbk
)
cos( 2k(N1)
N
1
sin( 2k
N )
2k2
sin( N )
..
.
)
sin( 2k(N1)
N
Hence the MVU estimate of the Fourier coecients is : = (HT H)1 HT x

(Linear Models)
Estimation Theory
Spring 2013
47 / 152
Example (Fourier Analysis)

After simplication (noting that (HT H)1 =
2
N I),
we have :
2
= [(ha1 )T x, . . . , (haM )T x , (hb1 )T x, . . . , (hbM )T x]T
N
which is the same as the standard solution :

N1
N1
2
2kn
2
2kn
, bk =
x[n] cos
x[n] sin
ak =
N
N
N
N
n=0
n=0
From the properties of linear models the estimates are unbiased.

The covariance matrix is :
C = 2 (HT H)1 =
2 2
I
N
Note that is Gaussian and C is diagonal (the amplitude estimates

are independent).
(Linear Models)
Estimation Theory
Spring 2013
48 / 152
Example (System Identication)

Consider identication of a Finite Impulse Response (FIR) model, h[k] for
k = 0, 1, . . . , p 1, with input u[n] and output x[n] provided for
n = 0, 1, . . . , N 1 :
x[n] =
p1

h[k]u[n k] + w [n]
n = 0, 1, . . . , N 1
k=0
FIR model can be represented by the linear model x = H + w where
u[0]
0
0
h[0]
u[1]
u[0]
0
h[1]
H
=
=
..
..
..
..
...
.
.
.
.
h[p 1] p1
u[N 1] u[N 2] u[N p] Np
The MVU estimate is = (HT H)1 HT x with C = 2 (HT H)1 .
(Linear Models)
Estimation Theory
Spring 2013
49 / 152
Linear Models with Colored Gaussian Noise

Determine the MVU estimator for the linear model x = H + w with w a
colored Gaussian noise with N (0, C).
Whitening approach : Since C is positive denite, its inverse can be
factored as C1 = DT D where D is an invertible matrix. This matrix acts
as a whitening transformation for w :
E [(Dw)(Dw)T ] = E (DwwT D) = DCDT = DD1 DT D = I
Now if we transform the linear model x = H + w to :
x = Dx = DH + Dw = H + w
where w = Dw N (0, I) is white and we can compute the MVU
estimator as :
= (HT H )1 HT x = (HT DT DH)1 HT DT Dx
so, we have :
= (HT C1 H)1 HT C1 x with C = (HT H )1 = (HT C1 H)1
(Linear Models)
Estimation Theory
Spring 2013
50 / 152
Linear Models with known components

Consider a linear model x = H + s + w, where s is a known signal. To
determine the MVU estimator let x = x s, so that x = H + w is a
standard linear model. The MVU estimator is :
= (HT H)1 HT (x s) with C = 2 (HT H)1
Example : Consider a DC level and exponential in WGN :

x[n] = A + r n + w [n] where r is known. Then we have :

w [0]
x[0]
1
1
x[1] 1
r w [1]

= .. A + .. +
..
..
.
.
.
.
x[N 1]
r N1
w [N 1]
The MVU estimator is :

N1

= (HT H)1 HT (x s) = 1
(x[n] r n )
A
N
=
with var(A)
n=0
(Linear Models)
Estimation Theory
Spring 2013
2
N
51 / 152
Best Linear Unbiased Estimators (BLUE)

Problems of nding the MVU estimators :
The MVU estimator does not always exist or impossible to nd.
The PDF of data may be unknown.
BLUE is a suboptimal estimator that :
restricts estimates to be linear in data ;
restricts estimates to be unbiased ;
= Ax
= AE (x) =
E ()
minimizes the variance of the estimates ;

needs only the mean and the variance of the data (not the PDF). As
a result, in general, the PDF of the estimates cannot be computed.
Remark : The unbiasedness restriction implies a linear model for the data.
However, it may still be used if the data are transformed suitably or the
model is linearized.
(Best Linear Unbiased Estimators)
Estimation Theory
Spring 2013
52 / 152
Finding the BLUE (Scalar Case)

1
Choose a linear estimator for the observed data

x[n]
=
N1

an x[n] = aT x
n = 0, 1, . . . , N 1
where a = [a0 , a1 , . . . , aN1 ]T
n=0
2
Restrict estimate to be unbiased :

=
E ()
N1

an E (x[n]) =
n=0
3
Minimize the variance

= E {[ E ()]
2 } = E {[aT x aT E (x)]2 }
var()
= E {aT [x E (x)][x E (x)]T a} = aT Ca
Estimation Theory
Spring 2013
53 / 152
Finding the BLUE (Scalar Case)

Consider the problem of amplitude estimation of known signals in noise :
x[n] = s[n] + w [n]
Choose a linear estimator : =
N1

an x[n] = aT x
n=0
2
= aT E (x) = aT s =
Restrict estimate to be unbiased : E ()
then aT s = 1 where s = [s[0], s[1], . . . , s[N 1]]T
Minimize aT Ca subject to aT s = 1.
The constrained optimization can be solved using Lagrangian Multipliers :

Minimize
J = aT Ca + (aT s 1)
The optimal solution is :

sT C1 x
= T 1
s C s
and
=
var()
Estimation Theory
1
sT C1 s
Spring 2013
54 / 152
Finding the BLUE (Vector Case)

Theorem (GaussMarkov)
If the data are of the general linear model form
x = H + w
with w is a noise vector with zero mean and covariance C (the PDF of w
is arbitrary), then the BLUE of is :
= (HT C1 H)1 HT C1 x
and the covariance matrix of is
C = (HT C1 H)1
Remark : If noise is Gaussian then BLUE is MVU estimator.
Estimation Theory
Spring 2013
55 / 152
Finding the BLUE

Example : Consider the problem of DC level in noise : x[n] = A + w [n],
where w [n] is of unspecied PDF with var(w [n]) = n2 . We have = A
and H = 1 = [1, 1, . . . , 1]T . The covariance matrix is :
1
2
0
0
02
0
0 0
0 1
0 2
0
0
12
1
C = ..
.. . .
.. C = ..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
0 0 N1
0 0 21
N1
and hence the BLUE is :

= (H C
T
H)
H C
N1 1 N1
1
x[n]
x=
2
n
n2
n=0
and the minimum covariance is :

T
C = (H C
H)
n=0
N1 1
1
=
n2
n=0
Estimation Theory
Spring 2013
56 / 152
Maximum Likelihood Estimation
(Maximum Likelihood Estimation)
Estimation Theory
Spring 2013
57 / 152

Problems : MVU estimator does not often exist or cannot be found.
BLUE is restricted to linear models.
Maximum Likelihood Estimator (MLE) :
can always be applied if the PDF is known ;
is optimal for large data size ;
is computationally complex and requires numerical methods.
Basic Idea : Choose the parameter value that makes the observed data,
the most likely data to have been observed.
Likelihood Function : is the PDF p(x; ) when is regarded as a variable
(not a parameter).
ML Estimate : is the value of that maximizes the likelihood function.
Procedure : Find log-likelihood function ln p(x; ) ; dierentiate w.r.t
and set to zero and solve for .
Estimation Theory
Spring 2013
58 / 152

Example : Consider DC level in WGN with unknown variance
x[n] = A + w [n]. Suppose that A > 0 and 2 = A. The PDF is :

N1
1
1
(x[n] A)2
p(x; A) =
N exp
2A
(2A) 2
n=0
Taking the derivative of the log-likelihood function, we have :

N1
N1
N
1
1
ln p(x; A)
=
+
(x[n] A) + 2
(x[n] A)2
A
2A A
2A
n=0
n=0
What is the CRLB ? Does an MVU estimator exist ?

MLE can be found by setting the above equation to zero :

N1

1
1
= +1
x 2 [n] +
A
2
N
4
n=0
Estimation Theory
Spring 2013
59 / 152
Stochastic Convergence
Convergence in distribution : Let {p(xN )} be a sequence of PDF. If
there exists a PDF p(x) such that
lim p(xN ) = p(x)
at every point x at which p(x) is continuous, we say that p(xN ) converges

d
in distribution to p(x). We write also xN x.
Example
Consider N independent RV x1 , x2 , . . . , xN with mean and nite variance
N
1
xn , then according to the Central Limit Theorem
2 . Let xN =
N
n=1
xN
converges in distribution to z N (0, 1).
(CLT), zN = N
Estimation Theory
Spring 2013
60 / 152
Stochastic Convergence
Convergence in probability : Sequence of random variables {xN }
converges in probability to the random variable x if for every > 0
lim Pr {|xN x| > } = 0
Convergence with probability 1 (almost sure convergence) :

Sequence of random variables {xN } converges with probability 1 to the
random variable x if and only if for all possible events
Pr { lim xN = x} = 1
N
Example
Consider N independent RV x1 , x2 , . . . , xN with mean and nite variance
N
1
xn . Then according to the Law of Large Numbers
2 . Let xN =
N
n=1
(LLN), zN converges in probability to .
Estimation Theory
Spring 2013
61 / 152
Asymptotic Properties of Estimators

Asymptotic Unbiasedness : Estimator N is an asymptotically unbiased
estimator of if :
lim E (N ) = 0
N
Asymptotic Distribution : It refers to p(xn ) as it evolves from

n = 1, 2, . . ., especially for large value of n (it is not the ultimate form of
distribution, which may be degenerate).
Asymptotic Variance : is not equal to lim var(N ) (which is the
N
limiting variance). It is dened as :

1
lim E {N[N lim E (N )]2 }
asymptotic var(N ) =
N
N N
Mean-Square Convergence : Estimator N converges to in a
mean-squared sense, if :
lim E [(N )2 ] = 0
N
Estimation Theory
Spring 2013
62 / 152
Consistency
Estimator N is a consistent estimator of if for every > 0
plim(N ) =
lim Pr [|N | > ] = 0
Remarks :
If N is asymptotically unbiased and its limiting variance is zero then
it converges to in mean-square.
If N converges to in mean-square, then the estimator is consistent.
Asymptotic unbiasedness does not imply consistency and vice versa.
plim can be treated as an operator, e.g. :
plim(xy ) = plim(x)plim(y ) ;
plim

plim(x)
x
=
y
plim(y )
The importance of consistency is that any continuous function of a

consistent estimator is itself a consistent estimator.
Estimation Theory
Spring 2013
63 / 152

Properties : MLE may be biased and is not necessarily an ecient
estimator. However :
MLE is a consistent estimator meaning that
lim Pr [| | > ] = 0
MLE asymptotically attains the CRLB (asymptotic variance is equal

to CRLB).
Under some regularity conditions, the MLE is asymptotically
normally distributed
a
N (, I1 ())
even if the PDF of x is not Gaussian.
If an MVU estimator exists then ML procedure will nd it.
Estimation Theory
Spring 2013
64 / 152

Example
Consider DC level in WGN with known variance 2 .
Sol. Then

N1
1
exp
(x[n] A)2
p(x; A) =
N
2
2
2
(2 ) 2
n=0
N1
1
ln p(x; A)
= 2
(x[n] A) = 0
A
n=0
N1

=0
x[n] N A
n=0
N1

= x = 1
x[n]
which leads to A
N
n=0
Estimation Theory
Spring 2013
65 / 152
MLE for Transformed Parameters (Invariance Property)

Theorem (Invariance Property of the MLE)
The MLE of the parameter = g (), where the PDF p(x; ) is
parameterized by , is given by
= g ()
where is the MLE of .
It can be proved using the property of the consistent estimators.
If = g () is a one-to-one function, then
= arg max p(x; g 1 ()) = g ()
If = g () is not a one-to-one function, then

pT (x; ) =
max
{:=g ()}
p(x; )
and
Estimation Theory
= arg max pT (x; ) = g ()
Spring 2013
66 / 152
MLE for Transformed Parameters (Invariance Property)

Example
Consider DC level in WGN and nd MLE of = exp(A). Since g () is a
one-to-one function then :
x)
= arg max pT (x, ) = arg max p(x; ln ) = exp(
Example
Consider DC level in WGN and nd MLE of = A2 . Since g () is not a
one-to-one function then :
= arg max{p(x; ), p(x; )}

0
2

= arg max {p(x; ), p(x; )}

0

=
arg
max
<A<
2
2 = x2
p(x; A) = A
Estimation Theory
Spring 2013
67 / 152
MLE (Extension to Vector Parameter)

Example
Consider DC Level in WGN with unknown variance. The vector parameter
= [A 2 ]T should be estimated.
We have :
N1
1
(x[n] A)
2
ln p(x; )
A
ln p(x; )
2
N1
N
1
= 2+ 4
(x[n] A)2
2
2
n=0
n=0
which leads to the following MLE :

= x
A
N1
1
2
(x[n] x)2
=
N
n=0
Estimation Theory
Spring 2013
68 / 152
MLE for General Gaussian Case

Consider the general Gaussian case where x N ((), C()).
The partial derivative of the PDF is :

1
C()
()T 1
ln p(x; )
1
= tr C ()
C ()(x ())
+
k
2
k
k
C1 ()
1
(x ())
(x ())T
2
k
for k = 1, . . . , p.
By setting the above equations equal to zero, MLE can be found.
A particular case is when C is known (the rst and third terms
become zero).
In addition, if () is linear in , the general linear model is obtained.
Estimation Theory
Spring 2013
69 / 152
MLE for General Linear Models

Consider the general linear model x = H + w where w is a noise vector
with PDF N (0, C) :

1
1
T 1
exp (x H) C (x H)
p(x; ) =
2
(2)N det(C)
Taking the derivative of ln p(x; ) leads to :
(H)T 1
ln p(x; )
=
C (x H)
Then
HT C1 (x H) = 0
= (HT C1 H)1 HT C1 x
which is the same as MVU estimator. The PDF of is :

N (, (HT C1 H)1 )
Estimation Theory
Spring 2013
70 / 152
MLE (Numerical Method)

NewtonRaphson : A closed form estimator cannot be always computed
by maximizing the likelihood function. However, the maximum value can
be computed by the numerical methods like the iterative NewtonRaphson
algorithm.

k+1
2 ln p(x; )
= k
T
1

ln p(x; )

=k
Remarks :
The Hessian can be replaced by the negative of its expectation, the
Fisher information matrix I().
This method suers from convergence problems (local maximum).
Typically, for large data length, the log-likelihood function becomes
more quadratic and the algorithm will produce the MLE.
Estimation Theory
Spring 2013
71 / 152
Least Squares Estimation
(Least Squares)
Estimation Theory
Spring 2013
72 / 152
The Least Squares Approach

In all the previous methods we assumed that the measured signal x[n] is
the sum of a true signal s[n] and a measurement error w [n] with known
probabilistic model. In least squares method
x[n] = s[n, ] + e[n]
where e(n) represents the modeling and measurement errors. The objective
is to minimize the LS cost :
J() =
N1

(x[n] s[n, ])2
n=0
We do not need a probabilistic assumption but only

a deterministic signal model.
It has a broader range of applications.
No claim about the optimality can be made.
The statistical performance cannot be assessed.
(Least Squares)
Estimation Theory
Spring 2013
73 / 152
The Least Squares Approach
Example
Estimate the DC level of a signal. We observe x[n] = A + [n] for
n = 0, . . . , N 1 and the LS criterion is :
J(A) =
N1

(x[n] A)2
n=0

J(A)
= 2
(x[n] A) = 0
A
N1
n=0
(Least Squares)
Estimation Theory
N1
1
A=
x[n]
N
n=0
Spring 2013
74 / 152
Linear Least Squares

Suppose that the observation model is linear x = H + then
J() =
N1

(x[n] s[n, ])2 = (x H)T (x H)
n=0
= xT x 2xT H + T HT H
where H is full rank. The gradient is
J()
= 2HT x + 2HT H = 0
= (HT H)1 HT x
The minimum LS cost is :

T (x H)
= xT [I H(HT H)1 HT ]x = xT (x H)
Jmin = (x H)
where I H(HT H)1 HT is an Idempotent matrix.
(Least Squares)
Estimation Theory
Spring 2013
75 / 152
Comparing Dierent Estimators for the Linear Model

Consider the following linear model
x = H + w
Estimator
Assumption
Estimate
No probabilistic assumption
= (HT H)1 HT x
BLUE
w is white with unknown PDF
= (HT H)1 HT x
MLE
w is white Gaussian noise
= (HT H)1 HT x
MVUE
w is white Gaussian noise
= (HT H)1 HT x
LSE
For MLE and MVUE the PDF of the estimate will be

Gaussian.
(Least Squares)
Estimation Theory
Spring 2013
76 / 152
Weighted Linear Least Squares

The LS criterion can be modied by including a positive denite
(symmetric) weighting matrix W :
J() = (x H)T W(x H)
That leads to the following estimator :
= (HT WH)1 HT Wx
and minimum LS cost :
Jmin = xT [W WH(HT WH)1 HT W]x
Remark : If we take W = C1 , where C is the covariance of noise then
weighted least squares estimator is the BLUE. However, there is no true
LS-based reason for this choice.
(Least Squares)
Estimation Theory
Spring 2013
77 / 152
Geometrical Interpretation
Recall the general signal model s = H. If we denote the column of H by
hi we have :
1
p
2
i hi
s = [h1 h2 hp ] . =
..
p
i =1
The signal model is a linear combination of the vectors {hi }.

The LS minimizes the length of the error vector between the data and
the signal model = x s :
J() = (x H)T (x H) = x H2
The data vector can lie anywhere in R N , while signal vectors must lie
in a p-dimensional subspace of R N , termed S p , which is spanned by
the column of H.
(Least Squares)
Estimation Theory
Spring 2013
78 / 152
Geometrical Interpretation (Orthogonal Projection)

Intuitively, it is clear that the LS error is minimized when s = H is
the orthogonal projection of x onto S p .
So the LS error vector = x s is orthogonal to all columns of H :
HT = 0
=0
HT (x H)
= (HT H)1 HT x
The signal estimate is the projection of x onto S p :

s = H = H(HT H)1 HT x = Px
where P is the orthogonal projection matrix.
Note that if z Range(H), then Pz = z. Recall that Range(H) is the
subspace spanned by the columns of H.
Now, since Px S p then P(Px) = Px. Therefore any projection
matrix is idempotent, i.e. P2 = P.
It can be veried that P is symmetric and singular (with rank p).
(Least Squares)
Estimation Theory
Spring 2013
79 / 152
Geometrical Interpretation (Orthonormal columns of H)

Recall that
HT H =
< h1 , h1 > < h1 , h2 >

< h2 , h1 > < h2 , h2 >
..
..
..
.
.
.
< hp , h1 > < hp , h2 >
< h1 , hp >
< h2 , hp >
..
.
< hp , hp >
If the columns of H are orthonormal then HT H = I and = HT x.

In this case we have i = hT x, thus
i
s = H =
p

i hi =
i =1
p

(hT
i x)hi
i =1
If we increase the number of parameters (the order of the linear

model), we can easily compute the new estimate.
(Least Squares)
Estimation Theory
Spring 2013
80 / 152
Choosing the Model Order

Suppose that you have a set of data and the objective is to t a
polynomial to data. What is the best polynomial order ?
Remarks :
It is clear that by increasing the order Jmin is monotonically
non-increasing.
By choosing p = N we can perfectly t the data to model. However,
we t the noise as well.
We should choose the simplest model that adequately describes the
data.
We increase the order only if the cost reduction is signicant.
If we have an idea about the expected level of Jmin , we increase p to
approximately attain this level.
There is an order-recursive LS algorithm to eciently compute a
(p + 1)-th order model based on a p-th order one (See 8.6).
(Least Squares)
Estimation Theory
Spring 2013
81 / 152
Sequential Least Squares

1] based on x[N 1] = x[0], . . . , x[N 1] is
Suppose that [N
available. If we get a new data x[N], we want to compute [N]

as a
function of [N 1] and x[N].
Example
Consider LS estimate of DC level :
N1
1
A[N 1] =
x[n]. We have :
N
n=0
1
1
x[n] =
N +1
N +1
N
A[N]
=
n=0

N1
1
N
x[n] + x[N]
N
n=0
1
N
A[N 1] +
x[N]
=
N +1
N +1
1])
1] + 1 (x[N] A[N
= A[N
N +1
new estimate = old estimate + gain prediction error
(Least Squares)
Estimation Theory
Spring 2013
82 / 152

Example (DC level in uncorrelated noise)
x[n] = A + w [n] and var(w [n]) = n2 . The WLS estimate (or BLUE) is :
N1
x[n]
n2
1
n=0 n2
1] = n=0
A[N
N1
Similar to the previous example we can obtain :

2

1]
1] + 1/N
x[N] A[N
A[N]
= A[N
N1 1
n=0 n2

1] + K [N] x[N] A[N
1]
= A[N
The gain factor K [N] can be reformulated as :
K [N] =
1])
var(A[N
1]) + 2
var(A[N
(Least Squares)
Estimation Theory
Spring 2013
83 / 152

Consider the general linear model x = H + w, where w is an uncorrelated
noise with the covariance matrix C. The BLUE (or WLS) is :
= (HT C1 H)1 HT C1 x and C = (HT C1 H)1
Lets dene :
C[n] = diag(02 , 12 , . . . , n2 )

H[n 1]
np
=
H[n] =
1p
hT [n]

x[n] = x[0], x[1], . . . , x[n]
The objective is to nd [n],

based on n + 1 data samples, as a function of
1] and new data x[n]. The batch estimator is :
[n
1] = [n 1]HT [n 1]C1 [n 1]x[n 1]
[n
with [n 1] = (HT [n 1]C1 [n 1]H[n 1])1
(Least Squares)
Estimation Theory
Spring 2013
84 / 152

Estimator Update :

1]
= [n
1] + K[n] x[n] hT [n][n
[n]
where
K[n] =
n2
[n 1]h[n]
+ hT [n][n 1]h[n]
Covariance Update :
[n] = (I K[n]hT [n])[n 1]
The following Lemma is used to compute the updates :
Matrix Inversion Lemma

(A + BCD)1 = A1 A1 B[C1 + DA1 B]1 DA1
with A = 1 [n 1], B = h[n], C = 1/n2 , D = hT [n].
Initialization ?
(Least Squares)
Estimation Theory
Spring 2013
85 / 152
Constrained Least Squares

Linear constraints in the form A = b can be considered in LS solution.
The LS criterion becomes :
Jc () = (x H)T (x H) + T (A b)
Jc
= 2HT x + 2HT H + AT
Setting the gradient equal to zero produces :

c
1
= (HT H)1 HT x (HT H)1 AT
2
= (HT H)1 AT
2
where = (HT H)1 HT x is the unconstrained LSE.

Now Ac = b can be solved to nd :
= 2[A(HT H)1 AT ]1 (A b)
(Least Squares)
Estimation Theory
Spring 2013
86 / 152
Nonlinear Least Squares

Many applications have nonlinear observation model : s() = H. This
leads to a nonlinear optimization problem that can be solved numerically.
Newton-Raphson Method : Find a zero of the gradient of the criterion
by linearizing the gradient around the current estimate.

1

g()

g()
k+1 = k

=k
where g() =
sT ()
[x s()] and

N1

2 s[n] s[n] s[n]
[g()]i
=
(x[n] s[n])
j
i j
j i
n=0
Around the solution [x s()] is small so the rst term in the Jacobian
can be neglected. This makes this method equivalent to the Gauss-Newton
algorithm which is numerically more robust.
(Least Squares)
Estimation Theory
Spring 2013
87 / 152

Gauss-Newton Method : Linearize signal model around the current
estimate and solve the resulting linear problem.

s()
( 0 ) = s(0 ) + H(0 )( 0 )
s() s(0 ) +
=0
The solution to the linearized problem is :
= [HT (0 )H(0 )]1 HT (0 )[x s(0 ) + H(0 )0 ]
= 0 + [HT (0 )H(0 )]1 HT (0 )[x s(0 )]
If we now iterate the solution, it becomes :
k+1 = k + [HT (k )H(k )]1 HT (k )[x s(k )]
Remark : Both the Newton-Raphson and the Gauss-Newton methods can
have convergence problems.
(Least Squares)
Estimation Theory
Spring 2013
88 / 152

Transformation of parameters : Transform into a linear problem by
seeking for an invertible function = g() such that :
s() = s(g1 ()) = H
where
= (HT H)1 HT x.
So the nonlinear LSE is = g1 ()
Example (Estimate the amplitude and phase of a sinusoidal signal)

s[n] = A cos(2f0 n + )
n = 0, 1, . . . , N 1
The LS problem is nonlinear, however, we have :

A cos(2f0 n + ) = A cos cos 2f0 n A sin sin 2f0 n
if we let 1 = A cos and 2 = A sin then the signal model becomes
linear s = H. The LSE is
= (HT H)1 HT x and g1 () is

!
2
2
2
and
= arctan
A = 1 + 2
1
(Least Squares)
Estimation Theory
Spring 2013
89 / 152

Separability of parameters : If the signal model is linear in some of the
parameters, try to write it as s = H(). The LS error can be minimized
wrt .
= [HT ()H()]1 HT ()x
The cost function is a function of that can be minimized by a

numerical method (e.g. brute force).

= xT I H()[HT ()H()]1 HT () x
J(, )
Then
= arg max

xT H()[HT ()H()]1 HT () x
Remark : This method is interesting if the dimension of is much less

than the dimension of .
(Least Squares)
Estimation Theory
Spring 2013
90 / 152

Example (Damped Exponentials)
Consider the following signal model :
s[n] = A1 r n + A2 r 2n + A3 r 3n
n = 0, 1, . . . , N 1
with = [A1 , A2 , A3 , r ]T . It is known that 0 < r < 1.

= [A1 , A2 , A3 ]T so we get s = H(r ) with :
1
1
1
2
r
r3
r
H(r ) = ..
..
..
.
.
.
Lets take
r N1 r 2(N1) r 3(N1)

Step 1 : Maximize xT H(r )[HT (r )H(r )]1 HT (r ) x to obtain r .
Step 2 : Compute = [HT (r )H(r )]1 HT (r )x.
(Least Squares)
Estimation Theory
Spring 2013
91 / 152
The Bayesian Philosophy
(The Bayesian Philosophy)
Estimation Theory
Spring 2013
92 / 152
Introduction
Classical Approach :
Assumes is unknown but deterministic.
Some prior knowledge on cannot be used.
Variance of the estimate may depend on .
MVUE may not exist.
In Monte-Carlo Simulations, we do M runs for each xed and then
compute sample mean and variance for each (no averaging over ).
Bayesian Approach :
Assumes is random with a known prior PDF, p().
We estimate a realization of based on the available data.
Variance of the estimate does not depend on .
A Bayesian estimate always exists
In Monte-Carlo simulations, we do M runs for randomly chosen and
then we compute sample mean and variance over all values.
Estimation Theory
Spring 2013
93 / 152
Mean Square Error (MSE)

Classical MSE
Classical MSE is a function of the unknown parameter and cannot be
used for constructing the estimators.

2
2 p(x; )dx
mse() = E {[ (x)] } = [ (x)]

Note that the E {} is wrt the PDF of x.
Bayesian MSE :
Bayesian MSE is not a function of and can be minimized to nd an
estimator.

2
2 p(x, )d xd
[ (x)]
Bmse() = E {[ (x)] } =
Note that the E {} is wrt the joint PDF of x and .
Estimation Theory
Spring 2013
94 / 152
Minimum Mean Square Error Estimator

Consider the estimation of A that minimizes the Bayesian MSE, where A is
a random variable with uniform PDF p(A) = U [A0 , A0 ] and independent
of w [n].

2
2
[A A] p(A|x)dA p(x)d x
Bmse() =
[A A] p(x, )dxdA =
where we used Bayes theorem : p(x, ) = p(A|x)p(x).
Since p(x) 0 for all x, we need only minimize the integral in brackets.
We set the derivative equal to zero :

2
p(A|x)dA = 2 Ap(A|x)dA + 2A
p(A|x)dA
[A A]
A
which results in
=
A
Ap(A|x)dA = E (A|x)
Bayesian MMSE Estimate : is the conditional mean of A given data x

or the mean of posterior PDF p(A|x).
Estimation Theory
Spring 2013
95 / 152

How to compute the Bayesian MMSE estimate :

= E (A|x) = Ap(A|x)dA
A
The posteriori PDF can be computed using Bayes Rule :
p(A|x) =
p(x|A)p(A)
p(x|A)p(A)
="
p(x)
p(x|A)p(A)dA
"
p(x) is the marginal PDF dened as p(x) = p(x, A)dA.
The integral in the denominator acts as a normalization of
p(x|A)p(A) such the integral of p(A|x) be equal to 1.
p(x|A) has exactly the same form as p(x; A) which is used in the
classical estimation.
The MMSE estimator always exists and can be computed by :
"
= " Ap(x|A)p(A)dA
A
p(x|A)p(A)dA
Estimation Theory
Spring 2013
96 / 152

Example : For p(A) = U [A0 , A0 ] we have :

A0
N1
1
1
1
A
exp
(x[n] A)2 dA
2A0 A0 (2 2 )N/2
2 2
n=0
=
A

A0
N1
1
1
1
2
exp
(x[n] A) dA
2A0 A0 (2 2 )N/2
2 2
n=0
Before collecting the data the mean of the prior PDF p(A) is the best
estimate while after collecting the data the best estimate is the mean
of posterior PDF p(A|x).
Choice of p(A) is crucial for the quality of the estimation.
Only Gaussian prior PDF leads to a closed-form estimator.
Conclusion : For an accurate estimator choose a prior PDF that can be
physically justied. For a closed-form estimator choose a Gaussian prior
PDF.
Estimation Theory
Spring 2013
97 / 152
Properties of the Gaussian PDF

Theorem (Conditional PDF of Bivariate Gaussian)
If x and y are distributed according to a bivariate Gaussian PDF :

T

1 x E (x)
1
x E (x)
1
exp
C
p(x, y ) =
y E (y )
y E (y )
2
2 det(C)

with covariance matrix :
C=
var(x)
cov(x, y )
cov(x, y ) var(y )
then the conditional PDF p(y |x) is also Gaussian and :

cov(x, y )
[x E (x)]
var(x)
cov2 (x, y )
var(y |x) = var(y )
var(x)
E (y |x) = E (y ) +
Estimation Theory
Spring 2013
98 / 152

Example (DC level in WGN with Gaussian prior PDF)
Consider the Bayesian model x[0] = A + w [0] (just one observation) with
a prior PDF of A N (A , A2 ) independent of noise.
First we compute the covariance cov(A, x[0]) :
cov(A, x[0]) = E {(A A )(x[0] E (x[0])}
= E {(A A )(A + w [0] A 0)}
= E {(A A )2 + (A A )w [0]} = var(A) + 0 = A2
The Bayesian MMSE estimate is also Gaussian with :
2
= A|x = A + cov(A, x[0]) [x[0] A ] = A + A (x[0] A )
A
var(x)
2 + A2
= 2 = A2
var(A)
A|x
2
cov2 (A, x[0])
= A2 (1 2 A 2 )
var(x)
+ A
Estimation Theory
Spring 2013
99 / 152

Theorem (Conditional PDF of Multivariate Gaussian)
If x (with dimension k 1) and y (with dimension l 1) are jointly
Gaussian with PDF :

T

1 x E (x)
1
x E (x)
1
exp
C
p(x, y ) =
k+l
y E (y)
y E (y)
2
(2) 2 det(C)

with covariance matrix :
C=
Cxx
Cyx
Cxy
Cyy
then the conditional PDF p(y|x) is also Gaussian and :

E (y|x) = E (y) + Cyx C1
xx [x E (x)]
Cy |x
= Cyy Cyx C1
xx Cxy
Estimation Theory
Spring 2013
100 / 152
Bayesian Linear Model

Let the data be modeled as
x = H + w
where is a p 1 random vector with prior PDF N ( , C ) and w is a
noise vector with PDF N (0, Cw ).
Since and w are independent and Gaussian, they are jointly Gaussian
then the posterior PDF is also Gaussian. In order to nd the MMSE
estimator we should compute the covariance matrices :
Cxx
= E {[x E (x)][x E (x)]T }

= E {(H + w H )(H + w H )T }
= E {(H( ) + w)(H( ) + w)T }
= HE {( )( )T }HT + E (wwT ) = HC HT + Cw
Cx
= E {( )[H( ) + w]T }
= E {( )( )T }HT = C HT
Estimation Theory
Spring 2013
101 / 152

Theorem (Posterior PDF for the Bayesian General Linear Model)
If the observed data can be modeled as
x = H + w
where is a p 1 random vector with prior PDF N ( , C ) and w is a
noise vector with PDF N (0, Cw ), then the posterior PDF p(x|) is
Gaussian with mean :
E (|x) = + C HT (HC HT + Cw )1 (x H )
and covariance
C|x = C C HT (HC HT + Cw )1 HC
Remark : In contrast to the classical general linear model, H need not be
full rank.
Estimation Theory
Spring 2013
102 / 152

Example (DC Level in WGN with Gaussian Prior PDF)
x[n] = A + w [n],
n = 0, . . . , N 1,
A N (A , A2 ),
w [n] N (0, 2 )
We have the general Bayesian linear model x = HA + w with H = 1. Then

E (A|x) = A + A2 1T (1A2 1T + 2 I)1 (x 1A )
var(A|x) = A2 A2 1T (1A2 1T + 2 I)1 1A2
We use the Matrix Inversion Lemma to get :
(I + 1
A2 T 1
1 ) =I
2
E (A|x) = A +
A2
A2 +
2
N
11T
2
2
A
+ 1T 1
=I
11T
2
2
A
+N
(
x A ) and var(A|x) =
Estimation Theory
2 2
N A
2
2
A + N
Spring 2013
103 / 152
Nuisance Parameters
Denition (Nuisance Parameter)
Suppose that and are unknown parameters but we are only interested
in estimating . In this case is called a nuisance parameter.
In the classical approach we have to estimate both but in the
Bayesian approach we can integrate it out.
Note that in the Bayesian approach we can nd p(|x) from p(, |x)
as a marginal PDF :

p(|x) = p(, |x)d
We can also express it as
p(x|)p()
p(|x) = "
p(x|)p()d

where
p(x|) =
p(x|, )p(|)d
Furthermore if is independent of we have :

p(x|) = p(x|, )p()d
Estimation Theory
Spring 2013
104 / 152
General Bayesian Estimators
(General Bayesian Estimators)
Estimation Theory
Spring 2013
105 / 152

Risk Function
A general Bayesian estimator is obtained by minimizing the Bayes Risk
= arg min R()
= E {C ()} is the Bayes Risk, C () is a cost function and

where R()
= is the estimation error.
Three common risk functions

1. Quadratic :
= E {2 } = E {( )
2}
R()
2. Absolute :
= E {||} = E {| |}
R()
3. Hit-or-Miss :
= E {C ()}
R()
where C () =
Estimation Theory
0 || <
1 ||
Spring 2013
106 / 152

The Bayes risk is

= E {C ()} =
R()
C ( )p(x,
)d xd =
g ()p(x)dx

where
=
g ()
C ( )p(|x)d
should be minimized.
Quadratic
= E {( )
2 } =Bmse()
which leads to the MMSE
For this case R()
estimator with
= E (|x)
so is the mean of the posterior PDF p(|x).
Estimation Theory
Spring 2013
107 / 152

Absolute
In this case we have :

= | |p(|x)d
g ()
=

( )p(|x)d +
( )p(|x)d
equal to zero and the use of Leibnitzs

By setting the derivative of g ()
rule, we get :

p(|x)d =
p(|x)d
Then is the median (area to the left=area to the right) of p(|x)
Estimation Theory
Spring 2013
108 / 152

Hit-or-Miss
In this case we have

=
g ()
= 1

1 p(|x)d +
1 p(|x)d
p(|x)d
For arbitrarily small the optimal estimate is the location of the

maximum of p(|x) or the mode of the posterior PDF.
This estimator is called the maximum a posteriori (MAP) estimator.
Remark : For unimodal and symmetric posterior PDF (e.g. Gaussian
PDF), the mean and the mode and the median are the same.
Estimation Theory
Spring 2013
109 / 152
Minimum Mean Square Error Estimators

Extension to the vector parameter case : In general, like the scalar
case, we can write :

i = 1, 2, . . . , p
i = E (i |x) = i p(i |x)d i
Then p(i |x) can be computed as a marginal conditional PDF. For
example :

p(|x)d 2 d p d 1 = 1 p(|x)d
1 = 1
In vector form we have :
"
p(|x)d
1
"
2 p(|x)d
=
= p(|x)d = E (|x)
..
.
"
p p(|x)d
Similarly
Bmse(i ) = [C|x ]ii p(x)d x
Estimation Theory
Spring 2013
110 / 152
Properties of MMSE Estimators

For Bayesian Linear Model, poor prior knowledge leads to MVU
estimator.
T 1
1 T 1
= E (|x) = + (C1
+ H Cw H) H Cw (x H )
For no prior knowledge 0 and C1

0, and therefore :
1 T 1
[HT C1
w H] H Cw x
Commutes over ane transformations.

Suppose that = A + b then the MMSE estimator for is
= E (|x) = E (A + b|x) = AE (|x) + b = A + b

Enjoys the additive property for independent data sets.
Assume that , x1 , x2 are jointly Gaussian with x1 and x2
independent :
1
= E () + Cx1 C1
x1 x1 [x1 E (x1 )] + Cx2 Cx2 x2 [x2 E (x2 )]
Estimation Theory
Spring 2013
111 / 152
Maximum A Posteriori Estimators

In the MAP estimation approach we have :
= arg max p(|x)
where
p(|x) =
p(x|)p()
p(x)
An equivalent maximization is :
= arg max p(x|)p()
or
= arg max[ln p(x|) + ln p()]
If p() is uniform or is approximately constant around the maximum

of p(x|) (for large data length), we can remove p() to obtain the
Bayesian Maximum Likelihood :
= arg max p(x|)
Estimation Theory
Spring 2013
112 / 152

Example (DC Level in WGN with Uniform Prior PDF)
The MMSE estimator cannot be obtained in explicit form due to the need
to evaluate the following integrals :

A0
N1
1
1
1
2
A
exp
(x[n] A) dA
2A0 A0 (2 2 )N/2
2 2
n=0
=
A

A0
N1

1
1
1
exp
(x[n] A)2 dA
2A0 A0 (2 2 )N/2
2 2
n=0
x < A0
A0
=
x
A0 x A0
The MAP estimator is given as : A
A0
x > A0
Remark : The main advantage of MAP estimator is that for non jointly
Gaussian PDFs it may lead to explicit solutions or less computational
eort.
Estimation Theory
Spring 2013
113 / 152

Extension to the vector parameter case
1 = arg max p(1 |x) = arg max

1
p(|x)d2 . . . dp
This needs integral evaluation of the marginal conditional PDFs.

An alternative is the following vector MAP estimator that maximizes the
joint conditional PDF :
= arg max p(|x) = arg max p(x|)p()
This corresponds to a circular Hit-or-Miss cost function.

In general, as N , the MAP estimator the Bayesian MLE.
If the posterior PDF is Gaussian, the mode is identical to the mean,
therefore the MAP estimator is identical to the MMSE estimator.
The invariance property in ML theory does not hold for the MAP
estimator.
Estimation Theory
Spring 2013
114 / 152
Performance Description
In the classical estimation the mean and the variance of the estimate
(or its PDF) indicates the performance of the estimator.
In the Bayesian approach the PDF of the estimate is dierent for each
realization of . So a good estimator should perform well for every
possible value of .
The estimation error = is a function of two random variables
( and x).
The mean and the variance of the estimation error, wrt two random
variables, indicates the performance of the estimator.
The mean value of the estimation error is zero. So the estimates are
unbiased (in the Bayesian sense).
= Ex, ( E (|x)) = Ex [E|x () E|x (|x)] = Ex (0) = 0
Ex, ( )
The variance of the estimate is Bmse :
2 ] = Bmse()
var() = Ex, (2 ) = Ex, [( )

Estimation Theory
Spring 2013
115 / 152
Performance Description (Vector Parameter Case)

The vector of the estimation error is = and has a zero mean.
Its covariance matrix is :
M = Ex, (T ) = Ex, {[ E (|x)][ E (|x)]T }
&
'
()
= Ex E|x [ E (|x)][ E (|x)]T = Ex (C|x )
if x and are jointly Gaussian, we have :
M = C|x = C Cx C1
xx Cx
since C|x does not depend on x.
For a Bayesian linear model we have :
M = C|x = C C HT (HC HT + Cw )1 HC
In this case is a linear transformation of x and and thus is
Gaussian. Therefore :
N (0, M)
Estimation Theory
Spring 2013
116 / 152
Signal Processing Example

Deconvolution Problem : Estimate a signal transmitted through a
channel with known impulse response :
x[n] =
n
s 1
h[n m]s[m] + w [n]
n = 0, 1, . . . , N 1
m=0
where s[n] is a WSS Gaussian process with known ACF, and w [n] is WGN
with variance 2 .
x[0]
x[1]
.
..
x[N 1]
h[0]
h[1]
.
..
h[N 1]
x = H + w
0
h[0]
.
..
h[N 2]
.
..
0
0
.
..
h[N ns ]
s[0]
s[1]
.
..
s[ns 1]
w [0]
w [1]
.
..
w [N 1]
s = Cs HT (HCs HT + 2 I)1 x
where Cs is a symmetric Toeplitz matrix with [Cs ]ij = rss [i j]

Estimation Theory
Spring 2013
117 / 152
Signal Processing Example

Consider that H = I (no ltering), so the Bayesian model becomes
x=s+w
Classical Approach The MVU estimator is s = (HT H)1 HT x = x.
Bayesian Approach The MMSE estimator is s = Cs (Cs + 2 I)1 x.
Scalar case : We estimate s[0] based on x[0] :
s [0] =
rss [0]
x[0]
x[0] =
2
rss [0] +
+1
where = rss [0]/ 2 is the SNR.

Signal is a Realization of an Auto Regressive Process : Consider
a rst order AR process : s[n] = a s[n 1] + u[n], where u[n] is
WGN with variance u2 . The ACF of s is :
rss [k] =
u2
(a)|k|
1 a2
Estimation Theory
[Cs ]ij = rss [i j]

Spring 2013
118 / 152
Linear Bayesian Estimators
(Linear Bayesian Estimators)
Estimation Theory
Spring 2013
119 / 152
Introduction
Problems with general Baysian estimators :
dicult to determine in closed form,
need intensive computations,
involve multidimensional integration (MMSE) or multidimensional
maximization (MAP),
can be determined only under the jointly Gaussian assumption.
What can we do if the joint PDF is not Gaussian or unknown ?
Keep the MMSE criterion ;
Restrict the estimator to be linear.
This leads to the Linear MMSE Estimator :
Which is a suboptimal estimator that can be easily implemented.
It needs only the rst and second moments of the joint PDF.
It is analogous to BLUE in classical estimation.
In practice, this estimator is termed Wiener lter.
Estimation Theory
Spring 2013
120 / 152
Linear MMSE Estimators (scalar case)

Goal : Estimate , given the data vector x. Assume that only the rst two
moments of the joint PDF of x and are available :

E ()
C Cx
,
Cx Cxx
E (x)
LMMSE Estimator : Take the class of all ane estimators
=
N1

an x[n] + aN = aT x + aN
n=0
where a = [a0 , a1 , . . . , aN1 ]T . Then minimize the Bayesian MSE :

= E [( )
2]
Bmse()
Computing aN : Lets dierentiate the Bmse with respect to aN :

E ( aT x aN )2 = 2E ( aT x aN )
aN
Setting this equal to zero gives : aN = E () aT E (x).
Estimation Theory
Spring 2013
121 / 152

Minimize the Bayesian MSE :
= E [( )
2 ] = E [( aT x E () + aT E (x))2 ]
Bmse()
(
' T
= E [a (x E (x)) ( E ())]2
= E [aT (x E (x))(x E (x))T a] E [aT (x E (x))( E ())]
E [( E ())(x E (x))T a] + E [( E ())2 ]
= aT Cxx a aT Cx Cx a + C
This can be minimized by setting the gradient to zero :
Bmse()
= 2Cxx a 2Cx = 0
a
T
which results in a = C1
xx Cx and leads to (Note that : Cx = Cx ) :
1
T 1
= aT x + aN = CT
x Cxx x + E () Cx Cxx E (x)
= E () + Cx C1
xx (x E (x))
Estimation Theory
Spring 2013
122 / 152

The minimum Bayesian MSE is obtained :
= CT C1 Cxx C1 Cx CT C1 Cx
Bmse()
x xx
xx
x xx
Cx C1
xx Cx + C
= C Cx C1
xx Cx
Remarks :
The results are identical to those of MMSE estimators for jointly

Gaussian PDF.
The LMMSE estimator relies on the correlation between random
variables.
If a parameter is uncorrelated with the data (but nonlinearly
dependent), it cannot be estimated with an LMMSE estimator.
Estimation Theory
Spring 2013
123 / 152

The data model is : x[n] = A + w [n]
n = 0, 1, . . . , N 1
where A U [A0 , A0 ] is independent from w [n] (WGN with variance 2 ).
We have E (A) = 0 and E (x) = 0. The covariances are :
Cxx
= E (xxT ) = E [(A1 + w)(A1 + w)T ] = E (A2 )11T + 2 I
Cx
= E (AxT ) = E [A(A1 + w)T ] = E (A2 )1T
Therefore the LMMSE estimator is :

2 T
2
T
2 1
= Cx C1
A
xx x = A 1 (A 11 + I) x =
where
A
A3 0
A20
1
= E (A ) =
A
dA =
=
2A0
6A0 A0
3
A0
A2
A20 /3
x
A20 /3 + 2 /N
A0
Estimation Theory
Spring 2013
124 / 152

Comparison of dierent Bayesian estimators :

A0
N1
1
2
A exp
(x[n] A) dA
2 2
A0
n=0
MMSE : A =

N1
A0
1
exp
(x[n] A)2 dA
2 2
A0
n=0
x < A0
A0
x
A0 x A0
MAP : A =
A0
x > A0
=
LMMSE : A
A20 /3
x
A20 /3 + 2 /N
Estimation Theory
Spring 2013
125 / 152
Vector space of random variables : The set of scalar zero mean random
variables is a vector space.
The zero length vector of the set is a RV with zero variance.
For any real scalar a, ax is another zero mean RV in the set.
For two RVs x and y , x + y = y + x is a RV in the set.
For two RVs x and y , the inner product is : < x, y >= E (xy )
The length of x is dened as : x = < x, x > = E (x 2 )

Two RVs x and y are orthogonal if :
< x, y >= E (xy ) = 0.
For RVs, x1 , x2 , y and real numbers a1 and a2 we have :

< a1 x1 + a2 x2 , y >= a1 < x1 , y > +a2 < x2 , y >
E [(a1 x1 + a2 x2 )y ] = a1 E (x1 y ) + a2 E (x2 y )
The projection of y on x is :
< y, x >
E (yx)
x=
x
2
x
x2
Estimation Theory
Spring 2013
126 / 152
The LMMSE estimator can be determined using the vector space
viewpoint. We have :
N1

an x[n]
=
n=0
where aN is zero, because of zero mean assumption.

belongs to the subspace spanned by x[0], x[1], . . . , x[N 1].
is not in this subspace.
A good estimator will minimize the MSE :
2 ] = 2
E [( )
where = is the error vector.
Clearly the length of the vector error is minimized when is
orthogonal to the subspace spanned by x[0], x[1], . . . , x[N 1] or to
each data sample :
&
)
E ( )x[n]
= 0 for n = 0, 1, . . . , N 1
Estimation Theory
Spring 2013
127 / 152
The LMMSE estimator can be determined by solving the following
equations :

N1

am x[m] x[n] = 0
n = 0, 1, . . . , N 1
E
m=0
or
N1

n = 0, 1, . . . , N 1
am E (x[m]x[n]) = E (x[n])
m=0
E (x 2 [0])
E (x[1]x[0])
..
.
E (x[N 1]x[0])
E (x[0]x[1])
E (x 2 [1])
..
.
E (x[N 1]x[1])
..
.
E (x[0]x[N 1])
E (x[1]x[N 1])
..
.
2
E (x [N 1])
Therefore
Cxx a = CT
x
a0
a1
..
.
aN1
E (x[0])
E (x[1])
..
.
E (x[N 1])
T
a = C1
xx Cx
The LMMSE estimator is :

= aT x = Cx C1
xx x
Estimation Theory
Spring 2013
128 / 152
The vector LMMSE Estimator

We want to estimate = [1 , 2 , . . . , p ]T with a linear estimator that
minimize the Bayesian MSE for each element.
i =
N1

ain x[n] + aiN

Bmse(i ) = E (i i )2
Minimize
n=0
Therefore :

i = E (i ) + Ci x C1
xx x E (x)
*+,- *+,- * +, 1N NN
i = 1, 2, . . . , p
N1
The scalar LMMSE estimators can be combined into a vector estimator :

x E (x)
= E () + Cx C1
xx
* +, - *+,- *+,- * +, p1
and similarly
Bmse(i ) = [M ]ii
pN NN
where
M = C Cx C1
Cx
*+,- *+,- *+,- xx *+,pp
N1
Estimation Theory
pp
pN
Np
Spring 2013
129 / 152
The vector LMMSE Estimator

Theorem (Bayesian GaussMarkov Theorem)
If the data are described by the Bayesian linear model form
x = H + w
where is a p 1 random vector with mean E () and covariance C , w
is a noise vector with zero mean and covariance Cw and is uncorrelated
with (the joint PDF of p(, w) is arbitrary), then the LMMSE estimator
of is :

= E () + C HT (HC HT + Cw )1 x HE ()
The performance of the estimator is measured by = whose mean is
zero and whose covariance matrix is

1
T 1
C = M = C C HT (HC HT + Cw )1 HC = C1
+ H Cw H
Estimation Theory
Spring 2013
130 / 152
Sequential LMMSE Estimation

1] based on x[n 1], update the new estimate
Objective : Given [n
[n] based on the new sample x[n].
Example (DC Level in White Noise)

Assume that both A and w [n] have zero mean :
x[n] = A + w [n]
Estimator Update :
1] =
A[N
A2
x
A2 + 2 /N
1] + K [N](x[N] A[N
1])
A[N]
= A[N
where
K [N] =
1])
Bmse(A[N
1]) + 2
Bmse(A[N
Minimum MSE Update :
1])
Bmse(A[N])
= (1 K [N])Bmse(A[N
Estimation Theory
Spring 2013
131 / 152

Vector space view : If two observations are orthogonal, the LMMSE
estimate of is the sum of the projection of on each observation.
:
1 Find the LMMSE estimator of A based on x[0], yielding A[0]
A2
= E (Ax[0]) x[0] =
x[0]
A[0]
E (x 2 [0])
A2 + 2
2
Find the LMMSE estimator of x[1] based on x[0], yielding x [1|0]

x[1|0] =
2
E (x[0]x[1])
x[0] = 2 A 2 x[0]
2
E (x [0])
A +
Determine the innovation of the new data :

x [1] = x[1] x[1|0].
This error vector is orthogonal to x[0]
Add to A[0]
the LMMSE estimator of A based on the innovation :
= A[0]
+
A[1]
E (A
x [1])
x[1]
E (
x 2 [1])
+,
*
the projection of A on
+ K [1](x[1] x[1|0])
= A[0]
x[1]
Estimation Theory
Spring 2013
132 / 152

Basic Idea : Generate a sequence of orthogonal RVs, namely, the
innovations :
/
.
x[2] , . . . ,
x[n]
x[0] , x[1] ,
*+,*+,*+,- *+,x[0] x[1]
x [1|0] x[2]
x [2|0,1]
x[n]
x [n|0,1,...,n1]
Then, add the individual estimators to yield :
A[N]
=
N

K [n]
x [n] = A[N1]+K
[N]
x [N]
where
K [n] =
n=0
E (A
x [n])
E (
x 2 [n])
It can be shown that :

1]
x[N] = x[N] A[N
and
K [N] =
1])
Bmse(A[N
1])
2 + Bmse(A[N
the minimum MSE be updated as :
1])
Bmse(A[N])
= (1 K [N])Bmse(A[N
Estimation Theory
Spring 2013
133 / 152
General Sequential LMMSE Estimation

Consider the general Bayesian linear model x = H + w, where w is an
uncorrelated noise with the diagonal covariance matrix Cw . Lets dene :
Cw [n] = diag(02 , 12 , . . . , n2 )

H[n 1]
np
=
H[n] =
1p
hT [n]

x[n] = x[0], x[1], . . . , x[n]
Estimator Update :

1]
= [n
1] + K[n] x[n] hT [n][n
[n]
where
M[n 1]h[n]
+ hT [n]M[n 1]h[n]
Minimum MSE Matrix Update :
K[n] =
n2
M[n] = (I K[n]hT [n])M[n 1]

Estimation Theory
Spring 2013
134 / 152
General Sequential LMMSE Estimation

Remarks :
For the initialization of the sequential LMMSE estimator, we can use
the prior information :
[1]
= E ()
M[1] = C
For no prior knowledge about we can let C . Then we have

the same form as the sequential LSE, although the approaches are
fundamentally dierent.
No matrix inversion is required.
The gain factor K[n] weights condence in the new data (measured
by n2 ) against all previous data (summarized by M[n 1]).
Estimation Theory
Spring 2013
135 / 152
Wiener Filtering
Consider the signal model : x[n] = s[n] + w [n]
n = 0, 1, . . . , N 1
where the data and noise are zero mean with known covariance matrix.
There are three main problems concerning the Wiener Filters :
Filtering : Estimate = s[n] (scalar) based on the data set
x = [x[0], x[1], . . . , x[n]]. The signal sample is estimated
based on the the present and past data only.
Smoothing : Estimate = s = [s[0], s[1], . . . , s[N 1]] (vector) based on
the data set x = x[0], x[1], . . . , x[N 1] . The signal sample
is estimated based on the the present, past and future data.
Prediction : Estimate = x[N 1 + ] for positive integer based on the
data set x = [x[0], x[1], . . . , x[N 1]].
Remarks : All these problems are solved using the LMMSE estimator :
with the minimum Bmse :
M = C Cx C1
= Cx C1
xx x
xx Cx
Estimation Theory
Spring 2013
136 / 152
Wiener Filtering (Smoothing)

Estimate
= s = [s[0], s[1], . . . , s[N 1]] (vector) based on the data set

x = x[0], x[1], . . . , x[N 1] .
Cxx = Rxx = Rss + Rww
Therefore :
Cx = E (sxT ) = E (s(s + w)T ) = Rss
and
1
s = Csx C1
xx x = Rss (Rss + Rww ) x = Wx
Filter Interpretation : The Wiener Smoothing Matrix can be interpreted

as an FIR lter. Lets dene :
W = [a0 , a1 , . . . , aN1 ]
where aT
n is the (n + 1)th raw of W. Lets dene also :
hn = [h(n) [0], h(1) [1], . . . , h(n) [N 1]] which is just the vector an when
ipped upside down. Then
s [n] =
aT
nx
N1

h(n) [N 1 k]x[k]
k=0
This represents a time-varying, non causal FIR lter.

Estimation Theory
Spring 2013
137 / 152
Wiener Filtering (Filtering)

Estimate = s[n] based on the data set x = x[0], x[1], . . . , x[n] . We
have again Cxx = Rxx = Rss + Rww and

Cx = E (s[n] x[0], x[1], . . . , x[n] ) = rss [n], rss [n 1], . . . , rss [0] = (rss )T
Therefore :
s [n] = (rss )T (Rss + Rww )1 x = aT x
Filter Interpretation : If we dene hn = [h(n) [0], h(1) [1], . . . , h(n) [N 1]]

which is just the vector a when ipped upside down. Then :
T
s [n] = a x =
n

h(n) [n k]x[k]
k=0
This represents a time-varying, causal FIR lter. The impulse response can
be computed as :
(Rss + Rww )a = rss
Estimation Theory
(Rss + Rww )hn = rss

Spring 2013
138 / 152
Wiener Filtering (Prediction)

Estimate = x[N 1 + ] based on the data set
x = [x[0], x[1], . . . , x[N 1]]. We have again Cxx = Rxx and
Cx
= E (x[N 1 + ][x[0], x[1], . . . , x[N 1]])

= rxx [N 1 + ], rxx [N 2 + ], . . . , rxx [] = (rxx )T
Therefore :
T
x[N 1 + ] = (rxx )T R1
xx x = a x
Filter Interpretation : If we let again h[N k] = ak , we have

x[N 1 + ] = a x =
T
N1

h[N k]x[k]
k=0
This represents a time-invariant, causal FIR lter. The impulse response

can be computed as :
Rxx h = rxx
For = 1, this is equivalent to an Auto-Regressive process.
Estimation Theory
Spring 2013
139 / 152
Kalman Filters
(Kalman Filters)
Estimation Theory
Spring 2013
140 / 152
Introduction
Kalman lter is a generalization of the Wiener lter.
In Wiener lter we estimate s[n] based on the noisy observation
vector x[n]. We assume that s[n] is a stationary random process with
known mean and covariance matrix.
In Kalman lter we assume that s[n] is a non stationary random
process whose mean and covariance matrix vary according to a known
dynamical model.
Kalman lter is a sequential MMSE estimator of s[n]. If the signal
and noise are jointly Gaussian, then the Kalman lter is an optimal
MMSE estimator, if not, it is the optimal LMMSE estimator.
Kalman lter has many applications in Control theory for state
estimation.
It can be generalized to vector signals and noise (in contrast to
Wiener lter).
(Kalman Filters)
Estimation Theory
Spring 2013
141 / 152
Dynamical Signal Models

Consider a rst order dynamical model :
s[n] = as[n 1] + u[n]
n0
where u[n] is WGN with variance u2 , s[1] N (s , s2 ) and s[1] is

independent of u[n] for all n 0. It can be shown that :
s[n] = a
n+1
s[1] +
n

ak u[n k]
E (s[n]) = an+1 s
k=0
and
Cs [m, n] = E
&
)

s[m] E (s[m]) s[n] E (s[n])
= am+n+2 s2 + u2 amn
n

a2k
for m 0
k=0
and Cs [m, n] = Cs [n, m] for m < n.

(Kalman Filters)
Estimation Theory
Spring 2013
142 / 152
Dynamical Signal Models

Theorem (Vector Gauss-Markov Model)
The Gauss-Markov model for a p 1 vector signal s[n] is :
s[n] = As[n 1] + Bu[n]
n0
A is p p and B p r and all eigenvalues of A are less than 1 in

magnitude. The r 1 vector u[n] N (0, Q) and the initial condition is a
p 1 random vector with s[n] N (s , Cs ) and is independent of u[n]s.
Then, the signal process is Gaussian with mean E (s[n]) = An+1 s and
covariance :
m+1
Cs [m, n] = A
m

n+1 T

T
Cs A
+
Ak BQBT Anm+k
k=mn
for m n and Cs [m, n] = Cs [n, m] for m < n. The covariance matrix

C[n] = Cs [n, n] and the mean and covariance propagation equations are :
E (s[n]) = AE (s[n 1]) , C[n] = AC[n 1]AT + BQBT
(Kalman Filters)
Estimation Theory
Spring 2013
143 / 152
Scalar Kalman Filter

Data model :
State equation
Observation equation
s[n] = as[n 1] + u[n]

x[n] = s[n] + w [n]
Assumptions :
u[n] is zero mean Gaussian noise with independent samples and
E (u 2 [n]) = u2 .
w [n] is zero mean Gaussian noise with independent samples and
E (w 2 [n]) = n2 (time-varying variance).
s[1] N (s , s2 ) (for simplicity we suppose that s = 0).
s[1], u[n] and w [n] are independent.
Objective : Develop a sequential MMSE
estimator to estimate s[n] based
on the data X[n] = x[0], x[1], . . . , x[n] . This estimator is the mean of
posterior PDF :

s [n|n] = E (s[n]x[0], x[1], . . . , x[n])
(Kalman Filters)
Estimation Theory
Spring 2013
144 / 152

To develop the equations of a Kalman lter we need the following
properties :
Property 1 : For two uncorrelated jointly Gaussian data vector, the
MMSE estimator (if it is zero mean) is given by :
= E (|x1 , x2 ) = E (|x1 ) + E (|x2 )
Property 2 : If = 1 + 2 , then the MMSE estimator of is :
= E (|x) = E (1 + 2 |x) = E (1 |x) + E (2 |x)
Basic Idea : Generate the innovation x[n] = x[n] x[n|n 1] which is
uncorrelated with previous samples X[n
1]. Then use x[n] instead of x[n]
for estimation (X[n] is equivalent to X[n 1], x[n] ).
(Kalman Filters)
Estimation Theory
Spring 2013
145 / 152

From Property 1, we have :

s [n|n] = E (s[n]X[n 1], x[n]) = E (s[n]X[n 1]) + E (s[n]x[n])

= s [n|n 1] + E (s[n]x[n])
From Property 2, we have :

s [n|n 1] = E (as[n 1] + u[n]X[n 1])

= aE (s[n 1]X[n 1]) + E (u[n]X[n 1])
= as [n 1|n 1]
The MMSE estimator of s[n] based on x[n] is :

E (s[n]
x (n))
x[n] = K [n](x[n] x[n|n 1])
E (s[n]x[n]) =
2
E (
x [n])
where x[n|n 1] = s [n|n 1] + w[n|n
1] = s [n|n 1].
Finally we have :
s [n|n] = as [n 1|n 1] + K [n](x[n] s [n|n 1])
(Kalman Filters)
Estimation Theory
Spring 2013
146 / 152
Scalar State Scalar Observation Kalman Filter

State Model :
Observation Model :
s[n] = as[n 1] + u[n]

x[n] = s[n] + w [n]
Prediction :
s [n|n 1] = as [n 1|n 1]
Minimum Prediction MSE :
M[n|n 1] = a2 M[n 1|n 1] + u2
Kalman Gain :
K [n] =
n2
M[n|n 1]
+ M[n|n 1]
Correction :
s [n|n] = s [n|n 1] + K [n](x[n] s [n|n 1])
Minimum MSE :
M[n|n] = (1 K [n])M[n|n 1]
(Kalman Filters)
Estimation Theory
Spring 2013
147 / 152
Properties of Kalman Filter

The Kalman lter is an extension of the sequential MMSE estimator
but applied to time-varying parameters that are represented by a
dynamic model.
No matrix inversion is required.
The Kalman lter is a time-varying linear lter.
The Kalman lter computes (and uses) its own performance measure,
the Bayesian MSE M[n|n].
The prediction stage increases the error, while the correction stage
decreases it.
The Kalman lter generates an uncorrelated sequence from the
observations so can be viewed as a whitening lter.
The Kalman lter is optimal in the Gaussian case and the optimal
LMMSE estimator if the Gaussian assumption is not valid.
In KF M[n|n], M[n|n 1] and K [n] do not depend on the
measurement and can be computed o line. At steady state (n )
the Kalman lter becomes an LTI lter (M[n|n], M[n|n 1] and K [n]
become constant).
(Kalman Filters)
Estimation Theory
Spring 2013
148 / 152
Vector State Scalar Observation Kalman Filter

State Model :
Observation Model :
Prediction :
s[n] = As[n 1] + Bu[n] with u[n] N (0, Q)

x[n] = hT [n]s[n] + w [n] with s[1] N (s , Cs )
s[n|n 1] = As[n 1|n 1]
Minimum Prediction MSE (p p) :

M[n|n 1] = AM[n 1|n 1]AT + BQBT
Kalman Gain (p 1) :
K[n] =
M[n|n 1]h[n]
n2 + hT [n]M[n|n 1]h[n]
Correction :
s[n|n] = s[n|n 1] + K[n](x[n] hT [n]s [n|n 1])
Minimum MSE Matrix (p p) :
M[n|n] = (I K[n]hT [n])M[n|n 1]
(Kalman Filters)
Estimation Theory
Spring 2013
149 / 152
Vector State Vector Observation Kalman Filter

State Model :
Observation Model :
Prediction :
s[n] = As[n 1] + Bu[n] with u[n] N (0, Q)

x[n] = H[n]s[n] + w[n] with w[n] N (0, C[n])
s[n|n 1] = As[n 1|n 1]
Minimum Prediction MSE (p p) :

M[n|n 1] = AM[n 1|n 1]AT + BQBT
Kalman Gain Matrix (p M) : (Need matrix inversion !)

K[n] = M[n|n 1]HT [n] C[n] + H[n]M[n|n 1]HT [n]
Correction :
s[n|n] = s[n|n 1] + K[n](x[n] H[n]s [n|n 1])
Minimum MSE Matrix (p p) :
M[n|n] = (I K[n]H[n])M[n|n 1]
(Kalman Filters)
Estimation Theory
Spring 2013
150 / 152
Extended Kalman Filter

The extended Kalman lter is a sub-optimal approach when the state
and the observation equations are nonlinear.
Nonlinear data model : Consider the following nonlinear data model
s[n] = f(s[n 1]) + Bu[n]
x[n] = g(s[n]) + w[n]
where f() and g() are nonlinear function mapping (vector to vector).
Model linearization : We linearize the model around the estimated value
of s using a rst order Taylor series.

f
s[n 1] s[n 1|n 1]
f(s[n 1]) f(s[n 1|n 1]) +
s[n 1]

g
s[n] s[n|n 1]
g(s[n]) g(s[n|n 1]) +
s[n]
We denote the Jacobians by :

g
f

H[n] =
A[n 1] =

s[n 1]
s[n]
s [n1|n1]
(Kalman Filters)
Estimation Theory
s [n|n1]
Spring 2013
151 / 152
Extended Kalman Filter

Now we use the linearized models :
s[n] = A[n 1]s[n 1] + Bu[n]
+f(s[n 1|n 1]) A[n 1]s[n 1|n 1]
x[n] = H[n]s[n] + w[n]
+g(s[n|n 1]) H[n]s[n|n 1]
There are two dierences with the standard Kalman lter :
1
2
A is now, time varying.

Both equations have known terms added to them. This will not change
the derivation of the Kalman lter.
The extended Kalman lter has exactly the same equations where A
is replaced with A[n 1].
A[n 1] and H[n] should be computed at each sampling time.
MSE matrices and Kalman gain can no longer be computed o line.
(Kalman Filters)
Estimation Theory
Spring 2013
152 / 152

Estimation Theory

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Estimation Theory

Transféré par

Droits d'auteur :

Formats disponibles

Estimation Theory

Applications : Image processing, communications, biomedicine, system

Fundamentals of Statistical Signal Processing

Review of Probability and Random Variables

(Probability and Random Variables)

Probability Distribution Function

Probability Density Function (PDF)

p(x)dx = P() P() = 1

Pr [{|X () x}] = Pr [X x] = P(x) =

(Probability and Random Variables)

Gaussian Probability Density Function

Some other common PDF

t z1 e t dt and u(x) is the unit step function.

(Probability and Random Variables)

Joint, Marginal and Conditional PDF

Conditional PDF : p(x|y ) is dened as the PDF of X conditioned on

Independent Random Variables

A direct conclusion is that :

p(y |x) = p(y )

which means conditioning does not change the PDF.

(Probability and Random Variables)

Expected Value of a Random Variable

Some properties of expected value :

Conditional expectation : The conditional expectation of X given that a

(Probability and Random Variables)

Moments of a Random Variable

The rst moment of X is its expected value or the mean ( = E (X )).

(Probability and Random Variables)

The second central moment (variance) is denoted by 2 or var(X ).

(Probability and Random Variables)

Some properties of Gaussian RVs

The Euclidian norm X 2 + Y 2 of two independent RVs with

(Probability and Random Variables)

If X and Y are zero mean then xy = E {XY }.

Independence, Uncorrelatedness and Orthogonality

Discrete White Noise

rxx [k] = rxx [k]

rxy [k] = ryx [k]

rxx [k] exp(j2fk)

Introduction and Minimum Variance Unbiased

(Minimum Variance Unbiased Estimation)

The Mathematical Estimation Problem

The rst step is to nd the PDF of data as a

The Mathematical Estimation Problem

(Minimum Variance Unbiased Estimation)

The Mathematical Estimation Problem

(Minimum Variance Unbiased Estimation)

Assessing Estimator Performance

Consider the following estimators :

An estimator is a random variable, so its

2 ) = var(x[0]) = 2 > var (A

Assuming that the estimators have the same variance, we have :

By increasing n, the variance will decrease (if n , ).

Minimum Variance Criterion

Instead of minimizing MSE we can minimize the variance of

Minimum Variance Unbiased Estimator

Existence of MVU Estimator : In general MVU estimator does not

Determine the Cramer-Rao lower bound (CRLB) and check to see if

Apply the Rao-Blackwell-Lehmann-Schee theorem (we will skip it).

Restrict to linear unbiased estimators.

(Minimum Variance Unbiased Estimation)

Cramer-Rao Lower Bound

(Cramer-Rao Lower Bound)

Cramer-Rao Lower Bound

2 = x2 is not even unbiased.