1

Lecture 2: Linear Optimum Filtering

Wiener Filters
Problem statement
Given the sequence of input samples [u(0), u(1), u(2), . . .] and the sequence of desired response samples
[d(0), d(1), d(2), . . .]
Given a family of lters computing their outputs according to
y(n) =

k=0
w
k
u(n k), n = 0, 1, 2, . . . (1)
Find the parameters {w
0
, w
1
, w
2
, . . .} such as to minimize the mean square error dened as
J = E[e(n)
2
] (2)
where the error signal is
e(n) = d(n) y(n) = d(n)

l=0
w
l
u(n l) (3)
2
The family of lters (1) is the family of linear discrete time lters (IIR or FIR).
Lecture 2 2
Principle of orthogonality
Dene the gradient operator , having its k-th entry

k
=

w
k
(4)
and thus, the kth entry of the gradient of criterion J is (remember, e(n) = d(n)

l=0
w
l
u(n l))

k
J =
J
w
k
= 2E
_
_
e(n)
e(n)
w
k
_
_
= 2E [e(n)u(n k)]
For the criterion to attain its minimum, the gradient of the criterion must be identically zero, that is

k
J = 0, k = 0, 1, 2, . . .
resulting in the fundamental
Principle of orthogonality: E [e
o
(n)u(n k)] = 0, k = 0, 1, 2, . . . (5)
Stated in words:
The criterion J attains its minimum i
the estimation error e
o
(n) is orthogonal to the samples u(i) which are used to compute the lter output.
We will index with o all the variables e.g. e
o
, y
o
computed using the optimal parameters {w
o0
, w
o1
, w
o2
, . . .}.
Let us compute the cross- correlation
E [e
o
(n)y
o
(n)] = E
_
_
e
o
(n)

k=0
w
ok
u(n k)
_
_
=

k=0
w
ok
E [u(n k)e
o
(n)] = 0
(6)
Lecture 2 3
Otherwise stated, in words, we have the following Corollary of Orthogonality Principle:
When the criterion J attains its minimum then
the estimation error e
0
(n) is orthogonal to the lter output y
0
(n).
Wiener Hopf equations
From the orthogonality estimation error input window samples we have
E [u(n k)e
0
(n)] = 0, k = 0, 1, 2, . . .
E
_
_
u(n k)(d(n)

i=0
w
oi
u(n i))
_
_
= 0, k = 0, 1, 2, . . .

i=0
w
oi
E [u(n k)u(n i)] = E [u(n k)d(n)] , k = 0, 1, 2, . . .
But
* E [u(n k)u(n i)] = r(i k) is the autocorrelation function of input signal u(n) at lag i k
* E [u(n k)d(n)] = p(k) is the cross-correlation between the lter input u(n k) and the desired signal
d(n)
and therefore

i=0
w
oi
r(i k) = p(k), k = 0, 1, 2, . . . WIENER HOPF (7)
Lecture 2 4
Solution of the Wiener Hopf equations for linear transversal lters (FIR)
y(n) =
M1

k=0
w
k
u(n k), n = 0, 1, 2, . . . (8)
and since only w
0
, w
1
, w
2
, . . . w
M1
are nonzero, Wiener-Hopf equations become
M1

i=0
w
oi
r(i k) = p(k), k = 0, 1, 2, . . . , M 1 WIENER HOPF (9)
which is a system of M equations with M unknowns: {w
o,0
, w
o,1
, w
o,2
, . . . , w
o,M1
}.
Matrix formulation of Wiener Hopf equations
Let us denote
u(n) =
_
u(n) u(n 1) u(n 2) . . . u(n M + 1)
_
T
R = E[u(n)u
T
(n)] = E
_

_
u(n)
u(n 1)
u(n 2)
u(n M + 1)
_

_
_
u(n) u(n 1) u(n 2) . . . u(n M + 1)
_
(10)
=
_

_
Eu(n)u(n) Eu(n)u(n 1) . . . Eu(n)u(n M + 1)
Eu(n 1)u(n) Eu(n 1)u(n 1) . . . Eu(n 1)u(n M + 1)
. . . .
. . . .
. . . .
Eu(n M + 1)u(n) Eu(n M + 1)u(n 1) . . . Eu(n M + 1)u(n M + 1)
_

_
=
_

_
r(0) r(1) . . . r(M 1)
r(1) r(0) . . . r(M 2)
. . . .
. . . .
. . . .
r(M 1) r(M 2) . . . r(0)
_

_
p = E[u(n)d(n)] =
_
p(0) p(1) p(2) . . . p(1 M)
_
T
(11)
w
0
=
_
w
o,0
w
o,1
. . . w
o,M1
_
T
(12)
Lecture 2 5
then Wiener Hopf equations can be written in a compact form
Rw
0
= p with solution w
o
= R
1
p (13)
Mean square error surface
Let us dene
e
w
(n) = d(n)
M1

k=0
w
k
u(n k) = d(n) w
T
u(n) (14)
Then the cost function can be written as
J
w
= E[e
w
(n)e
w
(n)] = E[(d(n) w
T
u(n))(d(n) u
T
(n)w)]
= E[d
2
(n) d(n)u
T
(n)w w
T
u(n)d(n) + w
T
u(n)u
T
(n)w]
= E[d
2
(n)] E[d(n)u
T
(n)]w w
T
E[u(n)d(n)] + w
T
E[u(n)u
T
(n)]w
= E[d
2
(n)] 2E[d(n)u
T
(n)]w + w
T
E[u(n)u
T
(n)]w
=
2
d
2p
T
w + w
T
Rw
=
2
d
2
M1

i=0
p(i)w
i
+
M1

l=0
M1

i=0
w
l
w
i
R
i,l
(15)
Thus, we can proceed in another way to nd the (same) optimal solution w
o
.
* J
w
is a second order function of the parameters { w
0
w
1
. . . w
M1
}
* J
[ w
0
w
1
. . . w
M1
]
is a bowl shaped M + 1- dimensional surface with M degrees of freedom.
Lecture 2 6
* J attains the minimum, J
min
, where the gradient is zero

w
J = 0
J
w
k
= 0, k = 0, 1, . . . , M 1
J
w
k
= 2p(k) + 2
M1

l=0
w
l
r(k l) = 0, k = 0, 1, . . . , M 1
which nally gives the same Wiener Hopf equations
M1

l=0
w
l
r(k l) = p(k) (16)
Minimum Mean square error
Using the form of the criterion
J
w
=
2
d
2p
T
w + w
T
Rw (17)
one can nd the value of the minimum criterion (remember, Rw
0
= p and w
o
= R
1
p):
J
w
o
=
2
d
2p
T
w
o
+ w
T
o
Rw
o
=
2
d
2w
T
o
Rw
o
+ w
T
o
Rw
o
=
2
d
w
T
o
Rw
o
=
2
d
w
T
o
p
=
2
d
p
T
R
1
p (18)
Lecture 2 7
Canonical form of the Error - performance surface
(Parenthesis: How to compute a scalar out of a vector w, containing the entries of w at power one (linear
combination) or at power two (quadratic form):
* linear combination (rst order form) a
T
w =

M1
l=0
a
l
w
l
;
* quadratic form w
T
Rw =

M1
l=0

M1
i=0
w
l
w
i
R
i,l
= w
2
0
R
0,0
+ w
0
w
1
R
1,0
+ . . . w
2
M1
R
M1,M1
)
How can we rewrite the criterion
J
w
=
2
d
2p
T
w + w
T
Rw (19)
in a quadratic form (how to complete a perfect square, encompassing 2p
T
w)?
Consider rst the case when w is simply a scalar (resulting also in scalars R, r, p)
J
w
= Rw
2
2pw +
2
d
= R(w
2
2w
p
R
) +
2
d
= R(w
2
2w
p
R
+
p
2
R
2
)
p
2
R
+
2
d
= R(w
p
R
)
2

p
2
R
+
2
d
In the case when w is a vector, the term corresponding to the one-dimensional
p
2
R
is p
T
R
1
p
J
w
= w
T
Rw 2p
T
w + p
T
R
1
p p
T
R
1
p +
2
d
= (w R
1
p)
T
R(w R
1
p) p
T
R
1
p +
2
d
= J
w
0
+ (w R
1
p)
T
R(w R
1
p)
= J
w
0
+ (w w
0
)
T
R(w w
0
)
2 (This was the solution of exercise 5.5 page 182 in [Haykin 91])
Lecture 2 8
Let
1
,
2
, . . . ,
M
be the eigenvalues and (generally the complex) eigenvectors
1
,
2
, . . . ,
M
of the matrix
R, thus satisfying
R
i
=
i

i
(20)
Then the matrix Q = [
1

2
. . .
M
] can transform R to a diagonal form as follows
R = QQ
H
(21)
where the superscript H means complex conjugation and transposition. Then
J
w
= J
w
0
+ (w w
0
)
T
R(w w
0
) = J
w
0
+ (w w
0
)
T
QQ
H
(w w
0
)
Introduce now the transformed version of the tap vector w as
= Q
H
(w w
o
) (22)
Now the quadratic form can be put into its canonical form
J = J
w
o
+
H

= J
w
o
+
M

i=1

i
= J
w
o
+
M

i=1

i
|
i
|
2
Lecture 2 9
Optimal Wiener Filter Design: Example
(Useful) Signal Generating Model The model is given by the transfer function
H
1
(z) =
D(z)
V
1
(z)
=
1
1 + az
1
=
1
1 + 0.8458z
1
or the dierence equation
d(n) + ad(n 1) = v
1
(n) d(n) + 0.8458d(n 1) = v
1
(n)
where
2
v
1
= r
v
1
(0) = 0.27
The channel (perturbation) model is more complex. It involves a low pass lter with a transfer function
H
2
(z) =
X(z)
D(z)
=
1
1 + bz
1
=
1
1 0.9458z
1
leading for the variable x(n) to the dierence equation
x(n) = 0.9458x(n 1) + d(n)
and a white noise corruption (x(n) and v
2
(n) are uncorrelated)
u(n) = x(n) + v
2
(n)
with
2
v
2
= r
v
2
(0) = 0.1 resulting in the nal measurable signal u(n).
FIR Filter The signal u(n) will be ltered in order to recover the original (useful) d(n) signal, using the
lter
y(n) = w
0
u(n) + w
1
u(n 1)
Lecture 2 10
Lecture 2 11
We plan to apply the Wiener Hopf equations
_
_
r
u
(0) r
u
(1)
r
u
(1) r
u
(0)
_
_
_
_
w
0
w
1
_
_
=
_
_
Ed(n)u(n)
Ed(n)u(n 1)
_
_
The signal x(n) obeys the generation model
H(z) =
X(z)
V
1
(z)
= H
1
(z)H
2
(z) =
1
1 + az
1
1
1 + bz
1
=
1
1 + a
1
z
1
+ a
2
z
2
=
1
1 0.1z
1
0.8z
2
and thus
x(n) + a
1
x(n 1) + a
2
x(n 2) = v
1
(n)
Using the fact that x(n) and v
2
(n) are uncorrelated and
u(n) = x(n) + v
2
(n)
it results
r
u
(k) = r
x
(k) + r
v
2
(k)
and consequently, since for white noise r
v
2
(0) =
2
v
2
= 0.1 and r
v
2
(1) = 0 it follows
r
u
(0) = r
x
(0) + 0.1, and r
u
(1) = r
x
(1)
Now we concentrate to nd r
x
(0), r
x
(1) for the AR process
x(n) + a
1
x(n 1) + a
2
x(n 2) = v(n)
Lecture 2 12
First multiply in turn the equation with x(n), x(n 1) and x(n 2) and then take the expectation
Ex(n) Ex(n)x(n) + Ex(n)a
1
x(n 1) + Ex(n)a
2
x(n 2) = Ex(n)v(n)
resulting in r
x
(0) + a
1
r
x
(1) + a
2
r
x
(2) = Ex(n)v(n) =
2
v
Ex(n 1) Ex(n 1)x(n) + Ex(n 1)a
1
x(n 1) + Ex(n 1)a
2
x(n 2) = Ex(n 1)v(n)
resulting in r
x
(1) + a
1
r
x
(0) + a
2
r
x
(1) = Ex(n 1)v(n) = 0
Ex(n 2) Ex(n 2)x(n) + Ex(n 2)a
1
x(n 1) + Ex(n 2)a
2
x(n 2) = Ex(n 2)v(n)
resulting in r
x
(2) + a
1
r
x
(1) + a
2
r
x
(0) = Ex(n 2)v(n) = 0
The equality Ex(n)v(n) =
2
v
can be obtained multiplying the AR model dierence equation with v(n) and
then taking expectations
Ev(n) Ev(n)x(n) + Ev(n)a
1
x(n 1) + Ev(n)a
2
x(n 2) = Ev(n)v(n)
resulting in Ev(n)x(n) =
2
v
since v(n) is uncorrelated with older values, x(n). We obtained the most celebrated Yule Walker equations:
r
x
(0) + a
1
r
x
(1) + a
2
r
x
(2) =
2
v
r
x
(1) + a
1
r
x
(0) + a
2
r
x
(1) = 0
r
x
(2) + a
1
r
x
(1) + a
2
r
x
(0) = 0
or as usually given in matrix form
_

_
r
x
(0) r
x
(1) r
x
(2)
r
x
(1) r
x
(0) r
x
(1)
r
x
(2) r
x
(1) r
x
(0)
_

_
_

_
1
a
1
a
2
_

_
=
_

2
v
0
0
_

_
Lecture 2 13
But we need to use the equations dierently:
_

_
1 a
1
a
2
a
1
1 + a
2
0
a
2
a
1
1
_

_
_

_
r
x
(0)
r
x
(1)
r
x
(2)
_

_
=
_

2
v
0
0
_

_
Solving for r
x
(0), r
x
(1), r
x
(2) we obtain
r
x
(0) =
_
1 + a
2
1 a
2
_

2
v
(1 + a
2
)
2
a
2
1
r
x
(1) =
a
1
1 + a
2
r
x
(0)
r
x
(2) =
_
_
a
2
+
a
2
1
1 + a
2
_
_
r
x
(0)
In our example we need only the rst two values, r
x
(0), r
x
(1), which result to be r
x
(0) = 1, r
x
(1) = 0.5.
Now we will solve for the cross-correlations Ed(n)u(n), Ed(n)u(n 1). First observe
Eu(n)d(n) = E(x(n) + v
2
(n))d(n) = Ex(n)d(n)
Eu(n 1)d(n) = E(x(n 1) + v
2
(n 1))d(n) = Ex(n 1)d(n)
and now take as a master dierence equation
x(n) + bx(n 1) = d(n)
Lecture 2 14
and multiply it in turn with x(n) and x(n 1) and then take the expectation
Ex(n) Ex(n)x(n) + Ex(n)bx(n 1) = Ex(n)d(n)
Ex(n)d(n) = r
x
(0) + br
x
(1)
Ex(n 1) Ex(n 1)x(n) + Ex(n 1)bx(n 1) = Ex(n 1)d(n)
Ex(n 1)d(n) = r
x
(1) + br
x
(0)
Using the numerical values, one obtains
Eu(n)d(n) = Ex(n)d(n) = 0.5272 Eu(n 1)d(n) = Ex(n 1)d(n) = 0.4458
Now we have all necessary variables needed to write the Wiener Hopf equations
_
_
r
u
(0) r
u
(1)
r
u
(1) r
u
(0)
_
_
_
_
w
0
w
1
_
_
=
_
_
Ed(n)u(n)
Ed(n)u(n 1)
_
_
_
_
1.1 0.5
0.5 1.1
_
_
_
_
w
0
w
1
_
_
=
_
_
0.5272
0.4458
_
_
resulting in
_
_
w
0
w
1
_
_
=
_
_
0.8360
0.7853
_
_