A Bayesian Approach to Problems in
StochasticEstimation and Control
Summary-In this paper, a general classof stochastic estimation
and control problems is formulatedfromthe BayesianDecision-
Theoretic viewpoint. A discussion as to how these problems can be
solved step by step in principle and practice from this approach is
presented. As a specific example, the closed form W i e n e r - K h a n
solution for linear estimation in Gaussian noise is derived. The pur-
pose of the paper is to show that the Bayesian approach provides;
1) ageneral unifying framework withinwhich topursuefurther
researches in stochastic estimation and control problems, and 2) the
necessary computations and difficulties that must be overcome for
these problems. An example of a nonlinear, non-Gaussian estimation
problem is also solved.
SINGLE
STAGE
ESTIYATIOS
PROBLEM
F O R T H E PURPOSE of illustrating the concepts
involved, the single-stage estimation problem will
be discussed first. Once this is accomplished, the
multistage problem can be treated straightforwardly.
Problem Statement
The following information is assumed, given;
set of measurements 21, 22, . . . , zk which are
denoted by the vector z.
The phsrsical relationship between the stateof na-
ture which is to be estimated and the measure-
ments. This is given by
2 = g,(. v), (1)
where
z is the measurement vector ( k x l ) ,
x is the state (signal) vector ( n X l ) , and
v is the noise vector ( q X 1).
The joint density function p ( x , a). From this the
respectivemarginaldensityfunctions, p ( r ) and
pb),are readily obtained.
is assumed that information for 3) is available in
analytical form or can be approximated by analytical
distributions. Item 2) can be either in closed form or
merely computable. The problem is to obtain an esti-
mate of x upon which t o base the best measurements
which will be defined later.
The Bayesian Solzrtion
The Bayesiansolution totheaboveproblem now
proceeds via the following steps:
1) Evaluate p(z)-This can be done analytically,
Manuscript received October 31, 1963; revised July 2, 1964.
Y . C. Ho is with Harvard Cniversity, Cambridge, Mass., and
-
also a Consultant a t t h e Minneapolis-Honeywell Regulator Co..
Boston, Mass.
R. C . K. Lee was formerly with the Minneapolis-Honeywell Regu-
lator Co., Minneapolis, Minn.
least in principle,orexperimentallybyhlonte
Carlomethodssince z = g ( x , v) and p ( x , 21) are
given. In t h e latter case, it is assumed possible t o
fit the experimental distribution again by a mem-
ber of a family of distributions.
A t thispoint,twoalternativesarepossible,one
maybesuperiortotheotherdepending on the
nature of the problem.
Evaluate p ( x , z ) . This is possible analytically if
v is of the same dimension asz and one can ob-
tain the functional relationship v = g * ( x , z ) from
(1). Then, using p ( x , v) and the theory of de-
rived distributions, one obtains
where
p(., 2) = p ( x , z' = g*(n, z ) ) J (2)
Evaluate p ( z / x ) . This conditional density func-
tion can always be obtained either analytically,
whenever possible, or experimentally from the
z = g ( x , v) and p ( x , v).
Note that 2a) may be difficult to obtain in gen-
eral since g* may not exist either because of the
nonlinear nature of g or the fact that z , v are of
different dimensions. Nevertheless, 2b) can always
be carried out. This fact will be demonstrated in
the nonlinear example in the sequel.
Evaluate p ( x / z ) using the following relationships:
a) Following2a)
b) Following 2b), use the Bayes' rule
Depending on the classof distributions one has as-
sumed or obtained for p ( ~n),, p ( ~ )p,( z / x ) ,this key
step may be easy or difficult to carry out. Several
classes of distribution which have nice properties
at for thispurposecanbefound in Raiffa and
Schlaifer [l]. The density
function f ( x / z ) is
known as the a posteriori density function of x. I t
is
is the knowledge about the state of nature after
the measurements z. By definition, it contains all
the information necessary for estimation.
333
334 TRANSA
I ECETEI O N S ON AUTOMATIC CONTROL October

4) Depending on the criterion function

for estimation, 1) Evaluate p ( z ) .
onecancomputeestimate f from p ( s / z ) . Some Since Z = H x f v and x, z are Gaussian and inde-
typical examples are pendent, one immediately gets
Criterion: XIaximize the Probability ( i = x ) .
p ( z ) is Gaussian
Solution: f = 1Iode of p ( x / z ) . (5)
This is defined as the most probable estimate.
When the a priori density function p ( x ) is uni-
form, this estimate is identical to the classical
maximum likelihood estimate. 2a) Evaluate P(X, z ) . Since (dg*,:'dz) = Identify matrix,
Criterion: AIinimize J I : x - . r l ~ z p ( . x ~ z ) d . ~ . i t follows
Solution : f = E ( x / z ) (6) p ( ~ Z), = p ( ~ ,= z - Hx)
This is the conditional mean estimate.
Criterion: LIinimize 1Iaximum x - 21. I = p(r)p,(z - Hx).2 (13)
Solution: ? = 1Iedium of p(x,'z). (7)
2b) Evaluate p ( ~ l x ) . ~
This can be defined as the minimax estimate.
Pictorially, the three estimates are shownin Fig. 1
for a general p ( x / z ) for a scalar case.

3) Evaluate p ( x / z ) . One gets from Bayes' rule,

By direct substitution of (to), (1I),

and (12), one
obtains
1 HP0HT + R
p(./z) =
( 2 ~ ) I 4Po
~ 1 'I2 I R I
x0 - Most Probable
Estimate
.exp [ -1/2[(x - f)Tpo-'(x - :T)
-
x,,
xc -
Condltional
Minimax
Mean
Estlmate
Estimate + (Z - H x ) ~ R - ' ( z- H x )
Fig. I-Estimates based on a posteriori density. - ( z - H f ) T ( H P o H T+ R)-'(z - H f ) ]f . (16)

Kow completing squares in the { ) , (16) simplifies

Clearly, other estimates, as well as confidence intervals to
can be derived from p ( x l z ) directly.
p(x/z)
I NPoHT + R I1!?
Speciul C a s e of the Wiener-Kalnzan F,ilter ( s i n g l e stage)
=
(2g)n12I Po j R 11:9 11!2

S o w a special case of the above estimation problem .exp { - 1:2(x - ? ) T P ' ( x - 2) 1 (15)
will be considered. Let there be given;
1) 4 set of measurements z = (21, 22, .. .,z~). where
2) T h e physical relationship
z = Hs + c. (8)
or equivalently,
3) The independent noise and state density functions
p(., = p(x)p(e) (9)
@(x) be Gaussian with and
.i. = 3 + PHTR-'(z - Hm). (20)
E(r) = 3
cov ( x ) = Po 1
p ( c ) be Gaussian with
4) Kow, since p ( x / z ) is Gaussian, the most probable,
conditional mean and minimax estimate all coin-
E(t) = 0 cide and is given by ?.

Cov (I')= R This is thederivation of the single stageWeiner-

Kalman filter [2], [SI. The pair ( P , a) is called a sufi-
Kow, following the steps for the Bayesian solution,
* p,(z--Hx) means substituting ( z - H x ) for v in p ( v ) .
I t is assumed that p ( x / z ) has finite second moment. Note: step (2b) is redundant.
1964 H
Stochastic oBayesian
inApproach
and Lee: Estimation 335

cientstatistic for the problem in the sense t h a t p ( x / z )

=p(x/P, i).

ESTIMATIOK
~IULTISTAGE PROBLEM
T h e problem formulation and the solutionin this case
is basicallysimilar tothe single-stageproblem. T h e
only additional complication is t h a t now the state is
changing from stage to stage according to some dynamicFrom this the marginal density functions p(xk+l/Z,)
relationship, and that thea posteriori density function is and p(zkTljZk) can be directly evaluated.
to be computed recursively. 4) Evaluate

from (22)

p(xk+l/Zk+l) exactly as in the single-stage case.

Thesystemequationsgoverningtheevolution
of the state. Special Case of the Wiener-Kalman Filter5
T h e given d a t a a t k+ 1 is specified as follows:
The physical model is given by

where
xk+l is the state vector atk 1 +
~ k + 1 is the measurement noise a t k 1, +
z g + l is the additional measurement available a t where w and v are independent, white, Gaussian random
k+1, sequences with
w k is the disturbance vector a t k. P(.~kiZk) is Gaussian
The complete set of measurements Z k + l 2 (zl? . . ,
Zk+l).
The density functions4
A
P(Xk:/Zl, . . * , z k ) = p ( x k / z k )
~ ( w Ro/;+dxx.)statistics
, of avectorrandomse-
quencewithcomponents wk and v!:+~which de-
pends on x k .
Now i t is required to estimate xp+l based on measure-
ments 2 1 , . . . , z k + l . 5 This development of themultistage IViener-Kalman filtering
method is very similar to a paper by H. Rauch, F. Tung, and C.
C. Striebelentitled "On The ?&ximum Likelihood Estimatefor
The Bayesian Solution LinearDynamicSystems"presented at the S1-431 Conference on
System Optimization, 1964, Monterey, Calif. The onlydifference be-
The procedure is analogous to the single-stage case, tween the two
developments is thatthe Rauch-Tung-Striebel
1) Evaluate p(xg+l,/xp). Thiscanbe accomplished paper does not explicitly compute p ( x / z ) but simply computes its
maximum and uses it as the estimate. In the author's approach, the
either experimentally or analytically from knowl- computation of the maximum plays a secondary role. The explicit
edge of p(wk,vk+l/xl;), p(xk/Z,) and (21). calculation of the a posleriori probability is emphasized as the
Bayesian viewpoint. The authors are indebted toProf. &A. E. Bryson
for bringing this reference to their attention. Similar development of
the Wiener-Kalman filter is also presented in XAS.\ TR-R-135 1962
by G. L. Smith, S. Schmidt, and L. A. Megee. This was brought to
the authors' attention after the publicationof t h e J a C Cpreprint.
336 TRANSA
I ECETEI O X S OAT A U T O M A T I C C O N T R O L October

Since in this case, the noise wk, vk+1 is not dependent on .A SIMPLE XONLINEXR NON-GAUSSIAK
the state, (24) simplifies t o ESTIMATIONPROBLEM
The discussionsin theabovesectionshavebeen
carried out in terms of continuous density functions.
However, it is obvious that the same process can be
applied to problems involving discrete density function
and discontinuous functional relationships. I t is worth-
while, a t this point, to carry out one such solution for
a simple contriced example which nevertheless illustrates
the application of the basic approach.
The problems can be visualized as an abstraction of
the following physical estimation problem. An infrared
detector followed by a thresholddevice is usedin a
satellite to detect hot targets on the ground. However,
extraneous signals, particularly reflection from clouds,
obscure the measurements. The problem is t o design a
multistage estimation process to estimate the presence
of hot targets through measurement of the outputof the
threshold detector.
Let SI; (target) be scalar independent Bernoulli proc-
ess with,

Combining (28-30) using (24), one gets

p(sd = (1 - 4)6(Sk) + @(I - Sk), (37Y
ZB (cloud noise) be a scalar Markov process with,

p(t21) = (1 - ~ ) ~ ( T z I ) + - TZI) (38)

p(fzk+d%k) = (I - a - - );' 6(12k+l)

and the scalar measurement,

Zk = Sk f3f2k (40)

where @ indicates the logical "OR" operation.

Essentially (37-40) indicate the fact that as the de-
tector sweeps across the field of view, cloud reflection
tends to appear in groups while targets appear in iso-
lated dots.
Kow to proceed to the Bayesian solution. First, there
is,
711 O j l 1

SI l o 1 1 0

Probability of ZI )(1- a ) ( l -q)l q(l - a ) I a( 1- q ) 1 aq

p(z1) = (1 - a ) ( l - q)6(21) + ( a + q - aq)d(z1 - 1). (41)

T h e notation
I1 x = o
6(x) = 4,
;o x #O
is used here. Also, p ( x ) is to be interpreted as mass functions.
I964 H o and Lee: Bayesian Approach in Stochastic Estimation 337

Also, which,afterstraightforwardbutsomewhatlaborious
P(Zl/?Zl) = 6(Zl - 1)m + (1 - q)6(21)
manipulations, becomes

+ @(el - 1)](1 - n1). (42)

Then by direct calculation,
2 (1 - U ( z l ) ) s ( n z ) + a(zl)s(n2 - 1).
Furthermore,
= (1 - a’(21))6(121) + a’(e1>6(n1- l ) , (43)
I? here

Eqs. (49) and (50) now take the place of ( 3 i ) and (38)
and by the same process, one can get, in general,

similarly, Eqs. (51-57) now represent the general recursion solu-

tion for the multistage estimation process.
As a check,twopossibleobservedsequencesfor z,
(45) namely (0, 1) and (1, 1) areconsidered.With a = 1/4
and and q = lj4 it is found t h a t P ( s z / z ~ ,21) =0.571and
0.337, respectively. This agrees with intuition since the
sequence (1, 1) has a higher probability of being cloud
reflections. On the other hand, the numbers also showed
= (1 - q’(z1))6(sd + q’(z1)6(s1 - 1) (46)
that under the circumstances, it is very difficult to de-
tecttargetswithaccuracyusingthesystemcontrived
where here.
Oftentimes,one is actuallyinterested in p(x~:/Zk+~)
qS(z1 - 1)
(4;) with T > O inordertoobtaintheso-called“smoothed”
q’(z1) =
(1 - 4 ( 1 - P W l ) + (a+q - aq)6(21 - 1) ’ estimatefor sk. T h e desireddensityfunctioncanbe
and a reasonable estimate is computed from p(sk/Zk) byfurthermanipulations.
However, the calculation becomes involved andwill not
1 if ~‘(zI)> e (Givenconstant) be done here.
$1 =
{0 if q‘(z1) <e
(48)
RELATIOSSHIP TO GENERAL BAYESIAN
STATISTICAL DECISIOSTHEORY
where $1 = 1 may be interpreted as an alarm.
NOWconsider a secondmeasurement z p has been I t is worthwhile to pointouttherelationship of the
made. One has above formulation and solution of the estimation prob-
lem to and its differencefrom thegeneralstatistical
p(n?ll-Zl)p(~Z]IZl)d72l?
(49) decisionproblem.Forsimplicity,thesingle-stagecase
is consideredagain. In the general statistical decision
338 IEEE T R A X SO
ANCTIOXS A UTOlIfATIC CONTROL Octobev

problem,theinputdata is somewhatdifferent.One theoretic technology, this fact is called the Gen-

typical form is,' eralized Decomposition Xxiom.
p(x)-a priori density of x As anexample,considerthe single stageWiener-
le1-a set of choices of experimentsfrom Kalman problem and the added requirement that,
which we canderivemeasurements z
with J ( e , z, u, x) = ~ ( ux), = E ] B.Y
( + 2

p ( z / x , e)- conditional density of e for given x and e.

{ z~}-a set of choices of decisions (60)
J ( e , z , 21, x)-a criterion function which is a possible
function of e, z , ZL and x. be a minimum. Expanding (60), one gets
The problem is then stated as the determination of e
and ZI so t h a t E ( J ) is optimized. The optimal J is given
by
clearly,
JOpt
= Max (Min)
e
uopt= U ( i ) zt(x(z)) A u(z) = - B f ( z ) , (62)

which is one of the fundamental results of linear sto-

chasticcontrol.Thus,thecontrolaction 21 is only a

Thus, the main differences between the estimation prob- function of the criterion J and the a posteriori density
lem and the general decision problem are as follows: function p ( x l z ) . In fact, in this case only .
t of p(x,;'z) is
needed. \f7e call .i: as the mitzimal szfficient statistic for
1) In the estimation problem there is no choice of ex- the control problem.
periment.One alwas-s makesthesametype of In the more general multistage case, the decomposi-
measurement z given by g(x, a). T o generalize the tion property clearly still holds, the on117 difference be-
estimation problem. one can specify, ing t h a t p(x~~+l:Zk+1) is now dependent on u A : . However,
z e = g e (x,a ) ; { e ]= I , 2 . . . . this dependence is entirely deterministic since, in a given
=possible sets of measurements (59) situation, onealways knows what u k ' s are. In fact,in the
ij?ener-E;alman control problem, i t is known t h a t u k is
and then require that a linear function of .i-n only.
32 = O p t [(.?le
e
e = I , 2,. . . 1. COKCLUSIOK
2) In the generaldecision problem,thefunction I n the above sections, the problemof estimation from
the Bas-esian viewpoint is discussed. I t is the author's
z =g(x, I?)is implicit in p(z/x, e). Hence step 2a)
thesis that this approachoffers a unifs-ing methodology,
and 2b) for the estimation solution is not required.
a t least conceptually, to the general problems of estima-
This is often a tremendous simplification.
tion and control.
3) In the estimation problem the criterion function J
The a posreriori conditional density function p ( x l z ) is
is a h a \ - s a simplefunction of x only. There is,
seen to be the key to the solution of the general prob-
furthermore, no choice of action (one has to make
lem. Difficulties associated with the solution of the gen-
an estimate by definition). On the other hand, the
eral problem now appear more specifically as difficulties
generaldecisionproblem is more analogousto a
in steps leading to the computationof p ( x l z ) . From the
combinedestimationandcontrolproblemwhere
abovediscussions, i t is relativelyobviousthatthese
one has a further choice of action aiter determin-
difficulties are
ing ~ ( x i ' z )and
, like a control problem. the criteria
iunction is generally more complex 1) Computation of p ( z / x ) . In both the single-stage or
4) I t is, hen-ever, to be noted that the kel- stepis the multistagecase,thisproblem is complicatedby
computation of p ( s : ' z ) forbothproblems. The thenonlinearfunctionalrelationshipsbetween z
choice of action is determined only after the com- and x. Except in t h e case when z and x are linearly
putation of p ( x , / z ) . Thus, a general decision prob- related or when z and x are scalars, very little can
lem can be composed into two problems. namely, be done in general, analyticall>-or experimentally.
determination of P(x,:z) (estimation problem) and -As was mentioned earlier? this dificulty does not
choice of action(controlproblem).Incontrol- appear in the usual decision problem, since there
it is assumed that p(z;x) is given as part of the
For other equivalent forms, see Raiffa and Schlaifer [lj.
problem.
* See [l]. 2) Requirementthat p ( x / z ) be in analyticalform.
I964 IEEE T R A M S A C T I OCL 0Y,SVO
T RSOTALIGC: T O A I A 4 339

This is an obvious requirement if we intend to use
the solution in real-time applications.
3) Requirements that p ( x ) , p ( z ) , p ( x / z ) be conjugate
distributions [I]. This is simply the requirement
t h a t p ( x ) and p(x/zj bedensityfunctionsfrom
the same family. Sote that all the examples dis-
cussed in this paper possess this desirable prop-
erty. This is precisely the reason that multistage
computation can be doneefficiently. This imposed
a further restriction on the functionsg, f and h.
T h e difficulties (1-3) listed above are formidableones.
I t is not likely that they can beeasily circumvented ex-
cept for special classes of problems such as those dis-
cussed. However, i t is worthwhile first to pinpoint these
difficulties. Research toward their solution can then be
effectively initiated. Finally, it is felt that the Bayesian
approach offers a unified and intuitive viewpoint par-
ticularlyadaptabletohandlingmodern-daycontrol
problems where the State and the -1larkoa assumptions
play a fundamental role.
REFERENCES
[ l ] H. Raiffa and R. Schlaifer, “Xpplied Statistical Decision Theory”
Hanrard University Press, Cambridge, Mass., chs. 1-3; 1961.
[2] A . E. Bryson and 3.1. Frazler, “Smoothing in linear and nonlinear
systems,” Proc. of Optimal System Synthesis Symp., \\-right Field,
Ohio; 1962.
[3] Y. C . Ho, “On stochasticapproximationandoptimalfiltering
method,” J . X a t h . Analysis and Applications, vol. 6 , pp. 152-151;
February, 1963.
[1]R. E. k h a n , “Sew hIethods and Results in Linear Prediction
and Filtering Theory,” Research Institute for XdvancedStudies,
Baltimore, xid,, No.-61-1.

The Minimization of Measurement Error in a General

Perturbation-Correlation Process
Identification System

Summary-A general identilication systemisstudied for an Althoughonlyone of the basicfunctions,the self-

important class of realistic, time-varying processes. This class con- adaptive subsystem is dominated by the identification
sists of those inwhich the process is nominally known, and the statis-
tical characteristics of its varying parameters andof the environment
technique employed. It has been shown t h a t regardless
are also known. The expressionfor identification error in terms of the of its particular form, the identificationprocess performs
spectralproperties of the parameter variations and of the output its function by measuring one of the following:
transducer noise is developed.Optimization procedures are given
tominimizetheperturbation-correlationsystem’smean-square
1) the coefficients of the differential equation,
identification error. 2) the impulse or step response,
3) the frequency response.
INTRODCCTION
Similarly, in order to identify a system, i t is necessary
I T H T H E I K C R E A S E in automation and the

Wd emand formoreself-contained,sophisticated
systems, a moregeneralapproach
to have available, either directly or indirectly, both the
input and the output signals. Either normal input
to s>-stem special test signals are employed. Usually, the latter is
measurement must be taken. This is especialll- true in preferred, since it guarantees system measurement
or

re-
the field of self-adaptive control [1]-[13]. Here, before gardless of the level of normal plant activity.
thesystemcan performself-adjustment i t firstmust T h e basic function of identification has been treated
identify itself. Then, this measurement must be evalu- extensively in theliterature.However,thecombined
ated in accordance with a predetermined criterion and consideration of measurement noise, u priori knowledge,
mathematical model of the dpamical system. Thus, oneand time-varving system parameters has notbeen given
sees thatthe basicfunctions for self-adaptionare: sufficient treatment. The major block of identification
identification, evaluation, and adjustment. techniquesdealswiththelearningprocessassociated
with abstractsystems.CooperandLindenlaub[14]
Manuscript received October 1.5, 1963; revised July 13, 1964. pointed out that no investigation had been made which
The author is with the Dept. of Electrical Engineering, Rutgers,
The State University, S e w Brunswick, N . J. incorporated the effects of noise and a priori knowledge.