16.584: Lecture 6: Moments of Random Variables

1
16.584: Lecture 6: Moments of Random Variables

Characterization of RVs using averages: Expectation Operator
Average or mean value of a set of data is

x
=
1
N
N
i=1
x
i
(1)
Average of data x
i
: Center of Gravity (CG) of the set
The number which is closest in distance to all other points in the set.
Minimize the distance given below with respect to z to yield
x
D
2
=
N
i=1
(z x
i
)
2
(2)
Average value provides an indication of the most likely value of the set
c Prof. K. Chandra, June 5, 2006 16.584: Probability and Random Processes; ECE, UMASS Lowell
2
Standard Deviation
To determine the spread or deviation of data from the average: the
standard deviation
x
of the set:

x
=
_
_
1
N
N
i=1
(x
i

x
)
2
_
_
1/2
(3)
Interpretation using Probability Measure
W.R.T. a probability space (, F, P):
Consider discrete RV x that can take on M distinct values x
1
, x
2
, ...x
M
.
In N trials of the experiment, assume x
1
occurs n
1
times, x
2
occurs n
2
... x
i
occurs n
i
times.
3
X =
1
N
M
i=1
n
i
x
i
=
M
i=1
n
i
N
x
i
=
M
i=1
x
i
P[X = x
i
] (4)
Expected or Average value of a discrete rv X taking on values x
i
with
PMF P
X
(x
i
) is:
E[X] =

i
x
i
P[X = x
i
] =

i
x
i
P
X
(x
i
) (5)
Cont. RV X: E[X] =
_
xf
X
(x)dx
4
Expected value Contd.
Function of a RV Y = g(X)
E[Y ] = E[g(X)] =
_
g(x)f
X
(x)dx =
_
y
yf
Y
(y)
X discrete: E[Y ] =

i
g(x
i
)P
X
(x
i
)
Examples:
X : N(,
2
): Use transformation z =
(x)
to show that E[X] =

X : e
x
E[X] =
_
0
xe
x
dx Integrate by Parts
E[X] =
_
x
e
x
_
0
+
_
0
e
x
dx
=
1
5
Expectation Contd.: Properties
E
_
N
i=1
g
i
(X)
_
=

N
i=1
E[g
i
(X)]
Function of two Random Variables: Z = g(X, Y )
E[Z] =
_
z f
Z
(z)dz
E[Z] = E[g(x, y)] =
_
g(x, y)f
XY
(x, y)dxdy
Ex: If Z = X + Y
E[X + Y ] =
_
(X + Y )f
XY
(x, y)dxdy
=
_
x
__
f
XY
(x, y)dy
_
+
_
y
__
f
XY
(x, y)dx
_
= E[X] + E[Y ]
NOTE: Independence is not required in showing that E[X + Y ] =
E[X] + E[Y ].
In general : E(aX + bY ) = aE(X) + bE(Y )
6
Conditional Expectations
To determine averages of a subset of RVs that are conditioned on an
event B: E(X|B) =
_
xf
X|B
(x|B)dx
For discrete X: E(X|B) =

i=1
x
i
P
X|B
(x
i
|B)
Example:B [X a]
F
X|B
(x|X a) = 0, x < a
=
F
X
(x) F
X
(a)
1 F
X
(a)
, x a
f
X|B
(x|X a) = 0, x < a
=
f
X
(x)
1F
X
(a)
x a
Therefore, E [X|X a] =
_
a
xf
X
(x)dx
_
a
f
X
(x)dx
7
Conditional Expectations Contd.
Conditional expectations often occur w.r.t. RVs that are related to
each other.
If X, Y : two discrete random variables, with joint PMF P
X,Y
(x
i
, y
j
)
Conditional expectation of Y given X = x
i
is E[Y |X = x
i
] is:
E[Y |X = x
i
] =

j
y
j
P
Y |X
(y
j
|x
i
) (6)
P
Y |X
(y
j
|x
i
) is conditional probability that Y = y
j
given X = x
i
has
occurred and
P
Y |X
(y
j
|x
i
) =
P
XY
(x
i
, y
j
)
P
X
(x
i
)
(7)
Conditional expectation : E[Y |X = x] is a random variable, ( a func-
tion of rv x ).
8
E[Y ] can be determined as:
E[Y ] =

x
E[Y |X = x]P[X = x]
=

y
x
yP[Y = y|X = x]P[X = x]
=

x
y
yP[Y = y, X = x]
=

y
y
x
P[Y = y, X = x]
=

y
yP[Y = y]
9
E[Y ] = E [E(Y |X)]
If Y = g(X) , where X is a discrete RV
E[Y ] =

i
g(x
i
)P
X
(x
i
) = E[g(x)] (8)
If E[Y |X] = g(X),
E[E(Y |X)] =

x
E[Y |X]f
X
(x) = E[Y ]
For continuous Rvs, : E[Y ] =
_
E[Y |X = x]f
X
(x)dx
Note: If X, Y are independent, E[Y |X] = E[Y ]
10
Moments of a RV
If k is a positive integer, the k
th
moment m
k
of X is dened as:
k
= E[X
k
]
The k
th
central moment is:
k
= E[(X
X
)
k
] where
X
= E[X]
For k = 2, m
2
=
2
X
the variance of X : V ar[X].
Show that:
2
X
= E[X
2
]
2
X
.
Note: Other notation for expectation operation: E[X
k
] = X
k
.
Under certain conditions possible to reconstruct pdf from knowledge
of all of the moments of a RV.
11
Examples
Bernoulli Variables
f
X
(x) = p(x 1) + (1 p)(x)
E[X] = p 1 + 0 (1 p) = p E[X
2
] = p
Variance:
2
= E[X
2
] E[X]
2
= p p
2
= p(1 p) = pq
Binomial Variables X : f
X
(k)
Find E[X] and V ar[X]
12
Joint Moments
Consider two rvs X and Y possibly related in some way. Y = g(X)
Classical Problem: Infer or Estimate Y based on observations of X
If X, Y independent, information of X gives no information of Y
Ex: Linear Prediction: Y = aX + b
Observe X, predict Y exactly with knowledge of a, b.
In absence of explicit relations between X, Y , calculate the joint
moments of X, Y to infer the dependence features
The ij
th
joint moment of X and Y is:

ij
= E[X
i
Y
j
] =
_
y
_
x
f
XY
(x, y)x
i
y
j
dxdy
Discrete:
ij
= E[X
i
Y
j
] = =

y
x
P
XY
(x, y)x
i
y
j
13
Central Moments
The ij
th
central moment of X and Y is:
m
ij
= E[(X X)
i
(Y Y )
j
] (9)
The order of the moment is i + j.
Consider the case i = 1, j = 1,
m
11
= E[(X X)(Y Y )] = E[XY ] XY (10)
m
11
: covariance of X and Y , denoted as Cov[X, Y ].

11
= E[XY ]: correlation of X and Y .
14
Correlation Coecient
Correlation coecient : (Normalized covariance) between Rvs (X,Y):
XY
=
m
11
m
20
m
02
(11)
|| 1
If Cov[X, Y ] = 0, then = 0 and X and Y are said to be uncorrelated.
15
Properties of uncorrelated Random Variables
If X and Y are uncorrelated
2
X+Y
= E[(X+Y )
2
](E[X + Y ])
2
=
2
X
+
2
Y
If X and Y are independent, they are also uncorrelated. Proof:
E[XY ] =
_
x
_
y
xyf
X
(x)f
Y
(y)dxdy = E[X]E[Y ] (12)
Since Cov[X, Y ] = E[XY ] E[X]E[Y ], Cov[X, Y ] = 0.
Independent random variables are uncorrelated; The re-
verse is not generally true.
16
Linear Prediction
Problem: Predicting values of a rv Y by observing the values of rv X.
Data corresponding to measurements of X are available.
The model to be applied is:
Y = aX + b (13)
Actual value of Y aected by other sources independent of X, such as
additive Gaussian noise.
Denote the error = Y

Y
Objective: Find a and b that minimize the mean square error, (MMSE):

2
= E[(Y

Y )
2
]
This problem termed optimum linear prediction or linear regression in
statistics.
17
Linear Prediction: Solution
Expand the mse after substituting

Y = aX + b

2
= Y
2
2aXY 2bY + 2abX + a
2
X
2
+ b
2
(14)

2
a
=

2
b
= 0 (15)
Denote optimal solutions as a
0
, b
0
,
a
0
=
Cov(X, Y )
2
X
=

XY
X
(16)
b
0
= Y
Cov(X, Y )
2
X
X = Y
XY
X
X (17)
Substituting for a
0
and b
0
, the best linear predictor of Y is
18
Y Y =
XY
X
(X X) (18)
It passes through the point (X, Y ). (Averages)
19
Moment Generating Functions (MGF)
MGF of a cont. rv X is dened in terms of a transformed variable t, in
general complex, as
(t) = E[e
tX
]
=
_
e
tx
f
X
(x)dx
For X a discrete rv,
(t) =

i
e
tx
i
P[X = x
i
] (19)
Similar to Laplace transforms: f(x) F(t)
F(t) =
_
f(x)e
tx
dx.
20
Application of MGF
MGF applied for estimating the moments in the absence of knowledge
of f
X
(x).
If
k
= E[X
k
],
(t) = E[e
tX
]
= E
_
_
1 + tX +
tX
2
2!
+ ... +
tX
n
n!
+ ...
_
_
= 1 + t
1
+
t
2
2!
2
+ ... +
t
n
n!
n
+ ..
(21)
Taking derivatives of (t) and setting t = 0:
k
(t)|
t=0
=
k
k = 1, 2, ...
21
Example: Poisson RV: Find (t)
P[X = n] =
e
n
n!
(t) = e
n=0
e
tn
n
n!
= e
e
e
t
(t) = e
(e
t
1)
Taking derivatives:
(t)
= e
t
e
(e
t
1)
(t) = (e
t
)
2
e
(e
t
1)
+ e
t
e
(e
t
1)
Substituting t = 0, E[X] = and E[X
2
] =
2
+
2
X
= E[X
2
] E[X]
2
=
22
Example: Exponential RV : f
X
(x) = e
x
(t) =
_
0
e
x
e
tx
dx
=

t
Moments:
E[X] =

( t)
2
t=0
=
1
E[X
2
] =
2
(t )
2
t=0
=
2
2
23
MGF of two random variables
XY
(t
1
, t
2
) = E
_
e
(t
1
X+t
2
Y )
_
=
_
e
t
1
x+t
2
y
f
XY
(x, y)dxdy
For discrete rvs: (i, j) with
ij
= E[X
i
Y
j
],
XY
(t
1
, t
2
) =
i=0
j=0
t
j
1
t
k
2
i!j!
ij
To obtain the joint moments, note that:

ln
=
(l,n)
XY
(0, 0)

1,0
= E[X]
0,1
= E[Y ]

2,0
= E[X
2
]
0,2
= E[Y
2
]

1,1
= E[XY ] = Cov[X, Y ] E[X]E[Y ]
24
Consider Z = X + Y , where X, Y are independent RVs,
X+Y
(t) = E[e
t(X+Y )
]
= E[e
tX
]E[e
tY
]
X+Y
(t) =
X
(t)
Y
(t)
Example: Y =

N
i=1
x
i
, where x
i
are iid rvs and N is a rv. Find E[Y ]
and V ar[Y ] in terms of the moments of X and N.
Consider the conditional expectation E[e
t
n
i=1
x
i
|
N=n
]
Y |N
(t) = E[e
t
n
i=1
x
i
|
N=n
]
= (
X
(t))
n
To obtain
Y
(t), take the expectation of
Y |N
(t) :

Y
(t) = E[
Y |N
(t)] = E[(
X
(t))
N
]
25
To compute E[Y ] and V ar[Y ], obtain the rst and second derivatives
of
Y
(t):
Y
(t) = E[N
X
(t)
N1
X
(t)]
E[Y ] =
Y
(0) = E[N(
X
(0))
N1
X
(0)]
= E[N(1)
N1
E[X]] = E[N]E[X]
Y
(t) = E N(N 1)
X
(t)
N2
X
(t)
X
(t) + N(
X
(t))
N1
X
(t)
Y
(0) = E N(N 1)E[X]
2
+ E[N]E[X
2
]
E[Y
2
] = E[X]
2
(E[N
2
] E[N]) + E[N]E[X
2
]
= E[N]V ar[X] + E[X]
2
E[N
2
]
V ar[Y ] = E[Y
2
] E[Y ]
2
E[N
2
] E[N]
2
E[X]
2
2
V ar[N]
26
Characteristic Functions of a RV
MGFs uniquely determine the PMF if all moments exist and are known.
This implies that (t) exists and is nite in some region about t = 0.
Useful to isolate a region t = j in the MGF.
Resulting function is referred to as the characteristic function
X
()
X
() = E[e
jX
]
=
_
x
f
X
(x)e
jx
dx
Other than a change of sign in the exponent,
X
() : Fourier Transform
of pdf f
X
(x).
Allows determination of f
X
(x) may be obtained by inverse transform
methods: f
X
(x) =
1
2
_

X
()e
jx
d
27
Charc. Functions Contd.
For discrete and integer valued random variable X = 0, 1, ....,
X
() =
k=0
P[X = k]e
jk
which represents the FT of the sequence p
k
= P[X = k].

X
() is periodic in with a period of 2, (e
jk
= e
jk(+2)
).
Example: Geometric RV X : p
k
= pq
k
Find
X
()
X
() =
k=0
pq
k
e
jk
= p
(qe
j
)
k
=
p
1 qe
j
28
Moment Estimation from
X
()
E[X
n
] =
1
j
n
d
n
X
()
d
n
|
=0
X
() =
_
f
X
(x)
_
_
1 + jx +
(jx)
2
2!
+ ....
_
_
dx
= 1 + jE[X]
(j)
2
2!
E[X
2
] + ...
29
Probability Generating Functions (PGF)
For non-negative rvs: convenient to use the Laplace transform (cont rv)
or Z transform (discrete rv).
PGF G
X
(z) of a discrete rv X is : G
X
(z) = E[Z
k
] =

k=0
p
k
z
k
Characteristic function of X obtained by substituting z = e
j
in the
PGF
Taking derivatives of G
X
(z):
d
dz
G
X
(z) =
k=0
kp
k
z
k1
d
dz
G
X
(0) = (1)p
1
d
2
dz
2
G
X
(0) = (2)(1)p
2
d
n
dz
n
G
X
(0) = n!p
n
30
p
n
=
1
n!
d
n
dz
n
G
X
(0)
Hence the name Probability Generating Function.
31
The moments can also be estimated from the PGF by substituting z = 1
after taking the derivatives:G
X
(1) =

k=0
kp
k
= E[X]
G
X
(1) =
k=0
k(k 1)p
k
=
k=0
k
2
p
k

k=0
kp
k
= E[X
2
] E[X]
V ar[X] = G
X
(1) + G
X
(1) (G
X
(1))
2
32
Laplace Transforms
For continuous non-negative random variables, the Laplace transform is
considered:
X
(s) =
_
0
f
X
(x)e
sx
dx
= E[e
sX
]
E[X
n
] = (1)
n
d
n
ds
n
X
(s)|
s=0

16.584: Lecture 6: Moments of Random Variables

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

16.584: Lecture 6: Moments of Random Variables

Transféré par

Droits d'auteur :

Formats disponibles

1

16.584: Lecture 6: Moments of Random Variables

to show that E[X] =

Vous aimerez peut-être aussi