Review of Probability

1/28

Random Variable
! Definition
Numerical characterization of outcome of a random
event

!Examples
1) Number on rolled dice
2) Temperature at specified time of day
3) Stock Market at close
4) Height of wheel going over a rocky road
2/28

Random Variable
! Non-examples
1) Heads or Tails on coin
2) Red or Black ball from urn

But we can make

these into RVs

! Basic Idea dont know how to completely

determine what value will occur
Can only specify probabilities of RV values occurring.

3/28

Two Types of Random Variables

Random Variable

Discrete RV
Die
Stocks

Continuous RV
Temperature
Wheel height

4/28

PDF for Continuous RV

Given Continuous RV X
What is the probability that X = x0 ?
Oddity : P(X = x0) = 0
Otherwise the Prob. Sums to infinity
Need to think of Prob. Density Function (PDF)
pX(x)

xo

The Probability density function

of RV X

xo +

P ( x0 < X < x0 + ) = area shown

=

xo +

p X ( x )dx

5/28

Most Commonly Used PDF: Gaussian

A RV X with the following PDF is called a Gaussian RV

p X ( x) =

( x m ) 2 / 2 2

m & are parameters of the Gaussian pdf

m = Mean of RV X

= Standard Deviation of RV
2 = Variance of RV X

X (Note:

> 0)

Notation: When X has Gaussian PDF we say X ~ N(m, 2)

6/28

Zero-Mean Gaussian PDF

" Generally: take the noise to be Zero Mean

p x ( x) =

x 2 2 2

7/28

Effect of Variance on Guassian PDF

pX(x)

Area within 1 of mean = 0.683

= 68.3%

x=m

pX(x)

Small

Small Variability
(Small Uncertainty)

pX(x)

Large

Large Variability
(Large Uncertainty)
8/28

Why Is Gaussian Used?

"Central Limit theorem (CLT)
The sum of N independent RVs has a pdf
that tends to be Gaussian as N
"So What! Here is what : Electronic systems generate
internal noise due to random motion of electrons in electronic
components. The noise is the result of summing the random
effects of lots of electrons.

CLT applies

Guassian Noise
9/28

p XY ( x , y )

Joint PDF of RVs X and Y

Describes probabilities of joint events concerning X and Y. For

example, the probability that X lies in interval [a,b] and Y lies in
interval [a,b] is given by:
bd

Pr{( a < X < b) and ( c < Y < d )} = p XY ( x, y )dxdy

ac

This graph shows the Joint PDF

Graph from B. P. Lathis book: Modern Digital & Analog Communication Systems

10/28

Conditional PDF of Two RVs

When you have two RVs often ask: What is the PDF of Y if X is
constrained to take on a specific value.
In other words: What is the PDF of Y conditioned on the fact X is
constrained to take on a specific value.
Ex.: Husbands salary X conditioned on wifes salary = $100K?
First find all wives who make EXACTLY $100K how are their
husbands salaries distributed.
Depends on the joint PDF because there are two RVs but it
should only depend on the slice of the joint PDF at Y=$100K.
Now we have to adjust this to account for the fact that the joint
PDF (even its slice) reflects how likely it is that X=$100K will
occur (e.g., if X=105 is unlikely then pXY(105,y) will be small); so
if we divide by pX(105) we adjust for this.

11/28

Conditional PDF (cont.)

Thus, the conditional PDFs are defined as (slice and normalize):
p XY ( x, y )
,

pY | X ( y | x ) = p X ( x )
0,

p X ( x) 0
otherwise

p XY ( x, y )
,

p X |Y ( x | y ) = pY ( y )
0,

x is held
fixed

pY ( y ) 0
otherwise

y is held
fixed

slice and
normalize
y is held fixed
This graph shows the Conditional PDF
Graph from B. P. Lathis book: Modern Digital & Analog Communication Systems

12/28

Independent RVs
Independence should be thought of as saying that:
neither RV impacts the other statistically thus, the
values that one will likely take should be irrelevant to the
value that the other has taken.
In other words: conditioning doesnt change the PDF!!!
pY | X = x ( y | x ) =

p XY ( x, y )
= pY ( y )
p X ( x)

p XY ( x, y )
p X |Y = y ( x | y ) =
= p X ( x)
pY ( y )
13/28

Independent and Dependent Gaussian PDFs

y

Independent
(zero mean)

Independent
(non-zero mean)

Contours of pXY(x,y).
x

y
x

Dependent

y
x

If X & Y are independent,

then the contour ellipses
are aligned with either the
x or y axis
Different slices
give
same normalized
curves
Different slices
give
different
normalized curves
14/28

An Independent RV Result
RVs X & Y are independent if:

p XY ( x, y ) = p X ( x ) pY ( y )
Heres why:
p XY ( x, y ) p X ( x ) pY ( y )
pY | X = x ( y | x ) =
=
= pY ( y )
p X ( x)
p X ( x)

15/28

Characterizing RVs
!

PDF tells everything about an RV

but sometimes they are more than we need/know

So we make due with a few Characteristics

Mean of an RV
Variance of an RV
Correlation of RVs

(Describes the centroid of PDF)

(Describes the spread of PDF)
(Describes tilt of joint PDF)

Mean = Average = Expected Value

Symbolically: E{X}
16/28

Motivating Idea of Mean of RV

Motivation First w/ Data Analysis View
Consider RV X = Score on a test Data: x1, x2, xN
Possible values of RV X : V0 V1 V2... V100
0 1 2 100
Test
Average = x =

N
x
i =1 i

N 0V0 + N1V1 + ... N nV100 100 N i

=
= Vi
N
N
i =0

Ni = # of scores of value Vi
n
N i (Total # of scores)
N =
i =1
This is called Data Analysis View
But it motivates the Data Modeling View

P(X = Vi)

Statistics
Probability

17/28

Theoretical View of Mean

Data Analysis View leads to Probability Theory:
" For Discrete random Variables :

Data Modeling

E{ X } = xi PX ( xi )
n =1

Probability Function

" This Motivates form for Continuous RV:

E{ X } =

x p X ( x )dx

Notation: E{ X } = X

Probability Density Function

Shorthand Notation

18/28

Aside: Probability vs. Statistics

Statistics
Given a set of Data
Determine how the
data did behave

Probability Theory
Given a PDF Model
Describe how the
data will likely behave

E{ X } =

x p X ( x )dx

PDF

Law of Large
Numbers

1
Avg =
N

xi
i =1

Data

Dummy Variable
There is no DATA here!!!
The PDF models how
the data will likely behave

There is no PDF here!!!

The Statistic measures how
the data did behave
19/28

Variance of RV
There are similar Data vs. Theory Views here
But lets go right to the theory!!

Variance: Characterizes how much you expect the

RV to Deviate Around the Mean
Variance: 2 = E{( X m x ) 2 }
= ( x m x ) 2 p X ( x )dx

Note : If zero mean

2 = E{ X 2 }
= x 2 p X ( x )dx

20/28

Motivating Idea of Correlation

Motivate First w/ Data Analysis View
Consider a random experiment that observes the
outcomes of two RVs:
Example: 2 RVs X and Y representing height and weight, respectively
y

Positively Correlated

21/28

Illustrating 3 Main Types of Correlation

Data Analysis View:
yy

C xy

( xi x )( yi y )
i =1

yy

xx
Positive Correlation
Best Friends
GPA
&
Starting Salary

1
=
N

yy

xx
Zero Correlation
i.e. uncorrelated
Complete Strangers
Height
&
$ in Pocket

xx
Negative Correlation
Worst Enemies
Student Loans
&
Parents Salary
22/28

Prob. Theory View of Correlation

To capture this, define Covariance :

XY = E{( X X )(Y Y )}
XY = ( x X )( y Y ) p XY ( x, y )dxdy

If the RVs are both Zero-mean :

If X = Y:

XY = {XY }

XY = X2 = Y2

If X & Y are independent, then:

XY = 0
23/28

If XY = E{( X X )(Y Y )} = 0
Then Say that X and Y are uncorrelated
If XY = E{( X X )(Y Y )} = 0
Then

E{ XY } = X Y
Called Correlation of X & Y

So RVs X and Y are said to be uncorrelated

if XY = 0
or equivalently if E{XY} = E{X}E{Y}
24/28

Independence vs. Uncorrelated

X & Y are
Independent

Implies

X & Y are
Uncorrelated
E{ XY }

f XY ( x, y )

= E{ X }E{Y }

= f X ( x ) fY ( y )

PDFs Separate

Means Separate
Uncorrelated
Independence

INDEPENDENCE IS A STRONGER CONDITION !!!!

25/28

Confusing Covariance and

Correlation Terminology
Covariance :

Correlation :

XY = E{( X X )(Y Y )}
E{XY }

Correlation Coefficient :

Same if zero mean

XY

XY
=
XY

1 XY 1
26/28

Covariance and Correlation For

Random Vectors
x = [ X 1 X 1 ! X N ]T
Correlation Matrix :
E {X 1 X 1 } E {X 1 X 2 }

E {X 2 X 1 } E {X 2 X 2 }
T
R x = E{xx } =

"
"

E {X N X 1 } E {X N X 2 }

E {X 1 X N }

! E {X 2 X N }

#
"

! E {X N X N }
!

Covariance Matrix :

C x = E{(x x )( x x )T }
27/28

A Few Properties of Expected Value

E{ X + Y } = E{ X } + E{Y }

E{aX } = aE{ X }

E{ f ( X )} = f ( x ) p X ( x )dx

2 + 2 + 2
Y
XY
X
var{ X + Y } =
2 + 2 , if X & Y are uncorrelated
Y
X

var{aX } = a 2 X2

{(
)}
= E {( X + Y ) } where X = X X
= E {( X ) + (Y ) +2 X Y }
= E {( X ) }+ E {(Y ) }+ 2 E { X Y }

var{ X + Y } = E X + Y X Y
z
z

2
2

z z

= X2 + Y2 + 2 XY

z z

28/28