Basic Probability Theory

Lecture 5
Lecturer: Ali Ghodsi
Notes: Jim Dowling
October 18, 2007

Repetition: Bayes Theorem

P (B|A) =
if we can calculate

P (A B),

P (A B)
P (A)

then do so,

If not, then consider Bayes

P (B|A) =
If we don't have

P (A),

P (B) P (A|B)
P (A)

we can use the total probability theorem

P (A) = P (B) P (A|B) + P B C P A|B C

Example: Suicide Helpline

When a person calls a suicide helpline, the probability that he/she will commit
suicide

A = {attempt suicide} is:

P (A) = 0.02

If the person on the call-line asks questions in a test to discern probability

that the caller is a higher-risk caller, then the probability that the caller has a
positive result for this test is

T = {positive result on test}.

Of those who attempted to commmit suicide and had a positive result on

the the test:

P (T |A) = 0.8


P T |AC = 0.05
We are looking for the probability that a caller who tested positive on the
test will attempt suicide,

P (A|T ) =

P (A|T ),

which is:

P (A T )
P (A) P (T |A)
=
P (A)
P (A) P (T |A) + P (AC ) P (T |AC )
1

0.02 0.8
0.24
0.02 0.8 + 0.98 0.05

Aside
Looking for (seeking): When does the following hold true:

P (A|B) = P (B|A)
P (A) = P (B)
(A|B)
P (A|B) = P (B)P
, when P (A) = P (B), then
P (A)
the fraction P (A) in the numerator and P (B) in the denominator become 1,
P (A|B) = P (B|A).

Answer: When

Proof: In the equation

Another Aside
For the

empty set, the following equations hold true:

Ai = 1

Ai =

Ai =

i
The reason for these denitions is to

conserve the value of the operation if

it is extended beyond the empty set. For example:

!
\

Ai =

i{1}

Ai

A1 = A1 = A1

Independence of a Collection of Events

Denition:

A1 , A2 , ..., An

are independent events i

!
P

Ai

iS
where

S 2{1,2,..,n} ,

P (Ai )

iS

and where

S 2{1,2,..,n} = set of all subsets of {1, 2, ..., n}

2.1

Example of Independence for 3 Events

A, B and C are independent i

A, B are independent:

P (A B) = P (A) P (B)

A, C are independent:

P (A C) = P (A) P (C)

B, C are independent:

P (B C) = P (B) P (C)

A, B, C are independent:

P (A B C) = P (A) P (B) P (C)

If (A, B), (A, C), and (B,C) are independent, then we say that the events A, B,

Independence of (A,B,C) does not necessarily

imply pairwise independence. Similarly, pairwise independence of events does
and C are pairwise independent.

not imply that all the events are independent. We demonstrate this now with
an example.
We have 2 independent toin cosses.

H1 = {1st coin toss}

H2 = {2nd coin toss}
D = {1st and 2nd coin tosses dif f erent}
1
1
1
2 , P (H2 ) = 2 and P (D) = 2 .
We can clearly see that is is not possible to have all events be true, since if
The probabilities of these events are:

P (H1 ) =

both cosses produce a heads, then the two coin tosses will have dierent results
and

will not be true:

P (H1 H2 D) = 0
which is not the same as the product of the probabilities of the individual events:

P (H1 H2 D) 6= P (H1 ) P (H2 ) P (D) =

1
1 1 1
=
2 2 2
8

We can calculate the pairwise probabilities of the events as follows.

Heads on the rst and second coin tosses:

P (H1 H2 ) = P (H1 ) P (H2 ) =

1 1
1
=
2 2
4

Heads on the rst coin toss, tails on the second:

P (H1 D) = P (H1 ) P (D) =

1 1
1
=
2 2
4

Tails on the rst coin toss, heads on the second:

P (H2 D) = P (H2 ) P (D) =

1 1
1
=
2 2
4

Therefore, we can see that although the events are pairwise independent, the 3
events

(H1 , H2 , D)

are not independent.

Figure 1: Example of how Pairwise Independence does not imply full Independence.

2.2

Example of How Pairwise Independence for a set of 3

Events does not always imply Independence for the 3
Events

Imagine 2 six-sided die, where the events are dened as:

A = {f irst roll 3}
B = {f irst roll {3, 4, 5}}
C = {sum is 9}
Firstly, the intersection of the three events is dened as:

P (A B C) =
where the outcome that intersects

A, B, C

1
36

is the roll

(3, 6).

We can now calculate the independence of the three events,

P (A B C) =

A, B, C

, as :

1
1 1 1
1
= P (A) P (B) P (C) = =
36
2 2 9
36

However, although the 3 events are independent, the events are not pairwise
independent. We prove this by showing that the intersection of the events
and

is not equal to the product of the events

and

B:

P (A B) = column 3 in diagram =
, but

P (A) .P (B) =

1 1
1
=
2 2
4

therefore,

P (A B) 6= P (A) P (B)

1
6

Figure 2: Serial and Parallel Set of Servers with known Failure Rates

2.3

Reliability Example

Independent components that have known failure rates can be put together to
form a serial system, see Figure 2 (a).
The system consisting of a NIS, DNS and NFS server only works when all
systems are working. The probability that the system consisting of a NIS, DNS
and NFS server is working is

P (serial system working) =

3
Y

Pi

i=1
For the parallel system, if any of the components are working, the system is
working, see Figure 2 (b). The probability the parallel system is working is:

P (parallel system working) = 1 P (parallel not system working)

P (parallel system working) = 1

3
Y

(1 Pi )

i=1

Homework:
Birthday problem:

assume birthdays are equally distributed throughout the

year. For a class, what is the probability that two pupils have the same birthday.
How many pupils do we need to have a probability greater than 50% that 2 or
more pupils have the same birthday.

Figure 3: Sequential Tree Representation for a 3-stage N Choose K example,

with p and q as the choices.

Binomial Probability

If we have an experiment that involves a sequence of Independent Identical Stages,

independent trials.

we can say we have a sequence of

In the special case of

an independent trial, where there are only two outcomes, then it is called a

Bernoulli Trial.

In a Bernoulli trial, the outcome of the experiment will al-

ways be success or failure, heads or tails, etc.

3.1
if

Independent Stages Example (Pg 42)

P ({success}) = p,

and

q =1p

Probability of any outcome with

successes in

stages is:

pk q nk
Looking for:

P ({k success in n stages}) = C pk q nk

where

is the number of outcomes with


C=
, where

i! = 1x2x3

, and

0! = 1,

n
k

and


=
i

successes in

C=

n
k

stages, gives us

which is

n!
k!(n k)!

is non-negative.

will give us the number of these outcomes.

choose


= comb(n, k) = n choose k

Aside:

n 
X
n
pk (1 p)nk = norm 1
k

k=0

Order Matters (Permutations)

Replacement

Order Doesn't matter (Combinations)


, where k>n is allowed

n!
(nk)!

No Replacement

n+k1
n 1


, where k>n is allowed

n
k


=

n!
(nk)!k!

Table 1: Dierent Ways of Counting Things

X
1
p=
2

k=0

n
k

1 k 1 nk X
=
2 2

k=0

1
2n

3.2

n 
X
k=0

  n

n 
X
1
n
=1
= 2n
k
2

n
k
n
k

k=0


=1

Example: Grade of Service

Imagine an ISP with

using a modem is

P,

modems,

and

C > m.

customers.

The probability of customer

What is the probability customers use more

modems than are available? This is a binomial probability problem. How many
stages is this Bernoulli trial?
Answer:



C
X
C
pi q Ci
i

i=m+1

Counting

Counting represents the number of ways to pick

objects out of a set of

objects, see Table 1.

Selection with Replacement

Consider counting the number of ways to make
of

picks of objects out of a set

objects, such that when an object is picked from the set, it is replaced, so

there are always

objects to choose from, for each of the

for the rst pick we select from

objects, and for the
of picking

k th

pick, we select from

objects from

objects with replacement is:

This represents the upper-left hand entry in Table 1.

Permutations: Counting Principle

If you have a process of picking

picks. In this case,

stages, where

objects. So the number of ways

n n n n = nk

4.1

objects, for the second pick we select from

stage 1 has

...

stage

has

then there are

n1

options,

nk

options

n1 n2 ...nk

permutations. We can model this picking

sequential problem, i.e., as a tree, with a

represents the

ni

k stages

as a

options at stage i. This permutation

lower-left quadrant in Table 1.

Permutations Example 1
If we pick a phone number with 5 digits, where each digit can be

{0, 1, ..., 9}.

In

this case, order of phone numbers matters, so 00001 is not the same as 10000,
and repetitions are allowed, so we can have 99999. The number of ways in which
you can construct these numbers is

105 .

Permutations Example 2
Unique number of 5 digits, with distinct digits.

10 9 8 7 6 =

4.2

10!
10 9 8 7 6 5 4 3 2 1
=
54321
(10 5)!

Combinations
n!
(nk)! . If we take
ways. For the rst digit in our

For permutations with no repetition, we have the formula

the number 12345, it can be rearranged in

5!

arrangement, we have 5 choices, 4 choices for the 2nd digit, 3 choices for the 3rd
digit, 2 choices for the 4th digit, and one choice for the 5th digit. This gives us
the following:

n!


n!
(n k)!
n
=
=
k
k!
(n k)!k!
Combinations represent the

lower-right quadrant in Table 1.

4.2.1 Ball Picking Example

How many ways can
you colour them ? There may have repetitions, and ordering is not important.
We have three colours (red, green, blue) and 7 golf balls.
Solution:
Pictorially, we could use markers to identify where the colors are placed in
the balls. In the sequence of '0's below, a '|' represents a marker for the end of a
particular colour. The rst marker represents the end of red colours, the second
marker represents the end of green colours. No third marker is necessary.
In the balls and markers below:
0

the rst 2 balls are red, the next 2 balls are green and the last 3 balls are
blue.
Here,
0

the rst 2 balls are red, and the last 5 balls are blue.
Here,
0

all 7 balls are red.

Another way of looking at this is that we have a position before the start,
and a position after the end of the '0' where the marker for the color '|' can be
placed.
We can see this as having 9 positions for the markers:

{P1 , ..., P9 }
Another way of looking at it is that we have a bag of 3 colors, we want to
pick 7 colors, there can be repetition and the ordering doesn't matter. This is
represented as, the

upper-right quadrant in Table 1.

Example

4 colors (n

= 4)

n+k1
n1

and 3 golf balls (k

= 3)

{P1 , P2 , P3 , P 4 , P5 , P 6 }
here k can be larger than n.

Identity

Relation

In a bag of n objects, how many ways can we partition the bag into two components, one containing

objects and the other one will have

answer is the same number of ways, the

n
k


=

objects. The

n
nk

Multinomials/Partitions
nobjects into k
nk parts:
X
n =n

How can you partition the set of

has

nk

lower-right quadrant in Table 1:

n1 objects, n2 objects,

...,

parts, where the 1st partition

(n over n1 ),
(n n1 over n2 )
!
n1
X
(n over n1 ) (n n1 over n2 ) (n n1 n2 over n3 ) n (
n ) over nk
If we have n objects, the rst set of

n1 objects

is selected using

while from the remaing set of objects, we now select

i=1

n!
(n n1 )!
(n n1 n2 )!
n!
n!

 =

Pn1
n1 ! (n n1 )! n2 ! (n n1 n2 )! n3 ! (n n1 n2 n3 )!
n
!...n
1
k!
nk ! n ( i=1 ni ) nk !

Multinomial rule:

5.1

n!
n1 !n2 !n3 ! nc !

Example: How many ways can we count the word

MISSIPSSIPPI?
M I1 I2 I3 I4 S1 S2 S3 S4 P1 P2

How many ways can we partition the set of 11 positions into 4 partitions
containing 1, 4, 4, 2 elements for the letters M, I, S, P, respectively:

P1 P2 P3 .....P11
If we apply the multinomial rule, we get:

11!
1!4!4!2!

Homework
How can you dene combinations recursively, and what is its connection to
Pascal's triangle. Try to prove the recursion.

10