Basic Probability Theory

Lecture 3
Lecturer: Ali Ghodsi
Notes: Ahmad Al-Shishtawy
October 5, 2007

Problems From the Previous Lecture

Problem 13. We are given three coins: one has heads in both faces, the second
has tails in both faces, and the third has a head in one face and a tail in
the other. We choose a coin at random, toss it, and it comes heads. What
is the probability that the opposite face is tails?
Solution: Let H and N be the events that we saw a head and it is a normal
coin, respectively.
H = {Saw a head},

N = {N ormal coin}.

Now we need to calculate the conditional probability that it is a normal

coin given that we saw a head:

P (N | H) =
=
=

P (N H)
P (H)
P (N )P (H | H)
P (N )
1 1
1
3 2
=
1
3
2

In this problem, there is a tendency to reason that since the opposite face
is either heads or tails, the desired probability is 1/2. This is, however,
wrong, because given that heads came, it is more likely that the twoheaded coin was chosen.
Lesson learned: Proceed axiomatically and never follow your intuition.
Remark: P (A)P (B | A) = P (A B) = P (B)P (A | BA)

Using Conditional Probability for Modeling

When constructing probabilistic models for experiments that have a sequential

character, it is often natural and convenient to first specify conditional probabilities and then use them to determine unconditional probabilities as shown in
the following example.
1

Figure 1: Sequential description of the experiment for the radar detection

Example 1.9 Radar Detection. If an aircraft is present in a certain area,

a radar correctly registers its presence with probability 0.99. If it is not
present, the radar falsely registers an aircraft presence with probability
0.10. We assume that an aircraft is present with probability 0.05. What
is the probability of false alarm (a false indication of aircraft presence),
and the probability of missed detection (nothing registers, even though an
aircraft is present)?
Solution: A sequential representation of the experiment is appropriate here,
as shown in Fig. 1. Let A and B be the events
A

= {an aircraf t is present},

= {the radar generates an alarm},

and consider also their complements

Ac
B

= {an aircraf t is not present},

= {the radar does not generate an alarm},

The given probabilities are recorded along the corresponding branches of

the tree describing the sample space, as shown in Fig. 1. Each possible
outcome corresponds to a leaf of the tree, and its probability is equal to
the product of the probabilities associated with the branches in a path
from the root to the corresponding leaf. The desired probabilities of false
alarm and missed detection are
P (not present, f alse alarm) = P (Ac B) = P (Ac )P (B | Ac )
= 0.95 0.10 = 0.095,
P (present, no detection) = P (A B c ) = P (A)P (B c | A)
= 0.05 0.01 = 0.0005.
2

Remark: Always draw a tree when you encounter sequential events

Multiplication Rule

Assuming that all of the conditioning events have positive probability, we have
n1
P (ni=1 Ai ) = P (A1 )P (A2 | A1 )P (A3 | A1 A2 ) . . . P (An | i=1
Ai ).

The multiplication rule can be verified by writing

P (ni=1 Ai )

P (ni=1 Ai )

P (ni=1 Ai )
n1
= P (i=1
Ai )
n1
P (i=1
Ai )
...
n1
P (i=1
Ai ) P (ni=1 Ai )

P (ni=1 Ai ) = P (A1 A2 A3 ) . . .
n2
n1
P (i=1
Ai ) P (i=1
Ai )
P (ni=1 Ai )

P (ni=1 Ai )

P (ni=1 Ai )

n1
P (A1 A2 A3 )
P (i=1
Ai ) P (ni=1 Ai )

...
n2
n1
P (A1 A2 )
P (i=1 Ai ) P (i=1
Ai )
P (A1 A2 ) P (A1 A2 A3 )
P (A1 )

P (A1 )
P (A1 A2 )

P (A1 A2 )

...
P (ni=1 Ai )

n1
P (i=1
Ai ) P (ni=1 Ai )

n2
n1
P (i=1
Ai ) P (i=1
Ai )

n1
P (A1 )P (A2 | A1 )P (A3 | A1 A2 ) . . . P (An | i=1
Ai ).

Example 1.10. Three cards are drawn from an ordinary 52-card deck without
replacement (drawn cards are not placed back in the deck). We wish to
find the probability that none of the three cards is a heart. We assume that
at each step, each one of the remaining cards is equally likely to be picked.
By symmetry, this implies that every triplet of cards is equally likely to
be drawn. A cumbersome approach, that we will not use, is to count
the number of all card triplets that do not include a heart, and divide it
with the number of all possible card triplets. Instead, we use a sequential
description of the experiment in conjunction with the multiplication rule.
Solution: We define the events
A1

{f irst not heart},

A2
A3

=
=

{second not heart},

{third not heart},

Now we are seeking P (A1 A2 A3 ), the probability that none of the

three cards is a heart, using the multiplication rule
P (A1 A2 A3 ) = P (A1 )P (A2 | A1 )P (A3 | A1 A2 )
We have
P (A1 ) =
3

39
52

because there are 39 cards that are not red in a 52 card deck (3/4). If we
know that the first card is not heart then we are left with 51 cards, 38 of
which are not hearts
38
P (A2 | A1 ) =
51
similarly
37
P (A3 | A1 A2 ) =
50
and using the multiplication rule we have
39 38 37
P (A1 A2 A3 ) =

52 51 50
Example 1.11 A class consisting of 4 graduate and 12 undergraduate students
is randomly divided into 4 groups of 4. What is the probability that each
group includes a graduate student?
Solution: We define the events
A1
A2

=
=

A3 =
A =

{student 1 and 2 are in dif f erent groups},

{student 1, 2, and 3 are in dif f erent groups},
{student 1, 2, 3, and 4 are in dif f erent groups},
3i=1 Ai = A3 .

We are seeking P (A) and we calculate it using the multiplication rule:

P (A)

= P (A1 A2 A3 )
= P (A1 )P (A2 | A1 )P (A3 | A1 A2 )
12 8 4

=
15 14 13

Example 1.12. The Monty Hall Problem. This is a much discussed puzzle, based on an old American game show. You are told that a prize is
equally likely to be found behind any one of three closed doors in front
of you. You point to one of the doors. A friend opens for you one of the
remaining two doors, after making sure that the prize is not behind it.
At this point, you can stick to your initial choice, or switch to the other
unopened door. You win the prize if it lies behind your final choice of a
door. Consider the following strategies:
1. Stick to your initial choice.
2. Switch to the other unopened door.
Which is the best strategy?
Solution: A simple solution is by using Figure 2. For example, if your initial
choice is 1 (inner circle) then the probability to win if you dont switch is
1/3 (shaded area on the middle circle). That is if the prize is behind door
1. While if you switch after door 2 or 3 is opened then the probability to
win is 2/3 (shaded area on the outer circle). This is because the probability
of the prize being behind door 2 or 3 P (2 3) = 2/3 but you know that
P (Opened) = 0 so P (closed) = 2/3 and P (initial choice) = 1/3 which
means that it is better to switch.

Inner Circle:
Your initial choice

2
2

Middle Circle:
Probability to win
if you dont switch

2
1

Outer Circle:
Probability to win
if you switch

Figure 2: The Monty Hall Problem. The shaded area shows the probability to
win.

Total Probability Theorem

Let Ai be disjoint events that form a partition of the sample space and assume
that P (Ai ) > 0, for all i. Then, for any event B, we have
P (B)

= P (A1 B) + P (A2 B) + . . . + P (An B)

= P (A1 )P (B | A1 ) + P (A2 )P (B | A2 ) + . . . + P (An )P (B | An ).

Hint: to prove it start with

P (B ) = P (B (A1 A2 . . . An ))
Remark: The total probability theorem is for both discrete and continuous
probabilistic models.
Example. Solve the Monty Hall problem using the total probability theorem.
Solution: Define the following events
W

{W in if switch}

IP P

{Initially pick prize}

We are seeking P (W ) that can be represented as follows (see Figure 4)

P (W ) = P (IP P )P (W | IP P ) + P (IP P c )P (W | IP P c ),
we have
P (IP P ) = 1/3,
5

A1
B

A3

A2

Figure 3: Visualization of the total probability theorem.

and
P (IP P c ) = 2/3,
if initially picked prize but switched then
P (W | IP P ) = 0,
similarly
P (W | IP P c ) = 1
thus
P (W ) =

2
2
1
0+ 1=
3
3
3

Not initially pick prize

Win if switch

Initially pick prize

Figure 4: Using the total probability theorem to solve the Monty Hall problem

Example 1.13. You enter a chess tournament where your probability of winning a game is 0.3 against half the players (call them type 1), 0.4 against
a quarter of the players (call them type 2), and 0.5 against the remaining quarter of the players (call them type 3). You play a game against a
randomly chosen opponent. What is the probability of winning?
Solution: Let Ai be the event of playing with an opponent of type i. We have
P (A1 ) = 0.5,

P (A2 ) = 0.25,

P (A3 ) = 0.25.

Let also B be the event of winning. We have

P (B | A1 ) = 0.3,

P (B | A2 ) = 0.4,

P (B | A3 ) = 0.5.

Thus, by the total probability theorem, the probability of winning is

P (B) =
=
=

P (A1 )P (B | A1 ) + P (A2 )P (B | A2 ) + P (A3 )P (B | A3 )

0.5 0.3 + 0.25 0.4 + 0.25 0.5
0.375.

Inference and Bayes Rule

Let A1 , A2 , . . . , An be disjoint events that form a partition of the sample space,

and assume that P (Ai) > 0, for all i. Then, for any event B such that P (B) > 0,
we have
P (Ai | B) =
=

P (Ai )P (B | Ai )
P (B)
P (Ai )P (B | Ai )
ni=1 P (Ai )P (B | Ai )

Example 1.18. The False-Positive Puzzle. A test for a certain rare disease
is assumed to be correct 95% of the time: if a person has the disease, the
test results are positive with probability 0.95, and if the person does not
have the disease, the test results are negative with probability 0.95. A
random person drawn from a certain population has probability 0.001 of
having the disease. Given that the person just tested positive, what is the
probability of having the disease?
Solution: If A is the event that the person has the disease, and B is the event
that the test results are positive, the desired probability, P (A|B), is

P (A|B)

P (A)P (B|A)
P (A)P (B|A) + P (Ac )P (B|Ac )
0.001 0.95
=
0.001 0.95 + 0.999 0.05
= 0.0187.
=

Problems for Next Lecture

Problem 9.* The inclusion-exclusion formula. Show the following generalizations of the formula
P (A B) = P (A) + P (B) P (A B).
1. Let A, B, and C be events. Then,
P (A B C)

P (A) + P (B) + P (C) P (A B)

P (B C) P (A C)
+P (A B C).

2. Let A1 , A2 , . . . , An be events. Let S1 = {i|1 i n}, S2 =

{(i1 , i2 )|1 i1 < i2 n}, and more generally, let Sm be the set
of all m-tuples (i1 , . . . , im ) of indices that satisfy 1 i1 < i2 < . . . <
im n. Then,
X
X
P (nk=1 Ak ) =
P (Ai1 Ai2 )
P (Ai )
iS1

(i1 ,i2 )S2

P (Ai1 Ai2 Ai3 )

(i1 ,i2 ,i3 )S3

. . . + (1)n1 P (nk=1 Ak ).
Problem 14. A batch of one hundred items is inspected by testing four randomly selected items. If one of the four is defective, the batch is rejected.
What is the probability that the batch is accepted if it contains five defectives?
Problem 15. Let A and B be events. Show that P (A B|B) = P (A|B),
assuming that P (B) > 0.
Modified Monty Hall Problem. Solve the Monty Hall problem again assuming that your friend randomly opens one of the two doors (without
making sure that there is no prize behind it). If the prize is behind the
door your friend picked then you start over the game.
Examples. Read examples 1.14, 1.15, and 1.16.