Unit 4 – Elements of Probability Theory

1. Random experiments, Sample Space and events.

2. Different Approaches to Probability: Classical, Frequency interpretation and
Axiomatic approach,
3. Deduction of simple properties from axioms. Counting techniques and their use
in Probability.
4. Conditional Probability: Independence of Events, Bayes’ Theorem and its
applications.
***
Probability
 Originated in the beginning of seventeenth century from the game of
chance.
e.g. Baits, gambling etc.
 Further developments lead to the application of probability in various
field of science and technology.
 Probability theory deals with the study of random or non-deterministic
experiments also called as random experiments.
 By random experiment we mean as experiment which
 All the outcomes of the experiment are known in advance.
 But a particular performance of the experiment results in an
outcome is not known in advance.
 The experiment can be repeated under identical conditions.
 For example
 The formula y(t)=1/2gt2 gives the position of a particle falling under
gravity from a position of rest at any time t. this is called a
deterministic experiment.
 But if a fair coin is tossed, if a die is rolled etc.

Again before defining probability, one must be cleared about the various
notations and terminologies used in the definition of probability

 Random Experiments: A random experiment is an experiment or a

process for which the outcome cannot be predicted with certainty. E.g.
tossing a coin, picking a card from deck of cards, etc.
 Sample space: The set of all possible outcomes of a given random
experiment is called sample space or outcome space.
 E.g. (a) if a coin is tossed S1={H, T}; H=Head and T=Tail
(b) if a coin is tossed twice or two coins are tossed simultaneously
S2={HH,HT,TH,TT}
If a coin is tossed thrice the sample space will have 2 3 and generally if a
coin is tossed r times then 2r outcomes will occur in Sr.
 Events: Outcome or result of a random experiment, (subsets of sample
space) is called an event, generally denoted as E.
o e.g. getting Head on tossing a coin, Getting tail on tossing a coin,
getting two heads on tossing two coins.
 Elementary event: If an event contains only one element (if E is
singleton). E.g. tossing a coin and getting a head.
 Compound event: an event which is not elementary is called
compound. E.g. tossing two coins and getting a head.
 Sure Event: An event which is certain is called sure event. E.g sunrise,
sunset, etc.
 Impossible Event: Which does not occur at any observation.

 Mutually Exclusive Events: Events A and B are called disjoint or

mutually exclusive if A and B are alternative possible outcomes of the
same trail. Or no two or more of the events can happen simultaneously in
the same trail. E.g. on tossing a coin either Head or Tail will come, i.e.
occurrence of one event will restrict occurrence of other events.
 Exhaustive Events: All possible elements of the sample space.
 Equally Likely Events: Occurrence of events has same possibility/chance.
 Independent Events: Several events are said to be independent if the
happening or non-happening of an event is not affected by the
supplementary knowledge concerning the occurrence of any number of the
remaining events.
Eg. In the tossing an unbiased coin the event of getting a head in the first
toss is independent of getting a head in the second, third or subsequent
throws.
Now let us define the probability.
There are three definitions of probability based on the different approaches and
usefulness.
(i) Classical/Mathematical/a priori definition (approach)
(ii) Statistical/Frequency/Posterior definition
(iii) Axiomatic definition (approach)

Classical Definition of Probability

If a trail results in ‘n’ exhaustive, mutually exclusive and equally likely cases and
‘m’ of them are favourable to an event ‘E’ then the probability ‘p’ of happening
of event E is given by
𝑓𝑎𝑣𝑜𝑢𝑟𝑎𝑏𝑙𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑠𝑒𝑠 𝑚
𝑝 = 𝑃 (𝐸 ) = = (𝐴)
𝐸𝑥ℎ𝑎𝑢𝑠𝑡𝑖𝑣𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑠𝑒𝑠 𝑛
From (A), one can interpret that ‘the odds in favour of E are 𝑚/(𝑛 − 𝑚) or the
odds against E are (𝑛 − 𝑚)/𝑛
Since the number of cases of favourable to non-happening of E are (n-m), the
probability ‘q’ that E will not happen is given by
𝑛−𝑚 𝑚
𝑞= = 1− =1−𝑝 ⇒𝑝+𝑞 =1
𝑛 𝑛
Note:
 Probability ‘p’ of happening of an event E is known as the probability of
success and probability ‘q’ of non-happening of the event as probability of
failure.
 ‘p’ and ‘q’ are non-negative and cannot exceed unity i.e. 0 ≤ 𝑝 ≤ 1 and
0 ≤ 𝑞 ≤ 1.
 If 𝑃(𝐸) = 1, E is called certain event and 𝑃(𝐸) = 0, E is called an
impossible event.

Statistical/ Frequency Definition of Probability

If a trail is repeated a number of times under essential homogeneous and identical
conditions, then the limiting value of the ratio of the number of times the event
happens to the number of trails, as the number of trails become indefinitely large
is called the probability of happening of the event.
Symbolically, if in ‘n’ trails an event E happens ‘m’ times, then the probability
‘p’ of happening of E is given by
𝑚
𝑝 = 𝑃(𝐸 ) = lim .
𝑛⟶∞ 𝑛

For example: consider the problem of finding out the chance that a new-
born baby is a male/female. Similar as in the coin toss experiment, one may
assume that the birth of male and female babies are equally likely and assert that
the new born baby is a boy has probability 0.5. But, upon examination of past
records of birth in the region is 0.51 (say). Thus observing new born babies birth
probability (0.51) may be viewed as tossing a biased coin in favour of getting
head with probability 0.51.

Axiomatic Approach:
This approach follows three axioms:
1. Axiom of non-negativity: Probability cannot be negative.
o 0 ≤ 𝑃(𝐸) ≤ 1
2. Axiom of certainty: Probability of a sure event is 1.
o 𝑃 (𝑆𝑢𝑟𝑒 𝑒𝑣𝑒𝑛𝑡) = 1
3. Axiom of additivity: Probability of two mutually exclusive events can be
added.
Theorem 1: Probability of impossible event is zero, i.e., 𝑃 (∅) = 0.
Theorem 2: Probability of the complementary event 𝐴̅ of A is given by
𝑃(𝐴̅) = 1 − 𝑃(𝐴)

𝐴̅ A

Theorem 3: If A and B are two events and are not independent, then
𝑃 (𝐴 ∪ 𝐵 ) = 𝑃 (𝐴) + 𝑃 (𝐵 ) − 𝑃(𝐴 ∩ 𝐵)

A A∩B B

Conditional Probability:
The conditional probability of an event B is the probability that the event will
occur given the knowledge that an event A has already occurred. This
probability is written
𝑃(𝐴⋂𝐵)
𝑃 (𝐵|𝐴) =
𝑃(𝐴)
𝑷(𝑩|𝑨): Probability of B given A.
Independence of Events:
An event A is said to be independent of another event B, if the conditional
probability of A given B is equal to the unconditional probability of A, i.e.,
𝑷(𝑨|𝑩) = 𝑷(𝑨).
Also, 𝑃 (𝐴 ∩ 𝐵 ) = 0. A B

Bayes’ Theorem:
It describes the probability of an event, based on prior knowledge of conditions
that might be related to the event, and represented as
𝑃(𝐴)𝑃(𝐵|𝐴)
𝑃(𝐴|𝐵 ) =
𝑃(𝐵)
For more than 2 cases:
𝑃(𝐴𝑘 )𝑃(𝐵|𝐴𝑘 )
𝑃(𝐴𝑘 |𝐵) =
𝑃(𝐴1)𝑃(𝐵|𝐴1) + 𝑃(𝐴2)𝑃 (𝐵|𝐴2 ) + ⋯ + 𝑃(𝐴𝑘 )𝑃(𝐵|𝐴𝑘 )

It is used in various fields such as in prediction of any future event, Naive

Bayes’ Classifiers in machine learning, Discriminant Functions and Decision
Surfaces, Bayesian Parameter Estimation etc.
Important Notations:
S: Universal set. Collection of all possible events under the experiment. P(S)=1
∅: Empty set. P(∅)=0.
A: Event A. P(A): Probability of occurrence of event A
̅ : Compliment of A
𝑨 ̅ ): Probability of non-occurrence of event A
𝑷(𝑨
∅= ̅
𝑺 ̅)
P(∅)= P(𝑺
̅
S= ∅ ̅)
P(S)= P(∅
B: Event B. P(B): Probability of occurrence for event B
∪: Union P(A∪B): Probability of occurrence for event A or B.
∩: Intersection P(A∩B): Probability of occurrence for event A and B.
***
Problems:
With Replacement:
A 2 3 4 5 6 7 8 9 10 J Q K: 13*4=52
(Club, Diamond, Spade, Heart)

1. Find the probability of selecting a Queen or King of Diamond from a deck of

52 cards.
-Consider the following events from deck of cards:
A: Drawing a Queen.
B: Drawing a King of Diamond.
P(A)=4/52
P(B)=1/52
P(A∩B)=P(A and B)=0/52
P(A∪B)= P(A or B)=5/52

2. Find the probability of selecting a Red card or a King of Diamond from a

deck of 52 cards.
-Consider the following events from deck of cards:
A: Drawing a Red Card.
B: Drawing a King of Diamond.
P(A)=26/52
P(B)=1/52
P(A∩B)=1/52
P(A∪B)=26/52
3. Find the probability of selecting a Black card or a King from a deck of 52
cards.
-Consider the following events from deck of cards:
A: Drawing a Black Card.
B: Drawing a King.
P(A)=26/52
P(B)=4/52
P(A∩B)=P(A and B)=2/52
P(A∪B)=P(A or B)=28/52

4. Find the probability of selecting a face card or a Jack from a deck of 52 cards.
-Consider the following events from deck of cards:
A: Drawing a Face Card. JJJJ,QQQQ,KKKK
B: Drawing a Jack. JJJJ
P(A)=12/52
P(B)=4/52
P(A∩B)=P(A and B)=4/52
P(A∪B)=12/52

5. Find the probability of selecting any card or a Face card from a deck of 52
cards.
-Consider the following events from deck of cards:
A: Drawing a Card.
B: Drawing a Face Card.
P(A)=52/52=1 Sure Event
P(B)=12/52
P(A∩B)=12/52
P(A∪B)=52/52=1 Sure Event
Without Replacement:
1. Find the probability of selecting a Queen and King of Diamond from a deck
of 52 cards.
Consider the following events from deck of cards:
A: Drawing a Queen.
B: Drawing a King of Diamond.
P(A)=4/52
P(B|A)=P(B given A)=1/51
P(A∩B)=0

2. Find the probability of selecting a Red card and a King of Diamond from a
deck of 52 cards.
-Consider the following events from deck of cards:
A: Drawing a Red Card.
B: Drawing a King of Diamond.
P(A)=26/52
P(B|A)=1/51+0/51=1/51
P(A∩B)=1/52

3. Find the probability of selecting a Black card and a King from a deck of 52
cards.
-Consider the following events from deck of cards:
A: Drawing a Black Card.
B: Drawing a King.
P(A)=26/52
P(B|A)=4/51+3/51=7/51
P(A∩B)=2/52

4. Find the probability of selecting a face card and a Jack from a deck of 52
cards.
-Consider the following events from deck of cards:
A: Drawing a Face Card. JJJJ, QQQQ, KKKK
B: Drawing a Jack. JJJJ
P(A)=12/52
P(B|A)=4/51+3/51
P(A∩B)=4/52

5. Find the probability of selecting any card and a Face card from a deck of 52
cards.
-Consider the following events from deck of cards:
A: Drawing a Card.
B: Drawing a Face Card.
P(A)=52/52
P(B|A)=12/51+11/51
P(A∩B)=12/52
A: prime- 2,3,5 P(A)=3/6
B: even- 2,4,6 P(B)=3/6
C: odd- 1,3,5 P(C)=3/6
A∩B: A and B- 2 P(A∩B)=1/6
A∩C: A and C- 3,5 P(A∩C)=2/6
B ∩C: B and C- 0 P(B∩C)=0/6
A∪B: A or B- 2,3,4,5,6 P(A∪B)=5/6
A∪C: A or C- 1,2,3,5 P(A∪C)=4/6
B∪C: B or C- 1,2,3,4,5,6 P(B∪C)=6/6=1 Sure event

Question for practice:

With Replacement:
1. What is the probability of getting a 2 or a 5 when a die is rolled?
2. What is the probability of getting a 2 or an odd number when a die is rolled?
3. Find the probability of selecting a black card or a 6 from a deck of 52 cards.
4. Find the probability of selecting a Red card or an even number from a deck of
52 cards.
5. Find the probability of selecting a face card or an odd number from a deck of
52 cards.
Without Replacement:
1. What is the probability of getting a 2 and a 5 when a die is rolled?
2. What is the probability of getting a 2 and an odd number when a die is rolled?
3. Find the probability of selecting a black card and a 6 from a deck of 52 cards.
4. Find the probability of selecting a Red card and an even number from a deck
of 52 cards.
5. Find the probability of selecting a face card and an odd number from a deck
of 52 cards.
Problems on Bayes’ Theorem:
𝑃(𝐴𝑘 )𝑃(𝐵|𝐴𝑘 )
𝑃(𝐴𝑘 |𝐵) =
𝑃(𝐴1)𝑃(𝐵|𝐴1) + 𝑃(𝐴2)𝑃 (𝐵|𝐴2 ) + ⋯ + 𝑃(𝐴𝑘 )𝑃(𝐵|𝐴𝑘 )

P(Pam)=15/30
P(Pia)=5/30
P(Pablo)=10/30
P(First|Pam)=4/100=0.04
P(First|Pia)=6/100=0.06
P(First|Pablo)=3/100=0.03
𝑃 (𝑃𝑎𝑚|𝐹𝑖𝑟𝑠𝑡 )
𝑃 (𝑃𝑎𝑚)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑚)
=
𝑃(𝑃𝑎𝑚)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑚) + 𝑃 (𝑃𝑖𝑎)𝑃(𝐹𝑖𝑟𝑠𝑡|𝑃𝑖𝑎 ) + 𝑃(𝑃𝑎𝑏𝑙𝑜) 𝑃(𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑏𝑙𝑜 )
𝑃 (𝑃𝑖𝑎|𝐹𝑖𝑟𝑠𝑡 )
𝑃 (𝑃𝑖𝑎)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑖𝑎 )
=
𝑃 (𝑃𝑎𝑚)𝑃(𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑚 ) + 𝑃 (𝑃𝑖𝑎)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑖𝑎 ) + 𝑃(𝑃𝑎𝑚)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑏𝑙𝑜 )
= (5*0.06)/1.2=3/12=1/4=25%
𝑃 (𝑃𝑎𝑏𝑙𝑜|𝐹𝑖𝑟𝑠𝑡 )
𝑃 (𝑃𝑎𝑏𝑙𝑜)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑏𝑙𝑜 )
=
𝑃 (𝑃𝑎𝑚)𝑃(𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑚 ) + 𝑃 (𝑃𝑖𝑎)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑖𝑎 ) + 𝑃(𝑃𝑎𝑚)𝑃 (𝐹𝑖𝑟𝑠𝑡|𝑃𝑎𝑏𝑙𝑜 )
=(10*0.03)/1.2=3/12=1/4=25%
P(Pablo|First)=1-0.5-0.25=0.25=25%

1. Suppose that a product is produced in three factories X, Y and Z. It is known

that factory X produces thrice as many items as factory Y, and that factories Y
and Z produce same number of items. Assume that 3 percent of the items
produced by each of the factories X and Z are defective while 5 percent of those
manufactured by factory Y are defective. All the items produced in the three
factories are stocked, and an item of product is selected at random.
i. What is the probability that item is defective?
ii. If an item selected at random is found to be defective, what is the probability
that it was produced by factory X, Y, and Z respectively?
---
E1: Item is from Factory X 3n
E2: Item is from Factory Y n
E3: Item is from Factory Z n
S=3n+n+n=5n
P(E1)=3n/5n=0.6
P(E2)=n/5n=0.2
P(E3)=n/5n=0.2
A: Item is defective.
P(A|X)=3%=0.03
P(A|Y)=5%=0.05
P(A|Z)=3%=0.03
P(A)= P(E1)*P(A|X)+P(E2)*P(A|Y)+P(E3)*P(A|Z)
=0.6*0.03+0.2*0.05+0.2*0.03=0.018+0.01+0.006=0.034
P(E1|A)=Known-Item is defective; E1=?
P(A|E1)=Known-Item is from E1(X); A=?

𝑃(𝐸1) 𝑃(𝐴|𝐸1)
𝑃 (𝐸1|𝐴) =
𝑃(𝐸1) 𝑃(𝐴|𝐸1) + 𝑃(𝐸2) 𝑃(𝐴|𝐸2) + 𝑃(𝐸3) 𝑃(𝐴|𝐸3)
0.6∗0.03 0.018 0.018 9
𝑃 (𝐸1|𝐴) = = = = = 0.529
0.6∗0.03+0.2∗0.05+0.2∗0.03 0.018+0.01+0.006 0.034 17

𝑃(𝐸2) 𝑃(𝐴|𝐸2)
𝑃 (𝐸2|𝐴) =
𝑃(𝐸1) 𝑃(𝐴|𝐸1) + 𝑃(𝐸2) 𝑃(𝐴|𝐸2) + 𝑃(𝐸3) 𝑃(𝐴|𝐸3)
0.2∗0.05 0.01 10 5
𝑃 (𝐸2|𝐴) = = = = = 0.294
0.6∗0.03+0.2∗0.05+0.2∗0.03 0.034 34 17

𝑃(𝐸3) 𝑃(𝐴|𝐸3)
𝑃 (𝐸3|𝐴) =
𝑃(𝐸1) 𝑃(𝐴|𝐸1) + 𝑃(𝐸2) 𝑃(𝐴|𝐸2) + 𝑃(𝐸3) 𝑃(𝐴|𝐸3)
0.2∗0.03 0.006 6 3
𝑃 (𝐸3|𝐴) = = = = = 0.176
0.6∗0.03+0.2∗0.05+0.2∗0.03 0.034 34 17

P(E3|A)=1- P(E1|A)- P(E2|A)=1-9/17-5/17=1-14/17=3/17

Question for Practice:

2. In 2002 there will be three candidates for the position of principle- Mr.
Chatterji, Mr. Ayangar and Dr. Singh –whose chances of getting the appointment
re in the proportion 4:2:3 respectively. The probability that Mr. Chatterji if
selected would introduce co-education in the college is 0.3. The probability of
Mr. Ayangar and Dr. Singh doing the same are 0.5 and 0.8.
i. What is probability that there will be co-education in the college in 2003?
ii. If there is co-education in the college in 2003, what is the probability that Dr.
Singh is the principle?