EE5110: Probability Foundations for Electrical Engineers
JulyNovember 2015
Lecture 3: Cardinality and Countability
Lecturer: Dr. Krishna Jagannathan
Scribe: Ravi Kiran Raman
3.1 Functions
We recall the following deﬁnitions.
Deﬁnition 3.1 A function f : A → B is a rule that maps every element of set A to a unique element in set B.
In other words, ∀x ∈ A, ∃y ∈ B and only one such element, such that, f (x) = y. Then y is called the image of x and x, the preimage of y under f . The set A is called the domain of the function and B, the codomain.
R = {y : ∃x ∈ A,
s.t. f (x) = y} is called as the range of the function f .
Deﬁnition 3.2 A function f : A → B is said to be an injective (onetoone) function, if every element in the range R has a unique preimage in A.
Deﬁnition 3.3 A function f ∀y ∈ B, ∃x ∈ A, s.t. f (x) = y.
:
A
→
B
is said to be a surjective
(onto)
function, if R =
B ,
i.e,
Deﬁnition 3.4 A function f : A → B is a bijective function if it is both injective and surjective.
Hence, in a bijective mapping, every element in the codomain has a preimage and the preimages are unique. Thus, we can deﬁne an inverse function, f ^{−}^{1} : B → A, such that, f ^{−}^{1} (y) = x, if f (x) = y. In simple terms, bijective functions have welldeﬁned inverse functions.
3.2 Cardinality and Countability
In informal terms, the cardinality of a set is the number of elements in that set. If one wishes to compare the cardinalities of two ﬁnite sets A and B, it can be done by simply counting the number of elements in each set, and declare either that they have equal cardinality, or that one of the sets has more elements than the other. However, when sets containing inﬁnitely many elements are to be compared(for example, N versus Q), this elementary approach is not eﬃcient to do it. In the late nineteenth century, Georg Cantor introduced the idea of comparing the cardinality of sets based on the nature of functions that can be possibly deﬁned from one set to another.
Deﬁnition 3.5
(i) Two sets A and B are equicardinal (notation A = B) if there exists a bijective
function from A to B.
(ii) B has cardinality greater than or equal to that of A (notation B ≥ A) if there exists an injective function from A to B.
31
32
Lecture 3: Cardinality and Countability
(iii) B has cardinality strictly greater than that of A (notation B > A) if there is an injective function, but no bijective function, from A to B.
Having stated the deﬁnitions as above, the deﬁnition of countability of a set is as follow:
Deﬁnition 3.6 A set E is said to be countably inﬁnite if E and N are equicardinal. And, a set is said to be countable if it is either ﬁnite or countably inﬁnite.
The following are some examples of countable sets:
1. The set of all integers Z is countably inﬁnite.
We can deﬁne the bijection f : Z → N as follows :
n = f(z) ∈ N 
z ∈ Z 
1 
0 
2 
+1 
3 
1 
4 
+2 
5 
2 
. 
. 
. 
. 
. 
. 
The existence of this bijective map from Z to N proves that Z is countably inﬁnite.
2. The set of all rationals in [0, 1] is countable.
Consider the rational number ^{p}
to the set, if it not already present. By this way, the set of rational
numbers in [0, 1] can be explicitly listed as: {0, 1,
6 ^{,} Clearly, we can deﬁne a bijection from Q ∩ [0, 1] → N where each rational number is mapped to its index in the above set. Thus the set of all rational numbers in [0, 1] is countably inﬁnite and thus countable.
0 ≤ p ≤ q, add the rational number ^{p}
= 0. Increment q in steps of 1 starting with 1. For each such q and
_{q}
where q
q
1
2 ^{,}
1
3 ^{,}
2
3 ^{,}
1
4 ^{,}
3
4 ^{,} ^{1}
5 ^{,} ^{2}
^{3}
^{4} ^{,} ^{1}
5
5
_{6} ^{,}
}
5 ^{,} 5
,
3. The set of all Rational numbers, Q is countable.
In order to prove this, we state an important theorem, whose proof can be found in [1].
Theorem 3.7 Let I be a countable index set, and let E _{i} be countable for each i ∈ I . Then ^{} _{i}_{∈}_{I} E _{i} is countable. More glibly, it can also be stated as follows: A countable union of countable sets is countable.
We will now use this theorem to prove the countability of the set of all rational numbers. It has been already proved that the set Q ∩ [0, 1] is countable. Similarly, it can be showed that Q ∩ [n, n + 1] is countable, ∀n ∈ Z. Let Q _{i} = Q ∩ [i, i + 1]. Thus, clearly, the set of all rational numbers, Q = ∪ _{i}_{∈}_{Z} Q _{i} – a countable union of countable sets – is countable.
Remark: For two ﬁnite sets A and B, we know that if A is a strict subset of B, then B has cardinality greater than that of A. As the above examples show, this is not true for inﬁnite sets. Indeed, N is a strict subset of Q, but N and Q are equicardinal!
4. The set of all algebraic numbers (numbers which are roots of polynomial equations with rational co
eﬃcients) is countable.
5. The set of all computable numbers, i.e., real numbers that can be computed to within any desired precision by a ﬁnite, terminating algorithm, is countable (see Wikipedia article for more details).
Lecture 3: Cardinality and Countability
33
Deﬁnition 3.8 A set F is uncountable if it has cardinality strictly greater than the cardinality of N.
In the spirit of Deﬁnition 3.5, this means that F is uncountable if an injective function from N to F exists, but no such bijective function exists.
An interesting example of an uncountable set is the set of all inﬁnite binary strings. The proof of the following theorem uses the celebrated ‘diagonal argument’ of Cantor.
Theorem 3.9 (Cantor) : The set of all inﬁnite binary strings, {0, 1} ^{∞} , is uncountable.
Proof: It is easy to show that an injection from N to {0, 1} ^{∞} exists (exercise: produce one!). We need to show that no such bijection exists.
Let us assume the contrary, i.e, let us assume that the set of all binary strings, A = {0, 1} ^{∞} is countably inﬁnite. Thus there exists a bijection f : A → N. In other words, we can order the set of all inﬁnite binary strings as follows:
a
a
a
11 
a 
12 
a 
13 
. 
. 
. 
21 
a 
22 
a 
23 
. 
. 
. 
31 
a 
32 
a 
33 
. 
. 
. 
. 
. 
. 

. 
. 
. 

. 
. 
. 
, 
where, a _{i}_{j} is the j ^{t}^{h} bit of the i ^{t}^{h} binary string, i, j ≥ 1.
Consider the inﬁnite binary string given by ¯a = a¯ _{1}_{1} a¯ _{2}_{2} a¯ _{3}_{3}
Since our list contains all inﬁnite binary strings, there must exist some k ∈ N such that the string a¯ occurs at the k position in the list, i.e., f (¯a) = k. The k ^{t}^{h} bit of this speciﬁc string is a¯ _{k}_{k} . However, from the above list, we know that the k ^{t}^{h} bit of the k ^{t}^{h} string is a _{k}_{k} . Thus, we can conclude that the string ¯a cannot occur in any position k ≥ 1 in our list, contradicting our initial assumption that our list exhausts all possible inﬁnite binary strings.
Thus, there cannot possibly exist a bijection from N to {0, 1} ^{∞} , proving that {0, 1} ^{∞} is uncountable.
Now using Cantor’s theorem, we will prove that the set of irrational numbers is uncountable.
, where a¯ _{i}_{j} is the complement of the bit a _{i}_{j} .
Claim 3.10 The sets [0, 1], R and {R \ Q} are uncountable.
Proof: Firstly, consider the set [0, 1]. Any number in this set can be expressed by its binary equivalent and
thus, there appears to be a bijection from [0, 1] → {0, 1} ^{∞} . However, this is not exactly a bijection as there
is a problem with the dyadic rationals (i.e., numbers of the form
For example, 0.01000
to produce an explicit bijection in the following way. For any real number x ∈ [0, 1], let
a _{b} , where a and b are natural numbers).
2
in binary is the same as 0.001111
. However we can tweak this “near bijection”
g(x) =
∞
k=1
x _{k} 2 ^{−}^{k}
The function g maps {0, 1} ^{∞} almost bijectively to [0, 1] but unfortunately, the dyadic rationals have two
preimages. For example we have g(0.1000 = g(0.0111 = ^{1} _{2} . To ﬁx this let the the set of dyadic rationals be diven by the list
)
)
D = d _{1} = ^{1}
_{2} , d _{2} =
_{4} 1 , d _{3} =
_{4} 3 , d _{4} =
1
_{8} , d _{5} = _{8}
^{3} , d _{6} =
5
_{8} , d _{7} = ^{7} ,
8
34
Lecture 3: Cardinality and Countability
Note that the dyadic rationals can be put in a list as given above as they are countable. Next, we deﬁne the following bijection f (x) from {0, 1} ^{∞} to [0, 1].
f(x) =
g(x)
d
_{2}_{n}_{−}_{1}
_{2}_{n}
d
if g(x) ∈/ D,
if 
g(x) = d _{n} 
for some n ∈ N and x _{k} 
terminates in 1, 
if 
g(x) = d _{n} 
for some n ∈ N and x _{k} 
terminates in 0. 
This is an explicit bijection from {0, 1} ^{∞} to [0, 1] which proves that the set [0, 1] is uncountable. (Why?)
Next, we can deﬁne a bijection from [0, 1] → R, for instance using the function tan(πx − ^{π} ), ∀x ∈ [0, 1]. Thus the set of all real numbers, R is uncountable.
Finally, we can write, R = Q ∪ {R \ Q}. Since Q is countable and R is uncountable, we can easily argue that
{R \ Q}, i.e, the set of all irrational numbers, is uncountable.
2
3.3
Exercises
1. Prove that 2 ^{N} , the power set of the natural numbers, is uncountable. (Hint: Try to associate an inﬁnite
binary string with each subset of N.)
2. Prove that the Cartesian product of two countable sets is countable.
3. Let A be a countable set, and B _{n}
and the elements a _{1} , a _{2} ,
be the set of all ntuples (a _{1} ,
,
a _{n} ), where a _{k}
, a _{n} need not be distinct. Show that B _{n} is countable.
∈ A(k = 1, 2,
, n)
4. Show that an inﬁnite subset of a countable set is countable.
5. A number is said to be an algebraic number if it is a root of some polynomial equation with integer
coeﬃcients. For example, ^{√} 2 is algebraic since it is a root of the polynomial x ^{2} − 2. However, it is known that π is not algebraic. Show that the set of all algebraic numbers is countable. Also, a transcendental number is a real number that is not algebraic. Are the transcendental numbers countable?
6. The Cantor set is an interesting subset of [0, 1], which we will encounter several times in this course.
One way to deﬁne the Cantor set C is as follows. Consider the set of all real numbers in [0, 1] written down in ternary (base3) expansion, instead of the usual decimal (base10) expansion. A real number x ∈ [0, 1]
belongs to C iﬀ x admits a ternary expansion without any 1s. Show that C is uncountably inﬁnite, and that it is indeed equicardinal with [0, 1].
References
[1]
Walter Rudin, “Principles of Mathematical Analysis,” McGraw Hill International Series, Third Edition.
EE5110: Probability Foundations for Electrical Engineers
JulyNovember 2015
Lecture 4: Probability Spaces
Lecturer: Dr. Krishna Jagannathan
Scribe:
Jainam Doshi, Arjun Nadh and Ajay M
4.1
Introduction
Just as a point is not deﬁned in elementary geometry, probability theory begins with two entities that are not deﬁned. These undeﬁned entities are a Random Experiment and its Outcome. These two concepts are to be understood intuitively, as suggested by their respective English meanings. We use these undeﬁned terms to deﬁne other entities.
Deﬁnition 4.1 The Sample Space Ω of a random experiment is the set of all possible outcomes of a random experiment.
An outcome (or elementary outcome) of the random experiment is usually denoted by ω. Thus, when a random experiment is performed, the outcome ω ∈ Ω is picked by the Goddess of Chance or Mother Nature or your favourite genie.
Note that the sample space Ω can be ﬁnite or inﬁnite. Indeed, depending on the cardinality of Ω, it can be classiﬁed as follows:
1. Finite sample space
2. Countably inﬁnite sample space
3. Uncountable sample space
It is imperative to note that for a given random experiment, its sample space is deﬁned depending on what one is interested in observing as the outcome. We illustrate this using an example. Consider a person tossing a coin. This is a random experiment. Now consider the following three cases:
• Suppose one is interested in knowing whether the toss produces a head or a tail, then the sample space is given by, Ω = {H, T }. Here, as there are only two possible outcomes, the sample space is said to be ﬁnite.
• Suppose one is interested in the number of tumbles before the coin hits the ground, then the sample space is the set of all natural numbers. In this case, the sample space is countably inﬁnite and is given by, Ω = N.
• Suppose one is interested in the speed with which the coin strikes ground, then the set of positive real numbers forms the sample space. This is an example of an uncountable sample space, which is given by, Ω = R ^{+} .
41
42
Lecture 4: Probability Spaces
Thus we see that for the same experiment, Ω can be diﬀerent based on what the experimenter is interested in.
Let us now have a look at one more example where the sample space can be diﬀerent for the same experiment and you can get diﬀerent answers based on which sample space you decide to choose.
Bertrand’s Paradox: Consider a circle of radius r. What is the probability that the length of a chord chosen at random is greater than the length of the side of an equilateral triangle inscribed in the circle?
This is an interesting paradox and gives diﬀerent answers based on diﬀerent sample spaces. The entire description of Bertrand’s Paradox can be found in [1].
Deﬁnition 4.2 (Informal) An event is a subset of the sample space, to which probabilities will be assigned.
An event is a subset of the sample space, but we emphasise that not all subsets of the sample space are necessarily considered events, for reasons that will be explained later. Until we are ready to give a more precise deﬁnition, we can consider events to be those “interesting” subsets of Ω, to which we will eventually assign probabilities. We will see later that whenever Ω is ﬁnite or countable, all subsets of the sample space can be considered as events, and be assigned probabilities. However, when Ω is uncountable, it is often not possible to assign probabilities to all subsets of Ω, for reasons that will not be clear now. The way to handle uncountable sample spaces will be discussed later.
Deﬁnition 4.3 An event A is said to occur if the outcome ω, of the random experiment is an element of A, i.e., if ω ∈ A.
Let us take an example. Say the random experiment is choosing a card at random from a pack of playing cards. What is the sample space in this case? It is a 52 element set as each card is a possible outcome. As the sample space is ﬁnite, any subset of the sample space can be considered as an event. As a result there will be 2 ^{5}^{2} events (Power set of n elements has 2 ^{n} elements). An event can be any subset of the sample space which includes the empty set, all the singleton sets (containing one outcome) and collection of more than one outcomes. Listed below are a few events:
• The 7 of Hearts (1 element)
• A face Card (12 elements)
• A 2 and a 7 at the same time (0 element)
• An ace of any color (4 elements)
• A diamond card (13 elements)
Next, let us look at some nice properties that we would expect events to satisfy:
• Since the sample space Ω always occurs, we would like to have Ω as an event.
• If A is an event (i.e., a “nice” subset of the sample space to which we would like to assign a probability), it is reasonable to expect A ^{c} to be an event as well.
• If A and B are two events, we are interested in the occurrence of at least one of them (A or B) as well as the occurrence of both of them (A and B). Hence, we would like to have A ∪ B and A ∩ B to be events as well.
The above three properties motivate a mathematical structure of subsets, known as an algebra.
Lecture 4: Probability Spaces
43
4.2 Algebra, F _{0}
Let Ω be the sample space and let F _{0} be a collection of subsets of Ω. Then, F _{0} is said to be an algebra (or a ﬁeld) if
i. ∅ ∈ F _{0} .
ii. A ∈ F _{0} , implies A ^{c} ∈ F _{0} .
iii. A ∈ F _{0} and B ∈ F _{0} implies A ∪ B ∈ F _{0} .
It can be shown that an algebra is closed under ﬁnite union and ﬁnite intersection (see Exercise 1(a)).
However, a natural question that arises at this point is “Is the structure of an algebra enough to study
events of typical interest?” Consider the following example:
}. Let us say that
we are interested in determining if the number of tosses before seeing a head is even. It is easy to see that this ‘event’ of interest will not be included in the algebra. This is because an algebra contains only ﬁnite
Example: Toss a coin repeatedly until the ﬁrst heads shows. Here, Ω = {H, TH, TTH,
unions of subsets, but the ‘event’ of interest entails a countably inﬁnite union. This motivates the deﬁnition
of a σalgebra.
4.3 Sigma Algebra, F
A collection F of subsets of Ω is called a σalgebra (or σﬁeld) if
i. ∅ ∈ F .
ii. A ∈ F , implies A ^{c} ∈ F.
iii.
If A _{1} , A _{2} , A _{3} ,
is a countable collection of subsets in F , then
∞
i=1
Ai ∈ F .
Note that unlike an algebra, a σalgebra is closed under countable union and countable intersection (see Exercise 1(b)). Some examples of σalgebra are:
i. {∅, Ω}
ii. {∅, A, A ^{c} , Ω}
iii. Power set of Ω, denoted by 2 ^{Ω} .
The 2tuple (Ω, F ) is called a measurable space. Also, every member of the σalgebra F is called an F  measurable set in the context of measure theory. In the speciﬁc context of probability theory, F measurable sets are called events. Thus, whether or not a subset of Ω is considered an event depends on the σalgebra that is under consideration.
4.4 Measure
We now proceed to deﬁne measures and measure spaces. We will see that a probability space is indeed a special case of a measure space.
44
Lecture 4: Probability Spaces
Deﬁnition 4.4 Let (Ω, F ) be a measurable space. A measure on (Ω, F ) is a function µ: F −→ [0,∞] such that
i.
µ(∅) = 0.
ii. If {A _{i} , i ≥ 1} is a sequence of disjoint sets in F , then the measure of the union (of countably inﬁnite disjoint sets) is equal to the sum of measures of individual sets, i.e.,
µ
i=1 A i =
∞
∞
i=1
µ(A _{i} )
(4.1)
The second property stated above is known as the countable additivity property of measures. From the deﬁnition, it is clear that a measure can only be assigned to elements of F . The triplet (Ω, F , µ) is called a measure space. µ is said to be a ﬁnite measure if µ(Ω) < ∞; otherwise, µ is said to be an inﬁnite measure. In particular, if µ(Ω) = 1, then µ is said to be a probability measure. Next, we state this explicitly for pedagogical completeness.
4.5 Probability Measure
A probability measure is a function P: F −→ [0,1] such that
i. 
P(∅) = 0. 
ii. 
P(Ω) = 1. 
iii. (Countable additivity:) If {A _{i} , i ≥ 1} is a sequence of disjoint sets in F , then
P
i=1 A i =
∞
∞
i=1
P(A _{i} )
The triplet (Ω, F , P) is called a probability space, and the three properties, stated above, are sometimes referred to as the axioms of probability.
Note: It is clear from the deﬁnition that probabilities are deﬁned only to elements of F , and not necessarily
to 
all subsets of Ω. In other words, probability measures are assigned only to events. Even when we speak 
of 
the probability of an elementary outcome ω, it should be interpreted as the probability assigned to the 
singleton set {ω} (assuming of course, that the singleton is an event).
4.6 Exercises
1.
a) Let A _{1} , A _{2} ,
, A _{n} be a ﬁnite collection of subsets of Ω such that A _{i} ∈ F _{0} (an algebra), 1 ≤ i ≤ n.
A _{i} ∈ F _{0} . Hence, infer that an algebra is closed under ﬁnite union
n n
i=1
A _{i} ∈ F _{0} and
i=1
Show that
and ﬁnite intersection.
b) Suppose A _{1} , A _{2} , A _{3} ,
∞
i=1
A _{i} ∈ F.
is a countable collection of subsets in the σalgebra F , then show that
Lecture 4: Probability Spaces
45
2. [σalgebra : Properties and Construction].
(a) 
Show that a σalgebra is also an algebra. 

(b) 
Given a sample space Ω and a σalgebra F of the subsets of Ω, show that if A, B ∈ F , A \ B and A B, the symmetric diﬀerence of A and B are present in F . 

(c) 
Consider the random experiment of throwing a die. If a statistician is interested in the occurrence of either an odd or an even outcome, construct a sample space and a σalgebra of subsets of this sample space. 

(d) 
Let A _{1} , A _{2} , 
, 
A _{n} be arbitrary subsets of Ω. Describe (explicitly) the smallest σalgebra F con 
taining A _{1} , A _{2} ,
under certain conditions). List all the sets in F for n = 2.
, A _{n} . How many sets are there in F ? (Give an upper bound that is attainable
3. Let F and G be two σalgebras of subsets of Ω.
(a) 
Is F ∪ G, the collection of subsets of Ω lying in either F or G a σalgebra? 

(b) 
Show that F ∩ G, the collection of subsets of Ω lying in both F and G is a σalgebra. 

(c) 
Generalize (b) to arbitrary intersections as follows. 
Let I be an arbitrary index set (possibly 
uncountable), and let {F _{i} } _{i}_{∈}_{I} be a collection of σalgebras on Ω.
σalgebra.
Show that
^{}
i∈I
F _{i}
is also
a
4. Let F be a σalgebra of subsets of Ω, and let B ∈ F . Show that
is a σalgebra of subsets of B.
G = {A ∩ B A ∈ F}
5. Let X and Y be two sets and let f : X → Y be a function. If F is a σalgebra over the subsets of Y and G = { A  ∃ B ∈ F such that f ^{−}^{1} (B) = A } , does G form a σalgebra of subsets of X ? Note that f ^{−}^{1} (N ) is the notation used for the preimage of set N under the function f for some N ⊆ Y . That is, f ^{−}^{1} (N ) = {x ∈ Xf (x) ∈ N } for some N ⊆ Y .
6. Let Ω be an arbitrary set.
(a) 
Is the collection F _{1} consisting of all ﬁnite subsets of Ω an algebra? 
(b) 
Let F _{2} consist of all ﬁnite subsets of Ω, and all subsets of Ω having a ﬁnite complement. Is F _{2} an algebra? 
(c) 
Is F _{2} a σalgebra? 
(d) 
Let F _{3} consist of all countable subsets of Ω, and all subsets of Ω having a countable complement. Is F _{3} a σalgebra? 
References
[1]
Sheldon Ross, “A First Course in Probability,” Pearson, 8th Edition.
EE5110: Probability Foundations for Electrical Engineers
JulyNovember 2015
Lecture 5: Properties of Probability Measures
Lecturer: Dr. Krishna Jagannathan
Scribe: Ajay M, Gopal Krishna Kamath M
5.1
Properties
In this lecture, we will derive some fundamental properties of probability measures, which follow directly from the axioms of probability. In what follows, (Ω, F , P) is a probability space.
• Property 1: Suppose A be a subset of Ω such that A ∈ F . Then,
(5.1)
Proof: Given any subset A ∈ Ω, A and A ^{c} partition the sample space. Hence, A ^{c} ∪ A = Ω and A ^{c} ∩ A = ∅. By the ”Countable Additivity” axiom of probability, P(A ^{c} ∪ A) = P(A) + P(A ^{c} ) =⇒ P(Ω) = P(A) + P(A ^{c} ) =⇒ P(A ^{c} ) = 1 − P(A).
• Property 2: Consider events A and B such that A ⊆ B and A, B ∈ F . Then P(A) ≤ P(B) Proof: The set B can be written as the union of two disjoint sets A and A ^{c} ∩ B. Therefore, we have P(A) + P(A ^{c} ∩ B) = P(B) =⇒ P(A) ≤ P(B) since P(A ^{c} ∩ B) ≥ 0.
P(A ^{c} ) = 1 − P(A).
• Property 3: (Finite Additivity) If A _{1} , A _{2} ,
, A _{n} are ﬁnite number of disjoint events, then
P
i=1 A i =
n
n
i=1
P(A _{i} ).
(5.2)
Proof: This property follows directly from the axiom of countable additivity of probability measures.
It is obtained by setting the events A _{n}_{+}_{1} , A _{n}_{+}_{2} ,
as empty sets. LHS will simplify as:
P
i=1 A ^{i} = P
∞
i=1 A i .
n
RHS can be manipulated as follows:
∞
i=1
P(A _{i} ) ^{(}^{a}^{)} =
lim
k→∞
k
i=1
P(A _{i} )
n k
=
i=1
n
(b)
=
i=1
n
=
i=1
P(A _{i} ) + lim
k→∞
i=n+1
P(A _{i} ) + lim
k→∞ ^{0}
P(A _{i} ).
P(A _{i} )
where (a) follows from the deﬁnition of an inﬁnite series and (b) is a consequence of setting the events from A _{n}_{+}_{1} onwards to null sets.
51
52
Lecture 5: Properties of Probability Measures
• Property 4: For any A, B ∈ F ,
P(A ∪ B) = P(A) + P(B) − P(A ∩ B).
In general, for a family of events {A _{i} } ^{n} i=1 ^{⊂} ^{F}^{,}
P
i=1 A i =
n
n
i=1
P(A _{i} ) − ^{} P(A _{i} ∩ A _{j} ) + i<j<k P(A _{i} ∩ A _{j} ∩ A _{k} ) +
i<j
(5.3)
n + (−1) ^{n}^{+}^{1} P i=1 A _{i} . (5.4)
This property is proved using induction on n. The property can be proved in a much more simpler way using the concept of Indicator Random Variables, which will be discussed in the subsequent lectures. Proof of Eq (5.3): The set A ∪ B can be written as A ∪ B = A ∪ (A ^{c} ∩ B). Since A and A ^{c} ∩ B are disjoint events, P(A∪B) = P(A)+P(A ^{c} ∩B). Now, set B can be partitioned as, B = (A∩B)∪(A ^{c} ∩B). Hence, P(B) = P(A ∩ B) + P(A ^{c} ∩ B). On substituting this result in the expression of P(A ∪ B), we will obtain the ﬁnal result that P(A ∪ B) = P(A) + P(B) − P(A ∩ B).
• Property 5: If {A _{i} , i ≥ 1} are events, then
P i=1 A _{i} =
∞
m→∞ ^{P}
lim
i=1 A _{i} .
m
This result is known as continuity of probability measures.
Proof: Deﬁne a new family of sets B _{1} = A _{1} , B _{2} = A _{2} \ A _{1} ,
Then, the following claims are placed:
Claim 1: B _{i} ∩ B _{j} = ∅, ∀i
Claim 2:
Since {B _{i} , i ≥ 1} is a disjoint sequence of events, and using the above claims, we get
n−1
i=1
,
B _{n} = A _{n} \
A _{i} ,
= j.
∞ ∞
i=1
A _{i} =
i=1
B _{i} .
Therefore,
P
i=1 A ^{i} = P
∞
P
i=1 A i
∞
=
^{(}^{a}^{)}
=
^{(}^{b}^{)}
=
^{(}^{c}^{)}
=
i=1 B i =
∞
∞
i=1
P(B _{i} )
lim
m→∞
m
i=1
m→∞ ^{P}
lim
m→∞ ^{P}
lim
∞
i=1
P(B _{i} )
P(B _{i} ).
m
i=1
B i
i=1 A i .
m
(5.5)
Here, (a) follows from the deﬁnition of an inﬁnite series, (b) follows from Claim 1 in conjunction with Countable Additivity axiom of probability measure and (c) follows from the intermediate result required to prove Claim 2. Hence proved.
Lecture 5: Properties of Probability Measures
53
• Property 6: If {A _{i} , i ≥ 1} is a sequence of increasing nested events i.e. A _{i} ⊆ A _{i}_{+}_{1} , ∀i ≥ 1, then
P i=1 A _{i} =
∞
_{m}_{→}_{∞} P (A _{m} ) .
lim
(5.6)
• Property 7: If {A _{i} , i ≥ 1} is a sequence of decreasing nested events i.e. A _{i}_{+}_{1} ⊆ A _{i} ∀i ≥ 1, then
P i=1 A _{i} =
∞
_{m}_{→}_{∞} P (A _{m} ) .
lim
(5.7)
Properties 6 and 7 are said to be corollaries to Property 5.
• Property 8: Suppose {A _{i} , i ≥ 1} are events, then
P
i=1 A i ≤
∞
∞
i=1
P(A _{i} ).
(5.8)
This result is known as the Union Bound. This is bound is trivial if ^{} _{i}_{=}_{1} P(A _{i} ) ≥ 1 since the LHS of (5.8) is a probability of some event. This is a very widely used bound, and has several applications. For instance, the union bound is used in the probability of error analysis in Digital Communications for complicated modulation schemes.
Proof: Deﬁne a new family of sets B _{1} = A _{1} , B _{2} = A _{2} \ A _{1} ,
∞
n−1
,
B _{n} = A _{n} \
A _{i} ,
i=1
Claim 1: B _{i} ∩ B _{j} = ∅, ∀i
Claim 2:
Since {B _{i} , i ≥ 1} is a disjoint sequence of events, and using the above claims, we get
= j.
∞ ∞
i=1
A _{i} =
i=1
B _{i} .
P
i=1 A ^{i} = P
∞
i=1 B i =
∞
∞
i=1
P(B _{i} ).
Also, since B _{i} ⊆ A _{i} ∀i ≥ 1, P(B _{i} ) ≤ P(A _{i} ) ∀i ≥ 1 (using Property 2). probabilities follow
_{n}
i=1
P(B _{i} ) ≤
n
i=1
P(A _{i} ).
Eventually, in the limit, the following holds:
Therefore, the ﬁnite sum of
∞ ∞
i=1
P(B _{i} ) ≤
i=1
P(A _{i} ).
Finally we arrive at the result,
P _{}
i=1 A i ≤
∞
∞
i=1
P(A _{i} ).
5.2
Exercises
1.
a) Prove Claim 1 and Claim 2 stated in Property 5.
54
Lecture 5: Properties of Probability Measures
b) Prove Properties 6 and 7, which are corollaries of Property 5.
2. A standard card deck (52 cards) is distributed to two persons: 26 cards to each person. All partitions are equally likely. Find the probability that the ﬁrst person receives all four aces.
3. Consider two events A and B such that P(A) > 1 − δ and P(B) > 1 − δ, for some very small δ > 0. Prove that P(A ∩ B) is close to 1.
4. [Grimmett] Given events A _{1} , A _{2} ,
A _{n} , prove that,
P(∪ _{1}_{≤}_{r}_{≤}_{n} A _{r} ) ≤ min
1≤k≤n
1≤r≤n
P(A _{r} ) − ^{} =k P(A _{r} ∩ A _{k} )
r:r
5. Consider a measurable space (Ω, F ) with Ω = [0, 1]. A measure P is deﬁned on the nonempty subsets of Ω (in F ), which are all of the form (a, b), (a, b], [a, b) and [a, b], as the length of the interval, i.e., P((a, b)) = P((a, b]) = P([a, b)) = P([a, b]) = b − a.
a) Show that P is not just a measure, but its a probability measure.
b) Let A _{n} = [ _{n}_{+}_{1} , 1] and B _{n} = [0, _{n}_{+}_{1} ], for n ≥ 1. P(∩ _{i}_{∈}_{N} B _{i} ).
1
1
Compute P(∪ _{i}_{∈}_{N} A _{i} ),
c) Compute P(∩ _{i}_{∈}_{N} (B
c
i
∪ A
c
i
)).
1
d) Let C _{m} = [0, _{m} ] such that P(C _{m} ) = P(A _{n} ). Express m in terms of n.
e) Evaluate P(∩ _{i}_{∈}_{N} (C _{i} ∩ A _{i} ))
and P(∪ _{i}_{∈}_{N} (C _{i} ∩ A _{i} )).
P(∩ _{i}_{∈}_{N} A _{i} ), P(∪ _{i}_{∈}_{N} B _{i} ) and
6. [Grimmett] You are given that at least one of the events A _{n} , 1 ≤ n ≤ N , is certain to occur. However, certainly no more than two occur. If P(A _{n} ) = p and P(A _{n} ∩ A _{m} ) = q, m = n, then show that p ≥ and q ≤
2
N ^{.}
1
N
EE5110: Probability Foundations in Signal Processing
JulyNovember 2015
Lecture 6: Discrete Probability Spaces
Lecturer: Dr. Krishna Jagannathan
Scribe: Ravi Kolla
6.1 Discrete Probability Spaces
In this lecture, we discuss discrete probability spaces. This corres ponds to the case when the sample space Ω is countable. This is the most conceptually straightforward case, since it is possible to assign probabilities to all subsets of Ω .
Deﬁnition 6.1 A probability space (Ω , F , P ) is said to be a discrete probability space if the following condi tions hold:
(a) 
The sample space Ω is ﬁnite or countably inﬁnite, 
(b) 
The σ algebra is the set of all subsets of Ω , i.e., F = 2 ^{Ω} , and 
(c) 
The probability measure, P , is deﬁned for every subset of Ω . In particular, it can be deﬁned in terms of the probabilities P ({ ω } ) of the singletons corresponding to each of the elementary ou tcomes ω , and satisﬁes for every A ∈ F , 
and
P (A) = ^{} P ({ ω } ) ,
ω ∈ A
_{}
ω ∈ Ω
P ({ ω } ) = 1 .
6.1.1 Examples of Discrete Probability Space
1. Let us consider a coin toss experiment with the probability of getting a head as p and the probability of getting a tail as (1 − p ). Then, the sample space and the σ algebra are
Ω = { H, T } ≡ { 0 , 1 },
F = 2 ^{Ω} = { Φ , { H } , { T } , { Ω }} .
respectively. The probability measure is
P 
({ H } ) ≡ P ({ 0 } ) = p, 

P 
({ T } ) ≡ P ({ 1 } ) = 
1 − p. 
In this case, we say that P (. ) is a Bernoulli measure on ^{} { 0 , 1 } , 2 ^{{} ^{0}^{,} ^{1}^{}} ^{} .
2. Let Ω = N , F = 2 ^{N} . Then, we can deﬁne the probability of a singleton as
under the constraint that
P ({ k } ) = a _{k} ≥ 0 , k ∈ N
_{}
k ∈ N
P ({ k } ) = 1 .
61
62
Lecture 6: Discrete Probability Spaces
For example, a _{k} = ^{1} _{k} , k ∈ N is a valid measure, since
2
k ∈ N
1
_{2} _{k} = 1 .
As another example, consider a _{k} = (1 − p ) ^{k} ^{−} ^{1} p, 0 < p < 1 , k ∈ N . This is known as a geometric measure with parameter p . It is a valid probability measure since
k ∈ N
(1 − p ) ^{k} ^{−} ^{1} p = 1 .
3. Let Ω = N ∪ { 0 } , F = 2 ^{Ω} . Let us deﬁne
P ({ k } ) =
_{e} − λ _{λ} k
k !
,
λ > 0 .
This probability measure is called a Poisson measure with parameter λ on ^{} Ω , 2 ^{Ω} ^{} . This is a valid probability measure, since
∞
k
=0
P ({ k } ) =
∞
k
=0
_{e} − λ _{λ} k
k
!
= e ^{−} ^{λ}
∞
k
=0
λ
^{k}
k
!
e ^{λ}
= 1 .
4. Let Ω = { 0 , 1 , 2 , · · · , N } , N ∈ N F = 2 ^{Ω} . Let
us deﬁne
P ({ k } ) = ^{N} p ^{k} (1 − p ) ^{N} ^{−} ^{k} , 0 < p < 1 .
k
This probability measure is called a Binomial measure with parameters (N, p ) on ^{} Ω , 2 ^{Ω} ^{} . This can be veriﬁed to be a valid probability measure as follows:
k ∈ Ω N
k
p ^{k} (1 − p ) ^{N} ^{−} ^{k} = (p + 1 − p ) ^{N} = 1
Note that in all the examples above, we have not explicitly speciﬁed an expression for P (A) for every A ⊂ Ω. Since the sample space is countable, the probability of any subset of the sample space can be obtained as the sum of probabilities of the corresponding elementary outcomes . In other words, for discrete probability spaces, it suﬃces to specify the probabilities of singletons corresp onding to each of the elementary outcomes.
6.2 Exercises
1. An urn contains a number of white balls and b number of black balls. Balls are drawn randomly from the urn without replacement. Find the probability that a white ball is draw n at the k th draw.
2. An urn contains white and black balls. When two balls are drawn witho ut replacement, suppose the probability that both the balls are white is (a) Find the smallest number of balls in the urn. (b) How small can the total number of balls be if the number of black balls is even?
1
3 ^{.}
Lecture 6: Discrete Probability Spaces
63
3. Consider the sample space Ω = N . Find the values of the constant C for which the following are
probability measures:
(a)
(b)
(c)
(d)
f (x)
f (x)
f (x)
= C 2 ^{−} ^{x}
= ^{C} ^{2} ^{−} ^{x}
x
= Cx ^{−} ^{2}
f (x) = ^{C} ^{2} ^{x}
x
!
4.
set of odd numbers? Prime numbers?
Recall the Poisson measure on (Ω , 2 ^{Ω} ), where Ω = N ∪ { 0 } . What is the probability assigned to the
EE5110: Probability Foundations for Electrical Engineers
JulyNovember 2015
Lecture 7: Borel Sets and Lebesgue Measure
Lecturer: Dr. Krishna Jagannathan
Scribes: Ravi Kolla, Aseem Sharma, Vishakh Hegde
In this lecture, we discuss the case where the sample space is uncountable. This case is more involved than the case of a countable sample space, mainly because it is often not possible to assign probabilities to all subsets of Ω. Instead, we are forced to work with a smaller σ algebra. We consider assigning a “uniform probability measure” on the unit interval.
7.1 Uncountable sample spaces
Consider the experiment of picking a real number at random from Ω = [0, 1], such that every number is “equally likely” to be picked. It is quite apparent that a simple strategy of assigning probabilities to singleton subsets of the sample space gets into diﬃculties quite quickly. Indeed,
(i) 
If we assign some positive probability to each elementary outcome, then the probability of an event with inﬁnitely many elements, such as A = {1, ^{1} _{2} , ^{1} _{3} , · · · }, would become unbounded. 
(ii) 
If we assign zero probability to each elementary outcome, this alone would not be suﬃcient to determine 
the probability of a uncountable subset of Ω, such as ^{}
2 ^{,} not additive over uncountable disjoint unions (of singletons in this case).
1
2
_{3} ^{} . This is because probability measures are
Thus, we need a diﬀerent approach to assign probabilities when the sample space is uncountable, such as Ω = [0, 1]. In particular, we need to assign probabilities directly to speciﬁc subsets of Ω. Intuitively, we would like our ‘uniform measure’ µ on [0, 1] to possess the following two properties.
(i) 
µ ((a, b)) = µ ((a, b]) = µ ([a, b)) 
= µ ([a, b]) 
(ii) 
Translational Invariance. That is, if A ∈ [0, 1], then for any x ∈ Ω, µ (A ⊕ x) = µ (A) where, the set A ⊕ x is deﬁned as 
A ⊕ x = {a + xa ∈ A, a + x ≤ 1} ∪ {a + x − 1a ∈ A, a + x > 1}
However, the following impossibility result asserts that there is no way to consistently deﬁne a uniform measure on all subsets of [0, 1].
Theorem 7.1 (Impossibility Result) There does not exist a deﬁnition of a measure µ (A) for all subsets of [0, 1] satisfying (i) and (ii).
Proof: Refer proposition 1.2.6 in [1].
Therefore, we must compromise, and consider a smaller σalgebra that contains certain “nice” subsets of the sample space [0, 1]. These “nice” subsets are the intervals, and the resulting σalgebra is called the Borel σalgebra. Before deﬁning Borel sets, we introduce the concept of generating σalgebras from a given collection of subsets.
Bien plus que des documents.
Découvrez tout ce que Scribd a à offrir, dont les livres et les livres audio des principaux éditeurs.
Annulez à tout moment.