Vous êtes sur la page 1sur 154

EE5110: Probability Foundations for Electrical Engineers

July-November 2015

Lecture 3: Cardinality and Countability

Lecturer: Dr. Krishna Jagannathan

Scribe: Ravi Kiran Raman

3.1 Functions

We recall the following deﬁnitions.

Deﬁnition 3.1 A function f : A B is a rule that maps every element of set A to a unique element in set B.

In other words, x A, y B and only one such element, such that, f (x) = y. Then y is called the image of x and x, the pre-image of y under f . The set A is called the domain of the function and B, the co-domain.

R = {y : x A,

s.t. f (x) = y} is called as the range of the function f .

Deﬁnition 3.2 A function f : A B is said to be an injective (one-to-one) function, if every element in the range R has a unique pre-image in A.

Deﬁnition 3.3 A function f y B, x A, s.t. f (x) = y.

:

A

B

is said to be a surjective

(onto)

function, if R =

B ,

i.e,

Deﬁnition 3.4 A function f : A B is a bijective function if it is both injective and surjective.

Hence, in a bijective mapping, every element in the co-domain has a pre-image and the pre-images are unique. Thus, we can deﬁne an inverse function, f 1 : B A, such that, f 1 (y) = x, if f (x) = y. In simple terms, bijective functions have well-deﬁned inverse functions.

3.2 Cardinality and Countability

In informal terms, the cardinality of a set is the number of elements in that set. If one wishes to compare the cardinalities of two ﬁnite sets A and B, it can be done by simply counting the number of elements in each set, and declare either that they have equal cardinality, or that one of the sets has more elements than the other. However, when sets containing inﬁnitely many elements are to be compared(for example, N versus Q), this elementary approach is not eﬃcient to do it. In the late nineteenth century, Georg Cantor introduced the idea of comparing the cardinality of sets based on the nature of functions that can be possibly deﬁned from one set to another.

Deﬁnition 3.5

(i) Two sets A and B are equicardinal (notation |A| = |B|) if there exists a bijective

function from A to B.

(ii) B has cardinality greater than or equal to that of A (notation |B| ≥ |A|) if there exists an injective function from A to B.

3-1

3-2

Lecture 3: Cardinality and Countability

(iii) B has cardinality strictly greater than that of A (notation |B| > |A|) if there is an injective function, but no bijective function, from A to B.

Having stated the deﬁnitions as above, the deﬁnition of countability of a set is as follow:

Deﬁnition 3.6 A set E is said to be countably inﬁnite if E and N are equicardinal. And, a set is said to be countable if it is either ﬁnite or countably inﬁnite.

The following are some examples of countable sets:

1. The set of all integers Z is countably inﬁnite.

We can deﬁne the bijection f : Z N as follows :

 n = f(z) ∈ N z ∈ Z 1 0 2 +1 3 -1 4 +2 5 -2 . . . . . .

The existence of this bijective map from Z to N proves that Z is countably inﬁnite.

2. The set of all rationals in [0, 1] is countable.

Consider the rational number p

to the set, if it not already present. By this way, the set of rational

numbers in [0, 1] can be explicitly listed as: {0, 1,

6 , Clearly, we can deﬁne a bijection from Q [0, 1] N where each rational number is mapped to its index in the above set. Thus the set of all rational numbers in [0, 1] is countably inﬁnite and thus countable.

0 p q, add the rational number p

= 0. Increment q in steps of 1 starting with 1. For each such q and

q

where q

q

1

2 ,

1

3 ,

2

3 ,

1

4 ,

3

4 , 1

5 , 2

3

4 , 1

5

5

6 ,

}

5 , 5

,

3. The set of all Rational numbers, Q is countable.

In order to prove this, we state an important theorem, whose proof can be found in [1].

Theorem 3.7 Let I be a countable index set, and let E i be countable for each i ∈ I . Then iI E i is countable. More glibly, it can also be stated as follows: A countable union of countable sets is countable.

We will now use this theorem to prove the countability of the set of all rational numbers. It has been already proved that the set Q [0, 1] is countable. Similarly, it can be showed that Q [n, n + 1] is countable, n Z. Let Q i = Q [i, i + 1]. Thus, clearly, the set of all rational numbers, Q = iZ Q i – a countable union of countable sets – is countable.

Remark: For two ﬁnite sets A and B, we know that if A is a strict subset of B, then B has cardinality greater than that of A. As the above examples show, this is not true for inﬁnite sets. Indeed, N is a strict subset of Q, but N and Q are equicardinal!

4. The set of all algebraic numbers (numbers which are roots of polynomial equations with rational co-

eﬃcients) is countable.

5. The set of all computable numbers, i.e., real numbers that can be computed to within any desired precision by a ﬁnite, terminating algorithm, is countable (see Wikipedia article for more details).

Lecture 3: Cardinality and Countability

3-3

Deﬁnition 3.8 A set F is uncountable if it has cardinality strictly greater than the cardinality of N.

In the spirit of Deﬁnition 3.5, this means that F is uncountable if an injective function from N to F exists, but no such bijective function exists.

An interesting example of an uncountable set is the set of all inﬁnite binary strings. The proof of the following theorem uses the celebrated ‘diagonal argument’ of Cantor.

Theorem 3.9 (Cantor) : The set of all inﬁnite binary strings, {0, 1} , is uncountable.

Proof: It is easy to show that an injection from N to {0, 1} exists (exercise: produce one!). We need to show that no such bijection exists.

Let us assume the contrary, i.e, let us assume that the set of all binary strings, A = {0, 1} is countably inﬁnite. Thus there exists a bijection f : A N. In other words, we can order the set of all inﬁnite binary strings as follows:

a

a

a

 11 a 12 a 13 . . . 21 a 22 a 23 . . . 31 a 32 a 33 . . . . . . . . . . . . ,

where, a ij is the j th bit of the i th binary string, i, j 1.

Consider the inﬁnite binary string given by ¯a = a¯ 11 a¯ 22 a¯ 33

Since our list contains all inﬁnite binary strings, there must exist some k N such that the string a¯ occurs at the k position in the list, i.e., f a) = k. The k th bit of this speciﬁc string is a¯ kk . However, from the above list, we know that the k th bit of the k th string is a kk . Thus, we can conclude that the string ¯a cannot occur in any position k 1 in our list, contradicting our initial assumption that our list exhausts all possible inﬁnite binary strings.

Thus, there cannot possibly exist a bijection from N to {0, 1} , proving that {0, 1} is uncountable.

Now using Cantor’s theorem, we will prove that the set of irrational numbers is uncountable.

, where a¯ ij is the complement of the bit a ij .

Claim 3.10 The sets [0, 1], R and {R \ Q} are uncountable.

Proof: Firstly, consider the set [0, 1]. Any number in this set can be expressed by its binary equivalent and

thus, there appears to be a bijection from [0, 1] → {0, 1} . However, this is not exactly a bijection as there

is a problem with the dyadic rationals (i.e., numbers of the form

For example, 0.01000

to produce an explicit bijection in the following way. For any real number x [0, 1], let

a b , where a and b are natural numbers).

2

in binary is the same as 0.001111

. However we can tweak this “near bijection”

g(x) =

k=1

x k 2 k

The function g maps {0, 1} almost bijectively to [0, 1] but unfortunately, the dyadic rationals have two

pre-images. For example we have g(0.1000 = g(0.0111 = 1 2 . To ﬁx this let the the set of dyadic rationals be diven by the list

)

)

D = d 1 = 1

2 , d 2 =

4 1 , d 3 =

4 3 , d 4 =

1

8 , d 5 = 8

3 , d 6 =

5

8 , d 7 = 7 ,

8

3-4

Lecture 3: Cardinality and Countability

Note that the dyadic rationals can be put in a list as given above as they are countable. Next, we deﬁne the following bijection f (x) from {0, 1} to [0, 1].

f(x) =

  g(x)

d

2n1

2n

d

if g(x) / D,

 if g(x) = d n for some n ∈ N and x k terminates in 1, if g(x) = d n for some n ∈ N and x k terminates in 0.

This is an explicit bijection from {0, 1} to [0, 1] which proves that the set [0, 1] is uncountable. (Why?)

Next, we can deﬁne a bijection from [0, 1] R, for instance using the function tan(πx π ), x [0, 1]. Thus the set of all real numbers, R is uncountable.

Finally, we can write, R = Q ∪ {R \ Q}. Since Q is countable and R is uncountable, we can easily argue that

{R \ Q}, i.e, the set of all irrational numbers, is uncountable.

2

3.3

Exercises

1. Prove that 2 N , the power set of the natural numbers, is uncountable. (Hint: Try to associate an inﬁnite

binary string with each subset of N.)

2. Prove that the Cartesian product of two countable sets is countable.

3. Let A be a countable set, and B n

and the elements a 1 , a 2 ,

be the set of all n-tuples (a 1 ,

,

a n ), where a k

, a n need not be distinct. Show that B n is countable.

A(k = 1, 2,

, n)

4. Show that an inﬁnite subset of a countable set is countable.

5. A number is said to be an algebraic number if it is a root of some polynomial equation with integer

coeﬃcients. For example, 2 is algebraic since it is a root of the polynomial x 2 2. However, it is known that π is not algebraic. Show that the set of all algebraic numbers is countable. Also, a transcendental number is a real number that is not algebraic. Are the transcendental numbers countable?

6. The Cantor set is an interesting subset of [0, 1], which we will encounter several times in this course.

One way to deﬁne the Cantor set C is as follows. Consider the set of all real numbers in [0, 1] written down in ternary (base-3) expansion, instead of the usual decimal (base-10) expansion. A real number x [0, 1]

belongs to C iﬀ x admits a ternary expansion without any 1s. Show that C is uncountably inﬁnite, and that it is indeed equi-cardinal with [0, 1].

References

[1]

Walter Rudin, “Principles of Mathematical Analysis,” McGraw Hill International Series, Third Edition.

EE5110: Probability Foundations for Electrical Engineers

July-November 2015

Lecture 4: Probability Spaces

Lecturer: Dr. Krishna Jagannathan

Scribe:

Jainam Doshi, Arjun Nadh and Ajay M

4.1

Introduction

Just as a point is not deﬁned in elementary geometry, probability theory begins with two entities that are not deﬁned. These undeﬁned entities are a Random Experiment and its Outcome. These two concepts are to be understood intuitively, as suggested by their respective English meanings. We use these undeﬁned terms to deﬁne other entities.

Deﬁnition 4.1 The Sample Space of a random experiment is the set of all possible outcomes of a random experiment.

An outcome (or elementary outcome) of the random experiment is usually denoted by ω. Thus, when a random experiment is performed, the outcome ω Ω is picked by the Goddess of Chance or Mother Nature or your favourite genie.

Note that the sample space Ω can be ﬁnite or inﬁnite. Indeed, depending on the cardinality of Ω, it can be classiﬁed as follows:

1. Finite sample space

2. Countably inﬁnite sample space

3. Uncountable sample space

It is imperative to note that for a given random experiment, its sample space is deﬁned depending on what one is interested in observing as the outcome. We illustrate this using an example. Consider a person tossing a coin. This is a random experiment. Now consider the following three cases:

Suppose one is interested in knowing whether the toss produces a head or a tail, then the sample space is given by, Ω = {H, T }. Here, as there are only two possible outcomes, the sample space is said to be ﬁnite.

Suppose one is interested in the number of tumbles before the coin hits the ground, then the sample space is the set of all natural numbers. In this case, the sample space is countably inﬁnite and is given by, Ω = N.

Suppose one is interested in the speed with which the coin strikes ground, then the set of positive real numbers forms the sample space. This is an example of an uncountable sample space, which is given by, Ω = R + .

4-1

4-2

Lecture 4: Probability Spaces

Thus we see that for the same experiment, Ω can be diﬀerent based on what the experimenter is interested in.

Let us now have a look at one more example where the sample space can be diﬀerent for the same experiment and you can get diﬀerent answers based on which sample space you decide to choose.

Bertrand’s Paradox: Consider a circle of radius r. What is the probability that the length of a chord chosen at random is greater than the length of the side of an equilateral triangle inscribed in the circle?

This is an interesting paradox and gives diﬀerent answers based on diﬀerent sample spaces. The entire description of Bertrand’s Paradox can be found in [1].

Deﬁnition 4.2 (Informal) An event is a subset of the sample space, to which probabilities will be assigned.

An event is a subset of the sample space, but we emphasise that not all subsets of the sample space are necessarily considered events, for reasons that will be explained later. Until we are ready to give a more precise deﬁnition, we can consider events to be those “interesting” subsets of Ω, to which we will eventually assign probabilities. We will see later that whenever Ω is ﬁnite or countable, all subsets of the sample space can be considered as events, and be assigned probabilities. However, when Ω is uncountable, it is often not possible to assign probabilities to all subsets of Ω, for reasons that will not be clear now. The way to handle uncountable sample spaces will be discussed later.

Deﬁnition 4.3 An event A is said to occur if the outcome ω, of the random experiment is an element of A, i.e., if ω A.

Let us take an example. Say the random experiment is choosing a card at random from a pack of playing cards. What is the sample space in this case? It is a 52 element set as each card is a possible outcome. As the sample space is ﬁnite, any subset of the sample space can be considered as an event. As a result there will be 2 52 events (Power set of n elements has 2 n elements). An event can be any subset of the sample space which includes the empty set, all the singleton sets (containing one outcome) and collection of more than one outcomes. Listed below are a few events:

The 7 of Hearts (1 element)

A face Card (12 elements)

A 2 and a 7 at the same time (0 element)

An ace of any color (4 elements)

A diamond card (13 elements)

Next, let us look at some nice properties that we would expect events to satisfy:

Since the sample space Ω always occurs, we would like to have Ω as an event.

If A is an event (i.e., a “nice” subset of the sample space to which we would like to assign a probability), it is reasonable to expect A c to be an event as well.

If A and B are two events, we are interested in the occurrence of at least one of them (A or B) as well as the occurrence of both of them (A and B). Hence, we would like to have A B and A B to be events as well.

The above three properties motivate a mathematical structure of subsets, known as an algebra.

Lecture 4: Probability Spaces

4-3

4.2 Algebra, F 0

Let Ω be the sample space and let F 0 be a collection of subsets of Ω. Then, F 0 is said to be an algebra (or a ﬁeld) if

i. ∅ ∈ F 0 .

ii. A ∈ F 0 , implies A c ∈ F 0 .

iii. A ∈ F 0 and B ∈ F 0 implies A B ∈ F 0 .

It can be shown that an algebra is closed under ﬁnite union and ﬁnite intersection (see Exercise 1(a)).

However, a natural question that arises at this point is “Is the structure of an algebra enough to study

events of typical interest?” Consider the following example:

}. Let us say that

we are interested in determining if the number of tosses before seeing a head is even. It is easy to see that this ‘event’ of interest will not be included in the algebra. This is because an algebra contains only ﬁnite

Example:- Toss a coin repeatedly until the ﬁrst heads shows. Here, Ω = {H, TH, TTH,

unions of subsets, but the ‘event’ of interest entails a countably inﬁnite union. This motivates the deﬁnition

of a σ-algebra.

4.3 Sigma Algebra, F

A collection F of subsets of Ω is called a σ-algebra (or σ-ﬁeld) if

i. ∅ ∈ F .

ii. A ∈ F , implies A c ∈ F.

iii.

If A 1 , A 2 , A 3 ,

is a countable collection of subsets in F , then

i=1

Ai ∈ F .

Note that unlike an algebra, a σ-algebra is closed under countable union and countable intersection (see Exercise 1(b)). Some examples of σ-algebra are:

i. {∅, }

ii. {∅, A, A c , Ω}

iii. Power set of Ω, denoted by 2 .

The 2-tuple (Ω, F ) is called a measurable space. Also, every member of the σ-algebra F is called an F - measurable set in the context of measure theory. In the speciﬁc context of probability theory, F -measurable sets are called events. Thus, whether or not a subset of Ω is considered an event depends on the σ-algebra that is under consideration.

4.4 Measure

We now proceed to deﬁne measures and measure spaces. We will see that a probability space is indeed a special case of a measure space.

4-4

Lecture 4: Probability Spaces

Deﬁnition 4.4 Let (Ω, F ) be a measurable space. A measure on (Ω, F ) is a function µ: F −→ [0,] such that

i.

µ() = 0.

ii. If {A i , i 1} is a sequence of disjoint sets in F , then the measure of the union (of countably inﬁnite disjoint sets) is equal to the sum of measures of individual sets, i.e.,

µ

i=1 A i =

i=1

µ(A i )

(4.1)

The second property stated above is known as the countable additivity property of measures. From the deﬁnition, it is clear that a measure can only be assigned to elements of F . The triplet (Ω, F , µ) is called a measure space. µ is said to be a ﬁnite measure if µ(Ω) < ; otherwise, µ is said to be an inﬁnite measure. In particular, if µ(Ω) = 1, then µ is said to be a probability measure. Next, we state this explicitly for pedagogical completeness.

4.5 Probability Measure

A probability measure is a function P: F −→ [0,1] such that

 i. P(∅) = 0. ii. P(Ω) = 1.

iii. (Countable additivity:) If {A i , i 1} is a sequence of disjoint sets in F , then

P

i=1 A i =

i=1

P(A i )

The triplet (Ω, F , P) is called a probability space, and the three properties, stated above, are sometimes referred to as the axioms of probability.

Note:- It is clear from the deﬁnition that probabilities are deﬁned only to elements of F , and not necessarily

 to all subsets of Ω. In other words, probability measures are assigned only to events. Even when we speak of the probability of an elementary outcome ω, it should be interpreted as the probability assigned to the

singleton set {ω} (assuming of course, that the singleton is an event).

4.6 Exercises

1.

a) Let A 1 , A 2 ,

, A n be a ﬁnite collection of subsets of Ω such that A i ∈ F 0 (an algebra), 1 i n.

A i ∈ F 0 . Hence, infer that an algebra is closed under ﬁnite union

n n

i=1

A i ∈ F 0 and

i=1

Show that

and ﬁnite intersection.

b) Suppose A 1 , A 2 , A 3 ,

i=1

A i ∈ F.

is a countable collection of subsets in the σ-algebra F , then show that

Lecture 4: Probability Spaces

4-5

2. [σ-algebra : Properties and Construction].

 (a) Show that a σ-algebra is also an algebra. (b) Given a sample space Ω and a σ-algebra F of the subsets of Ω, show that if A, B ∈ F , A \ B and A B, the symmetric diﬀerence of A and B are present in F . (c) Consider the random experiment of throwing a die. If a statistician is interested in the occurrence of either an odd or an even outcome, construct a sample space and a σ-algebra of subsets of this sample space. (d) Let A 1 , A 2 , , A n be arbitrary subsets of Ω. Describe (explicitly) the smallest σ-algebra F con-

taining A 1 , A 2 ,

under certain conditions). List all the sets in F for n = 2.

, A n . How many sets are there in F ? (Give an upper bound that is attainable

3. Let F and G be two σ-algebras of subsets of Ω.

 (a) Is F ∪ G, the collection of subsets of Ω lying in either F or G a σ-algebra? (b) Show that F ∩ G, the collection of subsets of Ω lying in both F and G is a σ-algebra. (c) Generalize (b) to arbitrary intersections as follows. Let I be an arbitrary index set (possibly

uncountable), and let {F i } iI be a collection of σ-algebras on Ω.

σ-algebra.

Show that

i∈I

F i

is also

a

4. Let F be a σ-algebra of subsets of Ω, and let B ∈ F . Show that

is a σ-algebra of subsets of B.

G = {A B |A ∈ F}

5. Let X and Y be two sets and let f : X Y be a function. If F is a σ-algebra over the subsets of Y and G = { A | ∃ B ∈ F such that f 1 (B) = A } , does G form a σ-algebra of subsets of X ? Note that f 1 (N ) is the notation used for the pre-image of set N under the function f for some N Y . That is, f 1 (N ) = {x X|f (x) N } for some N Y .

6. Let Ω be an arbitrary set.

 (a) Is the collection F 1 consisting of all ﬁnite subsets of Ω an algebra? (b) Let F 2 consist of all ﬁnite subsets of Ω, and all subsets of Ω having a ﬁnite complement. Is F 2 an algebra? (c) Is F 2 a σ-algebra? (d) Let F 3 consist of all countable subsets of Ω, and all subsets of Ω having a countable complement. Is F 3 a σ-algebra?

References

[1]

Sheldon Ross, “A First Course in Probability,” Pearson, 8th Edition.

EE5110: Probability Foundations for Electrical Engineers

July-November 2015

Lecture 5: Properties of Probability Measures

Lecturer: Dr. Krishna Jagannathan

Scribe: Ajay M, Gopal Krishna Kamath M

5.1

Properties

In this lecture, we will derive some fundamental properties of probability measures, which follow directly from the axioms of probability. In what follows, (Ω, F , P) is a probability space.

Property 1:- Suppose A be a subset of Ω such that A ∈ F . Then,

(5.1)

Proof:- Given any subset A Ω, A and A c partition the sample space. Hence, A c A = Ω and A c A = . By the ”Countable Additivity” axiom of probability, P(A c A) = P(A) + P(A c ) =P(Ω) = P(A) + P(A c ) =P(A c ) = 1 P(A).

Property 2:- Consider events A and B such that A B and A, B ∈ F . Then P(A) P(B) Proof:- The set B can be written as the union of two disjoint sets A and A c B. Therefore, we have P(A) + P(A c B) = P(B) =P(A) P(B) since P(A c B) 0.

P(A c ) = 1 P(A).

Property 3:- (Finite Additivity) If A 1 , A 2 ,

, A n are ﬁnite number of disjoint events, then

P

i=1 A i =

n

n

i=1

P(A i ).

(5.2)

Proof:- This property follows directly from the axiom of countable additivity of probability measures.

It is obtained by setting the events A n+1 , A n+2 ,

as empty sets. LHS will simplify as:

P

i=1 A i = P

i=1 A i .

n

RHS can be manipulated as follows:

i=1

P(A i ) (a) =

lim

k→∞

k

i=1

P(A i )

n k

=

i=1

n

(b)
=

i=1

n

=

i=1

P(A i ) + lim

k→∞

i=n+1

P(A i ) + lim

k→∞ 0

P(A i ).

P(A i )

where (a) follows from the deﬁnition of an inﬁnite series and (b) is a consequence of setting the events from A n+1 onwards to null sets.

5-1

5-2

Lecture 5: Properties of Probability Measures

Property 4:- For any A, B ∈ F ,

P(A B) = P(A) + P(B) P(A B).

In general, for a family of events {A i } n i=1 F,

P

i=1 A i =

n

n

i=1

P(A i ) P(A i A j ) + i<j<k P(A i A j A k ) +

i<j

(5.3)

n + (1) n+1 P i=1 A i . (5.4)

This property is proved using induction on n. The property can be proved in a much more simpler way using the concept of Indicator Random Variables, which will be discussed in the subsequent lectures. Proof of Eq (5.3):- The set A B can be written as A B = A (A c B). Since A and A c B are disjoint events, P(AB) = P(A)+P(A c B). Now, set B can be partitioned as, B = (AB)(A c B). Hence, P(B) = P(A B) + P(A c B). On substituting this result in the expression of P(A B), we will obtain the ﬁnal result that P(A B) = P(A) + P(B) P(A B).

Property 5:- If {A i , i 1} are events, then

P i=1 A i =

m→∞ P

lim

i=1 A i .

m

This result is known as continuity of probability measures.

Proof:- Deﬁne a new family of sets B 1 = A 1 , B 2 = A 2 \ A 1 ,

Then, the following claims are placed:

Claim 1:- B i B j = , i

Claim 2:-

Since {B i , i 1} is a disjoint sequence of events, and using the above claims, we get

n1

i=1

,

B n = A n \

A i ,

= j.

∞ ∞

i=1

A i =

i=1

B i .

Therefore,

P

i=1 A i = P

P

i=1 A i

=

(a)

=

(b)

=

(c)

=

i=1 B i =

i=1

P(B i )

lim

m→∞

m

i=1

m→∞ P

lim

m→∞ P

lim

i=1

P(B i )

P(B i ).

m

i=1

B i

i=1 A i .

m

(5.5)

Here, (a) follows from the deﬁnition of an inﬁnite series, (b) follows from Claim 1 in conjunction with Countable Additivity axiom of probability measure and (c) follows from the intermediate result required to prove Claim 2. Hence proved.

Lecture 5: Properties of Probability Measures

5-3

Property 6:- If {A i , i 1} is a sequence of increasing nested events i.e. A i A i+1 , i 1, then

P i=1 A i =

m P (A m ) .

lim

(5.6)

Property 7:- If {A i , i 1} is a sequence of decreasing nested events i.e. A i+1 A i i 1, then

P i=1 A i =

m P (A m ) .

lim

(5.7)

Properties 6 and 7 are said to be corollaries to Property 5.

Property 8:- Suppose {A i , i 1} are events, then

P

i=1 A i

i=1

P(A i ).

(5.8)

This result is known as the Union Bound. This is bound is trivial if i=1 P(A i ) 1 since the LHS of (5.8) is a probability of some event. This is a very widely used bound, and has several applications. For instance, the union bound is used in the probability of error analysis in Digital Communications for complicated modulation schemes.

Proof:- Deﬁne a new family of sets B 1 = A 1 , B 2 = A 2 \ A 1 ,

n1

,

B n = A n \

A i ,

i=1

Claim 1:- B i B j = , i

Claim 2:-

Since {B i , i 1} is a disjoint sequence of events, and using the above claims, we get

= j.

∞ ∞

i=1

A i =

i=1

B i .

P

i=1 A i = P

i=1 B i =

i=1

P(B i ).

Also, since B i A i i 1, P(B i ) P(A i ) i 1 (using Property 2). probabilities follow

n

i=1

P(B i )

n

i=1

P(A i ).

Eventually, in the limit, the following holds:

Therefore, the ﬁnite sum of

∞ ∞

i=1

P(B i )

i=1

P(A i ).

Finally we arrive at the result,

P

i=1 A i

i=1

P(A i ).

5.2

Exercises

5-4

Lecture 5: Properties of Probability Measures

b) Prove Properties 6 and 7, which are corollaries of Property 5.

2. A standard card deck (52 cards) is distributed to two persons: 26 cards to each person. All partitions are equally likely. Find the probability that the ﬁrst person receives all four aces.

3. Consider two events A and B such that P(A) > 1 δ and P(B) > 1 δ, for some very small δ > 0. Prove that P(A B) is close to 1.

4. [Grimmett] Given events A 1 , A 2 ,

A n , prove that,

P(1rn A r ) min

1kn

1rn

P(A r ) =k P(A r A k )

r:r

5. Consider a measurable space (Ω, F ) with Ω = [0, 1]. A measure P is deﬁned on the non-empty subsets of Ω (in F ), which are all of the form (a, b), (a, b], [a, b) and [a, b], as the length of the interval, i.e., P((a, b)) = P((a, b]) = P([a, b)) = P([a, b]) = b a.

a) Show that P is not just a measure, but its a probability measure.

b) Let A n = [ n+1 , 1] and B n = [0, n+1 ], for n 1. P(iN B i ).

1

1

Compute P(iN A i ),

c) Compute P(iN (B

c

i

A

c

i

)).

1

d) Let C m = [0, m ] such that P(C m ) = P(A n ). Express m in terms of n.

e) Evaluate P(iN (C i A i ))

and P(iN (C i A i )).

P(iN A i ), P(iN B i ) and

6. [Grimmett] You are given that at least one of the events A n , 1 n N , is certain to occur. However, certainly no more than two occur. If P(A n ) = p and P(A n A m ) = q, m = n, then show that p and q

2

N .

1

N

EE5110: Probability Foundations in Signal Processing

July-November 2015

Lecture 6: Discrete Probability Spaces

Lecturer: Dr. Krishna Jagannathan

Scribe: Ravi Kolla

6.1 Discrete Probability Spaces

In this lecture, we discuss discrete probability spaces. This corres ponds to the case when the sample space Ω is countable. This is the most conceptually straightforward case, since it is possible to assign probabilities to all subsets of Ω .

Deﬁnition 6.1 A probability space (Ω , F , P ) is said to be a discrete probability space if the following condi- tions hold:

 (a) The sample space Ω is ﬁnite or countably inﬁnite, (b) The σ -algebra is the set of all subsets of Ω , i.e., F = 2 Ω , and (c) The probability measure, P , is deﬁned for every subset of Ω . In particular, it can be deﬁned in terms of the probabilities P ({ ω } ) of the singletons corresponding to each of the elementary ou tcomes ω , and satisﬁes for every A ∈ F ,

and

P (A) = P ({ ω } ) ,

ω A

ω

P ({ ω } ) = 1 .

6.1.1 Examples of Discrete Probability Space

1. Let us consider a coin toss experiment with the probability of getting a head as p and the probability of getting a tail as (1 p ). Then, the sample space and the σ -algebra are

Ω = { H, T } ≡ { 0 , 1 },

F = 2 = { Φ , { H } , { T } , { }} .

respectively. The probability measure is

 P ({ H } ) ≡ P ({ 0 } ) = p, P ({ T } ) ≡ P ({ 1 } ) = 1 − p.

In this case, we say that P (. ) is a Bernoulli measure on { 0 , 1 } , 2 { 0, 1} .

2. Let Ω = N , F = 2 N . Then, we can deﬁne the probability of a singleton as

under the constraint that

P ({ k } ) = a k 0 , k N

k N

P ({ k } ) = 1 .

6-1

6-2

Lecture 6: Discrete Probability Spaces

For example, a k = 1 k , k N is a valid measure, since

2

k N

1

2 k = 1 .

As another example, consider a k = (1 p ) k 1 p, 0 < p < 1 , k N . This is known as a geometric measure with parameter p . It is a valid probability measure since

k N

(1 p ) k 1 p = 1 .

3. Let Ω = N ∪ { 0 } , F = 2 . Let us deﬁne

P ({ k } ) =

e λ λ k

k !

,

λ > 0 .

This probability measure is called a Poisson measure with parameter λ on , 2 . This is a valid probability measure, since

k

=0

P ({ k } ) =

k

=0

e λ λ k

k

!

= e λ

k

=0

λ

k

k

!

e λ

= 1 .

4. Let Ω = { 0 , 1 , 2 , · · · , N } , N N F = 2 . Let

us deﬁne

P ({ k } ) = N p k (1 p ) N k , 0 < p < 1 .

k

This probability measure is called a Binomial measure with parameters (N, p ) on , 2 . This can be veriﬁed to be a valid probability measure as follows:

k N

k

p k (1 p ) N k = (p + 1 p ) N = 1

Note that in all the examples above, we have not explicitly speciﬁed an expression for P (A) for every A Ω. Since the sample space is countable, the probability of any subset of the sample space can be obtained as the sum of probabilities of the corresponding elementary outcomes . In other words, for discrete probability spaces, it suﬃces to specify the probabilities of singletons corresp onding to each of the elementary outcomes.

6.2 Exercises

1. An urn contains a number of white balls and b number of black balls. Balls are drawn randomly from the urn without replacement. Find the probability that a white ball is draw n at the k th draw.

2. An urn contains white and black balls. When two balls are drawn witho ut replacement, suppose the probability that both the balls are white is (a) Find the smallest number of balls in the urn. (b) How small can the total number of balls be if the number of black balls is even?

1

3 .

Lecture 6: Discrete Probability Spaces

6-3

3. Consider the sample space Ω = N . Find the values of the constant C for which the following are

probability measures:

(a)

(b)

(c)

(d)

f (x)

f (x)

f (x)

= C 2 x

= C 2 x

x

= Cx 2

f (x) = C 2 x

x

!

4.

set of odd numbers? Prime numbers?

Recall the Poisson measure on (Ω , 2 ), where Ω = N ∪ { 0 } . What is the probability assigned to the

EE5110: Probability Foundations for Electrical Engineers

July-November 2015

Lecture 7: Borel Sets and Lebesgue Measure

Lecturer: Dr. Krishna Jagannathan

Scribes: Ravi Kolla, Aseem Sharma, Vishakh Hegde

In this lecture, we discuss the case where the sample space is uncountable. This case is more involved than the case of a countable sample space, mainly because it is often not possible to assign probabilities to all subsets of Ω. Instead, we are forced to work with a smaller σ -algebra. We consider assigning a “uniform probability measure” on the unit interval.

7.1 Uncountable sample spaces

Consider the experiment of picking a real number at random from Ω = [0, 1], such that every number is “equally likely” to be picked. It is quite apparent that a simple strategy of assigning probabilities to singleton subsets of the sample space gets into diﬃculties quite quickly. Indeed,

 (i) If we assign some positive probability to each elementary outcome, then the probability of an event with inﬁnitely many elements, such as A = {1, 1 2 , 1 3 , · · · }, would become unbounded. (ii) If we assign zero probability to each elementary outcome, this alone would not be suﬃcient to determine

the probability of a uncountable subset of Ω, such as

2 , not additive over uncountable disjoint unions (of singletons in this case).

1

2

3 . This is because probability measures are

Thus, we need a diﬀerent approach to assign probabilities when the sample space is uncountable, such as Ω = [0, 1]. In particular, we need to assign probabilities directly to speciﬁc subsets of Ω. Intuitively, we would like our ‘uniform measure’ µ on [0, 1] to possess the following two properties.

 (i) µ ((a, b)) = µ ((a, b]) = µ ([a, b)) = µ ([a, b]) (ii) Translational Invariance. That is, if A ∈ [0, 1], then for any x ∈ Ω, µ (A ⊕ x) = µ (A) where, the set A ⊕ x is deﬁned as

A x = {a + x|a A, a + x 1} ∪ {a + x 1|a A, a + x > 1}

However, the following impossibility result asserts that there is no way to consistently deﬁne a uniform measure on all subsets of [0, 1].

Theorem 7.1 (Impossibility Result) There does not exist a deﬁnition of a measure µ (A) for all subsets of [0, 1] satisfying (i) and (ii).

Proof: Refer proposition 1.2.6 in [1].

Therefore, we must compromise, and consider a smaller σ-algebra that contains certain “nice” subsets of the sample space [0, 1]. These “nice” subsets are the intervals, and the resulting σ-algebra is called the Borel σ-algebra. Before deﬁning Borel sets, we introduce the concept of generating σ-algebras from a given collection of subsets.