Probability Foundations for Electrical Engineers

Prof. Krishna Jagannathan

Department of Electrical Engineering
Indian Institute of Technology, Madras

Lecture - 11
The Infinite Coin Toss Model

Welcome back. In the previous lecture, we defined the lebesgue measure on 0 1 the
uniform probability measure on 0 1 which is an uncountable sample space. We also
defined on the real line the lebesgue measure which formalizes the notion of length. So,
these 2 measures are, so the lebesgue measure on 0 1 is a probability measure and the
lebesgue measure on r is not a probability measure it just it is an infinite measure. And
both these are defined on the corresponding borel sigma algebras on 0 1 or the real line
which are generated by open intervals. Today, we will consider another uncountable
sample space this model is called the infinite coin toss model.

(Refer Slide Time: 01:07)

In this case, the sample space is the uncountable set 0 1 power infinity. So, this
corresponds to, so each elementary outcome corresponds to an infinite coin toss alright.
So, each little omega, so if you take a little omega in omega it will be of the form omega
1 omega 2 dot. So, these are bits. So, each elementary outcome is the infinite binary state
and we know that by cantors argument we know that this is an uncountable sample
space. So, essentially what I am doing is calling head is 0 and tail is 1. So, this is an
uncountable sample space and we want to do we want to assign probability measure on
this uncountable sample space. And you know that, because the sample space
uncountable if you consider all possible subsets of this omega 2 power omega is too
large sigma algebra to assign meaningful probability measures on.

So, we have to again settle for some smaller sigma algebra only consisting of those
subsets of omega which are in some sense interest to us. And then we will try and then
what we will do is once you define appropriate sigma algebra on omega we will assign a
probability measure what is what we like be a uniform probability measure on this
omega? Ideally what you want is that all no particular string being preferred over any
other particular string that is the kind of you wants intuitively we want a uniform
measure on this space. So, that is what we are building towards. So, this is also a very
canonical probability uncountable probability space. And in some sense you can prove
that what I am going to say is has very close correspondence to the borel measure on
borel sigma algebra lebesgue measure on 0 1, if write it as binary string you will write.

So, it is fairly close to what we have done, but because it is in the language of coin tosses
the flavor is a little different. So, you have each outcome is a infinite binary string
alright. So, in this now, the question arises we have to put a sigma algebra on this. And
see say if you put 2 power omegas it is too large it is too complicated. So, what I want to
do is first start of with some collection of subsets which allows me to describe the
outcome of let say the first n tosses alright start with finite set of tosses. So, let me say
that F n, so this is going to be a collection of subsets of omega whose occurrence can be
decided by looking at the first n tosses.

So I have not really called it an event not I only called it a subset omega I have not it
build by sigma algebra on this. So, I have not deliberately avoided calling it the
collection of events, because by event I mean something very precise I mean F
measurable sets. So, this F n consists of collections of subsets of omega itself is all is
binary strings infinite binary strings. And all possible you take a certain collection of
subsets of omega whose occurrence or non-occurrence can be decided based only on the
first n coin tosses of any outcome. So, by this I mean so if I want to ask let us say an
example I mean see. So, I want let say I am interested in whether or not the
(Refer Slide Time: 06:01)

So, for example, I am going to look at least 2 heads in the first 10 tosses. This is
something I am interested in let us say I want to know if there are at least 2 heads in the
first. So, I have an elementary outcome which has any infinite bit strings I am looking at
the first 10 bits and asking if there are at least 2 heads in it. So, this is a, so if you called
this event some E or something then E is an element of F 10, because I can decide
whether E occurs or not by looking at the first 10 tosses. So, this is what I am looking
this is what F n contains. It contains all those subsets of omega whose occurrence or non-
occurrence can be decided for certain by looking at only the first 10 tosses.

So, this is a little bit of an English definition I will give a little more of a formal
definition now. So, one thing that is immediately is that something that can be decided in
10 tosses can also be decided in 11 tosses. So, if some E is an element of F n it is
necessarily an element of F n plus 1 F n plus 2 and so on. So, if you can call whether
element occurred or not in 10 tosses you can do, so in 20 tosses no big deal. So, these are
all nested, so I will write that down. So, you can so this is easy to show claim F n is
contained in F m for all n less than or equal to m this is very easy. So, this is, so I have
defined F n in certain colloquial kind of way.

So, the correct way to define F n mathematically is to say that n E subset let say E
belongs to F n if the n first n bits follow a certain conditions. In this case the condition
was 2 heads at least 2 heads out of 10 tosses. So, I am essentially looking at the first 10
bits and I mandating that to be a certain particular subset of the 2 power n possibilities in
the first 10 tosses. So, more formally I think you will understand what I am going to say.
Now, so if you want to like this more formally we will say that A belongs to F n if there
exists some A. I will write a superscript n that is a subset of 01 power n such that a
equals a set of all omega for which omega 1 omega 2 dot omega n if you look at the first
n bits here this is in this a n. So, is this clear? So, I am say this a index by I am
superscript by n. So, this is a subset of 01 power n.

So, 01 power n gives you all possibilities for the first n bits, but I am mandating that the
first n bits of my set of interest should be a fixed subset of 01 power n. So, it should have
a certain property for example, in this particular example, the first the A n consists of
those n bit patterns in which the at least there are 2 heads in the first tens. So, you should
look at 10 n equal to 10. And say that you look at a superscript n will include all those
patterns in which there are at least 2 zeros in the first 10. So, essentially this captures the
property of the first n bits that I want to decide whether it occurred or not. Actually this
is all this is very simple step I am couching it in a certain notation to be formal this
suppose this is clear.

So, as I say, so this is the definition of, so I am saying that A belongs to F n if there
exists some the first n bits have a certain property we can look at this a n as a subset of
01 power n that satisfies the property I want for the first 10 bits. So, remember this is an
element of omega see A is not element of omega a is subset of omega for which the coin
tosses satisfies the certain property for the first 10 bits. So for example, so if let me take
another example. So, if I want to decide whether my third toss is a head I want to decide
if my third toss is a head. So, it is an element of F 3 or F 4 F 5 everything. So, let say F 3,
so we say that F 3, so if an element of F 3. So, we will say a 3 will consist of the first 3
bits in which the third bit alone is a head. The other 2 can be anything there will be, so
how many possibilities? 4 possibilities the third bit fix as head the first you can be
anything.

So, this a superscript 3 will consist of those 4 strings of triples 3 bits, so is this clear. So,
it is very simple step I actually coughed it in certain notation, but I hope this will be
clear. So, what do we really want? So, I want to eventually say I want to eventually
assign some uniform probability measure. So, ideally I would like to say that is particular
n bit pattern should have probability 1 by 2 power n that is what I want to say. So, I want
to say that each bit being 0 or 1 should be probability half and many sting of a particular
string of n bit should have probability 1 by 2 power n that is what I want to say. But on
the other hand i have to put a proper sigma algebra on it and do it formally that is what
we are building towards.

(Refer Slide Time: 13:54).

Now, what you can show another claim F n is sigma algebra you can show this you try
this as a home work. So, F n the set of all events whose occurrence or non-occurrence
can be decided in at most 10 tosses is sigma algebra you can prove it try this as a home
work. Now, given that F n is already sigma algebra 1 might be 10 it to say that may
measurable space should be omega comma F n. So, if I take a big n right let us say I take
n equal to some million. So, I take a very big n and I say omega comma F n is going to
be my sigma algebra it is a valid measurable space. So, I can put some probability
measure on it to say that any n bit pattern has probability 1 by 2 power n the problem we
doing that. So, if you take let us say omega comma F n as a measurable space, so
elements subsets.

So, elements of F m for m greater than or equal to n cannot be assign probability

measures. So, what happens? So, if you take n equal to million if want to answer the
question what is whether my million plus 1-th tosses head or not I will immediately
cannot answer it. So, no matter what n you take if take F n as your sigma algebra I will
immediately ask you yes in the board how about my n plus one th bit oh my n plus one th
bit head or you know some question about something beyond n. So, then you will not be
able to answer it, because I am only taking events in well elements of F n as my
measurable sets. So, this is a little bit of a problem this is a sigma algebra there is no
problem. But it is not a nice in a sigma algebra it is not a rich enough sigma algebra to
help us characterize evens beyond n n-th coin toss occurrence beyond the n-th coin
tosses.

So, we have to do something else. So, we have to look beyond these finite horizons. So,
we do the following we define, so let us define F 0 equal to union i equal 1 through
infinity F i. You know I mean maybe I should actually write as I keep saying maybe
actually just write that is better notation is a keeps in i belongs to n F i the same thing.
Now you are what i doing? You are unioning all the all these sigma algebras, so F n
consists of those subsets. So, F n is the collection of those subsets of omega whose
occurrence can be decided in n coin tosses. So, when will this, so this is a this is another
collection of this is also a collection of subsets of omega what kinds of elements will it
have? Will contain all those subsets of omega whose occurrences can be whose
occurrence or non-occurrence can be decided in no see I have a union over all i

Student: Countable number.

Countable number; see it consist of all those, so this is something you have to understand
F naught consist of although subset of omega whose occurrence can be decided in some
finite number of tosses. So, why do I say that? So, it looks it though it should you know I
deliberately avoided putting infinity in order to not confuse i equal to 1 to infinity i
raised it and put i belongs to n. So, that you do not think of it as F infinity or some such
thing there is no F infinity F n is those subsets which can be decided in n coin tosses. So,
if, so let think of this way if a subset is contained in F naught it must be contained in 1 of
the F i is for some mile. So, it must be decided in some finite i in a number of tosses. So,
it is not something that can be decided in some infinite tosses no. So, F naught consist
you should properly write this down F naught consist of all subsets of omega which can
be decided in some finite number of tosses. So, this is the important thing the some finite
number of tosses, yes.

Student: We can take (refer time: 20:56) intersection like this.

Yeah.
Student: But that time.

It is a same interpretation it is the. So, if you intersect over collection index it is the set of
all elements contained in all the sets or if it is a union it should be contained in at least
one of the intersects.

Student: like there if it was finite still there will be a even smaller element which is now
contained in ((Refer Time: 21:23)).

So, I think you are confusing things that are not be related, so it let us this clarify this
first. So, I want to make sure that this is clear in everybodys mind. So, if F naught is
some collection of subsets of omega we want to know what kind of subsets are in this F
naught. I am saying that if some subset is in F naught let say if some subset a is in F
naught then it must exist in at least 1 of the A is, but, you know that see this F is also
nested increasing.

So, if it is it must been one of the F is lets say if it is let us say it is an F j there it will be
F j F j plus 1 F j plus 2 and so on that is because of an nested structure. So, it consist of
all those subsets of omega whose occurrence can be decided in some finite j number of
tosses, but that j can be million or billon does not matter that j can that j can be as large
as you want. But it should be able to all that event in some finite number of tosses. It can
be very large as an example of something if I ask you if the millionth bit to the 2
millionth bit are all heads that is a subset in F naught, because I can call it looking at the
first 2 million bits. Can somebody think of an example of something that is not contain in
F naught? Actually very easy all heads this whether my string is all heads it is all zeros
cannot be called. We will looking at the F naught, because F naught consist of those
events it can be called in some finite number of tosses.

But whether it is all heads I have to I can never stop looking at it beyond the point I
mean I have to look at everything. So, an example of something that is not contain in F
naught is whether my sequences all the all head sequence or if I want answer if all my
odd tosses are tails first toss, third toss, fifth toss is there all tails that is what I want to
know that is also now contain in F naught. Now, I want to know what is, so this is F
naught I hope everybody understands F naught. So, now, I want to know what kind of an
object this F naught is see this F is are all sigma algebras. But we had to move away
from it, because it only gives a finite characterization of the first n bits F n gives only a
characterization of the n bits. So, we have move away from it now, I come up with some
object F naught which consist of those subsets which can be decided in a finite number
of tosses any finite number of tosses.

Now, what kind of object is F naught? This is the question, see if this where in
intersection we know that intersection of sigma algebras is always sigma algebra, but
union of sigma algebra need not be sigma algebra it is not true. So in fact, it turns out
that F naught is not a sigma algebra, but it is a algebra we can prove that it is an algebra
its closed under finite unions. So, by which what I mean is that if I have 1 subset whose
occurrence can be called in let us say i tosses I have another subset which whose
occurrence can be called in j tosses with let say j bigger than i. Then that union of the 2
can be called in the bigger of the 2 j tosses again. So, it is can say that it is closed under
finite unions, but you can also show that it is not closed under countable union countable
infinite unions. So, here I go I will put that down.

(Refer Slide Time: 26:08)

So, we can show that F naught is an algebra and then we can show that F naught is not a
sigma algebra. So, this I will leave as a home work this is just chasing up all the
definition. So, it chasing up the definition of an algebra. Can somebody tell why F
naught is not a sigma algebra? I just give an example actually odd let say odd tosses. So,
let me say for an example that let, so I am just going to prove 2 I am going to prove that
it is not a sigma algebra proof of 2. In order to prove that it is not a sigma algebra I can
show an I can produce an example where it is not closed under countable unions it is
closed in the finite unions we will show. Let A i be the event that the i-th toss is a head i-
th toss is a 0 then and let E equal to, so be the event is hold not is a event you know why
it should not an event? I have not defined the sigma algebra I should say these are subset
of omega such that.

So, for example, a 2 will consist of all those coin tosses all those infinite bits strings in
which the second string is a head. Similarly, a 10 will consist of all those strings for
which the tenth is the head. So, I want to know I want to consider the following E equal
to subset of omega such that it is even all even index tosses are heads. So, A i, so you
will know that A i belongs to what F i A i is clearly element of F i, because I can call it
an i tosses F i implies A i is a element of F naught for all i. So, A i is element of F i
therefore, A i must be element of F naught, because if your a is in even 1 of the F i is
then it will be in F naught. So, this much after all A i is whether the i-th toss is head it
can be decided in finitely many coin tosses, but I am claiming that my e. So, i can
defined by E as a countable union of the A i is not it? Do will intersection probably is not
it? True or no?

(Refer Slide Time: 30:15)

So, if I write E I can write E note that E equal to intersection a i; i is equal to 246 dot. I
am intersecting all the even index A is. So, A i is the corresponds to the i-th bit being
heads I want all even index tosses to be heads. So, which means E must be the
intersection of all is A is where the index turns over only the even numbers if you want
all yours tosses to be heads you will say i equal to 1 through i belongs to n for example, I
am just mandating that all my even tosses are heads. So, this is, so these are all elements
of F naught, but E is not an element of F naught. So, A i belongs to F naught E is not an
element of F naught implies F naught is not a sigma algebra. So, I am, so I what I argued
here is for each i A i is an F naught, because whether the i-th toss is a head i can decide
infinite number of tosses which is what F naught is. But E says all my even index tosses
must be heads and that is clearly not an element of F naught, because F naught is only
consist of those subsets which are decidable in finitely many tosses.

And I cannot decide in E in finitely may tosses, but E is a countable intersection of

elements of F naught. So, really it is the proof, so here is get another example of an
algebra which is not a sigma algebra. So, just think about it little be you are I mean I
guess it is a little bit easier on the interval and, so on. So, this is a little I am doing this
specifically to make you think of about this kind of sample spaces as well. So, my F
naught is an algebra, but it is not a sigma algebra. But this something nice about an
algebra as well yesterday we stated a theorem that says if I tell you the measure on an
algebra some pseudo measure on an algebra then I can extended under certain
circumstances. So, if I tell you the measure I would like to put on those subsets which
can be decided infinite may tosses.

Then may I can invoke caratheodory and then transfer the measure on to exchange the
measure on to sigma algebra what is the sigma algebra now? Sigma of F naught sigma of
we do not know what sigma algebra additives some sigma algebra. So, we can what we
will try on do is put a p naught some pseudo measure on F naught and then used
caratheodory theorem. and extend the measure on to the sigma algebra generated by F
naught remember F naught is not a sigma algebra. So, you have make it sigma algebra by
taking countable unions and so on. So, here is the program alright I do the following F
naught is not a sigma algebra, so this is done lemma is done. So, I want to define p
naught from F naught to 01 satisfying p naught of omega equal to 1 and countable
additivity on F naught that is what caratheodory wants to do.

But, what we should initially do is we have to set our p naught to be right on the algebra
on the first place just like we did not on the intervals. We put p naught as the length of
the interval and then decided we will extended. Now, we have to say that if you give me
an element of F naught which can be decided say an n tosses. I will basically make all
these n 2 power n possibilities equal that is what I am going to do to correspond to a
uniform measure. So, we want to define p naught to correspond to fair coin tosses. So,
we do as follows let us say I give you a n F naught, so if A is an F naught. So, I want to
decide what probability of some a n in F naught is? So, if A is in F naught then it must be
in yeah it will be in one of the F is. Then A is in F i for some i in natural numbers see
you know that if A is in F i it must also be in F i plus 1 and so on.

So, if you want you can put that i for which you can put the smallest i if you want, but
we will see the that does not matter you can take any I, because F is are nested
increasing. So, if you find A i you can also find other i. So, never mind, so A belongsto F
i some i. So, if A is in F i then what can you say about the first i bits it must be a certain
form then a must be written of a must have a representation A remember that guy a super
script i must belong to wait a second omega. So, I must really write omega 1 omega 2
omega i is in a super script i is not it this is my characterization of F i. So, I am saying
that the first i bit should have a certain pattern it could be any one of the subsets of 2
power n possibilities this A i in some subset of 01 power n 01 power i n.

(Refer Slide Time: 38:17).

Next define; I am going to now define a measure; next define p naught of A is equal to
cardinality of a super script i over 2 power i you see what I am doing. So, this A i will
corresponds to some pattern among the first i bits in the example I gave I was 10. And I
was looking at all lows possibilities where they were at least 2 heads remember. So, in
that case my I will be 10. So, I want to assign the probability of well p naught which is
the pseudo measure as the cardinality of that set. So, what is the cardinality of that set?
Now, of the first 10 tosses you want at least 2 heads what I said. So, will have 10 chose 2
plus 10 chose 3 plus that will be the cardinality of that set divide by 2 power 10 in this
case that is what I am doing. So, I am again couching it some notation, but it is actually
very simple.

For example, to take a very simple example I am looking at the set of all tosses where the
third tosses a head then I will have it will be an element of F 3. So, I will have 1 by 2
power 3 in the denominator will be 2 power 3 and here set of a is will be the 4
possibilities you are fixing the third bit, but first 2 can be anything. So, there are 4
possibilities 4 over 8 half. So, I am basically saying that the probability of some bit being
whatever heads or tail is half. And I also say that if you give me any specific n bit
patterns. So, I may be I tell you that my first 10 tosses should be a very specific string
then there will only be once one such thing here and here will be 1 by 2 power 10. So,
this is what you would normally call a fair coin tosses except, now there is a infinitely
many coin tosses.

So, this is perfectly under I mean intuitive it just couch in some notation. Now, there is
one little issue here, so remember if some A is in F i it is also an element of F i plus 1
and, so 1. So, I may for example, chose some I which is not the smallest I then you may
ask me is this is this measure going to change? It would not change come to think of it let
say I take the third coin toss being head let say I call it F 4 now it is an it is an F 3, but it
is also in F 4 clearly. Now I am looking at 4 strings in which the third bit alone is heads.
So, there 8 possibilities now and the denominator are 60. So, it is again the same as you
can verify that in fact, this is this definition of probability of p naught of a this would
have measure is independent of whether E actually manage to chose the smallest i or
naught.

So, you let us say you start define saying this is my smallest, but it does not really matter
you can even chose some bigger i. This is invariant under choosing bigger i for reason.
So, this is perfectly well defined uniquely defined, so this is what I want. Now I have to
verify p naught of omega what happens now p naught of omega? I have to verify it is 1
this is a true. So, I will look at p naught of omega then what will be my first i bits will be
all 2 power possibilities. So, it will be 2 power i over 2 power i. So, it is my omega it
consist of all possible binary strings infinite binary strings therefore, my first i bits can be
all possible 2 power i strings. So, p naught of omega is 1. Now, there is see in order to
invoke caratheodory we need just one more thing we have an algebra F naught you know
that p naught of omega is one you just need countable additivity over is F naught, now
that is non-trivial, but it is true.

So, now, you can invoke caratheodory and say this p naught can be extent to a unique
measure on sigma off naught. So, you can verify p naught of omega equal to 1 this is
easy this is 1 p naught is countably additive on F naught this is non-trivial it is non-
trivial, but true therefore, caratheodory holds. So, by caratheodory p naught can be
uniquely extended to a measure on what? Omega comma sigma of F naught extended to
probability measure p on omega comma sigma of F naught. And that is my probability
measure that is what there is do it and that probability measure will agree perfectly with
p naught on this F naught sets. If you ask me under this measure p what is the
probability, but my thirteenth toss is head would be half, because p naught says it half.

So, p will agree with it except now, this p is also defined on sets like E see remember E
was never on F naught E is the event that there all event tosses are head. Let us say this E
was not in F naught. So, my p naught will not assign any value to it, but my p will be
defined on y is now, y is p defined on E it is a countable intersection of F naught sets. So,
this must be E is not in the algebra, but it is in the sigma algebra. So, p naught is not
define for e, but p is defined for e. So, now what do you think is p of E is? P of E is half
p of E will be 0 try showing the p of E is 0. So, p of E is a p of a countable intersection
you have to use continuity of probabilities. So, do I have a running out of time, so you
can write. So, in this case I just put a little box and give you a hint for this p of e equal to
p of intersection i is equal to 246 dot A i.
(Refer Slide Time: 47:01)

So, this is like you can write it as i equal to i belongs to 2 n or whatever. So, you can
essentially what you can do? May be you can put i equal to 2 i A 2 i may be and put i
equal to 1 to infinity it is same thing. Now, what do you do? So, this by continuity of
probability is limit n tend into infinity probability intersection i equals 1 through n a 2 i
this is by continuity of probability. There is no nested decreasing business here, but this
is true by continuity of probabilities. Now what is that? 1 by 2 power n limit of that is, so
this is this guy is looking at there are certain n bits have to be heads. So, probability of
that is you know is 1 by 2 power n. So, this is 1 by 2 power n. So, this will be 0 is a
probability of all even bits being heads is 0, so that is an example. So, the bit p naught is
not define for E. So, p is defined for E and is equal to 0. I will stop here.