Académique Documents
Professionnel Documents
Culture Documents
Syed Ali Khayam School of Electrical Engineering & Computer Science National University of Sciences & Technology (NUST) Pakistan
Course Information
Lecture Timings:
Office Hours
Wednesdays: 4:00pm-5:30pm Fridays: 4:00pm-5:30pm Office is located on top floor, last room on your left in the north wing
Textbook
Course Outline
Syllabus
Introduction to Probability Theory Functions of Random Variables Limits and Inequalities Stochastic Processes Prediction and Estimation Markov Chains and Processes (time permitting) Assorted Topics (time permitting)
Grading
Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%
Course Outline
Grading
Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%
We will also have a voluntary user study as well Anyone volunteering and seeing it through will get 10 extra credit points
Policies
Quizzes are announced and will take place at the start of Friday classes Exams will be closed book, but you will be allowed to bring an A4-sized cheat sheet to the exam
I would like to thank Dr. Garcia for providing online lecture notes on the textbooks website Throughout this course, I will be borrowing examples and explanations from the Stochastic Systems Course taught by Professor Hayder Radha at Michigan State University
Stochastic theory is an extension of probability theory This course on Stochastic theory will teach mathematical tools that are commonly-used in a variety of engineering, computer science and IT disciplines We will focus solely on performance modeling of phenomena observed in communications engineering Applications and examples will be provided as required
Random Experiments and Random Variables Axioms of Probability Mutual Exclusivity Conditional Probability Independence Law of Total Probability Bayes Theorem
Definition of Probability
Probability: [m-w.org]
1 : the quality or state of being probable 2 : something (as an event or circumstance) that is probable 3 a (1) : the ratio of the number of outcomes in an exhaustive set of equally likely outcomes that produce a given event to the total number of possible outcomes (2) : the chance that a given event will occur b : a branch of mathematics concerned with the study of probabilities 4 : a logical relation between statements such that evidence confirming one confirms the other to some degree
10
Definition of Probability
Do you know which famous person was so opposed to probability theory that he said: God does not play dice with the universe.?
11
Definition of Probability
And do you know which famous person said: God does play dice with the universe. All the evidence points to him being an inveterate gambler, who throws the dice on every possible occasion.?
12
A procedure An outcome
Outcome (e.g., the value observed [head, tail] after flipping the coin)
13
outcome
event
sample space
14
s1
s5 s6
s2
s4 s3
15
Experiment: Roll a fair die once and record the number of dots on the top face
S = {1, 2, 3, 4, 5, 6} A = the outcome is even = {2, 4, 6} B = the outcome is greater than 4 = {5, 6}
16
Axioms of Probability
Probability of any event A is non-negative: Pr{A} 0 The probability that an outcome belongs to the sample space is 1: Pr{S} = 1 The probability of the union of mutually exclusive events is equal to the sum of their probabilities: If A1 A2=, => Pr{A1 U A2} = Pr{A1} + Pr{A2}
17
Mutual Exclusivity
Pr
i 1
Ai
i 1
Pr Ai
A1
s1
s5
s6
A2
s2
s4 s3
S
Copyright Syed Ali Khayam 2008 18
Mutual Exclusivity
In general, we have:
Pr{A1 U A2} = Pr{A1} + Pr{A2} Pr{A1 A2}
s1
s5
s6
s2
s4
s3
S
Copyright Syed Ali Khayam 2008 19
S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6), (2,1), (2,2), (2,3), (2,4), (2,5), (2,6), (3,1), (3,2), (3,3), (3,4), (3,5), (3,6), (4,1), (4,2), (4,3), (4,4), (4,5), (4,6), (5,1), (5,2), (5,3), (5,4), (5,5), (5,6), (6,1), (6,2), (6,3), (6,4), (6,5), (6,6)
20
21
22
S={
(1,1), (1,2), (1,3), (1,4), (1,5), (1,6), (2,1), (2,2), (2,3), (2,4), (2,5), (2,6), (3,1), (3,2), (3,3), (3,4), (3,5), (3,6), (4,1), (4,2), (4,3), (4,4), (4,5), (4,6), (5,1), (5,2), (5,3), (5,4), (5,5), (5,6), (6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
A1
23
24
A2
Pr A1 Pr A2 Pr A1 1 1 2
Pr A1 Pr A2 1 Pr A1 Pr A2
Pr A2
25
Conditional Probability
Given that event B has already occurred, what is the probability that event A will occur? Given that event B has already occurred, reduces the sample space of A
s1
s1
s5
s6
s2
s4
s3
s4
s3
26
Conditional Probability
Given that event B has already occurred, we define a new conditional sample space that only contains Bs outcomes The new event space for A is the intersection of A and B: EA|B = AB
s1
s5
Event B has already occurred
s1 s6
s5
s6
s2
s4 s3
s2
s4
s3
Conditional Probability
Consider that the example below corresponds to an experiment where we throw a fair dice and record the number of dots on its face For this experiment, what is Pr {A|B} in the example below? Pr{A|B} = Pr{s6|B}= 1/3 We need to normalize all probabilities in a conditional sample space with Pr{B}
s1
s1
s6
s5
s6 s4
s2
s4 s3
S
Copyright Syed Ali Khayam 2008
s3
28
Conditional Probability
We need to normalize all probabilities in a conditional sample space with Pr{B} Pr{s1|B} = Pr{s1}/Pr{B} = (1/6)/(1/2) = 1/3 Pr{s5|B} = Pr{s5}/Pr{B} = (1/6)/(1/2) = 1/3 Pr{s6|B} = Pr{s6}/Pr{B} = (1/6)/(1/2) = 1/3
s1
s1 s6
s5
s6 s4
s2
s4 s3
S
Copyright Syed Ali Khayam 2008
s3
Conditional Probability
s1
s1 s6
s5
s6 s4
s2
s4 s3
S
Copyright Syed Ali Khayam 2008
s3
30
Independence
Two events are independent if they do not provide any information about each other
Pr A B
Pr A
In other words, the fact that B has already happened does not affect the probability of As outcomes
31
Independence
Note that
Pr A B
Pr A only when Pr A
Pr A Pr B
Pr A B
Pr A
Pr B A
Pr B
32
Independence
Pr
i 1
Ai
i 1
Pr Ai
Pr Ak Ai1, Ai 2 ,
, Aip
Pr Ak ,
Aij
Ak
33
Independence: Example
s2 s4 s6
s5
s1
s3
34
Independence: Example
Are events A and C independent? Yes: Pr{A C} = Pr{s5} = 1/6 Pr{A}Pr{C} = (3/6)x(2/6) = 1/6
s2 s4 s6
s5 s1 s3
S
Copyright Syed Ali Khayam 2008 35
Independence: Example
s2 s4 s6
s5 s1
s3
36
Independence: Example
Are events A and B independent? NO: Pr{A B} = Pr{s5} = 1/6 Pr{A}Pr{B} = (3/6)x(3/6) = 1/4
s2 s4 s6
s5 s1 s3
S
Copyright Syed Ali Khayam 2008 37
Independence: Example
s1
s5
s6
s2
s4 s3
38
Independence: Example
Are events A and B independent? NO: Pr{A B} = Pr{} = 0 Pr{A}Pr{B} = (2/6)x(3/6) = 1/6 Recall that A and B are mutually exclusive
s1
s5
s6
s2
s4 s3
S
Copyright Syed Ali Khayam 2008 39
outcome
event
sample space
2.
Pr A1
3.
A2
Pr A1
Pr A2
In general, we have:
Pr A1
A2
Pr A1
Pr A2
Pr A1
A2
40
5.
Pr A B
Pr A
6.
Pr A
Pr A Pr B
41
T1
Pr{R1|T1}
R1
42
Pr{R0|T0} T0 R0
T1
Pr{R1|T1}
R1
43
T1
Pr{R1|T1}
R1
44
T1
Pr{R1|T1}
R1
45
T0
R0
T1
Pr{R1|T1}
R1
46
T1
Pr{R1|T1}
R1
47
T1
Pr{R1|T1}
R1
48
4. Whenever you see two events which have an AND relationship (i.e., A B), check if they are independent. If so, set Pr{A B} = Pr{A}Pr{B} Example 2: On a binary channel, find the probability that a 0 is transmitted and a 1 is received? A: at time n+1, a 1 is received when a 0 is transmitted AND B: at time n, a 0 is received when a 1 is transmitted Probability of above event is: Pr{A B} Are A and B independent? YES! Pr{A|B}=Pr{R1|T0}=Pr{A} => Pr{A B} = Pr{A}Pr{B} = Pr{R1|T0} Pr{R0|T1}
Pr{R0|T0} T0 R0
T1
Pr{R1|T1}
R1
49
S = B 1 U B2 U U BN Bi Bj = , i j
B2 B3
s1 B4 s2
s5 s6
B1
s4
s3
50
Total Probability
B2
s1
B1
s5
A
B3
A
B4 s2 s4
s6
s3
Copyright Syed Ali Khayam 2008 51
Total Probability
Thus event A can be expressed as the union of mutually exclusive events: A = (A B1) U (A B2) U U (A BN) => Pr{A} = Pr{A B1} + Pr{A B2} + + Pr{A BN}
B2
s1
B1
s5
A
B3
A
B4 s2 s4
s6
s3
Copyright Syed Ali Khayam 2008 52
Total Probability
If B1, B2,, BN form a partition then for any event A: Pr{A} = Pr{A B1} + Pr{A B2} + + Pr{A BN}
B2
s1
B1
s5
A
B3
A
B4 s2 s4
s6
s3
Copyright Syed Ali Khayam 2008 53
Total Probability
Using the definition of conditional probability: Pr{A| Bi} = Pr{A Bi} / Pr{Bi} => Pr{A Bi} = Pr{A| Bi} Pr{Bi}
B2
s1
B1
s5
A
B3
A
B4 s2 s4
s6
s3
Copyright Syed Ali Khayam 2008 54
B2 s1 B1 s5
A
B3
A
B4 s2 s4
s6
s3
55
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability
B2
s1
B1
s5
A
B3
A
B4 s2 s4
s6
s3
Copyright Syed Ali Khayam 2008 56
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability Pr{Bi|A} = Pr{A Bi} / Pr{A} Since Pr{A Bi} = Pr{A|Bi} Pr{Bi}, we obtain Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
B2 s1 B1 s5
A
B3
A
B4 s2 s4
s6
s3
57
Bayes Theorem
Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A} From the Law of Total Probability, we have:
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + + Pr{A|BN} Pr{BN}
Pr Bi A Pr A Bi Pr Bi
N
Pr A B j Pr B j
j 1
Bayes Rule
B2 s1 B1 s5
A
B3
A
B4
s2
s4
s6
s3
Copyright Syed Ali Khayam 2008 58
Bayes Theorem
Experiment is performed; Event A is observed; now what is the probability that Bi has occurred
B2 s1 B1 s5
A
B3
A
B4
s2
s4
s6
s3
Copyright Syed Ali Khayam 2008 59
Bayes Theorem is best understood through a classical example of a memory-less binary channel shown below What we already know about this channel is:
A priori probabilities: Pr{T0}, Pr{T1} Channel probabilities: Pr{R0|T0}, Pr{R1|T0} = 1 - Pr{R0|T0}, Pr{R1|T1}, Pr{R0|T1} = 1 - Pr{R1|T1}
T0
Pr{R0|T0}
R0
T1
Pr{R1|T1}
R1
60
Pr{Ti|Ri}: The probability that Ti was transmitted given that Ri has been received, i = 0, 1; or the probability of successful symbol transmission Pr{Ti|Rj}: The probability that Ti was transmitted given that Rj has been received, i j: or the probability of symbol error
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
61
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
62
Lets first focus on finding Pr{T0|R0} From Bayes Rule, we know that Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
63
We know these
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
64
Lets now focus on finding Pr{R0} in terms of what we already know From the Law of Total Probability, we have Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}
We know all of these terms
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
65
Lets now focus on finding Pr{R0} in terms of what we already know From the Law of Total Probability, we have Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
66
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
67
Using similar computations, we can show that Pr{T1|R1} = Pr{R1|T1} Pr{T1} / (Pr{R1|T1} Pr{T1} + Pr{R1|T0} Pr{T0})
T0
Pr{R0|T0}
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
68
T0
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
69
A Priori Probabilities
Pr{T0} = 0.45 Pr{T1} = 1 - Pr{T0} = 0.55 Pr{R0|T0} = 0.94 Pr{R1|T0} = 1 - Pr{R0|T0} = 0.06 Pr{R1|T1} = 0.91 Pr{R0|T1} = 1 - Pr{R1|T1} = 0.09
Channel probabilities:
T0
Pr{R0|T0}=0.94
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}=0.91
R1
70
T0
Pr{R0|T0}=0.94
R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}=0.91
R1
71
Lecture 1: Appendix A
72
Memoryless: Bit transmission at time n+i, i>0 has no dependence on bit transmission at time n Binary: Only two symbols are transmitted, represented by T0 and T1 Prior Probabilities: Probabilities of T0 and T1 are calculated ahead of time from the data; Pr{T1}= 1 - Pr{T0} Crossover Probabilities: Pr{R0|T1} and Pr{R1|T0} are called crossover or biterror probabilities. These probabilities are also calculated ahead of time by sending training signals on the channel; Pr{R0|T0}=1-Pr{R1|T0}, Pr{R0|T1}=1-Pr{R1|T1}
Pr{R0|T0} T0 R0
T1 Pr{R1|T1}
Copyright Syed Ali Khayam 2008
R1
73
Example 1
On a binary channel, find the probability that a 0 is transmitted and a 1 is received? A: a 1 is received AND B: a 0 is transmitted Probability of above event is: Pr{A B} Are A and B independent?
Pr{R0|T0} T0 R0
T1
Pr{R1|T1}
R1
74
Example 1
What is the sample space of our experiment?
Pr{R0|T0} T0 R0
T1
Spring 2008 Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
75
Example 1
What is the sample space of our experiment? Sample Space, S = {(T0 R0), (T0 R1), (T1 R0), (T1 R1)}
Pr{R0|T0} T0 R0
T1
Spring 2008 Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
76
Example 1
A: a 1 is received AND B: a 0 is transmitted Are A and B independent? A and B are independent when only when Pr{A B} = Pr{A}Pr{B}. So the main question is thefollowing: Is Pr{A B} = Pr{A}Pr{B}?
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
77
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{A B} = Pr{A}Pr{B} ? The LHS of the above equation is: Pr{A B} = Pr{A|B}Pr{B} = Pr{R1|T0}Pr{T0} The RHS is: Pr{A}Pr{B} = Pr{R1}Pr{T0} So the above question can be rephrased as: Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ?
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
78
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ? We can get rid of Pr{T0} from both sides, so we are left with the following question: Is Pr{R1|T0} = Pr{R1} ?
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
79
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{R1|T0} = Pr{R1} ? It can be intuitively deduced that the above equality relation does not hold in general because from the figure below we can see that Pr{R1} should be a function of both Pr{R1|T0} and Pr{R1|T1}.
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
80
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{R1|T0} = Pr{R1} ?
It can be intuitively deduced that the above equality relation does not hold in general. Mathematically, we can show this by computing the Pr{R1}: Pr{R1} = Pr{ (T0 is txd AND R1 is recd) OR (T1 is txd AND R1 is recd)} Pr{R1} = Pr{ (T0 R1) U (T1 R1)}
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
81
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{R1|T0} = Pr{R1} ?
Pr{R1} = Pr{ (T0 R1) U (T1 R1)} Clearly, (T0 R1) and (T1 R1) are mutually exclusive events. => Pr{R1} = Pr{T0 R1} + Pr{T1 R1} or Pr{R1} = Pr{R1 T0} + Pr{R1 T1} which gives Pr{R1} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} Now we rephrase the question posed on the top of this slide as: Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ?
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
82
Example 1
A: a 1 is received AND B: a 0 is transmitted Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ? For the above relation to be satisfied, the following relationship must be satisfied: Pr{T0} = 1 => Pr{T1} = 1- Pr{T0} = 0
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
83
Example 1
A: a 1 is received AND B: a 0 is transmitted Final Result: Pr{T0} = 1 => A and B are independent So events A and B are independent when T0 is the only symbol being transmitted
Pr{R0|T0} T0 R0
T1
Copyright Syed Ali Khayam 2008
Pr{R1|T1}
R1
84