Lecture Notes EE322 020916

Probability statistics and random
processes
Instructor: Intekhaab Siddiquee
2016 SPR
Contents
1
Review of set theory

Definitions
Venn diagram
Operation on sets
Union
Intersection
partition
Complement
Subtraction
Cartesian product
Finite and infinite sets
Countable and uncountable sets
Distributive law
De Morgans law
Functions
Probability Theory
Combinatorics
Counting methods
Discrete random variables

Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
March 15, 2016
2 / 93
Contents (cont.)
Uniform distributions
Bernoulli distribution
Geometric distribution
Binomial distribution
Multinomial distribution
Negative Binomial(Pascal) distribution
Poisson distribution
Expectation
Bernoulli random variable
Geometric random variable
Poisson random variable
binomial variable
Variance of a discrete random variable
Cumulative Distribution Fuction
Continuous random variables
March 15, 2016
3 / 93
Review of Set Theory
March 15, 2016
4 / 93
Definitions
Definitions
Set: A set is a collection of elements. One way to define a set is by explicitly listing all the
elements belonging to this set. Sets are represented by capital letters and the elements are
enclosed by curly braces and separated by commas. E.g
A = {, , , , 5, 7}
When an element belongs to a set it is represented by the symbol , e.g.
A
the symbol meaning belongs to. If an element does not belong to a set, it is represented by
the symbol .
/ Example
8
/A
Below are some examples of commonly used sets:
March 15, 2016
5 / 93
Definitions
1
Set of natural numbers, N = {1, 2, 3, }
Set of integers, Z = { , 3, 2, 1, 0, 1, 2, 3, }
Set of rational numbers, Q
Set of real numbers , R
Set of complex numbers, C
Closed interval on the real line, e.g. [a, b] is the set of real numbers x such that a x b,
where a and b are real numbers.
Open interval on the real line, e.g. (a, b) is the set of real numbers x such that a < x < b,
where a and b are real numbers.
The second way to define a set is by defining the property of the elements of the set. In this
case, they are represented the following way:
A = {x|x satisfies some property}

If more than one property is defined, they are separated by commas. Examples of some sets
represented by properties of the elements are given below:
March 15, 2016
6 / 93
Definitions
C = {x|x Z, 2 x 2} defines the set C = {2, 1, 0, 1, 2}
C = {x3 |x N, 2 x < 5} defines the set C = {8, 27, 64}
Set of rational numbers, Q = { ab |a, b Z, b 6= 0}
Set of complex numbers C = {a + ib | a, b R, i =
Open interval (a, b) = {x| a < x < b}
1}
Subset: If all the elements of set A belong to set B, then A is a subset of B. Set B is the
superset of A. This is mathematically represented as A B, where the symbol means
subset of.
Example: Suppose A = {1, 2, 3}, B = {1, 2}. Then B A.
Null Set: A set that contains no elements is called a null set or an empty set. It is usually
defined by the symbol . i.e. = {}.
Note: A null set will be a subset of any set.
Universal Set: It is the set of all elements in the context that is being considered. In
probability theory, a universal set is called sample space. Universal set is usually
represented by the symbol S.
March 15, 2016
7 / 93
Definitions
Example: Suppose A, B, C, D, F are all possible grades of an exam. So the universal set is
S = {A, B, C, D, F }.
Equality of sets: Two sets A and B are said to be equal if A B and B A, i.e. they
have the same elements.
Example: Sets A = {1, 2, 3}, B = {2, 3, 1} are equal while C = {1, 2, 4}, D = {1, 2} are not.
March 15, 2016
8 / 93
Definitions
Venn diagram
Venn diagram is a useful way to graphically represent sets. This helps to visually
understand relationship among different sets. A set is represented by a closed region. Two sets
having some common elements are represented by overlapping regions. A subset is
represented by a region completely inside the region representing the superset. A universal
set is usually represented by a rectangular region inside which all the sets reside.
In the figure beside, S is the universal

set, sets A and B have some common
elements and D C.
Figure 1: Sets represented by Venn diagram
March 15, 2016
9 / 93
Definitions
Operation on sets: Union

Union of two sets A and B is defined as the set containing all the elements of both A and B.
Union of sets A and B is mathematically represented by A B. An element x belongs to A B
if and only if either x A or x B. Union of three or more sets is defined the same way.
Example: Suppose A = {1, 2, 3} and B = {2, 3, 5}. Then A B = {1, 2, 3, 5}.
Example: Suppose A = {1, 5}, B = {3, 7}. Then A B = {1, 3, 5, 7}.
The figure beside shows, in the

Venn diagram, the union of sets
A and B by the shaded region.
Figure 2: Union of sets

March 15, 2016
10 / 93
Definitions
Operation on sets: Intersection

Intersection of two sets A and B is defined as the set containing the elements which belong
to both A and B. Intersection of sets A and B is mathematically represented by A B. An
element x belongs to A B if and only if x A as well as x B. Intersection of three or
more sets is defined the same way.
Example: Suppose A = {1, 2, 3} and B = {2, 3, 5}. Then A B = {2, 3}.
Example: Suppose A = [1, 5], B = [3, 7]. Then A B = .
The figure beside shows, in the

Venn diagram, the intersection
of sets A and B by the shaded
region.
Figure 3: Intersection of sets

March 15, 2016
11 / 93
Definitions
Operation on sets: Disjoint or Mutually exclusive sets

Two sets A and B are called disjoint or mutually exclusive if A B = .
Example: Suppose A = {1, 2, 4}, B = {3, 5, 7, 9}. Then A B = and A and B are disjoint.
In the figure below, sets A, B and C are all disjoint.
Figure 4: Disjoint sets
March 15, 2016
12 / 93
Definitions
Operation on sets: Partition

If set A is the union of a collection of n non-empty disjoint sets A1 , A2 , An , then the collection
of these sets is called a partition of A. Mathematically, the sets A1 , A2 , An are partition of A if
A = A1 A2 An and Ai Aj = (1 i n, 1 j n, i 6= j)
Example: Set A = {1, 2, 3, 4, 6, 8, 0} can have partition of sets A1 , A2 , A3 defined as
A1 = {1, 2}, A2 = {3, 4, 6}, A3 = {8, 0}.
Figure 5 shows partition of S into A, B, C, D.
Figure 5: Set partitions
March 15, 2016
13 / 93
Definitions
Example: Let A = {1, 2, 4, 7}. Write all the partitions of A.

Solution: The partitions are the following:
1.
{1}, {2}, {4}, {7}
2.
{1, 2}, {4, 7}}
3.
{1, 4}, {2, 7}}
4.
{1, 7}, {2, 4}}
5.
{1}, {2, 4, 7}
6.
{2}, {1, 4, 7}
7.
{4}, {1, 2, 7}
8.
{7}, {1, 2, 4}
9.
{1, 2, 4, 7}
March 15, 2016
14 / 93
Definitions
Operation on sets: Complement

Complement of set A, denoted by A is the set containing elements which are in the
universal set but not in set A. Mathematically, A = {x| x S, x
/ A}. In the figure below, set
complementary to A is shown by the shaded region.
Figure 6: Complement set
A set and its complement are related by the equation A A = S.

March 15, 2016
15 / 93
Definitions
Operation on sets: Subtraction

Given sets A and B, The difference A B is defined as the set whose elements are in A
but not in B. Mathematically, A B = {x| x A, x
/ B}
Example: Suppose A = {1, 2, 3, 4, 5}, B = {3, 4, 7, 9}. Then the subtraction is given as
A B = {1, 2, 5}
The figure below shows the subtraction set by the shaded region.
Figure 7: Subtraction of sets
(Note: A B = A B)
March 15, 2016
16 / 93
Definitions
Example: Suppose that the universal set S is defined as S = {1, 2, , 10} and A = {1, 2, 3},
B = {x S : 2 x 7}, and C = {7, 8, 9, 10}.
(a) Find A B
(b) Find (A C) B
(c) Find A (B C)
(d) Do A, B, and C form a partition of S?
Solution:
(a)
A B = {1, 2, 3, 4, 5, 6, 7}
(b)
A C = {1, 2, 3, 7, 8, 9, 10}
B = {2, 3, , 7}
thus:
(A C) B = {1, 8, 9, 10}
March 15, 2016
17 / 93
Definitions
(c)
A = {4, 5, , 10}
B C = {2, 3, 4, 5, 6}
thus: A (B C) = {2, 3, , 10}
(d) No, since they are not disjoint. For example,
A B = {2, 3} 6=
March 15, 2016
18 / 93
Definitions
Operation on sets: Cartesian product

A cartesian product of two sets A and B is the set containing all the ordered pairs from A
and B.
C = A B = {(x, y)| x A, y B}
Example: If A = {1, 2, 3}, B = {4, 5} then A B = {(1, 4), (1, 5), (2, 4), (2, 5), (3, 4), (3, 5)}
Multiplication principle: If set A has m elements and set B has n elements, then the set
A B has m n elements.
The multiplication principle can be extended two more than two sets. cartesian product of n
sets, A1 , A2 , , An is denoted by
A1 A2 An = {(x1 , x2 , , xn )|x1 A1 , x2 A2 , , xn An }
One important cartesian product is
Rn = {(x1 , x2 , , xn )|x1 R, x2 R, , xn R}
March 15, 2016
19 / 93
Definitions
Operation on sets: Finite and infinite sets

Cardinality: Number of elements of a set is called its cardinality and is denoted by two
vertical lines on two sides of the set. E.g., cardinality of set A is denoted by |A|.
Example: Cardinality of set A given by A = {1, 2, 3, 5} is 4.
Finite and infinite sets: A set with finite number of elements, i.e. with finite cardinality, is
called a finite set. A set whose cardinality is not a finite number is an infinite set.
Example: A = {1, 2, 3} is a finite set. R, N, Z are infinite sets.
An infinite set may be countable when the elements can be listed, or it may be uncountable
when the elements cannot be listed.
March 15, 2016
20 / 93
Definitions
Operation on sets: Countable and uncountable sets

A set A is countable if any one of the following two conditions hold:
a. if it is a finite set, i.e. |A| is finite.
b. it can be put in one-to-one correspondence with set of natural numbers N. In this case, the
set is called countably infinite.
A set is uncountable if it not countable.
Example: Sets N, Z, Q and any of their subsets are countable.
Any set containing an interval, e.g. [2, 4] is uncountable.
March 15, 2016
21 / 93
Definitions
Operation on sets: Distributive law in set theory

The law states that for any sets A, B, C,
A (B C) = (A B) (A C)
(1)
A (B C) = (A B) (A C)
(2)
Example: S = {1, 2, 3, 4, 5, 6}, A = {1, 2}, B = {2, 4, 5}, C = {1, 5, 6}.

LHS of equation 1 is
A (B C) = {1, 2} {1, 2, 4, 5, 6} = {1, 2}
RHS of equation 1 is
(A B) (A C) = {2} {1} = {1, 2}
LHS of equation 2 is
A (B C) = {1, 2} {5} = {1, 2, 5}
RHS of equation 2 is
(A B) (A C) = {1, 2, 4, 5} {1, 2, 5, 6} = {1, 2, 5}
March 15, 2016
22 / 93
Definitions
De Morgans law
De Morgans law, sometimes called De Morgans theorem, can be stated by the following
two formulas, given n sets A1 , A2 , , An
A1 A2 An = A1 A2 An
(3)
A1 A2 An = A1 A2 An
(4)
March 15, 2016
23 / 93
Definitions
Proof of De Morgans law: A B = A B

Let P = A B and Q = A B
Let x be an arbitrary element of P. Then
/ (A B)
xP xAB x
x
/ A and x
/B
x A and x B
x (A B) x Q P Q
Let y be an arbitrary element of Q. Then
y Q y A and y B y
/ A and y
/By
/ (A B)
y AB y P QP
Therefore P = Q.
March 15, 2016
24 / 93
Definitions
Proof of De Morgans law: A B = A B

Let P = A B and Q = A B
Let x be an arbitrary element of P. Then
/ (A B)
xP xAB x
x
/ A or x
/B
x A or x B
x (A B) x Q P Q
Let y be an arbitrary element of Q. Then
y Q y A or y B y
/ A or y
/By
/ (A B)
y AB y P QP
Therefore P = Q.
March 15, 2016
25 / 93
Definitions
Example of De Morgans Law

Example: Suppose U = {j, k, l, m, n}, X = {j, k, m} and Y = {k, m, n}. Need to show that
X Y =X Y.
Solution:
X Y = {j, k, m} {k, m, n} = {k, m} X Y = {j, l, n}
X = {j, k, m} X = {l, n}
Y = {k, m, n} Y = {j, l}
X Y = {l, n} {j, l} = {j, l, n}
Therefore X Y = X Y .
March 15, 2016
26 / 93
Definitions
Example of De Morgans Law

Example: Let U = {1, 2, 3, 4, 5, 6, 7, 8}, P = {4, 5, 6} and Q = {5, 6, 8}. Show that
P Q) = P Q.
Solution:
P Q = {4, 5, 6} {5, 6, 8} = {4, 5, 6, 8} P Q = {1, 2, 3, 7}
P = {4, 5, 6} P = {1, 2, 3, 7, 8}
Q = {5, 6, 8} Q = {1, 2, 3, 4, 7}
P Q = {1, 2, 3, 7, 8} {1, 2, 3, 4, 7} = {1, 2, 3, 7}
So, we get P Q) = P Q.
March 15, 2016
27 / 93
Definitions
Inclusion-exclusion principle
For two finite sets A and B, the following is true
|A B| = |A| + |B| |A B|
(5)
The principle is extended to three sets as

|A B C| = |A| + |B| + |C| |A B| |B C| |C A| + |A B C|
(6)
As a general case,
|n
i=1 Ai | =
n
X
|Ai |
i=1
n
X
i<j
|Ai Aj |+
n
X
|Ai Aj Ak | +(1)n+1 |A1 A2 An | (7)
i<j<k
March 15, 2016
28 / 93
Definitions
Example:(verification ) Given three subsets

A1 = {2, 3, 7, 9, 10}, A2 = {1, 2, 3, 9}, and A3 = {2, 4, 9, 10} of S = {1, 2, ..., 10}.
Following table may be created:
A1 = {2, 3, 7, 9, 10}
|A1 |
=5
A2 = {1, 2, 3, 9}
|A2 |
=4
A3 = {2, 4, 9, 10}
|A3 |
=4
A1 A2 = {2, 3, 9}
|A1 A2 |
=3
A1 A3 = {2, 9, 10}
|A1 A3 |
=3
A2 A3 = {2, 9}
|A2 A3 |
=2
|A1 A2 A3 |
=2
A1 A2 A3 = {2, 9}
|A1 A2 A3 | = (5 + 4 + 4) (3 + 3 + 2) + 2 = 7. Now
A1 A2 A3 = {1, 2, 3, 4, 7, 9, 10} and the cardinality is 7 which verifies the principle.
March 15, 2016
29 / 93
Definitions
Example:(problem solving ) Suppose in a party we have guests with white or red shirts and
black or non-black shoes. Below are the combinations:
- 10 people with white shirts and 8 people with red shirts;
- 4 people have black shoes and white shirts;
- 3 people have black shoes and red shirts;
- Total number of people with white or red shirts or black shoes is 21. Need to find the number
of people with black shoes.
Solution: To simplify, we designate people of red shirts as set R, black shoes as set B and
white shirts as set W. Now from the given conditions, we have the following:
|W | = 10, |R| = 8, |B W | = 4, |B R| = 3, |W R B| = 21
and need to find |B|. From exclusion-inclusion principle, we can write
|W R B| = |W | + |R| + |B| |W R| |R B| |W B| + |W R B|
21 = 10 + 8 + |B| 0 3 4 + 0 (|W R| = 0)
From
above, we get |B| = 10.
March 15, 2016
30 / 93
Definitions
Functions
A function f is a rule that maps an element from a specific set, called the domain to an
element of another set, called co-domain. Mathematically represented as
f :AB
If x is an element from the domain, the corresponding element of the co-domain to which x is
mapped, is represented as f (x), The set of all the elements of the co-domain that are
obtained by the mapping of all the elements of the domain form a subset of the co-domain and
is called the range of the function.
March 15, 2016
31 / 93
Definitions
Definitions
Random Experiment is a process by which certain uncertain is observed. An outcome is
the result of the random experiment. Set of all possible outcomes is called the sample space.
Example: Tossing a coin - sample space is {H, T }.
Example: Rolling a dice - sample space is {1, 2, 3, 4, 5, 6}.
Example: Tossing a coin two times the sequence of heads and tails is to be observed sample space is
S = {(H, T ), (H, H), (T, T ), (T, H)}.
When a random experiment is repeated, each experiment is called a trial.
An event is a subset of the sample space. If, e.g. the event is the set of outcomes of even
numbers when die is rolled, event is the set {2, 4, 6}.
Intersection and union of events are also events. A B occurs if either A or B occurs.
A B occurs if both A and B occur.
March 15, 2016
32 / 93
Definitions
Axioms of Probability Theory

A probability measure P(A) is assigned to an event A. The value of the measure is between 0
and 1 and it measures the likeliness of an event to occur. Below are the axioms related to
probability:
1
For any event A, P (A) 0.
Probability of sample space S, P (S) = 1.
If A1 , A2 , A3 , are disjoint events, then

P (A1 A2 A3 ) = P (A1 ) + P (A2 ) + P (A3 ) +
Example: We assume that any outcome between 1 and 6 is equally likely when you roll a dice,
P (1) = P (2) = P (3) = P (4) = P (5) = P (6). Because sum of probabilities of these outcomes
comprising the universal set must be equal to 1,
P (1) = P (2) = P (3) = P (4) = P (5) = P (6) =
1
.
6
March 15, 2016
33 / 93
Definitions
Axioms of Probability Theory...cntd.

Based on the axioms of the probability theory, the following can be deducted:
1
For any event A, P (A) = 1 P (A).
Probability of empty set is zero, i.e. P () = 0.
For any event A, P (A) 1.
P (A B) = P (A) P (A B)
P (A B) = P (A) + P (B) P (A B)
If A B, then P (A) P (B).
Item 5 above is, in fact, inclusion-exclusion principle, which can be generalized as
n
[
!
Ai
i=1
n
X
i=1
P (Ai )
P (Ai Aj )+
P (Ai Aj Ak ) + (1)n1 P
n
\
!
Ai
(9)
i=1
i<j<k
(8)
i<j
March 15, 2016
34 / 93
Definitions
Discrete Probability Models

If the sample space S is countable, the probability model is discrete. If A S, then A is also countable and
P (A) =
P (sj )
sj A
When sample space is finite and the outcomes are equally likely,
P (sj ) =
|A|
|S|
Example: In a roll of dice, what is the probability of getting an even outcome?

Answer: In this case, S = {1, 2, 3, 4, 5, 6}, and A = {2, 4, 6}. Hence probability of the event under consideration is
P (A) = P (2) + P (4) + P (6) =
1
1
1
1
+ + =
6
6
6
2
Example: In a roll of dice two consecutive times, what is the probability of the sum of the rolls being 11?
Answer: The sample space in this case will have 36 elements, each element is of type (ai , bi ) where ai {1, 2, 3, 4, 5, 6}
and bi {1, 2, 3, 4, 5, 6}.:
A = {(6, 5), (5, 6)}
Hence the probability of the event under consideration is
P (A) =
|A|
2
1
=
=
|S|
36
18
March 15, 2016
35 / 93
Definitions
Conditional Probability
Conditional probability: If A and B are events in a sample space S, then conditional
probability of A given B is the probability of A after outcomes in B has been observed. So,
we can write the following two equations:
P (B)P (A|B) = P (A B)
(10)
P (A)P (B|A) = P (A B)
(11)
which will produce the following:

P (A B)
P (B)
P (A B)
P (B|A) =
P (A)
(12)
P (A|B) =
(13)
If A and B are disjoint, P (A B) = 0. Hence P (A|B) = P (B|A) = 0.

When B A, P (A|B) =
When A B, P (A|B) =
P (AB)
P (B)
P (AB)
P (B)
=
=
P (B)
P (B)
P (A)
P (B)
= 1.
March 15, 2016
36 / 93
Definitions
Chain Rule of Conditional Probability

The chain rule is an extension of the conditional probability definition and can be expressed
as
P (A1 A2 An ) = P (A1 )P (A2 |A1 )P (A3 |A1 , A2 ) P (An |A1 , A2 , , An1 )
Example: A box contains 3 red balls, 3 black balls and 3 white balls. If 3 balls are picked at
randomly from the box, what is the probability of picking all balls red?
Answer: Let A1 = {R}, A2 = {R}, A3 = {R}.
Now, P (A1 ) =
3
9
1
, P (A2 |A1 )
3
2
8
1
, P (A3 |A1 , A2 )
4
1
.
7
Hence
P (A1 A2 A3 ) = P (A1 ) P (A2 |A1 ) P (A3|A1 , A2 ) =
1 1 1
1
=
3 4 7
84
March 15, 2016
37 / 93
Definitions
Independent Events
Two events A and B are independent if P (A|B) = P (A) or equivalently, P (B|A) = P (B).
When events A and B are independent, we see that
P (A B) = P (A)P (B)
Three events A, B and C are independent if P (A B C) = P (A) P (B) P (C) and any
two of the events are mutually independent, i.e.
P (A B) = P (A) P (B); P (A C) = P (A) P (C); P (B C) = P (B) P (C)
n events A1 , A2 , A3 , An are independent if
P (A1 A2 An ) = P (A1 ) P (A2 ) P (An ) and any combination of n 1 events are
mutually independent.
March 15, 2016
38 / 93
Definitions
Union of Independent Events

If A is and event and A is its complementary, then P (A) + P (A) = 1. This gives
P (A) = 1 P (A)
To find probability of union of independent events, the union is converted to intersection
using De Morgans law and then principle of independence of events is applied to find the
probability. (Note: If A and B are independent events, then A and B, A and B, and A and B
are independent too.)

P (A1 A2 An ) = P A1 A2 An
= 1 P A1 A2 An
= 1 P (A1 )P (A2 ) P (An )

= 1 (1 P (A1 ))(1 P (A2 )) (1 P (An ))
March 15, 2016
(14)
39 / 93
Definitions
Union of Events- Examples

Example: What is the probability that there will be at least one head in five coin tosses?
Answer: Need to find P (H) P (H) P (H) P (H) P (H). From Eqn. 14,

P (H) P (H) P (H) P (H) P (H) = 1
1
2
5
= 0.96875
To note that the probability, in fact, is 1 minus the probability of not getting a head in any of the
tosses.
March 15, 2016
40 / 93
Definitions
Law of Total Probability

If B1 , B2 , , Bn is a partition of the sample space S, then for any event A,
P (A) =
X
i
P (A Bi ) =
(15)
P (A|Bi )P (Bi )
Figure 9: Law of Total Probability
March 15, 2016
41 / 93
Definitions
Law of Total Probability - Examples

Example: Suppose we have 4 boxes containing red and black balls. Total balls in each box
is 100.
Box 1 - 75 black and 25 red
What is the probability that a ball chosen randomly from one of the boxes is red?
Answer: If the boxes are designated as B1 , B2 , B3 , B4 , the probability of choosing a box, i.e.
P (B1 ) = P (B2 ) = P (B3 ) = P (B4 ) = 0.25. Probability of choosing red ball from the boxes
are the following:
P (R|B1 ) = 25/100; P (R|B2 ) = 30/100; P (R|B3 ) = 20/100; P (R|B4 ) = 69/100
Hence
P (R) = P (R|B1 )P (B1 ) + P (R|B2 )P (B2 ) + P (R|B3 )P (B3 ) + P (R|B4 )P (B4 )
= 0.25(25/100 + 30/100 + 20/100 + 69/100) = 0.36
March 15, 2016
42 / 93
Definitions
Bayes Rule
Bayes Rule provides a way to find P (B|A) from P (A|B). From equations. 12 and 13, the
following can be obtained (on condition that P (A) 6= 0):
P (A|B)P (B) = P (B|A)P (A)
(16)
P (A|B)P (B)
P (A)
(17)
P (B|A) =
If B1 , B2 , B3 , form a partition of A, law of total probability can be used to get the following
equation:
P (A|Bj )P (Bj )
P (Bj |A) = P
P (A|Bi )P (Bi )
(18)
Equations 17 and 18 are called Bayes Rules.
March 15, 2016
43 / 93
Definitions
Bayes Rule - Examples

In the example of law of total probability, suppose the ball that was randomly picked, appears
to be red. What is the probability that box 3 was chosen?
Answer: Need to find P (B3 |B). From Bayes rule we have
P (R|B3 )P (B3 )
P (R)
(20/100) 0.25
5
=
=
0.36
36
P (B3 |R) =
March 15, 2016
44 / 93
Definitions
Combinatorics
March 15, 2016
45 / 93
Definitions
Counting methods
Multiplication principle: Suppose number of outcomes in a series of r experiments are
n1 , n2 , , nr respectively. Then there are total of n1 n2 nr possible outcomes in
the series of r experiments.
Example: How many different words containing three different letters from English alphabet
can be formed?
Answer: Suppose the experiments are 1. choosing the first letter, 2. choosing the second
letter and 3. choosing the third letter.
Outcome of first experiment (possible ways to choose the first letter)=26
Outcome of second experiment (possible ways to choose the second letter)=25
Outcome of third experiment (possible ways to choose the third letter)=24
Hence, the total number of outcomes, i.e. ways to form the words is 26 25 24 = 15600.
March 15, 2016
46 / 93
Definitions
Sampling
Sampling: Sampling from a set is the process of choosing an element from that set.
Random sampling is the process where each element of the set has equal probability of being
chosen. Two kinds of sampling are possible:
a. sampling with replacement (no restriction on number of times an element from the set may
be chosen) and
b. sampling without replacement (an element, once chosen, is removed from the set).
If ordering matters, i.e. a1 , a2 , a3 6= a1 , a3 , a2 , e.g. the sampling is called ordered
sampling. If order does not matter, the sampling is unordered.
March 15, 2016
47 / 93
Definitions
Ordered Sampling with Replacement

Suppose r elements are sampled from a set of n elements and we number the positions in
the sample containing r elements as 1, 2, , r.
The number of choices for the first position is n. Number of choices for the second and the
subsequent positions will also be n. This is because an element may be chosen for any
sampling, irrespective of whether it was chosen earlier or not.
Hence the total number of ways sampling can be done is
n n n (r times) = nr
March 15, 2016
48 / 93
Definitions
Ordered Sampling without Replacement (Permutations)

Suppose r elements are sampled from a set of n elements and we number the positions in
the sample containing r elements as 1, 2, , r (same way that we did with sampling with
replacement).
The number of choices for the first position is n.
Number of choices for the second position is n 1.
Number of choices for the third position is n 2.
Number of choices for the r-th position is n r + 1.

Hence the total number of ways sampling can be done is
Prn = n (n 1) (n r + 1)
Prn =
n(n 1)(n 2) (n r + 1)
(n r)(n r 1) 3 2 1
n!
(n r)!
(19)
March 15, 2016
49 / 93
Definitions
Permutation of n elements
Suppose we have a set of n different elements. We want to find out how many ways they can
be ordered. Obviously, the total number of ordering is given by
(n)(n 1)(n 2) (3)(2)(1) = n!
(20)
This is also called n-permutation.

Note: If r is substituted by n in eqn. 19, we get Pnn =
n!
.
0!
Hence from eqn. 20, we have 0! = 1.)
Example: Number of ways 10 elements in a set can be arranged to get different ordered sets
is given by
10! = 3628800
March 15, 2016
50 / 93
Definitions
Unordered sampling without replacement (Combinations)

Now suppose, we are only interested only in sets where order does not matter. From
eqn. 19, we have the number of ordered sets, each containing r elements. If ordering is not of
concern, r! ordered sets will be equivalent to one unordered set of r elements. Hence the
number of ways r elements can be chosen from n elements will be given by
Crn =
n!
Prn
=
r!
r!(n r)!
The term Crn is called binomial coefficient and is also represented as
(21)
n
.
r
Binomial theorem: For an integer n 0,

(a + b)n =
n
X
n
k=0
ak bnk
(22)
March 15, 2016
51 / 93
Definitions
Combinations...examples
Example: Suppose we have a set A with 10 partitions A1 , A2 , , A10 . How many different
subsets of A are unions of 3 partitions of A?
Answer: The number of subsets as union of 3 partitions from A will be given by
C310 =
10!
= 120
3!7!
Example: What is the probability of drawing 3 cards from a deck of 52 cards such that at least
one of them is an ace?
Answer: Total number of ways 3 cards can be drawn from 52 cards is
52
.
3
Suppose we
separate the 4 aces and draw 3 cards out of 48. Total number of ways this can be done is
48
.
3
This is the number of draws from 52 cards that will have no aces at all. So the number of draws
having at least one ace is
52
3
52
3
48
3
22100 17296
= 0.2174
22100
March 15, 2016
52 / 93
Definitions
Identities related to combinations

Following are commonly used identities:
n
k
n
X
n
k=0
n
nk
(23)
= 2n
(24)
n + 1 n n
=
+
(0 k n)
k+1
k+1
k
k
m + n X
m
n
=
Vandermondes identity
k
i
ki
i=0
(25)
(26)
March 15, 2016
53 / 93
Definitions
Bernoulli trial and Distribution

If a trial has only one of two outcomes, success and failure, the trial is called Bernoulli trial.
If each trial has a success probability p, the probability of k outcomes in a series of n trials is
given by
P (k) =
n
k
pk (1 p)nk
(27)
Suppose we have n elements and we want to divide these elements into r groups containing
n1 , n2 , , nr elements in the groups. The number of ways this can be done is given by

n
n!
=
n1 , n2 , , nr
n1 !n2 ! nr !
(28)
Multinomial theorem: It is generalization of binomial theorem and expressed by the

equation:
(x1 + x2 + + xn )n =
X
n1 +n2 ++nr =n

n
r
x n1 x n2 x n
r
n1 , n2 , , nr 1 2
March 15, 2016
(29)
54 / 93
Definitions
Unordered sampling with replacement

Suppose we want ot find out how many ways a total of k samples be chosen from an
n-element set such that repetition is allowed and ordering does not matter. Suppose element i
is chosen xi times. Hence the number of choices is the number of distinct solutions of the
equation
x1 + x2 + + xn = k xi {0, 1, 2, 3, }
Number of solutions to the equation x1 + x2 + + xn = k xi {0, 1, 2, 3, } is given by
n + k 1 n + k 1
=
k
n1
(30)
March 15, 2016
55 / 93
Definitions
Summary of different sampling methods

Different sampling methods are summarized in the table 1:
Ordered sampling with replacement
Ordered sampling without replacement
Unordered sampling without replacement
Unordered sampling with replacement
nk
n!
Pkn = (nk)!
n
n!
= k!(nk)!
k
n+k1
k
Table 1: Counting results for different sampling methods
March 15, 2016
56 / 93
Definitions
Discrete random variables

Let X be a discrete random variable, i.e. a variable that can assume values from a countable
finite or infinite range RX = {x1 , x2 , , xn , }. The function
PX (xk ) = P (X = xk ), for k = 1, 2, 3,
is called the probability mass function (PMF) of the variable X.
Properties of PMF:
0 PX (x) 1 for all x
X
PX (x) = 1
(31)
(32)
xRX
P (x A) =
PX (x) for all x A; A RX
(33)
xA
March 15, 2016
57 / 93
Definitions
Discrete random variables- Examples

Example: A dice is rolled two times. Suppose X be the variable representing sum of the two rolls . Calculate
the probability mass function of the variable X.
Answer: Let x be the outcome of the first roll and y - outcome of the second roll. S = x + y. The event can be
described by the ordered set with ordered elements (x, y), 1 x 6, 1 y 6. We create the following table:
Value of X(S)
(x , y)
Number of Outcome
P(X=S)
(1,1)
1
36
(1,2) , (2,1)
2
36
(1,3) , (2,2) , (3,1)
3
36
(1,4) , (2,3) , (3,2) , (4,1)
4
36
(1,5) , (2,4) , (3,3) , (4,2) , (5,1)
5
36
(1,6) , (2,5) , (3,4) , (4,3) , (5,2) , (6,1)
6
36
(2,6) , (3,5) , (4,4) , (5,3) , (6,2)
5
36
(3,6) , (4,5) , (5,4) , (6,3)
4
36
10
(4,6) , (5,5) , (6,4)
3
36
11
(5,6) , (6,5)
2
36
12
(6,6)
1
36
March 15, 2016
58 / 93
Definitions
Independent Random Variable

Two discrete random variables X and Y are said to be independent if
P (X = x, Y = y) = P (X = x)P (Y = y)
If X and Y are independent then
P (Y = y|X = x) = P (Y = y)
The idea can be generalized and the following statement about independentness can be made:
Discrete random variables X1 , X2 , , Xn are said to be independent if
P (X1 = x1 , X2 = x2 , , Xn = xn ) = P (X1 = x1 )P (X2 = x2 ) P (Xn = xn )
March 15, 2016
59 / 93
Definitions
A Few discrete random distributions

A list of commonly used discrete probability distributions:
Uniform distribution
Negative binomial distribution
Poission distribution
March 15, 2016
60 / 93
Definitions
Uniform distributions
Uniform Distribution: A random variable X that assumes the values x1 , x2 , , xk has a
uniform distribution if it assumes each of the values x1 , x2 , , xk with equal probability. The
random variable X is said to be uniformly distributed with the following probability mass
function:
P (X = xi ) =
1
k
1ik
Example: The random variable X that takes on the value of a single dice roll has a uniform
distribution, since X can assume any of the 6 values in the set {1, 2, 3, 4, 5, 6} with an equal
probability for each. This probability is 1/6.
March 15, 2016
61 / 93
Definitions
A random variable X is said to be a Bernoulli random variable with parameter p, written as
X Bernoulli(p) if its PMF is given by
PX (x) = 1 p
for x = 1
(34)
for x = 0
otherwise
March 15, 2016
62 / 93
Definitions
A random variable X is said to be a Geometric random variable with parameter p, written as
X Geometric(p) if its PMF is given by
PX (k) =
p(1 p)k1
for k = 1, 2, 3,
otherwise
(35)
Interpretation of k is the following: It is total number of trials to get the first success, if the
success probability of a trial is p.
March 15, 2016
63 / 93
Definitions
A random variable X is said to be a Binomial distributed random variable with parameters n
and p, written as X Binomial(n, p) if its PMF is given by
PX (k) =
n k
p (1
k
p)nk
for k = 1, 2, 3,
(36)
otherwise
A binomial distribution may be considered as number of ways the population can be divided
into two groups of k and n k each.
March 15, 2016
64 / 93
Definitions
Relationship between binomial and Bernoulli random variables

If X1 , X2 , , Xn are independent Bernoulli random variables, then X defined by
X = X1 + X2 + + Xn has a Binomial(n,p) distribution.
March 15, 2016
65 / 93
Definitions
In a multinomial distribution, a set of variables X1 , X2 , . . . , Xn have a probability mass
function defined as
N!
P (X1 = x1 , ..., Xn = xn ) = Qn
i=1
where xi are nonnegative integers such that
n
Y
xi !
xi = N and
pi i
(37)
i=1
pi = 1.
Multinomial coefficients provide the number of ways N elements of a set can be grouped into
n groups having x1 , x2 , , xn elements in them (x1 + x2 + + xn = N )
Example: How many ways cooking items may be divided among 11 people so that 6 of them
get to prepare the main dish, 2 of them desert and 3 of them the appetizer?
Answer: It can be done in
11!
6!2!3!
= 4620 ways.
March 15, 2016
66 / 93
Definitions
Negative Binomial(Pascal) distribution

A random variable X is said to be a Pascal random variable with parameters m and p,
written as X P ascal(m, p) if its PMF is given by
PX (k) =
k1 m
p (1
m1
p)km
for k = m, m + 1, m + 2,
(38)
otherwise
Pascal distribution is a generalization of geometric distribution. It describes repeated

independent trial until m successes are observed.
March 15, 2016
67 / 93
Definitions
Poisson distribution
A random variable X is said to be a Poisson random variable with parameter , written as
X P oisson() if its range is RX = 0, 1, 2, 3, and PMF is given by
PX (k) =
e k
for k RX
otherwise
k!
(39)
March 15, 2016
68 / 93
Definitions
Expectation
Let X be a discrete random variable with range RX = {x1 , x2 , x3 , }, where RX is finite
or countably infinite. The expected value of X, is defined as the weighted average of the values
in the range.
E[X] =
X
xk RX
xk P (X = xk ) =
(40)
xk PX (xk )
xk RX
March 15, 2016
69 / 93
Definitions
Linearity of Expectation
Theorem:
E[aX + b] = aE[X] + b for all a, b R
(41)
E[X1 + X2 + + Xn ] = E[X1 ] + E[X2 ] + + E[Xn ]
(42)
(for any set of random variables X1 , X2 , , Xn )
March 15, 2016
(43)
70 / 93
Definitions
Expected value - Bernoulli random variable

Expected value of a Bernoulli random variable is given by
E[X] = 0 (1 p) + 1 p = p
March 15, 2016
71 / 93
Definitions
Expected value - Geometric random variable

Expected value will be given by
E[X] =
X
k=1
xk PX (xk )
kp(1 p)k1
k=1
= p + 2p(1 p) + 3p(1 p)2 +

= p(1 + 2q + 3q 2 + 4q 3 + ) (q = 1 p)
1
(1 q)2
1
=p 2
p
1
=
p
=p
March 15, 2016
72 / 93
Definitions
Expected value - Poisson random variable

Expected value will be given by
E[X] =
X
k=0
xk PX (xk )
k=0
= e
e k
k!
k
(k 1)!
k=1
= e
k=1
= e
k1
(k 1)!
(1 + +
2
3
+
+ )
2!
3!
= e e
=
March 15, 2016
73 / 93
Definitions
Expected value- binomial distribution

Calculation of expected value for binomial distribution may be simplified using the fact that
binomial variable may be considered as sum of Bernoulli variables. If X is the binomial
distribution(n,p), then X = X1 + X2 + + Xn , where X1 , X2 , , Xn are independent
Bernoulli variables.
From the theorem on linearity of expectations, we have
E[X] = E[X1 ] + E[X2 ] + + E[Xn ]
= p + p + + p
= np
March 15, 2016
74 / 93
Definitions
Variance of a discrete random variable

Variance of a random variable X , with mean E[X] is defined as
V ar(X) = E[X E[X]]2
(44)
From property of linearity,

E[X E[X]]2 = E[X 2 2XE[X] + [E[X]]2 ]
= E[X 2 ] 2E[X]E[E[X]] + [E[X]]2
= E[X 2 ] [E[X]]2 = E[X 2 ] 2X
E[X 2 ] may be expressed as
E[X 2 ] =
x2k PX (xk )
xk RX
Standard deviation of a random variable X is defined as

SD(X) = X =
p
V ar(X)
March 15, 2016
75 / 93
Definitions
Variance ... cntd

Theorem: For a random variable X and real numbers a and b,
V ar(aX + b) = a2 V ar(X)
(45)
Theorem: For a independent random variables X1 , X2 , , Xn and

X = X1 + X2 + + Xn ,
V ar(X) = V ar(X1 ) + V ar(X2 ) + + V ar(Xn )
(46)
March 15, 2016
76 / 93
Definitions
Cumulative Distribution Fuction

Cumulative distribution function (CDF) of a random variable X is defined as
FX (x) = P (X x), for all x R
(47)
Example: Suppose a coin is tossed 3 times and X is the variable representing sum of the
number of heads in the tosses. The possible outcomes are in the set {0, 1, 23}. The
probabilities are the following:
P (X = 0) = (1/2)3 = 1/8
3
P (X = 1) =
(1/2)3 = 3/8
1
3
P (X = 2) =
(1/2)3 = 3/8
2
3
P (X = 3) =
(1/2)3 = 1/8
3
FX (x) =
1/8 x 0
1/2 x 1
7/8 x 2
1
x3
March 15, 2016
77 / 93
Definitions
Cumulative Distribution Fuction...cntd.

Following is an important relationship related to CDF:
P (a < X < b) = FX (b) FX (a)
(48)
March 15, 2016
78 / 93
Definitions
Continuous random variables

A continuous random variable X has range in the form of an interval or union of a number of
non-overlapping intervals on the real line.
Cumulative distribution function (CDF) of a continuous variable is defined as the function
FX (x) defined by the relationship
FX (x) = P (X x) x R
(49)
Probability density function fX (x) is defined as the derivative of the cumulative

distribution function
fX (x) =
d
[FX (x)]
dx
(50)
We have the following relationships:

Z
FX (x) =
(51)
fX (u)du
(52)
fX (x)dx = 1
March 15, 2016
79 / 93
Definitions
Examples
Example: Let the probability density function of a random variable X be defined as
fX (x) =
cebx
x0
otherwise
where c and b are positive constants. Find (a) possible values of c and b, (b) CDF, (c)
P (1 X 3).
Answer: (a) From the property that
Hence =
c
b
fX (x)dx = 1, we get
cebx dx =
c
=1
b
= 1, i.e. c = b.
March 15, 2016
80 / 93
Definitions
(b) CDF will be given by

Zx
ceby dy = 1
FX (x) =
cebx
= 1 ebx
b
(c) the probability P (1 X 3) will be given by

P (1 X 3) = FX (3) FX (1) = eb e3b
Z3
P (1 X 3) =
cebx dx = eb e3b
(using CDF)
(using PDF)
March 15, 2016
81 / 93
Definitions
Expected Value
Expected value of a continuous random variable is defined as
Z
E[X] = X =
(54)
xfX (x)dx
The expected value of the sum of two random variables X and Y is the sum of their expected
values:
X+Y = X + Y
March 15, 2016
82 / 93
Definitions
Expected value of a function of a random variable

Expected value of function of a continuous random variable is given by the formula
Z
E[g(x)] = X =
(55)
g(x)fX (x)dx
If variable Y is defined as Y = aX + b, expected value of Y will be given by:

Z
Y =
(ax + b)fX (x)dx
Z
=a
Z
xfX (x)dx + b
fX (x)dx
= a X + b
March 15, 2016
83 / 93
Definitions
Variance of of a random variable

The variance of a discrete random variable X measures the spread, or variability, of the
distribution, and is defined by
2
V ar(X) = X
=
(x X )2 fX (x)dx =
x2 fX (x)dx 2X
(56)
If a random variable Y is defined as Y = aX + b, a, b R, the variance of Y will be given by

2
Y
= a2
For independent random variables X and Y, the variance of their sum or difference is the sum
of their variances:
2
2
2
X+Y
= X
+ Y
2
2
2
XY
= X
+ Y
March 15, 2016
84 / 93
Definitions
Functions of continuous random variable

If X is a random variable having domain RX and Y = g(X), then Y is also a random
variable with some domain RY . It is possible to find the CDF and PDF of Y from the functional
relationship and the PDF/CDF of X with the help of the following theorem.
Theorem: If y = g(x) is monotonic and differentiable, PDF of Y is given by
fY (y) =
fX (x)
fX (g 1 y)
= 0 1
g 0 (x)
g (g y)
(57)
If corresponding to an increment of x for variable X, increment for Y is y, we can write:

fY (y)y = fX (x)x
If x 0, then x 0. and for the limiting case, we get
fY (y) = lim
x0
fX (x)
y
x
fX (x)
fX (g 1 (y))
= 0 1
g 0 (x)
g (g (y))
March 15, 2016
85 / 93
Definitions
Functions of continuous random variable...cntd

Suppose X and Y are related by Y = aX + b. In this case y = g(x) = ax + b. Hence
0
g (x) = a and x = g 1 (y) =
yb
.
a
We can write
fY (y) =
1
fX
a
yb
a
March 15, 2016
86 / 93
Definitions
Functions of continuous random variable...cntd.

If X is a random variable having domain RX and Y = g(X), then Y is also a random
variable with some domain RY . It is possible to find the CDF and PDF of Y from the functional
relationship and the PDF/CDF of X with the help of the following theorem.
Theorem: If it is possible to partition RX into a finite number of partitions which are
monotonic and differentiable, PDF of Y is given by
fY (y) =
n
n
X
X
fX (xi )
=
0
|g (xi )|
i=1
i=1
f (g 1 (yi ))
X0

g (g 1 (yi ))
(58)
where x1 , x2 , , xn are x-variables for different partitions, y1 , y2 , , yn are corresponding

y-variables and n is the number of partitions.
March 15, 2016
87 / 93
Definitions
Expected value and variance for uniform PDF

A PDF is said to be uniform in the interval (a, b), b > a, if it is defined as
fX (x) = c, a X b where c is a constant
(59)
From the property that

R
fX (x)dx = 1, we get
c(b a) = 1 c =
1
ba
Variance of uniform random variable

is 0, as can be easily seen from the
definition.
March 15, 2016
88 / 93
Definitions
Normal Distribution
The probability density of the normal distribution is:
2
fX (x | , 2 ) =
(x)
1
2 2
e
2
(60)
is the mean of the distribution. The parameter is its standard deviation with its variance
2 . A random variable with a Gaussian distribution is said to be normally distributed and is
called a normal deviate.
March 15, 2016
89 / 93
Definitions
Rayleigh distribution function

Rayleigh distribution arises in the case of random complex numbers whose real and
imaginary components are independently and identically distributed Gaussian with equal
variance and zero mean. In that case, the absolute value of the complex number is
Rayleigh-distributed.
The probability density function of the Rayleigh distribution is
f (x; ) =
x x2 /(22 )
e
,
2
x 0,
(61)
where is the scale parameter of the distribution. The cumulative distribution function is
2
F (x; ) = 1 ex
/(2 2 )
x [0, ).
(62)
Expectation and variance of Rayleigh distribution are given by

r
X =
' 1.253;
2
2
X
=
4 2
' 0.429 2
2
March 15, 2016
90 / 93
Definitions
Figure 10: Rayleigh distribution
March 15, 2016
91 / 93
Definitions
Exponential distribution function

The probability density function (pdf) of an exponential distribution is
f (x; ) =
ex
x 0,
x < 0.
(63)
March 15, 2016
92 / 93
Definitions
The cumulative distribution function is given by
F (x; ) =
1 ex
x 0,
x < 0.
(64)
The mean or expected value of an exponentially distributed random variable X with rate
parameter is given by E[X] =
1
.
The variance of X is given by Var[X] =

1
.
2
March 15, 2016
93 / 93

Lecture Notes EE322 020916

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Lecture Notes EE322 020916

Transféré par

Droits d'auteur :

Formats disponibles

Probability statistics and random

Review of set theory

Discrete random variables

March 15, 2016

Cumulative Distribution Fuction

Continuous random variables

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

Review of Set Theory

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

March 15, 2016

Set of natural numbers, N = {1, 2, 3, }

Set of rational numbers, Q

Set of real numbers , R

Set of complex numbers, C

A = {x|x satisfies some property}

March 15, 2016

C = {x|x Z, 2 x 2} defines the set C = {2, 1, 0, 1, 2}

C = {x3 |x N, 2 x < 5} defines the set C = {8, 27, 64}

Set of rational numbers, Q = { ab |a, b Z, b 6= 0}

Set of complex numbers C = {a + ib | a, b R, i =

Open interval (a, b) = {x| a < x < b}

March 15, 2016

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

In the figure beside, S is the universal

Figure 1: Sets represented by Venn diagram

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

Operation on sets: Union

The figure beside shows, in the

Figure 2: Union of sets

March 15, 2016

Operation on sets: Intersection

The figure beside shows, in the

Figure 3: Intersection of sets

March 15, 2016

Operation on sets: Disjoint or Mutually exclusive sets

Figure 4: Disjoint sets

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

Operation on sets: Partition

Figure 5: Set partitions

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

Example: Let A = {1, 2, 4, 7}. Write all the partitions of A.

Instructor: Dr. Intekhaab Siddiquee

{1}, {2}, {4}, {7}

{1, 2}, {4, 7}}

{1, 4}, {2, 7}}

{1, 7}, {2, 4}}

March 15, 2016

Operation on sets: Complement

Figure 6: Complement set

A set and its complement are related by the equation A A = S.

March 15, 2016

Operation on sets: Subtraction

Figure 7: Subtraction of sets

March 15, 2016

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016

Instructor: Dr. Intekhaab Siddiquee

March 15, 2016