Operations Research II

Things to remember

ESI 4313 is not a soap opera

Study regularly

Can not compress time

Understand, not memorize

Do not filter content

Ignoring problems wont help

Take notes

Probability Review

Experiment
I
I

Experiment performed and an outcome observed

Sample space : the set of all possible outcomes
I
I

Event: subset of
I

Coin flip: = {H, T }

Die roll: = {1, 2, 3, 4, 5, 6}

A = {2, 4, 6} and B = {2, 3, 5} are events defined on sample

space = {1, 2, 3, 4, 5, 6}

Occurrence
I
I
I

Event A occurs if the observed outcome is some member of A

If the observed outcome is not a member of A, then A occurred
Every trial of an experiment results in only one outcome but
many events
Example: die roll, outcome is 4.
Occurred: A = {2, 4, 6}, B = {1, 4, 6},
Did not occur: A = {1, 3, 5}, B = {2, 3, 5}

Events
I

Set operations
I
I
I
I

Empty set:
I

Union: A B = {x : x A or x B}
Intersection: A B = {x : x A and x B}
Complement: A = {x : x 6 A}
Inclusion: A B
A A =

Can consider intersections and unions of many (even infinitely

many) sets:

[
n=1

\
n=1

An = A1 A2 . . . = {x : x An for some n}
An = A1 A2 . . . = {x : x An for all n}

Algebra of Events
I

Algebra
I
I
I
I
I
I
I
I
I
I

AB =B A
AB =B A
A (B C ) = (A B) C
A (B C ) = (A B) C
A (B C ) = (A B) (A C )
A (B C ) = (A B) (A C )
A = A
A A =
A=
A=A

De Morgans Law
I
I

A B = A B
A B = A B

More Details...

A list of events in mutually exclusive or disjoint if there is no

point in which is included in more than one event
I

Events {1, 2} and {1, 4} are not disjoint

A list of events is collectively exhaustive if each point in is

included in at least one event in the list

Sample space
I
I

All possible distinguishable outcomes allowed by the model

Finest grain, mutually exclusive, collectively exhaustive listing
of all possible outcomes

Probability Law
I

Probability law P
I
I

Assigns numbers to events

Specifies the likelihood of events

3 axioms for P:
I
I
I

Non-negativity: P[A] 0 for every event A

Normalization: P[] = 1 ( = sample space)
Additivity: If A1 , A2 , . . . is a sequence of disjoint events, then
"
#

[
X
P
An =
P[An ]
n=1

Not so simple...

n=1

Consequences
I

P[] = 0

P[A] 1 for all events A

Let A, B and C be events

Can use Venn diagrams to visualize

If A B, then P[A] P[B]

P[A B] = P[A] + P[B] P[A B]

P[A B] P[A] + P[B]

P[A B C ] = P[A] + P[A B] + P[A B C ]

Examples

Suppose A and B are events on . If P[A] = 0.3, P[B] = 0.4

= 0.5, what is P[A B]?
and P[A B]

A desk drawer contains 5 pens, 3 of which are dry

I

Pens are selected at random one-by-one until a good pen is

found. The sequence of Gs and Bs for good and bad pens is
recorded. = ?
Suppose that only the number, and not the sequence of pens
tested is noted. = ?
Suppose that the pens are selected one-by-one and tested until
both good and bed pens are identified. = ?

Conditional Probability
I
I

A way to reason about the outcome of an experiment based

on partial information
Examples
I

The first letter is a t; what is the chance that the second

letter is an h?
How likely is it that a person has a disease given that a
medical test was negative?

Conditional probability of A given B

I
I
I

P[A|B] = probability of A given that B occured

B is the new universe (sample space)
Definition: Assuming P[B] 6= 0,
P[A|B] = P[A B]/P[B]

Exercise: Show that

I
I
I

0 P[A|B] 1
P[|B] = 1
If the events A1 , A2 , . . . are disjoint, then
"
#

[
X

P
An  B =
P[An | B]
n=1

n=1

Conditional Probability
I

Example
I
I
I
I
I

Roll a tetrahedral die twice

X = first roll, Y = second roll
Event B = {min{X , Y } = 2}
Let M = max{X , Y }
Then
P[M = 1 | B] =?
P[M = 2 | B] =?

By symmetry: P[A B] = P[A|B] P[B] = P[B|A] P[A]

Multiplication rule or chain rule

P[A1 A2 . . .An ] = P[A1 ] P[A2 |A1 ] P[A3 |A1 , A2 ] P[An |n1
i=1 Ai ]

Proof...

Example

Suppose a box contains a double-headed coin, a double-tailed

coin, and a conventional coin. A coin is picked at random and
flipped the result is a head. What is the probability it is a
double-headed coin?
I
I
I

Let D = event that the coin is double-headed

Let A = event that the coin shows a head
P[D|A] =?

Independence
I

Independence of two events A and B:

P[A B] = P[A|B] P[B] = P[A]P[B]

Independence of a collection of events

I

I
I

Intuition: Information on some of the events tells us nothing

about probabilities related to the remaining events
Example: P[A1 (A2 A3 ) | A5 A6 ] = P[A1 (A2 A3 )]
Mathematical definition: For any distinct i, j, . . . , q
P[Ai Aj . . . Aq ] = P[Ai ] P[Aj ] P[Aq ]

Example: 3 events A1 , A2 , A3 . Conditions for independence

P[A1 A2 ] = P[A1 ] P[A2 ]
P[A1 A3 ] = P[A1 ] P[A3 ]
P[A2 A3 ] = P[A2 ] P[A3 ]
P[A1 A2 A3 ] = P[A1 ] P[A2 ] P[A3 ]

Conditional Independence
I

Given an event C , events A and B are conditionally

independent if
P[A B | C ] = P[A|C ] P[B|C ]

If A and B are independent, they are not necessarily

conditionally independent

If A and B are conditionally independent, they are not

necessarily independent
Exercise

I
I
I
I
I

Roll a die twice

A = {lower score is a 2}, B = {higher score is a 5}
C = {1 roll shows less than 3 and 1 roll shows more than 3}
Are A and B independent?
Are A and B conditionally independent given C ?

Two Tools
I

Bayes rule
P[A|B] =

P[A] P[B|A]
P[B]

Total Probability Law: If A1 , A2 , . . . are disjoint, collectively

exhaustive events, then for any event B:
X
P[B] =
P[B|Ai ] P[Ai ]
i

I Example: A CD storing information in binary form has been corrupted.

Imagine that you are to make an effort to save as much information as

possible. Due to the damage on the CD, you know that there will be
errors in the reading. You know that if there was a 0 on the uncorrupted
CD, the probability that you correctly detect it on the corrupted CD is
0.9. The probability that you correctly detect a 1 is 0.85. Given that on
the uncorrupted CD the each digit is a 1 or a 0 with equal probability,
and given that you read in a 1 from the corrupted CD, what is the
probability that this is a correct reading?

Random Variables
I

Example
I
I
I
I

Roll a fair die

Bet $1 that the outcome is a perfect square
Let X denote winnings in this game
X () = 0 or 1

A random variable is a function from the sample space to the

real numbers

Two main types

I
I

Continuous: possible values form a continuum (interval of R)

Discrete: takes values on a countable set

Possible to define several r.v.s on the same sample space

Notation: random variable X , numerical value x

Discrete Random Variables

I

How to describe a random variable X ?

Probability mass function (PMF): pX (x) = P[X = x]

Example
I
I

Biased coin, P[H] = p, independent coin tosses

X = # of coin tosses until the 1st H
pX (x) = P[X = x] = P[TT TH] = (1p)x1 p, x = 1, 2, . . .

Geometric PMF

How to compute pX (x)

1. Collect all possible outcomes for which X = x:
Ax = { : X () = x}
2. Evaluate P[Ax ]
3. Repeat for all x

Discrete Random Variables

Example
I
I
I
I

Two independent rolls of a fair tetrahedral die

F = outcome of the first roll, S = outcome of the second roll
L = min{F , S}
pL (1) =?, pL (2) =?, pL (3) =?, pL (4) =?

Properties of the PMF

I
I
I

pX (x) is defined for all real x

pX (x) 0
P
If X takes values {xi }, then i pX (xi ) = 1

PMF
I

Binomial
I
I

X = number of Hs in n independent coin tosses

p = P[H]
 
n x
pX (x) =
p (1 p)nx
x

Poisson
I

For > 0 and x = 0, 1, 2, . . .

pX (x) = e

x
x!

Bernoulli
I

only two values: 0 and 1

pX (1) = p = 1 pX (0)

Uniform
I

on {1, 2, . . . , n}
pX (x) = 1/n, x = 1, 2, . . . , n

Expectation
I

Definition
EX =

xi pX (xi )

EX does not have to be one of xi

If probability = mass, then expectation = center of gravity:

X
(x EX )pX (x) = 0
x

Examples
I
I

X is uniform on {0, 1, 2, . . . , n}
X is geometric

Expectation
I

Let X be a r.v. and Y = g (X ):

X
EY =
y pY (y )
y

Theorem
EY =

g (x) pX (x)

x
I

Examples
I
I

nth moment of X : EX n
nth central moment X : E[(X EX )n ]

IMPORTANT: In general
E[g (X )] 6= g (EX )

Expectation
I

Variance: Var(X ) = E[(X EX )2 ]

I
I
I

Properties
I
I

p
Standard deviation: Var(X ) =
Small variance = mass closer to EX
0 variance = all mass at EX
If b is a constant: Eb = b
If a is a constant: E[aX ] = a EX

Expectation is linear:
E[g (X ) + h(X )] = E[g (X )] + E[h(X )]

Example
E[(X EX )2 ] = E[X 2 2X EX + (EX )2 ]
= E[X 2 ] 2(EX )2 + (EX )2 = E[X 2 ] (EX )2

Exercise: Var(aX + b) = a2 Var(X )

Example
I

Traverse a d = 200 mile distance at constant but random

speed V

pV (1) = pV (200) = 1/2

Travel time T = T (V ) = d/V

EV = 1/2 + 200/2 = 100.5
Var(V ) = EV 2 (EV )2 = 9900.25

Thus
ET EV 6= d = E[TV ]

Random variables V and T are dependent

Need a way to describe them jointly

Joint PMF
I

Joint PMF of X and Y is defined by

pX ,Y (x, y ) = P[{X = x} {Y = y }] = P[X = x, Y = y ]

Joint description of X and Y

Normalization
X

pX ,Y (x, y ) = 1

x,y
I

Marginals
pX (x) =

pX ,Y (x, y )

pY (y ) =

X
x

Amount of information

pX ,Y (x, y )

Joint PMF
I

Exercise: Let X and Y have the joint distribution

pX ,Y (x, y ) = cx y , where x, y {1, 2, . . .}, and
, (0, 1). Find
I
I
I

c
P[X > Y ]
P[X = Y ]

Functions: Let X and Y be discrete r.v.s. Define

Z = g (X , Y ). What is the PMF of Z ?
X
pZ (z) = P[Z = z] =
pX ,Y (x, y )
x,y : g (x,y )=z

Example: Suppose two numbers X and Y are picked at

random from {1, 2, . . . , 100} without replacement. Let
Z = max{X , Y }. Find pZ (z).

Expectation

E[g (X , Y )] =

XX
x

g (x, y ) pX ,Y (x, y )

In general
E[g (X , Y )] 6= g (EX , EY )

Example: Suppose two numbers X and Y are picked at

random from {1, 2, . . . , 100} without replacement. Let
Z = max{X , Y }. Find EZ .

Expectation

Expectation is linear:
E[g (X ) + h(Y )] = E[g (X )] + E[h(Y )]

Example
I
I
I
I

n people throw their hats in a box and then pick one at random
X = # of people who get their own hats
EX = ?
Var(X ) = ?

Conditioning
I

On an event A such that P[A] > 0:

I

Conditional PMF
pX |A (x) = P[X = x | A] =

P[{X = x} A]
P[A]

Conditional expectation
E[g (X ) | A] =

g (x) pX |A (x)

Example
I
I
I
I

X = the roll of a six-sided die

A = {the roll is an even number}
pX |A (x) = 1/3 if x = 2, 4, 6 and 0 otherwise
E[X |A] = 4

Conditioning
I

Conditioning on a random variable: A = {Y = y }

I

Conditional PMF
pX |Y (x|y ) = P[X = x | Y = y ] =

P[X = x, Y = y ]
P[Y = y ]

Conditional expectation
X
E[X | Y = y ] =
x pX |Y (x|y ) = h(y )

(a number)

E[X | Y ] = h(Y )

(random variable)

Double conditioning
X
E[E[X |Y ]] =
E[X |Y = y ] pY (y )
I

XX
y

x pX |Y (x|y ) py (y ) =

XX
x

x pX ,Y (x, y ) = EX

Conditioning
I

Example
I
I
I

Pick Y uniformly at random from {1, . . . , n}

Pick X uniformly at random from {1, . . . , Y }
EX = ?

Example
S=

N
X

Xi

n=1
I

Total Expectation Theorem: If {Ai } are disjoint events such

that i Ai = , then
X
EX =
E[X |Ai ] P[Ai ]
i

Example: pX (x) = (1 p)x1 p. Find EX .

Independence
I

A discrete r.v. X is independent of event A if

P[{X = x} A] = P[X = x] P[A]

Two discrete r.v.s X and Y are independent if

pX ,Y (x, y ) = pX (x) pY (y )

for all x and y

If X and Y are independent then

I
I

for all x

E[XY ] = EX EY
Var(X + Y ) = Var(X ) + Var(Y )

If E[XY ] = EX EY then X and Y not always independent

Continuous Random Variables

I

Example: uniform distribution on [0, 1]

PMF description does not work: pX (x) = 0 for all x

Probability Density Function (PDF) fX (x):

Z
fX (u) du
P[X B] =
xB

Properties of the PDF:

I
I

(x) 0
RfX
f (x) dx = 1
X

Interpretation: for small

Z
P[x X x + ] =

x+

fX (u) du fX (x)
x

PDF
I
I

fX (x) is not probability

Example: fX (x) = 1/(2 x) for 0 < x 1

Z

Z
fX (x) dx =

1
dx = 1
2 x

Mean and Variance

I
I
I

R
EX = x fX (x) dx
R
E[g (X )] = g (x) fX (x) dx
R
Var(X ) = X2 = (x EX )2 fX (x) dx

Example: fX (x) = 1/(b a), a x b

I
I

EX = (a + b)/2
Rb
Var(X ) = a (x (a + b)/2)2 /(b a) dx = (b a)2 /12

Cumulative Distribution Function

I

CDF:
FX (x) = P[X x]

Well defined for all random variables

I
I

P
Discrete: FX (x) = ux pX (u)
Rx
Continuous: FX (x) = fX (u) du

Continuos random variables:

fX (x) =

Mixed random variables

dFX (x)
dx

Examples
I

Exponential r.v. with parameter

fX (x) = e x , x 0
FX (x) = 1 e x , x 0

Gaussian (normal) r.v. with parameters and 2

fX (x) =

I
I
I
I

1
2 2

R
normalization: fX (x) = 1
mean =
variance = 2
standard normal r.v.: = 0, = 1

(x)2
2 2

Moment Generating Function

I

MGF of random variable X :

X (t) = E[e tX ]

Moments
I
I
I

0X (t) = E[Xe tX ] and 0X (0) = E[X ]

00X (t) = E[X 2 e tX ] and 00X (0) = E[X 2 ]

FX X

Example: exponential X

Joint PDF
I

X and Y are jointly continuous with joint density fX ,Y (x, y ) if

ZZ
P[(X , Y ) B] =
fX ,Y (x, y ) dx dy
B

Interpretation
P[x X x + , y Y y + ] fX ,Y (x, y ) 2

Properties
I
I

fRR
X ,Y (x, y ) 0 for all x and y

f (x, y ) dx dy = 1
X ,Y

Example: Two people are meeting for lunch. Each arrives

between 12PM and 1PM. All pairs of arrival times are equally
likely. The first to arrive will wait for 15 minutes. What is the
probability that they meet.

Joint PDF
I

Expectation
ZZ

E[g (X , Y )] =

g (x, y ) fX ,Y (x, y ) dx dy

Marginal densities
Z

fX (x) =

fX ,Y (x, y ) dy
Z

fY (y ) =

fX ,Y (x, y ) dx

Exercise: Let (X , Y ) be uniformly distributed over the disc of

radius 1 centered at the origin. Find: (i) joint density of X
and Y , and (ii) marginal density of X .

Joint CDF
I

Joint CDF for continuous X and Y

Z

FX ,Y (x, y ) = P[X x, Y y ] =

fX ,Y (u, v ) du dv

Then

... and

2
FX ,Y (x, y ) = fX ,Y (x, y )
x y

FX (x) = FX ,Y (x, )

FY (y ) = FX ,Y (, y )

Exercise: Suppose fX ,Y (x, y ) = c(x + y ) for 0 x + y 1,

x 0 and y 0. Show: (i) c = 3, (ii) P[X > Y ] = 1/2, (iii)
fX (x) = 3(1 x 2 )/2 for 0 x 1, (iv) P[Y < 1/2] = 11/16.

Independence
I

Two jointly continuous random variables are independent if

fX ,Y (x, y ) = fX (x) fY (y )

for all x, y

Conditional density
I
I
I

P[x X x + , y Y y + ] fX ,Y (x, y ) 2
P[x X x + ] fX (x)
Would like to have
P[x X x + | Y = y ] fX |Y (x|y )

Thus
fX |Y (x|y ) =

fX ,Y (x, y )
fY (y )

Example: Let (X , Y ) be uniformly distributed over the disc of

radius 1 centered at the origin. Find fX |Y (x|y ).

Derived Distributions
I

Z = g (X ) or Z = g (X , Y )

Distribution of Z

X , Y discrete:
P[Z = z] =

pX ,Y (x, y )

x,y : z=g (x,y )

I

X , Y continuous:
I
I

Find the CDF of Z

Differentiate FZ (z) to get fZ (z)

Examples
I
I

Let X be uniform on [0, 2]. Find the PDF of Y = X 3 .

X , Y are uniformly distributed in the square [0, 1] [0, 1]. Find
the PDF of Z = Y /X .

Sums of R.V.s
I

Independent X and Y

Discrete X and Y
P[X + Y = z] =

P[X + Y = z | X = x] P[X = x]

P[Y = z x] P[X = x]

x
I

Poisson r.v.s

Continuous X and Y
Z
fX +Y (z) =

fX +Y |X (z|x) fX (x) dx
Z

Uniform r.v.s

fY (z x) fX (x) dx

Example

X exponential with rate

Y exponential with rate

B Bernoulli, P[B = 0] = / (0, 1)

X , Y and B are independent

Z = X + BY

Two ways to solve

Poisson Process

Model
I
I

Would like to model (random) arrivals

Counting process
I
I
I

Nt 0
Nt is integer valued
Nt is non-decreasing

Applications:
I
I
I
I
I
I

Transportation
Manufacturing
Service industry
Supply chains
Healthcare
...

Data

Arrival
Rate

May 1959!

Time
24 hrs

(Lee A.M., Applied Q-Th)

% Arrivals

Dec 1995!

Time
24 hrs
(Help Desk Institute)

Bernoulli Process
I

Start with discrete time

Simplest stochastic process

An i.i.d. sequence of 0s and 1s

I
I

Xn = 1 arrival at time n
Xn = 0 no arrival at time n

P[Xn = 1] = 1 P[Xn = 0] = p [0, 1]

Sample space = {(X1 , X2 , . . .) : Xi {0, 1}}

3 Quantities of Interest
I

Nn number of successes in n trials

 
n k
P[Nn = k] =
p (1 p)nk
k
I

Yk time of the kth success



n1 k
P[Yk = n] =
p (1 p)nk
k 1
I
I

ENn = np and Var(Nn ) = np(1 p)

EYk = k/p and Var(Yk ) = k(1 p)/p 2

not independent

Tk = Yk Yk1 the kth interarrival time

I
I

geometric
i.i.d.

Definition

Independent increments
The numbers of arrivals in disjoint time intervals are
independent

Stationary increments
The distribution of the number of arrivals in any interval of
time depends only on the length of the interval

Poisson Process

Counting process: Nt number of arrivals in [0, t]

Continuous-time version of Bernoulli process

Simplest continuous-time counting process

Most relevant

One parameter: rate

Multiple (equivalent) definitions

Definitions
I

A definition: {Nt , t 0} is a Poisson process if

I
I
I

N0 = 0
stationary and independent increments
for 0

1 , k = 0
P[N = k] ,
k=1

0,
k 2

Another definition: {Nt , t 0} is a Poisson process if

I
I

N0 = 0
stationary and independent increments

P[Nt = k] =
I

(t)k t
e
k!

Yet another definition: {Nt , t 0} is a Poisson process if

I
I

N0 = 0
interarrival times are i.i.d. exponential with rate

Bernoulli/Poisson Relationship

Partition [0, t] in m intervals of length t/m

Bernoulli process: pm = t

... as 0
 
n k
p (1 p)nk
k

Poisson Process
I

Arrival rate :
ENt =

X
X
(t)i
(t)i t
e
= e t
i
i!
(i 1)!
i=0

i=1

= t e t

X
(t)i
i=0

= t

Example: You receive e-mail according to a Poisson process at

a rate of = 0.4 messages per hour. You check email every
30 minutes.
I
I

i!

P[no new message] = e 0.40.5 = e 0.2

P[one new message] = 0.2 e 0.2

Interarrival times: Ti
P[Ti > t] = P[no arrivals in (Yi1 , Yi1 + t]] = P[Nt = 0] = e t

Exponential Distribution

fX (x) = e x , x 0

Tail probability:
P[X > x] = e x

Memoryless property (t, s 0):

P[X > t + s | X > t] =

P[X > t + s]
= e s = P[X > s]
P[X > t]

Example

Poisson catches: Poisson process = 0.6 fish/hour.

Fish for 2 hours. If no catch, continue until the first catch.

I
I
I
I
I
I

P[fish for more than 2 hours] =

P[fish for more than 2 hours, but less than 5 hours] =
P[catch at least 2 fish] =
E[number of fish] =
E[future fishing time | fish for 4 hours] =
E[total fishing time] =

Arrival Times
I

Arrival times: Yk time of the kth arrival

Approach one:
I

{Yt k} = {Nt k}
P[Nt n] =

X
(t)i
i=k

e t

Approach two:
I
I

P[y < Yk y + ] fYk (y )

{y < Yk y + } =
fYk (y ) =

i!

Approach three:

k y k1 y
(y )k1 y
e
=
e
(k 1)!
(k 1)!

Merging of Poisson Processes

I

Two independent Poisson processes with rates 1 and 2

Merge the two processes

Focus on a small interval of length

I

P[0 arrivals in the merged process] (1 1 )(1 2 )

1 (1 + 2 )
P[1 arrival in merged process]
1 (1 2 ) + 2 (1 1 ) (1 + 2 )
P[1 arrival in process #1|1 arrival] =
P[1 arrival in process #1]/P[1 arrival] 1 /(1 + 2 ) =
1 /(1 + 2 )
(1)

Consider interarrival times: T1

(1)

(2)

(1)

P[min{T1 , T1 } > t] = P[T1

I

Sum of two Poisson r.v.s

(2)

and T1

(2)

> t] P[T1

> t] = e 1 t e 2 t

Examples

Bulbs
I
I

Each bulb has independent exponentially distributed lifetime

Install 3 light bulbs. Find expected time until the last light
bulb dies out

2 2 switch
I
I

Exponential weights
MaxWeight scheduling

Splitting of Poisson Processes

I
I

Poisson process with rate

For each arrival:
I
I

assign it to process #1 with probability p

assign it to process #2 with probability 1 p

Processes #1 and #2 are independent Poisson processes

Justification
(1)

P[Nt
I

(2)

= i, Nt

= j] =

Consider interarrival times:

(1)

T1

K
X
i=1

Ti

Examples

Infinite-server model
I
I

Poisson arrivals with rate

Independent holding times with distribution G

Highway encounters
I
I
I
I

One-way highway distance d

Poisson arrivals of cars
Random speed for each car distribution G
Select speed of a given car to minimize the number of
encounters

Conditional Arrivals
I

Suppose {Nt = n}

Distribution of arrivals on [0, t]?

Easier case: {Nt = 1}

P[T1 x | Nt = 1] =

General case: {Nt = n}

fY1 ,...,Yn |{Nt =n} (t1 , . . . , tn ) =

Simulation

Infinite-server model

Random Incidence for Poisson Process

Look at Poisson process that has been running forever

Show up at some random time (random incidence)

Residual life = time until the next arrival

Age = time from the last arrival

Duration = Age + Residual life

Distribution of the duration?

Large Population Limit

I
I

Poisson process = infinite-source model

T with general distribution G
I

nth model
I
I

ET = 1/

n sources
a source characterized by T (n) = nT

Residual life T (n) :

P[T (n) t] =

1
ET (n)

Z
0

P[T (n) > u] du

Markov Chains

Stochastic Process
I
I

Discrete time: n = 0, 1, . . .
Simplest example
I
I
I

Slightly more complicated

I
I

I
I

Bernoulli process: an i.i.d. sequence of 0s and 1s

P[Xn = 1] = 1 P[Xn = 0] = p [0, 1]
0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0 . . .

{Xn } i.i.d. with Xn E

... no memory

Incorporate memory
Example
I
I
I

Daily temperatures
70, 72, 71, 74, 75, ?
Simplest way to take into account the past: tomorrow depends
only on today

Markov Chain
I

Definition
I
I
I

Stochastic process X = {Xn , n = 0, 1 . . .}

Countable state space E : Xn E
X is called a Markov chain provided that
P[Xn+1 = j | X0 = i0 , X1 = i1 , . . . , Xn = i] = P[Xn+1 = j | Xn = i]

I
I

for all ik , i, j E and n = 0, 1, . . .

The future is independent of the past given the present
X0 either given or random

Time-homogeneous Markov chain

P[Xn+1 = j | Xn = i] = P(i, j) = pi,j
for all n

Matrix P describes the chain

Transition Matrix
I

P is a Markov matrix if
I
I

P(i,
P j) 0 for any i, j E
jE P(i, j) = 1 for each i

Example: Coins flips

Example
I

I
I
I

Jack is taking a probability class. Each week he can be either

up-to-date or he may have fallen behind. If he is up-to-date in
a given week, the probability that he will be up-to-date in the
next week is 0.8. If he is behind in a given week, then he will
be up-to-date in the next week with probability 0.6.
P=?
1 = up-to-date, 2 = behind
Graph representation

Example
I

Number of successes in independent trials

I
I

Nn = number of successes in n trials; N0 = 0

P[success] = p

Is {Nn } a MC? If so, P = ?

E = {0, 1, 2, . . .}

Markov property
P[Nn+1 = j | N0 = 0, N1 = i1 , . . . , Nn = i] =?

Matrix P
P(i, j) = P[Nn+1 = j | Nn = i] =?

Memory

Markov chain X on E
X0 , X1 , X2 , X3 , X4 , . . . , Xn , . . .

Suppose Xn+1 depends both on Xn and Xn1

X not a Markov chain

Define Yn = (Xn+1 , Xn )

Y is a MC

n-step Transition Probabilities

I

P describes one-step transitions

n-step transition probabilities

ri,j (n) = P[Xn = j | X0 = i]

ri,j (1) = pi,j ,

ri,j (2) = P[X2 = j|X0 = i] =

P[X2 = j, X1 = k|X0 = i]

pi,k pk,j = P 2 (i, j)

(not (P(i, j))2 )

k
I

In general
ri,j (n) =

X
k

ri,k (n 1) pk,j = P n (i, j)

Example

An urn always contains 2 balls

Red or blue
A ball is randomly chosen and replaced by a new ball

I
I

same color p probability

different color (1 p) probability

Initially both balls are red

Probability the kth ball selected is red?

Example

Markov chain with E = {1, 2, 3, 4, 5}

Find
P[X4 = 2, X3 2, X2 2, X1 2 | X0 = 1]

Example

E = {1, 2},

P=

Initial condition: X0 = 1

r1,1 (n)
r1,2 (n)


0.5 0.5
0.2 0.8

n=0
1
0

n=1
0.5
0.5

Does ri,j (n) converge?

n=2
0.5 0.5 + 0.5 0.2 = 0.35
0.5 0.5 + 0.5 0.8 = 0.65

n=3
0.305
0.695

Examples
I

Example 1

0 1 0
P = 0.5 0 0.5
0 1 0
I

r2,2 (n) does not converge

Example 2

1
0
0
0
0.3 0.4 0.3 0

P=
0
0 0.5 0.5
0
0 0.5 0.5
I

r1,1 (n) = 1, r3,1 (n) = 0, r2,1 (n) 1/2

Convergence results depend on the structure of P

Classification of States

State j is accessible (reachable) from state i if P n (i, j) > 0 for

some n 0

Two states communicate if they are accessible to each other

Two states that communicate are in the same class

Markov chain is irreducible if there is only one class

State i is recurrent if starting from i the process returns to

state i with probability 1

In the state is not recurrent, then it is called transient

Recurrent States
I

State i is
I
I

P
recurrent if P P n (i, i) =
transient if
P n (i, i) <

In a finite-state MC not all states can be transient

If state i is recurrent and communicates with state j, then

state j is also recurrent
P m+n+k (j, j) P m (j, i) P n (i, i) P k (i, j)

Recurrent class: A collection of recurrent states that

communicate with each other and no other state

Example

Recurrent states:

Transient states:

Recurrent classes:

P[X1 = 2, X2 = 6, X3 = 7|X0 = 1] =

P[X4 = 7|X0 = 2] =

Classification

I
I

Transient
Recurrent
I
I

Positive (expected return time < )

Null (expected return time = )

Positive/null recurrence is a class property

Finite MC

Periodic States
I

States in a recurrent class are periodic if they can be grouped

in d > 1 groups so that all transitions from one group lead to
to the next

Tj be the time of a visit to j after 0

P[Tj = n for some n 1 | X0 = j] = 1

Example
5

6
1

4
3
I

Groups: {1, 2, 5, 8} and {2, 4, 6, 7, 9}

8
9

Steady-State Probabilities
I

Q: Do ri,j (n) converge to some j independent of i?

Yes, if
I
I
I

irreducible chain
positive recurrent
no periodicity (aperiodic)

Start with
ri,j (n) =

ri,k (n 1) pk,j

k
I

Take n : j =

k pk,j or
= P

where = [1 , 2 , . . .]
I

Additional equation
X
k

k = 1

Example

Example



0.5 0.5
P=
0.2 0.8
Then

1 = 0.51 + 0.22
2 = 0.51 + 0.82

1 + 2 = 1

Thus 1 = 2/7, 2 = 5/7

I

Does it make sense?

Interpretations
I

i
I
I
I

probability of being in i after a long time

long-run fraction of time the chain spends in state i
steady state of the chain
I
I

if the chain is in state at time n

then the chain is in state at time n + 1

reciprocal of the expected time between visits to state i

= P
I
I
I
I

eigenvector
one column of P is redundant
need an extra equation to find
frequency interpretation
I
I
I

k frequency of being in k
k pk,j frequency or transitioning from k to j
P
k k pk,j frequency of entering j

Birth-Death Processes
I

Special structure of P
p0

1p0

p1

q1

= P works

Local balance:

p2

q2

q3

i pi = i+1 qi+1
I

pm1

p3

Example: pi = p, qi = q, p + q < 1:
i = 0 (p/q)i
(p/q)i
i = Pm
j
j=0 (p/q)

qm1

qm

1qm

Example
I

For each of the chains below determine if each statement is

true or false
(i) ri,j (n) converges to a limit as n for each (i, j)
(ii) statement (i) is true and ri,j (n) j for each (i, j)
(iii) statement (iii) is true and j > 0 for each state j

Example

Theorem
Suppose {Xi } is irreducible and aperiodic. Then all states are
recurrent positive if and only if the system of linear equations
X
j =
i P(i, j)
iE

i = 1

iE

has a solution . If there exists a solution , then it is stricly

positive, there are no other solutions, and
j = lim P n (i, j).
n

Examples

Linear list & transposition rule

Random walk
I
I

E = {0, 1, 2, . . .}
Transition matrix

q
q

P=

p
0
q

p
0

Absorption Probabilities
I

Markov chain

Each state is either absorbing or transient

Fix an absorbing state s

ai probability of reaching state s, staring from i

Then
as = 1
ai = 0 for all absorbing i 6= s
X
ai =
aj P(i, j) for all transient i
j

Minimal solution

Example: Random walk

Expected Time to Absorption

I

Expected number of steps until a recurrent state is entered

Let
i = E[min{n 0 : Xn is recurrent } | X0 = i]

Then
i = 0

for all recurrent states i

X
i = 1 +
j P(i, j) for all transient states i
j

Minimal solution

Example: Random walk

Expected Time Spent in Transient States

I

Let

"
S(i, j) = E

#

1{Xk =j}  X0 = i

k=0

and Q be a sub-matrix of P specifying transitions between

transient states.

Then
S = (I Q)1

Example

Continuous-Time Markov Chains

CTMC
I

Definition
I
I
I

Stochastic process X = {X (t), t 0}

Countable state space E : X (t) E
X is called a CTMC provided that
P[X (t + s) = j | X (s) = i, X (u) = x(u), 0 u < s]
= P[X (t + s) = j | X (s) = i]

for all s, t 0 and i, j, x(u) E

The future is independent of the past given the present

Time-homogeneous Markov chain:

P[X (t + s) = j | X (s) = i] = Pt (i, j)
for all t, s 0

Transition Function
I
I

Pt (i, j)
Properties
I
I
I

Pt (i, j) 0
P
PjE Pt (i, j) = 1
kE Pt (i, k) Ps (k, j) = Pt+s (i, j)

Example
I
I

Poisson process {Nt , t 0}

Markov property
P[Nt+s = j | Ns = i, Nu = x(u), 0 u < s] = P[Nt+s = j | Ns = i]

Transition function
(
Pt (i, j) =

0,
(t)ji
(ji)!

j <i
e

, j i

Time Spent in a State

I

How to specify/simulate a CTMC?

Let Wt be the length of time the process X remains in state

being occupied at time t

Then
P[Wt > u + v | Xt = i] = P[Wt > u, Wt+u > v | Xt = i]
= P[Wt > u | Xt = i] P[Wt+u > v | Xt+u = i]

Hence
P[Wt > u | Xt = i] = e i u
for u 0 and some i [0, ].

A Way to Specify CTMC

I

Two components
I
I

A vector of rates
A stochastic matrix Q with Q(i, i) = 0

Example: Consider a shoe-shine shop consisting of two chairs,

Chair 1 and 2. Upon arrival, a customer goes initially to Chair 1
where his shoes are cleaned and polish is applied. When done, the
customer moves to Chair 2 where the shoes are buffed. The service
times at the two chairs are independent exponential random
variables with rates 1 and 2 . Potential customers arrive according
to a Poisson process with a rate and enter the shop only when
both chairs are empty.

Example

M/M/1

Single server

Infinite waiting room

Poisson arrivals

Exponential service times

A Different Way to Specify CTMC

I

In state i:
I
I
I
I

Independent exponential timers

Timer (i, j) rate is i Q(i, j)
The process jumps once a timer expires
The process jumps from i to j 6= i with rate
R(i, j) = i Q(i, j)

Define R(i, i) = i , then

X
jE

R generator matrix

Example: M/M/1

R(i, j) = 0

Transition Functions
I

Start with
Pt+h (i, j) =

Pt (i, k) Ph (k, j)

k
I

Consider h 0 to obtain
d
Pt = Pt R = RPt
dt

Solution:
Pt = e tR
where
e tR =

n
X
t
n=0

n!

Rn

Example: Single machine/repairman

Steady State
I

dPt /dt = 0

If the process is irreducible recurrent:

R = 0
X
i = 1

Interpretations
I

Limit as t :
j = lim P[X (t) = j | X (0) = i]
t

I
I

I
I

Fraction of time spent in a particular state

Steady-state:
Pt =
Balance equations
Time discretization

Examples

M/M/1

M/M/

Queueing Theory

Queueing Theory
I

Queue = line

Many processes require that people or items wait for service

Questions

I
I
I
I

How much time spent in line?

What proportion of time a server is idle?
What is the average number of people/items waiting?
What is the probability distribution of the number of
customers?
What is the probability distribution of the waiting time?

Arrival process: arrivals are called customers

Service process: service time distribution

Queueing discipline: rules for order of service

I
I
I

first come, first served (FCFS)

last come, first served (LCFS)
service in random order (SIRO)

Queueing System Notation

I

Notation 1/2/3/4/5

First parameter: arrival process

I
I
I

M = i.i.d. exponential interarrival times

D = identical deterministic interarrival times
G = i.i.d. interarrival times with a general distribution

Second parameter: service times

Third parameter: number of parallel servers

Fourth parameter: max allowable customers in the system

Fifth parameter: service discipline

Example: M/M/1 exponential interarrival and service times

with a single server

M/M/1/n
I

Arrival rate and service rate

How to track system state over time?

Need a state of the system at time t

N(t) number of customers in the system at time t

{N(t), t 0} continuous time process how to analyze?

M/M/1/n
I

Arrival rate and service rate

How to track system state over time?

Need a state of the system at time t

N(t) number of customers in the system at time t

{N(t), t 0} continuous time process how to analyze?

CTMC
Transition rates

I
I

R(i, i + 1) =
R(i, i 1) =

M/M/1/n
I

Equations for steady-state probabilities:

0 = 1
1 = 2
...
n1 = n

Let = /

Then i = 0 i and
01

n
X

i=0
I

i fraction of time there are i customers in the system

(1 0 ) fraction of time the server is busy

Littles Theorem
I

Fundamental theorem for queueing systems

Let
I
I
I

N(t) # of customers in the system at time t

(t) # of customers who arrived in [0, t]
Ti time spend in the system by the ith arriving customer

Time averages (as t )

Z
1 t
N(u) du EN
Nt =
t 0
(t)
t =

t
P(t)
Ti
Tt = i=1
ET
(t)

Littles Theorem
I

Theorem: EN = ET

High ET implies high EN

Proof (simplified version: N(0) = 0 and FIFO):

I
I
I

Let (t) be the number of departures in [0, t]

N(t) = (t) (t)
Then
(t)
X

(t)
t
I

i=1
P(t)
i=1 Ti

(t)

Z
Ti

N(u) du
0

(t)
X

Ti

i=1

Z
0

(t)
N(u) du
t

As t
ET EN ET

P(t)

i=1 Ti
(t)

M/M/1
I

Balance equations
0 = 1
1 = 2
...

Thus i = i 0 and the solution exists only of

i <

i=0
I

If < 1 then the chain is recurrent positive,

1
0 = P

i
i=0

=1

and the number of customers in the system is geometric

PASTA

Under FCFS, it is important what arriving customers

encounter

Arriving customers do not necessarily see time averages

Consider a D/D/1 system

Poisson Arrival See Time Averages (PASTA)

For M/M/1: i probability that an arriving customer sees

i customers in the system

Back to M/M/1
I

Stationary number in the system: P[N = i] = i = (1 ) i

Average number of customers in the system

EN =

ii =

i=0
I

Average number of customers awaiting service

EK =

(i 1)i = EN (1 0 ) =

i=1
I

Average time spent in the system

ET =

1X
1
(i + 1) i =

i=0

2
1

Back to M/M/1

Average time spent waiting

EW =

1X
1
ii =

1
i=0

Littles law
I
I
I

EN = ET
EK = EW
1 0 = /

Multiple Servers
I

M/M/2
I
I
I

Analysis
I
I
I
I
I

FCFS
2 identical servers (service rate )
work-conserving
Similar to M/M/1
Consider the number of customers in the system
Transition rates
Balance equations
System stable only if < 2

Easy to generalize for > 2 servers

Example
I

System
I
I
I

Analysis
I
I
I
I

Poisson arrivals (rate )

One exponential server (rate )
If the server is idle, 2 customers are need for the server to start
working
If the server is busy, it services customers until the system is
empty
What is different when compared to M/M/1?
State of the system?
Transition probabilities
Steady-state probabilities

Suppose the cost of processing a customer is $x, and the cost

of staring the server up is $y . How much the operator needs
to charge per customer in order to break even in the long run?

Example

I
I

Finite source model

Machine repair model
I
I
I

K machines, R repair people

Machines breaks after exponential random times (with rate )
It takes an exponential amount of time (with rate ) to fix a
machine

Analysis
I
I
I
I

State: number of broken machines

Transition rates
Steady-state probabilities
Average arrival rate to the repair facility

Exponential Queues in Series

k stages of service

Poisson arrivals to stage 1

Exponential service at stage k with rate k

Infinite waiting rooms

Arrival processes to all stages are Poisson

Product form solution

Example: /M/1 in tandem

Open Network of /M/1 Queues

I

k nodes

Single server + infinite waiting room

Exogenous Poisson arrivals to node i with rate ri

Exponential service times at node i with rate i

Ni (t) the number of customers at node i at time t

Customer routing:
I
I
I
I

probabilistic routing
move from node i to node j with probability pij
P
after node i depart the network with probability 1 j pij
every customer eventually leaves the system

Example

Open Network of /M/1 Queues

I

i total arrival rate to node i

Flow conservation equations:

i = ri +

j pji

Stability:
i =

i
<1
i

Theorem:
k
Y
lim P[N1 (t) = n1 , . . . , Nk (t) = nk ] =
(1 i )ni i

Arrivals are not always Poisson! Example

i=1

Economies of Scale

Efficiency vs. Quality tradeoff

M/M/1

Dynamic Programming

Dynamic Programming

A technique for solving certain types of optimization problems

Idea: break up a large problem into smaller ones

Usually applied to problems which sequential decisions

Dynamic program
I
I

shortest path problem in an acyclic network

how to construct the network?

Motivating Application: Shortest Path

I
I

Travel from coast to coast drive from NYC (1) to LA (10)

The structure of stopovers is as follows
I
I
I
I

In 1 day can reach Columbus (2), Nashville (3) or Louisville (4)

On day 2 can reach Kansas City (5), Omaha (6) or Dallas (7)
On day 3 can reach San Antonio (8) or Denver (9)
On day 4 can reach LA

Let ci,j be the distance between cities i and j

Minimize the route length

Shortest Path
I

Classify the cities: All cities that can be reached in n days are
stage n cities

Start with easy problems work backward

For t-stage city i, let ft (i) be the shortest path length to LA

Would like to find f0 (1)

Sequential solution procedure

I
I
I
I

Find the shortest path to LA from cities 1 day away from LA

Find the shortest path to LA from cities 2 day away from LA
...
Find the shortest path to LA from NYC

In general
ft (i) =

min

j: j is a t + 1 city

ci,j + ft+1 (j)

Why?
I

Computational efficiency

Can enumerate all paths example

I
I
I
I

destination at stage 6
5 choices for stages 1-5
55 paths in total
Computational cost: 56 additions and (55 1) comparisons

Dynamic programming
I
I

Last stage trivial

Other stages for each node: 5 additions (potential choices for
the next hop) and 4 comparisons
Total: 4 5 5 + 5 = 105 additions and 4 5 4 + 4 = 84
comparisons

Characteristics of Dynamic Programming

I

The problem should have stages each stage corresponds to a

point at which a decision needs to be made

Each stage should have a number of associated states The

state contains all information that is needed to make an
optimal decision for the remaining problem

The decision chosen at each stage describes how the state at

the current stage is transformed in the state at the next stage

The optimal decision at the current state should not depend

on previously visited states or previous decisions

There must be a recursion that relates the cost or reward for

stages t, t + 1, . . . , T to the cost or reward for stages
t + 1, t + 2, . . . , T . This recursion formalizes the procedure
of working backwards from the last stage to the first stage

Four Applications

Shortest path

Production and inventory planning

Resource allocation

Equipment replacement

Production and Inventory Planning

I
I

Single item
Planning period of T periods
I
I

the demand for the item in each of the periods is known

the initial inventory level is known

At the start of each period decide how many items to

produce; production capacity is limited

Each periods demand must be met on time

Limited storage space

Minimize the total production and inventory cost over the

planning horizon

Periodic review model

Production and Inventory Planning: Variables

I
I

Demand in period t (t = 1, . . . , T ): dt
Cost of producing x units in period t: ct (x)
I

Cost of storing inventory I at the end of period t: ht (I )

I

often the cost is independent of t, i.e., ct (x) = c(x)

often ht (I ) = h(I )

Examples
I
I

If c() and h() are linear, then LP

If the production costs have a fixed-charge structure:
(
0,
x =0
c(x) =
a + bx, x > 0
then mixed IP

Production and Inventory Planning: Approaches

I

NLP formulation
min

T
X

ct (xt ) +

t=1

T
X

ht (It )

t=1

s.t. It1 + xt = dt + It ,

t = 1, . . . , T

0 It B,

t = 1, . . . , T

0 xt C ,

t = 1, . . . , T

Dynamic programming
I
I
I

general cost functions

units of demand, inventory and production must be integers
can be efficient of the magnitude of numbers involved is not
large

Production and Inventory Planning: DP

I

Stages: t = 1, . . . , T (time)

States: I = 0, . . . , B (starting inventory level)

Decisions: x = 0, . . . , C (production quantity)

ft (I ) minimum cost from start of stage t

... looking for f1 (I0 )

Recursion
I

Cost at the beginning of stage T :

fT (I ) = min {cT (x) + hT (I + x dT )}
0xC

I
I

Want to end up with 0 inventory at the end: x = dT I .

Thus
fT (I ) = cT (dT I )

Production and Inventory Planning: DP

Recursion
I

ft (I ) =

Cost at the beginning of stage t:

min

max(0,dt I )xmin(C ,dt +BI )

I

{ct (x) + ht (I + x dt ) + ft+1 (I + x dt )}

Sufficient storage capacity:

I + x dt B

Demand met:
I + x dt 0

Production and Inventory Planning: Example

I

Setup
I
I
I
I

T = 4 periods
Demands: 1,4,2,3
Inventory holding cost: $0.50 per unit
Production costs:
I
I

I
I

fixed setup cost: $3

variable cost: $1 per unit

Production capacity C = 5 units

Inventory capacity B = 4 units

Initialization: T = 4
I
I

d4 = 3
Cost at the beginning of stage T = 4:
f4 (I ) = c4 (d4 I )

f4 (0) = 6, f4 (1) = 5, f4 (2) = 4, f4 (3) = 0, f4 (4) =?

Production and Inventory Planning: Example

I

Next stage: t = 3
I
I

d3 = 2
Cost at the beginning of stage t = 3:

f3 (I ) =
I

min
max(0,2I )xmin(5,6I )

{c3 (x) + h3 (I + x 2) + f4 (I + x 2)}

I = 0:
f3 (0) = min {c3 (x) + 0.5 (x 2) + f4 (x 2)}
2x5

= min {2 + 1.5 x + f4 (x 2)} = 9.5

2x5

(x = 5)

I = 2:

f3 (2) = min {c3 (x) + 0.5 x + f4 (x)}

0x4


= min f4 (0), min {3 + 1.5 x + f4 (x)} = 7.5
1x4

(x = 3)

Production and Inventory Planning: Representation

Network representation

Nodes: stage/state pairs (t, I )

Arcs: decisions x

Arc from (t, I ) corresponding to decision x leads to

(t + 1, I + x dt )

Cost of this arc is ct (x) + ht (I + x dt )

Resource Allocation
I

Stockco is considering n investments

I

I
I

Investment i yields a NPV of ri (di ) when di $1,000 is

invested
Invest only in multiples of $1,000
Budget of B $1,000

NLP formulation:
max

n
X

ri (di )

i=1

s.t.

n
X

di B

i=1

di {0, 1, . . .}

Resource Allocation

Example
I
I
I
I

n = 3, B = 6
r1 (d1 ) = 7d1 + 2 (d1 > 0), r1 (0) = 0
r2 (d2 ) = 3d2 + 7 (d1 > 0), r1 (0) = 0
r3 (d3 ) = 4d3 + 5 (d1 > 0), r1 (0) = 0

DP formulation
I
I
I
I
I

Stages: i = 1, 2, 3 (investment categories)

States: y = 0, 1, . . . , 6 (budget available)
Decision: d = 0, 1, . . . , 6 (investment amount)
Recursion: fi (y ) (maximal return from inv. categories)
Goal: f1 (6)

Resource Allocation
I

Return from investment in category 3 only:

f3 (y ) = max r3 (d)
0dy
(
0,
y =0
= r3 (y ) =
4y + 5, y = 1, . . . , 6

Return from investment in categories 2 and 3:

f2 (y ) = max {r2 (d) + f3 (y d)}
0dy

I
I
I

f2 (0) = 0
f2 (1) = max{r2 (0) + f3 (1), r2 (1) + f3 (0)} = 10
f2 (2) = max{r2 (0) + f3 (2), r2 (1) + f3 (1), r2 (2) + f3 (0)} = 19

Resource Allocation: Representation

Network representation

Nodes: stage/state pairs (i, y )

Arcs: decisions d

Arc from (i, y ) corresponding to decision x leads to node

(i + 1, y d)

Reward on this arc is ri (d)

Equipment Replacement Problem

I

A company faces the problem of how long a machine should

be utilized before it should be traded in for a new one

Example
I

I
I

A new machine costs p =$1,000 and has a useful lifetime of 3

years
Maintaining a machine during the first 3 years costs m1 = $60,
m2 = $80 and m3 = $120, respectively
If a machine is traded in, a salvage value is obtained: s1 =
$800, s2 = $600 and s3 = $500, respectively, after the first 3
years
We currently have a y year old machine
Find a policy that minimizes total net costs over the next 5
years

Equipment Replacement Problem

I

DP formulation
I
I
I
I
I

I
I

Stages: t = 0, 1, . . . , 5 (time)
States: y = 0, 1, 2, 3 (age of machine)
Decisions: d = 0, 1 (keep or trade-in)
Recursion: ft (y ) (minimal net cost after period t)
Goal: f0 (y )

Always salvage at the end of year 5: f5 (y ) = sy , y = 1, 2, 3

At the end of period t < 5: keep or trade in
I

If y = 3 must trade in: ft (3) = s3 + p + ft+1 (1),

t = 0, 1, , 4
If y < 3 choice
(
my + ft+1 (y + 1), x = 0
ft (y ) = min
sy + p + ft+1 (1), x = 1
y = 1, 2; t = 0, 1, . . . , 4

Secretary Problem
I

There is a single secretarial position to fill.

There are n applicants for the position.

The applicants, if seen altogether, can be ranked from best to

worst unambiguously.

The applicants are interviewed sequentially in random order,

with each order being equally likely.

Immediately after an interview, the interviewed applicant is

either accepted or rejected, and the decision is irrevocable.

The decision to accept or reject an applicant can be based

only on the relative ranks of the applicants interviewed so far.

Objective:

Example
I

The owner of a lake must decide how many bass to catch and
sell each year.

If she sells x bass during year t, a revenue rt (x) is earned.

The cost of catching x bass during a year is a function

ct (x, b) of the number of bass caught during year t and of b,
the number of bass in the lake at the beginning of the year.

Bass reproduce in a non-deterministic manner so that, if s

bass are in the lake at the end of year t, then the number of
bass at the beginning of year (t + 1) will be Ms, where M is a
random number with the following PMF:
P[M = 1.2] = 0.4,

P[M = 1.5] = 0.4,

P[M = 2] = 0.2.

Maximize the owners net profits over T years assuming that

there are 104 bass in the lake currently.

Example
I

A supermarket chain has purchased 6 gallons of milk from a

local diary at $1 per gallon.

Each gallon is sold at the chains three stores for $2 per gallon.

Unsold milk at the end of the day can be sold back to the
diary at $0.50 per gallon.

The demand at each of the three stores (D1 , D2 , D3 ) are

distributed as follows:

P[D1 = 1] = 0.6

P[D1 = 2] = 0.0 P[D1 = 3] = 0.4

P[D2 = 1] = 0.5

P[D2 = 2] = 0.1 P[D2 = 3] = 0.4

P[D3 = 1] = 0.4

P[D3 = 2] = 0.3 P[D3 = 3] = 0.3

Maximize the expected profit

Infinite Horizon Problems

No terminal stage

What to do?

Markov Decision Processes (MDP)

Cost/Reward
I
I

Discounting
Averaging

MDP
I
I

State space S
Decision set D(i)
I

D(i) describes allowable actions when in state i S

Transition probabilities
I

state i S + decision d D(i) state

j with probability pi,j (d)
control the transition matrix
P[Xn+1 = j | X0 , d0 , . . . , Xn = i, dn = d] = pij (d)

Expected rewards
I

state i S + decision d D(i) expected reward/cost ri,d

Policy
I
I
I
I

A rule for selecting decisions

Stationary policy
Randomized/Deterministic
Optimal

Discounting
I

Discounting factor

Expected discounted reward:

"
#
X
V (i) = E
t rXt ,dt | X0 = i
t=0

Either
V (i) = max V (i)

or
V (i) = min V (i)

Policy Iteration
I

Stationary policy

One-step analysis:
V (i) = ri,(i) +

pi,j ((i)) V (j)

j
I

system of linear equations

Policy improvement:
V (i)

X
max ri,d +
pi,j (d) V (j)

dD(i)
j

Value Iteration

Consider a sequence of increasing intervals

V (t, i) = max expected discounted reward that can be earned

during t periods

X
pi,j (d)Vt1 (j)
V (t, i) = max ri,d +

dD(i)
j

V (0, i) = 0

Let t