Homework 6

PSTAT 160A FALL 2014

D UE T HURSDAY, D EC 4, 2014

(1) Let (Xn ) be a random walk on {0, 1, . . . , M } with two reflecting barrier at 0 and at M > 1. That is,
the transition matrix of (Xn ) is

p
if 0 < i < M and j = i + 1,

1 p if 0 < i < M and j = i 1,

Pi,j =

1
if i = M and j = M 1 or i = 0 and j = 1,

0
otherwise.
(a) Draw the transition graph for M = 3.
(b) Compute the invariant distribution for arbitrary M using the weighted graph method.
(c) Let T = inf {n 1 : Xn = 0} be the first return time to 0. Compute E [T | X0 = 0] and
E [T | X0 = 1].
Solution.
(a)
p
p
1
0

1
1p

2
1p

3
1

(b) Let the vertices of the graph be V = {0, . . . , M } and the edges E = {(i, i + 1) : i = 0, 1, . . . , M
1}, the weights w(i, i + 1) = i+1 . Then the transition probabilities of random walk on this weighted
i+1
graph are Qi,i+1 = i+i+1 if i = 1, . . . , M 1 and Q0,1 = QM,M 1 = 1. With = p/(1 p) P = Q. The
invariant distribution is
w(i, i + 1) + w(i, i 1)
P
= ci (1 + ), i = 1, . . . , M 1
i =
w(j,
k)
j,k
w(0, 1)
0 = P
= c,
j,k w(j, k)
w(M 1, M )
= cM ,
M = P
w(j,
k)
j,k
P
where c is the normalizing constant ( j,k w(j, k))1 .
(
M
2(M +1 1)
X
X
i
1
w(j, k) = 2
=
2M
i=1
j,k

6= 1,
= 1.

(c) We know from class that E [T | X0 = 0] = 1/0 . Since P [X1 = 1 | X0 = 0] = 1, that is, the first
0
.
step from the state leads to 1, E [T | X0 = 0] = E [T | X0 = 1] + 1 and E [T | X0 = 1] = 1
0
In this problem
(
1
6= 1,
M 1)
0 = 2(
1
= 1,
( 2M M
2( 1)
6= 1,
1
E [T | X0 = 0] =
2M
= 1,
( M
2 1
6= 1,
1
E [T | X0 = 1] =
2M 1 = 1,
1

where = p/(1 p).

(2) Let Z1 , Z2 , . . . be independent identically distributed, positive integer valued random variables.
These are the inter-arrival times of a bus service at a given bus stop. So, there is a bus at time 0,
the next bus arrives at Z1 , the next one at Z1 + Z2 and so on. Denote Un the time elapsed since the
last bus at time n and Vn the waiting time for the next bus at time n. The aim of this exercise is the
computation of the PMF Un + Vn for large n.
(a) What is the state space of the Markov chain Xn = (Un , Vn ), n 1?
(b) What are the transition probabilities of the Markov chain (Xn )n1 ?
(c) Is (Xn ) irreducible?
(d) What is the invariant distribution of (Xn ) in terms of the PMF of Z1 ?
(e) Compute
n
1X
P [Uk + Vk = j ] .
lim
n n
k=1
(f) If the law of Z1 is a so called non-lattice distribution, that is, the greatest common divisor of
{n : P [Z1 = n] > 0} is 1, then Xn is aperiodic. What is
lim P [Un + Vn = j ]

in this case?
Solution.
(a) If the set of possible values of Z1 is denoted by RZ then the set of possible states is
{(i, j) : i + j RZ , 0 i, 1 j } .
The reason for j 1 is that when the bus arrives, then U becomes 0 and V becomes the next interarrival time.
(b) When the process is in state (i, j) where j > 1 then the waiting time for the next bus is bigger then
1, so in the next time unit, the time elapsed since the last bus increases by one and the waiting time
until the next bus decreases by one unit. That is, from (i, j) the system can move only to (i + 1, j 1)
when j > 1. When the system is in the state (i, 1) then at the next time point a bus arrives, so the
system jumps to (0, Z) where Z is the next inter-arrival time. So the transition probabilities are

k = i + 1, ` = j 1 1
1
P(i,j),(k,`) = P [Z = `] j = 1, k = 0

0
otherwise.
(c) Yes, The states of the form (0, j) are all reachable with positive probability from any other state,
and if (i, j) is in the state space then it is reachable form (0, i + j) in i steps.
(d) A state of the form (0, `) can be reached in one step from any state of the form (k, 1) and the
transition probability does not depend on k, hence
X
0,` = P [Z = `]
k,1 .
k

On the other hand a state (i, j) with i > 0 can only be reached from (i 1, j + 1) in one step, so
X
i,j = i1,j+1 = = 0,i+j = P [Z = i + j ]
k,1 .
k

The sum of this probabilities is 1

X

`P [Z = `]

k,1 = 1

From this
X

k,1 = P

1
1
=
E [Z ]
` `P [Z = `]

From this we finally get

i,j =

P [Z = i + j ]
.
E [Z ]

where the distribution of Z is the common distribution of the inter-arrival times.

(e) According to the ergodic theorem
j

X 
X


1X
jP [Z = j ]
.
1(Uk +Vk =j)
ji,i =
E 1(U0 +V0 =j)  X0 = (k, `) k,` =
n kn
E [Z ]
i=1
k,`
Then the expectation which is
1X
jP [Z = j ]
P [Uk + Vk = j ]
n kn
E [Z ]
also has this limit.
(f) In this case the Markov chain is irreducible, aperiodic, and has an invariant distribution, so the
transition probabilities converge without taking time average, that is
P [Un + Vn = j ]

jP [Z = j ]
.
E [Z ]

(3) A particle performs a random walk on the non-negative integers {0, 1, . . .} as follows. At each step,
it moves right with probability p and left with probability q = 1 p, except that, if it is at 0 now,
it moves right with probability p and remains at 0 with probability q. This called a simple random
walk on D = {0, 1, 2, . . .} with a (not instantaneously) reflecting barrier at 0.
Let Xn be the particles position after the nth step. Then, (Xn )n0 is a Markov chain with state space
D.
(a) Write down the transition probabilities Pi,j = P [Xn+1 = j | Xn = i].
(b) Classify the states when p > q.
(c) Classify the states when p < q.
(d) Find the invariant distribution = (i )i0 when p < q.
(e) Let T = min {n 0 : Xn = 0}. Compute E [T | X0 = i] for i 1, when p < q.
Hint. What is the relationship between E [T | X0 = i] and E [T | X0 = 1]?
Solution.
(a)

j =i+1
p
Pi,j = 1 p j = i 1 or i = j = 0

0
otherwise
The chain is irreducible as each state communicates with 0.
3

(b) When p > q = 1 p then we can use a result from class. We have computed that the probability
that a random walk ever returns to zero when it is started from one is (1 p)/p < 1.
If the reflected random walk is starts from zero, then after a geometric time it reaches state 1, and
from state 1 it continues as random walk (without reflection) until it reaches 0, at this state it feels
again the effect of the reflection.
So there is a positive probability that the reflected random walk does not return to zero when it is
started from zero. It means that zero is a transient state, and since the chain is irreducible all states
are transient.
(c) When p < q = 1 p then we can apply the weighted graph method to get the invariant distribution. When the chain is irreducible, then the existence of the invariant distribution is equivalent
with the positive recurrence of the chain.
(d) The only modification needed compared to the class, is that there is a loop at zero in the graph. To
remind you, V = {0, 1, . . .}, E = {(0, 0)} {(i, i + 1) : i = 0, 1, . . .} and w(0, 0) = 1, w(i, i + 1) = i+1 .
Then if = p/(1p) < 1 then the sum of the weights is finite, and the random walk on the weighted
graph is the same as the reflected random walk defined above. So the invariant distribution is.
0 = c(w(0, 0) + w(0, 1))
i = c(w(i, i + 1) + w(i 1, i)),

i = 1, 2, . . .

where c is the normalizing constant. After substitution we get

i = ci (1 + ), i = 0, 1, 2, . . .
P
P
i
c is obtained from the condition i i = c(1 + )
i=0 = 1, that is
c(1 + ) = 1
and


i =

p
1
1p



p
1p

i
,

i = 0, 1, 2, . . .

(e) Let Mi = E [T | X0 = 0]. Then M0 is the mean return time to 0, therefore 0 = 1/M0 . So M0 =
Then we can write conditioning on the first step

1p
.
12p

M0 = (1 p) + p(1 + M1 ) = pM1 + 1
p
1
From this M1 = (M0 1)/p = (1 + 1=2p
1)/p = 12p
.
To get from state i to zero the random walk must through the states i1, i2, . . . 1. The average time
needed to reach i 1 from i is the same as the average time needed to reach 0 from 1. The reason is
that the transition probabilities from j + i to k + i is the same as from j to k as long as j > 0. So if
we denote by ti = E [time need to reach i 1 | X0 = i] then ti = M1 and Mi = ti + ti1 + + t1 =
i
iM1 = 12p
.

(4) Let (Xn )n0 be a Markov chain with state space D = {a, b, c, d, e} and transition matrix

0 0 0 0 1
1 0 0 0 0

1/2 1/2
0
0
0
P =

1/3 1/3 1/3 0 0

1/4 1/4 1/4 1/4 0
(a) Classify the states.
4

(b) Compute the invariant distribution = (a , b , c , d , e ).

(c) Let T = min {n 1 : Xn is b or c}. Compute

" T 1
#

X

E
f (Xn )  X0 = a .

n=0

for f with f (a) = 2, f (d) = 3, f (e) = 1.

(d) Define the reward function g by

2 if i = a and j = e,
g(i, j) = 3 if i = e and j = a,

0 otherwise.
Compute
n

1X
lim
g(Xk1 , Xk ).
n n
k=1
Solution.
(a) The chain is irreducible as the walk a e d c b a has positive probability. Since
it is a finite state Markov chain each state is positive recurrent. The period is 1, as a e a and
2
3
a e b a both has positive probability, so both Pa,a
and Pa,a
are positive.
(b) We have to solve the equations
1
1
1
a = b + c + d + e
2
3
4
1
1
1
b = c + d + e
2
3
4
1
1
c = d + e
3
4
1
d = e
4
e = a
1 = a + b + c + d + e

Using the special form of the equations

3
b = c =
2
4
c = d =
3
1
d = e =
4
e = a

1
a
2
1
a
3
1
a
4


1 = a + b + c + d + e = a

1 1 1
1+ + + +1
2 3 4


= a

Hence a = 12/37, b = 6/27, c = 4/37, d = 3/37, e = 12/37.

12 + 6 + 4 + 3 + 12
37
= a
12
12

(c) To compute the expectation in question, let Yn = Xmin(n,T ) . It is somewhat simpler if the two
absorbing states for Y , that is b and c, are considered as one new state b . Then the state space of Y
is {a, b , d, e} and its transition matrix is

0 0 0 1
0 1 0 0

Q=
1/3 2/3 0 0
1/4 1/2 1/4 0
Then with the convention that f (b ) = 0


" T 1
#
"
#


X
X


E
f (Xn )  X0 = a = E
f (Yk )  Y0 = a


n=0

The reason is that

Pwhen X hits b or c then Y hits b and remains there forever.
Denote mi = E [ k f (Yk ) | Y0 = i] for i = a, b , d, e. Then conditioning on the first step gives the next
system of equations

ma = f (a) + me
mb = 0
2
1
1
md = f (d) + ma + mb = f (d) + ma
3
3
3
1
1
1
1
1
me = f (e) + ma + mb + md = f (e) + f (d) + ma
4
2
4
4
3
So

1
1
3
ma = f (a) + me = f (a) + f (e) + f (d) + ma =
4
3
2

1
f (a) + f (e) + f (d)
4

and


#
"
#




X
3
1
21


E
f (Xn )  X0 = a = E
f (Yk )  Y0 = a = ma =
f (a) + f (e) + f (d) = .


2
4
8
n=0
k
" T 1
X

(d) To answer this question we can use the ergodic theorem for Markov chains.
n

X
X
1X
g(Xk1 , Xk ) =
E [g(X0 , X1 ) | X0 = i] i =
g(i, j)Pi,j i
lim
n n
i
i,j
k=1
= a g(a, e)Pa,e + e g(e, a)Pe,a =

12
1
33
12
21+
3 = .
37
37
4
37

Each problem is worth 25 points. Write your name at the top right corner of each page you submit.
6