Stochastic Processes 2011/2012

Assignment 2
1
Practical information
This is the last of two assignments you must pass to qualify for the written
exam in the course Stochastic Processes. You are allowed to carry out and
hand in your solution to the assignment in groups of no more than three
persons. The assignment must be handed in to Anders Tolver no later than
Thursday, January 5th, 2012 at 9:15 am.
Unless you have made a special arrangement with the lecturer assignments
handed in after the deadline above will not be considered and you will be
excluded from the exam. You may write the assignment in Danish or in
English according to your preference.
Requirements for passing the assignment
To pass the assignment you must answer questions 1.-14. below. You are
expected to argue for your answers by referring to relevant concepts and
results from the lectures and the text book. It is not sucient just to give
the result without providing an argument. If you do not pass the assignment
you will be asked to hand in selected parts of the assignment again. The
revised version of the assignment must be handed in no later than
Thursday, January 12th, 2012 at 9:15 am.
Only if you do a serious attempt to solve all of the questions 1.-14. when
you hand in your rst version of the solution to the assignment, you will be
allowed to hand in a revised version.
1
(2,0)
1

ED
1

(1,0)
1

ED
BC
2

(0,0)
1

(1,1)
(2,2)
1

(2,1)
3

Figure 1: Transition diagram of Markov chain in questions 1.-4.

We consider in questions 1.-4. the continuous-time Markov chain, {X(t)}
t0
S = {(0, 0), (1, 0), (1, 1), (2, 0), (2, 1), (2, 2)}
with transition diagram given by Figure 1. The initial distribution of the
Markov chain is given by P(X(0) = (0, 0)) = 1.
1. Write down the matrix, Q, of transition intensities and the transition
matrix, P, for the Markov chain of jumps. Find the communication
classes and classify each class as either recurrent or transient.
2. Find the expected number of visits to each of the transient states before
the Markov chain leaves the transient states forever.
[Hint: Compute (I P
0
)
1
where I is the identity matrix and P
0
is
the submatrix of P corresponding to the transient states.]
3. What is the expected time (=mean) it takes before the Markov chain
leaves the transient states?
[Hint: For any visit to state i the time spend in state i is exponentially
distributed with rate q
i
.]
4. Find the limiting distribution lim
t
P(X(t) = j) for any state j S.
2
For the remaining questions 5.-14. we consider the continuous-time Markov
chain on the countable state space
S = {(i, j)|i N
0
, j = 0, 1, . . . , i}.
We assume that the only non-zero transition intensities are given by
q
(i,0),(i+1,0)
=
i
, i N
0
,
q
(i,0),(i1,j)
=
i
, i N,
q
(i,j),(i,j+1)
=
i,j
, i N, j = 0, 1, . . . , i 1,
q
(i,i),(i,0)
=
i,i
, i N.

(3,0)

3,0

(2,0)

2,0

(1,0)

1,0

(3,3)

3,3

(0,0)

(3,1)

3,1

(1,1)

1,1

(2,2)

2,2

(2,1)

2,1

(3,2)

3,2

Figure 2: Transition diagram of Markov chain in questions 5.-14.

3
The transition diagram for the Markov chain in questions 5.-14. is shown
in Figure 2. The interpretation is that the states (i, 0) constitutes a ladder
where the Markov chain can always move from (i, 0) to either (i + 1, 0) or
(i 1, 0) (i > 0!). Further, from any of the states (i, 0) (i > 0!) the Markov
chain can make a loop through the states
(i, 0) (i, 1) (i, 2) . . . (i, i) (i, 0).
5. Find the transition probabilities for the Markov chain of jumps.
6. For what values of the parameters
i
, i N
0
,
i
, i N, and
i,j
, i
N, j = 0, . . . , i, is the Markov chain irreducible?
In the following we shall use Theorem 5 in the lecture notes to nd out
exactly when the Markov chain is recurrent. From Theorem 5 we get (by
choosing i = (0, 0)) that the Markov chain is recurrent if and only if we can
show that the only bounded solution to
(j) =

k=(0,0)
P
j,k
(k) (1)
is given by (j) = 0, j S\{(0, 0)}.
7. Write down the system of equations from (1) for any state of the form
(i, 0), i N. Be careful about the case i = 1!
8. Argue that for any state (i, j), j = 1, . . . , i 1 then (1) takes the form
(i, j) = (i, j + 1)
and that for states (i, i) we must have
(i, i) = (i, 0).
9. Use question 8. to deduce that the system of equations from question
7. may be written as
(i, 0) =

i

i
+
i
(i + 1, 0) +

i

i
+
i
(i 1, 0), i 2, (2)
and
(1, 0) =

1

1
+
1
(2, 0). (3)
4
10. Use question 7.-9. to show that in the case where
i
= > 0,
i
= > 0
and = then the only solution to (1) is given by
(i, j) = C

, i N, j = 0, . . . , i.
[Hint: Use Chapter 5.6 to show that any solution to (2) has the form
(i, 0) = C
1
+ C
2

i
, i 1.
Then use the boundary condition (3) to nd a constraint between C
1
and C
2
. ]
11. Use question 10. to argue that the Markov chain is recurrent for <
and transient for > .
In the following we assume that
i
= and
i
= with > > 0. Thus we
know from question 10. that the Markov chain is recurrent.
12. Use Theorem 17 form the lecture notes to write down the system
of equations that must be satised by an invariant probability =
((i))
iS
.
[Hint: Start by writing down the system of equations for (i, j) for
i N and j = 1, . . . , i. Then deal with the equations for (i, 0) i N
0
.]
13. Deduce from the equations in question 12. that for an invariant prob-
ability we must have
(i, j) =

i,0

i,j
(i, 0), j = 0, . . . , i
and
(i, 0) =

+
(i 1, 0) +

+
(i + 1, 0), i 1. (4)
14. You can use without proof (or argue from Chapter 5.6!) that (4) implies
that may have
(i, 0) = C

i
, i 0.
5
Find the condition that the Markov chain is positive recurrent and
determine if there exists an invariant distribution if

i,0
=
i,1
= . . . =
i,i
:=
i
for some
i
> 0, i N.
[Hint: The only thing to check is whether the constant C may be
chosen such that becomes a probability, i.e. such that

lS
(l) = 1.]
The following questions are not required to pass assignment 2.
15. Show that in the recurrent case where
i
= and
i
= and 0 < <
then there are choices of
i,j
> 0 that lead to a null-recurrent Markov
chain
16. Consider the case where
i
= and
i
= and = > 0. Dis-
cuss whether the Markov chain is transient, null-recurrent or positive
recurrent. The answer may depend on the choice of the intensities
i,j
.
6