Finitas cadenas de Markov

Aquí se introduce el concepto de un proceso estocástico de tiempo discreto, investigando su comportamiento para tales
procesos que poseen la propiedad de Markov ( para hacer predicciones sobre el comportamiento de un sistema que su fi
cinas a considerar sólo el estado actual del sistema y no de su historia). A continuación, añadir una restricción adicional
de la homogeneidad - insistimos en que las probabilidades de transición no haga dependerá del tiempo. Nuestra tarea
principal consiste en determinar cuáles son las características de una matriz de transición para una infinita cadena de
Markov de tiempo homogéneo requiere que tenga una distribución de probabilidad invariante. Para ello tenemos en
cuenta el comportamiento a largo plazo de una cadena de Markov tales. También tenemos en cuenta reducibilidad, la
transitoriedad, la recurrencia y la periodicidad; así como otras investigaciones que implican tiempos de retorno y número
esperado de pasos de un estado a otro.

0.1 Introducción de Cadenas de Markov finitos

Considere un proceso estocástico de tiempo discreto, X norte, n = 0, 1, 2, . . ., dónde X norte toma valores en el conjunto finito S
= { 1, . . . , NORTE} o {0, . . . , norte - 1}. (Un proceso estocástico es sólo una colección de variables aleatorias.) Llamamos a
los valores posibles de X norte la
estados del sistema. Nos gustaría describir las probabilidades para un proceso de este tipo, y esto se
puede lograr dando a los valores de P {X 0 = yo 0, X 1 = yo 1, . . . , X n =
yo norte} para cada norte y cada secuencia finito de estados ( yo 0, . . . , yo norte). Alternativamente, podríamos dar la distribución
de probabilidad inicial

φ (i) = P {X 0 = yo}, i = 1, . . . , norte (0.1.1)

y el probabilidades de transición,

q norte( yo n | yo 0, . . . , yo norte - 1) = P {X n = yo n | X 0 = yo 0, . . . , X norte - 1 = yo norte - 1}, (0.1.2)

1
y así mediante el uso de probabilidades condicionales obtenemos

P {X 0 = yo 0, . . . , X n = yo n} = φ (i 0) q 1 ( yo 1 | yo 0) q 2 ( yo 2 | yo 0, yo 1) · · · q norte( yo n | yo 0, . . . , yo norte - 1). ( 0.1.3)

Ahora nos limitamos nuestra atención a una clase particular de tales procesos, los que satisfacen la propiedad de
Markov.

Los estados de la característica de Markov que para hacer predicciones sobre el comportamiento de un sistema en
el futuro, es suficiente considerar sólo el estado actual del sistema y no su historia. En otras palabras, el estado
actual del sistema es importante, pero la forma en que llegó a ese estado no lo es. Para formalizar esta
matemáticamente, podemos decir que

P {X n = yo n | X 0 = yo 0, . . . , X norte - 1 = yo norte - 1} = P {X n = yo n | X norte - 1 = yo norte - 1}. (0.1.4)

Nos simplificar aún suponiendo que las probabilidades de transición no dependen del tiempo - esto se conoce
como homogeneidad.

Formalmente, podemos decir que para un proceso estocástico de tiempo discreto, X norte, n = 0, 1, 2, . . ., dónde X norte toma valores
en el conjunto finito S = { 1, . . . , NORTE} o {0, . . . , norte - 1}, una cadena de Markov timehomogeneous es un proceso tal que

P {X n = yo n | X 0 = yo 0, . . . , X norte - 1 = yo norte - 1} = Pi norte - 1, yo norte), (0.1.5)

para alguna función p: S × S → [ 0, 1].

De ahora en adelante - a no ser que se indique expresamente lo contrario - vamos a convenientemente referirse a una cadena de Markov

de tiempo homogénea tan sólo una cadena de Markov.

Con el fin de dar a las probabilidades de una cadena de Markov, se requiere una distribución de probabilidad inicial φ
(i) = P {X 0 = yo} y el probabilidades de transición p (i, j). Refiriéndose de nuevo a (0.1.3) tenemos que

P {X 0 = yo 0, . . . , X n = yo n} = φ (i 0) Pi 0, yo 1) Pi 1, yo 2) · · · Pi norte - 1, yo norte). (0.1.6)

Considere una norte × norte matriz denotada por PAG cuyo ( i, j) entrada PAG ij es p (i, j), esta matriz se conoce como el matriz
de transición para la cadena de Markov. El valor PAG ij representa la probabilidad de que el proceso, cuando en el
estado de yo, junto hacer una transición al estado j. Dado que las probabilidades son no negativos y dado que el
proceso debe hacer una transición a otro, tenemos que PAG es un matriz estocástica y por lo que satisface

0 ≤ P ij ≤ 1, 1 ≤ i, j ≤ NORTE, (0.1.7 una)

Σnorte
PAG ij = 1, 1 ≤ yo ≤ NORTE. (0.1.7 segundo)

j=1

Cualquier matriz satisface (0.1.7 una) y (0.1.7 segundo) puede ser una matriz de transición para una cadena de Markov.

2
Consideremos ahora el problema de determinar las probabilidades de que la cadena de Markov estará en un cierto estado yo
en un momento dado norte. ( Asumamos que tenemos una matriz de transición PAG y una distribución de probabilidad
inicial φ).

los norte- probabilidades de paso pag norte( i, j) se definen de la siguiente manera:

pag norte( i, j) = P {X n = j | X 0 = i} = P {X n + k = j | X k = yo}, (0.1.8)

señalando que la igualdad anterior es válida debido a la homogeneidad del tiempo. Así,

P {X n = j} = Σ φ (i) P {X n = j | X 0 = yo}. (0.1.9)

yo ∈ S

De hecho, la norte- probabilidad de transición paso es la ( i, j) entrada de la matriz PAG norte. Vamos a mostrar esto utilizando un argumento

de inducción. En primer lugar, tenga en cuenta que esto es cierto para trivialmente n = 1. Supongamos ahora que es cierto para cualquier

dado norte. Considerar,

P {X n + 1 = j | X 0 = i} = Σ P {X n = k | X 0 = i} P {X n + 1 = j | X n = k}
k

=Σ pag norte( i, k) p (k, j) (0.1.10)

k

pero si pag norte( i, k) es el ( i, k) ingreso de PAG norte, la última suma es exactamente el ( i, j) ingreso de
PAG norte P = P n + 1 y hemos terminado.

Por último, se introduce un poco más de notación. Podemos expresar una distribución de probabilidad inicial como un
vector
φ̄ 0 = ( ¯ φ 0 ( 1), . . . , ¯φ 0 ( NORTE)).
φ (0.1.11)

Nota: ¯ v denotará el vector ( v ( 1), . . . , v (N)) y esta notación se utilizará si ¯

v se ha de considerar como una fila o un vector columna. Por ejemplo, podemos escribir PAG ¯
vo¯ v PAG a pesar de que v es un vector columna en el primer caso y un vector fila de
el segundo.

Si sabemos ¯ φ 0, entonces la distribución en el tiempo de n, φ norte( i) = P {X n = yo} es dado por

φ̄ n = ¯
φ φ 0 PAG norte.

0.2 Algunos ejemplos

Estos son algunos ejemplos clásicos de tiempo homogénea finitas cadenas de Markov. Más ejemplos e
información adicional se puede encontrar haciendo referencia a [?,?,?,?,?].

3
0.2.1 de dos estados de la cadena de Markov

Considere el espacio de estados de un teléfono donde X n = 0 significa que el teléfono está libre en el momento norte
y X n = 1 significa que el teléfono está ocupado. Suponemos que durante cada intervalo de tiempo que existe una
probabilidad pag that a call comes in. (For ease we will assume that no more than one call comes in during any
particular time interval.) If the phone is busy during that period, the incoming call does not get through. We also
assume that if the phone is busy during a time interval, there is a probability q

that it will be free during the next interval. This gives a Markov chain with state space S = { 0, 1} and matrix

[1−p ]
p
P= . (0.2.1)
q 1−q

0.2.2 Simple Queuing Model

We adapt the previous example by assuming that the phone system can put one caller on hold. Hence at
any time the number of callers in the system is in the set S = { 0, 1, 2}. Again, any call will be completed
during a time interval with probability q and a new caller will come in with a probability p, unless the system
is already full.

We set
p( 0, 0) = 1 − p, p( 0, 1) = p, p( 0, 2) = 0, (0.2.2 a)

since a caller comes in with probability p ( again we are assuming only one caller arrives during any time
period). Also,

p( 2, 0) = 0, p( 2, 1) = q, p( 2, 2) = 1 − q, (0.2.2 b)

since no new callers may arrive if there are two callers in the system, and both calls may not end
simultaneously. If there is exactly one caller in the system, it is a little bit more complicated.

The state of the system goes from 1 to 0 if the current call is completed and no new callers enter the
system, that is, p( 0, 1) = q( 1 − p). Similarly, the state goes from 1 to 2 if the current call is not completed but
a new call arrives, that is,
p( 1, 2) = p( 1 − q). Since the rows must add to 1, p( 1, 1) = 1 − q( 1 − p) − p( 1 − q)
and hence
• •
1−p p 0
P= • q( 1 − p) 1 − q( 1 − p) − p( 1 − q) p( 1 − q) • . (0.2.2 c)
0 q 1−q

0.2.3 Random Walk with Reflecting Boundary

Consider a “random walker” moving along the sites {0, 1, . . . , N}. At each time step the walker moves one step,
to the right with probability p and to the left with

4
probability 1 − p. We want to confine the walker to the state space and so we need to impose boundary
conditions to ensure that they do not “walk through” 0 and/or N. If the walker is at one of the boundary points
{0, N}, the walker moves with probability 1 toward the inside of the interval. The transition matrix P for this
Markov chain is given by

p(i, i + 1) = p, p(i, i − 1) = 1 − p, 0 < i < N,

p( 0, 1) = 1, p(N, N − 1) = 1,
with p(i, j) = 0 for all other values of i, j. (0.2.3 a)

• If p = 1 2 we call this symmetric or unbiased random walk with reflecting boundaries.

• If p 6= 1
2 it is called a biased random walk.

Sometimes it is more convenient to consider partially reflecting boundaries where the walker at the
boundary moves the same as on the inside except that if the walker tries to leave the states {0, . . . , N} he runs
into a wall and goes nowhere. This corresponds to the boundary conditions

p( 0, 0) = 1 − p, p( 0, 1) = p; p(N, N − 1) = 1 − p, p(N, N) = p. ( 0.2.3 b)

Take, for example, N = 4 then the transition matrix for a random walk with reflecting boundaries, and with
partially reflective boundaries would look like this
• • • •
0 1 0 00 1−p p 0 0 0
• 1−p 0 p 00 • • 1−p 0 p 0 0•
• • • •
P= • 0 1−p 0 p0 • , P= • 0 1−p 0 p 0 ••
• • •
• 0 0 1−p0p • • 0 0 1−p 0 p•
0 0 0 10 0 0 0 1−pp
(0.2.3 c)
respectively.

0.2.4 Random Walk with Absorbing Boundaries

The chain is like the previous example except that when the walker reaches 0 or
N, the walker stays there forever.

These confined chains are sometimes referred to as “gambler’s ruin”, based on the interpretation of the
sums of independently identically distributed (i.i.d.) random variables, say, X n = ∑ n

k= 0 Z k where { Z n} is a i.i.d. integer valued sequence as the

winning/loss of a gamble. This chain is a birth and death chain since the Z n s take only the values 0, ± 1; and
so in this case we refer to the chain as simple random

5
walk. Here p = P{Z n = 1} could represent the probability of winning one pound and 1 − p = P{Z n = − 1} could
represent the probability of losing one pound. Since the state space is {0, 1, . . . , N} we can interpret this as two
gamblers in opposition, where N is the sum of their stakes, and 0 and N are absorbing states (one player
goes broke). (When one gambler plays the house - considered infinitely wealthy - the state space is {0, 1, 2, . . .}:
a countable Markov chain with 0 as an absorbing state.)

The transition matrix is given by

p(i, i + 1) = p, p(i, i − 1) = 1 − p, , 0 < i < N,

p( 0, 0) = 1, p(N, N) = 1. (0.2.4 a)

Again, take for example, N = 4 then the transition matrix for this type of random walk would look as follows:

• •
1 0 0 00
• 1−p 0 p 00 •
• •
P= • 0 1−p 0 p0 • . (0.2.4 b)
• •
• 0 0 1−p0p •
0 0 0 01

Note: From now on we adopt the convention that if p(i, j) is not specified for a particular i, j, then it is assumed
to be 0.

0.3 Long Term Behaviour

Now that we have P n we consider this stochastic matrix for n large in hope that we can develop an
understanding of the long term behaviour of a Markov chain.

0.3.1 Preliminaries

Recall, a probability vector is a vector with nonnegative components that sum to

1. Suppose ¯ π is a limiting probability vector, that is, for some initial probability
vector ¯ v,
π̄ = lim
π v P n, (0.3.1)
n →∞ ¯

and so it follows that

π̄ = lim
π v P n+ 1 = ( lim v P n) P = ¯ π P. (0.3.2)
n →∞ ¯ n →∞ ¯

If ¯π = ¯ π P then ¯ π is a invariant probability distribution. It can also be referred to

as a stationary, equilibrium or steady-state probability distribution.

6
Note: An invariant probability vector is a left eigenvector of P with eigenvalue 1.

Three questions naturally arise when dealing with invariant probability distributions for stochastic matrices.

1. Can we say that every stochastic matrix P has an invariant probability

distribution ¯ π?;

2. If P has an invariant distribution ¯ π, is it unique?

3. When is it true that • •

π̄
• π̄...•
lim • • , (0.3.3 a)
n →∞ Pn= • •

π̄

and hence that for all initial probability distributions ¯ v,

lim v P n = ¯ π? (0.3.3 b)
n →∞ ¯

Suppose P is any stochastic matrix. We can easily verify that the vector ¯1 = (1, 1, . . . , 1) is a right eigenvector
with eigenvalue 1, and so at least one left eigenvector for eigenvalue 1 exists.

Condition 0.3.1 If we can show that:

1. The left eigenvector can be chosen to have all nonnegative entries.

2. The eigenvalue 1 is simple and all other eigenvalues have absolute value less than 1.

Then it can be shown [?] that there exists a matrix Q such that D = Q − 1 P Q,
where the first row of Q − 1 is the unique invariant probability vector ¯ π; the first
column of Q contains all 1s. The matrix D is not necessarily diagonal but it can be obtained by using a
Jordan decomposition [?] and so it will have the form
• •
10···0
• 0... •
D= • • , where M n → 0. (0.3.4)
• M •

By raising P to powers and taking the limit, we obtain,

• • • •
10···0 π̄
• 0... • • π̄...•
lim • • Q−1= • • . (0.3.5)
n →∞ P n = lim n →∞ QD n Q − 1 = Q • 0
• • •

0 π̄

7
We now wonder which matrices satisfy (1) and (2) of condition (0.3.1). There is definitely one large class of
matrices for which this is true and can be obtained by referring to the Perron-Frobenius Theorem.

Suppose that P is a stochastic matrix such that all of the entries are strictly positive. The Perron-Frobenius
Theorem implies that:

• 1 is a simple eigenvalue for P;

• the left eigenvector of 1 can be chosen to have all positive entries (and hence can be made into a
probability vector by multiplying by an appropriate constant);

• and all the other eigenvalues have absolute value strictly less than 1.

A large number of matrices fall into this category but it does not cover all stochastic matrices with the
appropriate limit behaviour. Consider, for example, a matrix
P that satisfies most of the conditions laid out by the theorem except not all of its entries are positive,
although it may not satisfy all of the conditions, P 2, for instance, could well do. From this we obtain a general
rule which follows after the example.

Example:
For example, consider the matrix P given by
• • • •
1/6 5/6 0 1/4 1/2 1/4 17/72 5/9 5/24 1/6 2/3 1/6
P= • 0 5/6 1/6 • giving P2= • 5/24 5/9 17/72 • (0.3.6)

although P does not have all positive entries P 2 does.

Rule: If P is a stochastic matrix such that for some n, P n has all entries strictly positive, then P satisfies (1)
and (2) of condition (0.3.1).

Let us now investigate the following problem: under what conditions on a stochastic matrix P can we
conclude that P n has all positive entries for some sufficiently large n?

We approach this by considering how a Markov chain can fail to fulfill (1) and (2) of condition (0.3.1).

0.3.2 Motivation

Before we discuss how a Markov chain can fail to fulfil (1) and (2) of condition (0.3.1) we shall first consider
some examples.

8
Example with an Invariant Probability Distribution

Simple random walk with partially reflecting boundary on {0, . . . , 4}. In this case
• •
1/2 1/2 0 0 0
• 1/2 0 1/2 0 0 •
• •
P= • 0 1/2 0 1/2 0 0 • . (0.3.7)
• •
• 0 1/2 0 1/2 0 •
0 0 1/2 1/2

Taking powers of this matrix we see that for large n

• •
1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5
• 1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5 •
• •
Pn' • 1/5 1/5 1/5 1/5 1/5 1/5 1/5 • (0.3.8)
• •
• •

where ¯ π = ( 1 5 , 1 5 , 1 5 , 1 5 , 1 5)

Periodic Example

Simple random walk with reflecting boundary on {0, . . . , 4}. In this case
• •
0 1 0 0 0
• 1/2 0 1/2 0 0 •
• •
P= • 0 1/2 0 1/2 0 0 • . (0.3.9)
• •
• 0 1/2 0 1/2 0 •
0 0 1 0

By taking powers of this matrix we see that P n depends on whether n is odd or even. For large n, if n is even,

• •
1/4 0 1/2 0 1/4 0 1/2 0 1/2 0 1/4 0
• 1/2 0 1/4 0 1/2 0 1/2 0 1/4 0 1/2 0 ••
•
Pn' • 1/4 • , (0.3.10)
• •
• •

whereas, if n is odd,
• •
0 1/2 0 1/2 0 1/4 0 1/2 0 1/4 0 1/2
• 0 1/2 0 1/4 0 1/2 0 1/4 0 1/2 0 1/2 ••
•
Pn' • 0 • . (0.3.11)
• •
• •

At each step the random walker moves from an “even step” to an “odd step” or vice versa. In this example
we can say that P has period 2.

9
Transient Example

Simple random walk with absorbing boundary on {0, . . . , 4}. Here

• •
1 0 0 0 0
• 1/2 0 1/2 0 0 •
• •
P= • 0 1/2 0 1/2 0 0 • . (0.3.12)
• •
• 0 1/2 0 1/2 0 •
0 0 0 1

If n is large, • •
1 0 0 0 0 3/4 0 0 0 1/4
• 1/2 0 0 0 1/2 1/4 0 0 0 3/4•
• •
Pn' • 00001 • . (0.3.13)
• •
• •

In this case the random walker eventually gets to 0 or 4 and then stays at that state forever. Note that p n( 1, 0)
→3
4 , p n( 1, 4) → 1 4 . This implies that the probability
that a random walker will eventually stay at 0 is 3 4 , whereas, with probability 1 4,
they will eventually stay at 4. States 1,2,3 are called transient states of the Markov chain.

Reducible Example

Suppose S = { 1, 2, 3, 4, 5} and
• •
1/2 1/2 0 0 0
• 1/2 1/2 0 0 0 •
• •
P= • 0 0 1/2 1/2 0 0 • . (0.3.14)
• •
• 0 1/2 0 1/2 0 •
0 0 1/2 1/2

For large n, • •
1/2 1/2 0 0 0
• 1/2 1/2 0 0 0 •
• •
Pn' • 0 0 1/3 1/3 1/3 0 • . (0.3.15)
• •
• 0 1/3 1/3 1/3 0 •
0 1/3 1/3 1/3

In this case the chain splits into two smaller, non-intersecting chains: a chain with state space {1, 2} and a
chain with state space {3, 4, 5}. Each “sub chain” converges to an equilibrium distribution but one cannot
change from a state in
{ 1, 2} to a state in {3, 4, 5}. We say such a system is a reducible Markov chain.

10
0.3.3 Reducibility

We say two states of a Markov chain communicate with each other if and only if each state has a positive
probability of eventually being reached by a chain starting in the other state. More formally, states i and j communicate,
written i ↔ j, if there exists m, n ≥ 0 such that p m( i, j) > 0 and p n( j, i) > 0. The relation ↔ is an equivalence
relation on the state space:

• reflexive, i ↔ j ( since p 0( i, i) = 1 > 0);

• symmetric, i ↔ j implies that j ↔ i, ( this is immediate by the definition);

• transitive, i ↔ j and j ↔ k implies i ↔ k. ( To see this note that if

p m 1( i, j) > 0 and p m 2( j, k) > 0 then

p m 1+ m 2( i, k) = P{X m 1+ m 2 = k|X 0 = i}
≥ P{X m 1+ m 2 = k, X m 1 = j|X 0 = i}
= P{X m 1 = j|X 0 = i}P{X m 1+ m 2 = k|X m 1 = j}
= p m 1( i, j)p m 2( j, k) > 0,

and similarly, p n 1( j, i) > 0, p n 2( k, j) > 0 imply p n 1+ n 2( k, i) > 0).

This equivalence relation partitions the state space into disjoint sets called communication classes.

If only one communication class exists, that is, if for all i, j there exists an n = n(i, j) with p n( i, j) > 0, then the
chain is called irreducible. Any matrix satisfying (1) and (2) of condition (0.3.1) is irreducible. However, other
matrices can also possess this property.

If there exists more than one communication class then the chain splits into as many smaller, interacting or
noninteracting (depends on their behaviour) chains, splitting up the state space. Such a system is known as
a reducible Markov chain. (One cannot change from one reduced state space to another if the two state
spaces in question are noninteracting.)

0.3.4 Transience and Recurrence

We shall now take a closer look at matrices P with more than one communication class. When a chain
starts in one class and then with probability 1 it eventually leaves this class and never returns we call such
a class a transient class and the states are called transient states. Other classes are known as recurrent classes,
these have recurrent states. (A chain starting in a recurrent class never leaves that class.)

11
Suppose P is a matrix for a reducible Markov chain with recurrent communication classes R 1, . . . , R r and
transient classes T 1, . . . , T s. ( We observe that there must be at least one recurrent class.) For each recurrent
class R, the submatrix P obtained from considering only the rows and columns for states in R is a stochastic
matrix. Hence we can express P in the following form (possibly after rearranging the order of the states):

• ∣∣∣∣∣∣∣∣∣∣∣∣ 0•
P1
• P2 0 •
• •
• P3 •
P= • • (0.3.16 a)
• .. •
• 0 . •
• •
Pr
S Q

where P k is the matrix associated with R k. Raising P to powers, we obtain,

• ∣∣∣∣∣∣∣∣∣∣∣∣ 0 •
P n1 P n
• 0 •
• 2
•
• P n3 •
P= • • (0.3.16 b)
• .. •
• 0 . •
• •
P nr
Sn Qn

for some matrix S n.

0.3.5 Periodicity

Now let P be the matrix for an irreducible Markov chain; and note that if P was actually reducible we could
equally well consider separately each of the recurrent communication classes. We define the period of a
state i, d = d(i), to be the greatest common divisor of

J i = { n ≥ 0 : p n( i, i) > 0}. (0.3.17)

Let J be any nonempty subset of the nonnegative integers that is closed under addition, that is, m, n ∈ J ⇒ m
+ n ∈ J. An example of such a J is the set
J i since p m+n( i, i) ≥ p m( i, i)p n( i, i). Let d be the greatest common divisor of the elements of J. Then J ⊂ { 0, d, 2 d, .
. .}. It can be shown that J must contain all but a finite number of the elements of {0, d, 2 d, . . .}, that is, there
is some m such that md ∈ J for all m > M. Hence J i contains md for all m greater than some

M = m i.

Now consider another state j with m, n such that p m( i, j) > 0, p n( j, i) > 0, and so
m+n ∈ J i, m+n ∈ J j. Hence m+n = kd for some integer k. Also if l ∈ J j then
p m+n+l( i, j) ≥ p m( i, j)p l( j, j)p n( j, i) > 0, and so d divides l. In words, if d divides

12
every element of J i then it divides every element of J j. We can conclude from this that all states have the
same period and so we can talk about the period of P.

Note: If P is reducible it is possible for states in different communication classes to have different periods.

0.3.6 Further Investigations

If we are given an irreducible matrix P with period 1, that is, d = 1, we call it aperiodic. We shall now show
the following: if P is irreducible and aperiodic, then there exists an m > 0 such that for all n ≥ m, all entries of
P n are strictly positive. Take any i, j. Since P is irreducible there exists some m(i, j) such that p m( i, j) > 0.
Moreover, since P is aperiodic, there exists some M(i) such that for all n ≥ M(i), p n( i, i) > 0. Hence for all n ≥ M(i),
p n+m(i,j)( i, j) ≥

p n( i, i)p m(i,j)( i, j) > 0. Let M be the maximum value - we know a maximum exists because the state space is
finite - of M(i)+m(i, j) over all pairs ( i, j). Then
p n( i, j) > 0 for all n ≥ M and all i > j. To see this, consider the following example.

Example:
We saw earlier that the matrix P given by
• • • •
1/6 5/6 0 1/4 1/2 1/4 17/72 5/9 5/24 1/6 2/3 1/6
P= • 0 5/6 1/6 • giving P2= • 5/24 5/9 17/72 • . (0.3.18)

In fact, every P n with n ≥ 2 has strictly positive entries. The limiting probabilities are given by

• •
0.1875 0.625 0.1875
P2' • 0.1875 0.625 0.1875 • . (0.3.19)
0.1875 0.625 0.1875

Here m = 2 so that for n ≥ 2 all entries of P n are strictly positive.

Combining the last proof with the rule mentioned before that, we can now summarise with the following
theorem.

13
Theorem 0.3.1 If P is the transition matrix for an irreducible, aperiodic Markov chain, then there exists a
unique invariant probability vector ¯ π satisfying

π̄ P = ¯
π π. (0.3.20 a)

If ¯ φ is any initial probability vector,

lim φ Pn= ¯ π. (0.3.20 b)

n →∞ ¯

Moreover, ¯ π(i) > 0 for each i.

Now let us observe how P n behaves when P is not irreducible and aperiodic. First assume P is reducible
with recurrent classes R 1, . . . , R r and transient classes
T 1, . . . , T s. Each recurrent class acts as a small Markov chain and so there exists
r different invariant probability vectors ¯ π 1, . . . , ¯ π r with ¯ π k concentrated on R k
( π k( i) = 0 if i / ∈ R k). In other words, the eigenvalue 1 has multiplicity r with
one eigenvector for each recurrent class. To simplify the matter, assume that the submatrix P k for each
recurrent class is aperiodic because then if i ∈ R k,

lim j ∈ R k, p n( i, j) = 0, j / ∈ R k. (0.3.21)
n →∞ p n( i, j) = π k( j),

If i is any transient state, then the chain starting at i eventually ends up in a recurrent state. This means that
for each transient state j, lim n →∞ p n( i, j) = 0.

Denote the probability that the chain starting in state i eventually ends up in the recurrent class R k by α k( i), k
= 1, . . . , r Once the chain reaches a state in R k it will settle down to the equilibrium distribution on R k. From this
we note that if
j ∈ R k,
lim (0.3.22)
n →∞ p n( i, j) = α k( i)π k( j).

Furthermore, if ¯ φ is an initial probability vector,

lim φ Pn (0.3.23)
n →∞ ¯

exists but depends on ¯ φ.

Finally, suppose that P is irreducible but has period d > 1. In this case the state space splits nicely into d sets,
A 1, . . . , A d, such that the chain always moves from
A i to A i+ 1 ( or A d to A 1).

In general, if P is irreducible with period d, P will have d eigenvalues with absolute value 1, the d complex
numbers z with z d = 1. Each is simple; in particular, the eigenvalue 1 is simple and there exists a unique
invariant probability ¯ π. Given any
initial probability distribution ¯ φ, for large n, ¯ φ P n will cycle through d different
distributions, but they will average to ¯ π,

1
lim φ P n+ 1 + · · · + ¯ φ P n+d] = ¯ π. (0.3.24)
n →∞ d[ ¯

14
0.4 Return Times

Let X n be an irreducible (but perhaps periodic) Markov chain with transition matrix P. In addition, let I represent
the indicator function of an event (this is a random variable which equals 1 if the event occurs and 0
otherwise). Consider the amount of time spent in state j up through time n,

∑n
Y (j, n) = I{ X m= j}. (0.4.1)
m= 0

If π denotes the invariant probability distribution for P, from our previous results we can conclude that

1 1 ∑n
lim P{X m = j|X 0 = i} = π(j),
n →∞ n + 1 E(Y (j, n)|X 0 = i) = lim n →∞ n+1
m= 0
(0.4.2)
that is, π(j) represents the fraction of time that the chain spends in state j. We shall relate π(j) to the first
return time to the state j.

Fix a state i and assume that X 0 = i. Allow T to be the first time after 0 such that the Markov chain is in state i,

T = min{ n ≥ 1 : X n = i}. (0.4.3)

We originally assumed that the chain is irreducible and so we know that T < ∞
with probability 1. It is also true that E(T) = ∑ ∞ m= 0 nP(T = n) < ∞.

Consider the time until the k th return to the state i. This time is given by a sum of independent random
variables, T 1 + · · ·+ T k, each with the distribution T . Here
T m denotes the time between the ( m − 1)th and the m th return. For large values of k the law of large
numbers gives that

1
(0.4.4)
k (T 1 + · · · + T k) ≈ E(T),

and so there are about k visits to the state i. By setting n = kE(T) we obtain the relation

E(T) = [π(i)] − 1. (0.4.5)

This tells us that the expected number of steps to return to i, assuming that the chain starts at i, is given by
the inverse of the invariant probability. (The above argument is not completely rigorous, it just provides an
intuitive sketch.)

Note: Equation (0.4.5) only gives the expected value of the random variable T -
it does not mention anything else about its distribution. In general, one can glean very little information
regarding the distribution of T when presented with only the invariant probability ¯
π.

15
0.5 Transient States

Let us now draw our attention toward the task of computing the expected number of steps from i to j when i 6=
j.

Let P be the transition matrix for a Markov chain X n. Suppose P has some transient states (recall that a state
i is called transient if with probability 1 the chain visits i only a finite number of times) and let Q be the
submatrix of P which includes only the rows and columns for the transient states. After rearranging the
order of the states accordingly, we can write

[˜ ∣∣∣∣ 0 ] [˜ ∣∣∣∣ 0 ]
PS Pn
P= , Pn= . (0.5.1)
Q Sn Qn

The matrix Q is a substochastic matrix - a matrix with nonnegative entries whose row sums are less than or
equal to 1. We know that the states represented by Q
are transient and so Q n → 0. This implies that all of the eigenvalues of Q have absolute values strictly less
than 1. Hence, I − Q ( where I is the identity matrix) is an invertible matrix and we can safely define the
matrix M = (I − Q) − 1.

Consider two different transient states, i and j, we now show that the expected number of visits to i starting
at j is given by M ij, the ( j, i) entry of M. Consider
Y i, the total number of visits to i,

∑∞
Yi= I{ X n= i}. (0.5.2)
n= 0

Since i is transient, Y i < ∞ with probability 1. Suppose X 0 = j. Then,

∑∞ ∑∞ ∑∞
E(Y i| X 0 = j) = E[ I{ X n= i}| X 0 = j] = P{X n = i|X 0 = j} = p n( j, i).
n= 0 n= 0 n= 0
(0.5.3)

In other words, E(Y i| X 0 = j) is the (j,i) entry of the matrix I + P+P 2+ · · · which is exactly the ( j, i) entry of the matrix
I + Q + Q 2 + · · ·. A little manipulation gives us

(1+ Q+Q 2 + · · ·)( I − Q) = I, or, I +Q+Q 2 + · · · = ( I − Q) − 1 = M. ( 0.5.4)

So, as long as we assume X 0 = j, we can compute the expected number of steps until the chain enters a
recurrent class by simply summing M ij over all transient states i.

This technique can be used to determine the expected number of steps that an irreducible Markov chain
takes to go from one state j to another state i. Firstly, write the transition matrix P for the chain with i being
the first site:
[ p(i, i) ∣∣∣∣ R Q ]
P= . (0.5.5)
S

16
Now make i and absorbing site, we now obtain a new matrix,
[1 ∣∣∣∣ 0 ]
P̃ = . (0.5.6)
S Q

Let T i represent the number of steps needed to reach state i, that is, T i is the smallest time n such that X n = i. For
any other state k let T i,k be the number of visits to k before reaching i, noting that if we start at state k, we
include this as one visit to k. With this we have

E(T i| X 0 = j) = E( ∑ T i,k| X 0 = j) = ∑ M jk, (0.5.7)

k 6= i k 6= i

that is, M ¯1 gives a vector whose j th component is the number of steps starting at
j until reaching i.

Example:
Suppose P is the matrix for a random walk with reflecting boundary.
• •
0 1 0 0 0
• 1/2 0 1/2 0 0 •
• •
P= • 0 1/2 0 1/2 0 0 • . (0.5.8)
• •
• 0 1/2 0 1/2 0 •
0 0 1 0

If we let i = 0 then
• • • •
0 1/2 0 0 22212442
• 1/2 0 1/2 0 0 1/2 0 1/2 0 • • 24632464 •
Q= • • , M = (I − Q) − 1 = • • . (0.5.9)
• • • •
0 1 0

M ¯1 = (7, 12, 15, 16) giving the expected number of steps to get from 4 to 0 is 16.

Now suppose there are at least two different recurrent classes. We look for the probability that given a
transient state j, the Markov chain eventually ends up in a particular recurrent class. Let us assume that the
recurrent classes consist of single points r 1, . . . , r k with p(r i, r i) = 1 and order the states so that the recurrent states
r 1, . . . , r k precede the transient states t 1, . . . , t s giving

[I ∣∣∣∣ 0 ]
P= . (0.5.10)
S Q

For i = 1, . . . , s, j = 1, . . . , k, let α(t i, r j) be the probability that the chain starting at t i eventually ends up in the
recurrent state r j. Set α(r i, r i) = 1 and α(r i, r j) = 0 if i 6= j. For any transient state t i,

α(t i, r i) = P{X n = r j eventually | X 0 = t i}

17
 =∑ P{X 1 = x|X 0 = t i} P{X n = r j eventually | X 1 = x}
x∈S

=∑ p(t i, x)α(x, r j). (0.5.11)

x∈S

Let A be the s × k matrix with entries α(t i, r j) a continuación de lo anterior puede ser expresado en forma matricial como

A = S + control de calidad, o, A = (I - Q) - 1 S = METRO S. (0.5.12)

Ejemplo:
Considere un andador al azar con la absorción de límite en {0, . . . , 4}. Si ordenamos el estado {0, 4, 1, 2, 3} de manera que los
estados recurrentes preceden a los estados transitorios a continuación,

• •
1 0 0 0 0
• 0 1 0 0 0 •
• •
P= • 1/2 0 0 1/2 0 • (0.5.13)
• •
• 0 0 1/2 0 1/2 0 1/2 0 1/2 0 •

y entonces

• • • • • •
1/2 0 0 3/2 1 1/2 1 2 1 1/4 1/2 1/2
S= • 0 0• , M = • 1/2 1 3/2 • ,MS= • 3/4 1/4 3/4 • . (0.5.14)
1/2

Comenzando en el estado 1, la probabilidad de que la caminata, finalmente, se absorbe en el estado 0 es 3

18