Stat 150 Stochastic Processes

Spring 2009

Lecture 19: Stationary Markov Chains

Lecturer: Jim Pitman

Symmetry Ideas: General idea of symmetry: make a transformation, and something stays the same. In probability theory, the transformation may be conditioning, or some rearrangement of variables. What stays the same is the distribution of something. For a sequence of random variables X0 , X1 , X2 , . . . , various notions of symmetry: (1) Independence. X0 and X1 are independent. Distribution of X1 given X0 d does not involve X0 ; that is, (X1 |X0 A) = X1 . (2) Identical distribution. The Xn are identically distributed: X0 = Xn for every n. Of course IID = LLN / CLT. Stationary: (X1 , X2 , . . . ) = (X0 , X1 , . . . ) By measure theory, this is the same as
d d d

(shifting time by 1)

(X1 , X2 , . . . , Xn ) = (X0 , X1 , . . . , Xn1 ), for all n. Obviously IID = Stationary. But the converse is not necessarily true. E.g., Xn = X0 for all n, for any non-constant random variable X0 . Another example: Stationary MC, i.e., start a MC with transition matrix P with initial distribution . Then easily, the following are equivalent: P = X1 = X0 (X1 , X2 , . . . ) = (X0 , X1 , . . . ) . Otherwise put, (Xn ) is stationary means (X1 , X2 , . . . ) is a Markov chain with exactly the same distribution as (X0 , X1 , . . . ). Other symmetries: Cyclic symmetry: (X1 , X2 , . . . , Xn ) = (X2 , X3 , . . . , Xn , X1 )
n
d d d d

Reversible: (Xn , Xn1 , . . . , X1 ) = (X1 , X2 , . . . , Xn ) 1

Lecture 19: Stationary Markov Chains

Exchangeable: (X(1) , X(2) , . . . , X(n) ) = (X1 , X2 , . . . , Xn ) for every permutation of (1, . . . , n). Application of stationary idea to MCs. Think about a sequence of RVs X0 , X1 , . . . which is stationary. Take a set least n 1(if any) s.t. Xn A A in the state space. Let TA = ; that is, if no such n TA := least n 1 : 1A (Xn ) = 1. Consider the 1A (Xn ) process. E.g., 0 0 0 0 1 0 0 1 1 0 0 0 . . . To provide an informal notation: P(TA = n) = P(? 0 0 . . . 0 0 1)
n1 zeros

P(TA n) = P(? 0 0 . . . 0 0 ?)
n1 zeros

P(X0 A, TA n) = P(1 0 0 . . . 0 0 ?)
n1 zeros

Lemma: For every stationary sequence X0 , X1 , . . . , P(TA = n) = P(X0 A, TA n) With the informal notation, the claim is: P(? 0 0 . . . 0 0 1) = P(1 0 0 . . . 0 0 ?)
n1 zeros n1 zeros

for n = 1, 2, 3, . . .

Notice. This looks like reversibility, but reversibility of the sequence X0 , X1 , . . . , Xn is not being assumed. Only stationarity is required! Proof

P (? 0 0 . . . 0 0 1)
n1 zeros

P (1 0 0 . . . 0 0 ?)
n1 zeros

+ P (? 0 0 . . . 0 0 0)
n1 zeros

stationarity

+ P (0 0 0 . . . 0 0 ?)
n1 zeros

P (? 0 0 . . . 0 0 ?)
n1 zeros

P (? 0 0 . . . 0 0 ?)
n1 zeros

Lecture 19: Stationary Markov Chains

Take the identity above, sum over n = 1, 2, 3, . . . ,

P(TA < ) =
n=1

PA (T = n) P(X0 A, TA n)
n=1

= =E

1(X0 A, TA n)
n=1

= E(TA 1(X0 A)) This is Marc Kacs identity: For every stationary sequence (Xn ), and every measurable subset A of the state space of (Xn ): P(TA < ) = E(TA 1(X0 A)) Application to recurrence of Markov chains: Suppose that an irreducible transition matrix P on a countable space S has a stationary probability measure , that is P = , with i > 0 for some state i. Then is the unique stationary distribution for P j > 0, j . Ej Tj < , j . Ej Tj = Proof: Apply Kacs identity to A = {i}, with P = P governing (Xn ) as d a MC with transition matrix P started with X0 = . 1 P (Ti < ) = E [Ti 1(X0 = i)] = i Ei Ti Since i > 0 this implies rst Ei Ti < , and hence Pi (Ti < ) = 1. Next, need to argue that Pj (Ti < ) = 1 for all states j . This uses irreducibility, which gives us an n such that P n (i, j ) > 0, hence easily an m n such that it is possible to get from i to j in m steps without revisiting i on the way, i.e. Pi (Ti > m, Xm = j ) > 0 But using the Markov property this makes Pj (Ti < ) = Pi (Ti < | Ti > m, Xm = j ) = 1
Kac 1 , j . j

Lecture 19: Stationary Markov Chains

Finally P (Ti < ) =

j

j Pj (Ti < ) =
j

j = 1

and the Kac formula becomes 1 = i Ei Ti as claimed. Lastly, it is easy that j > 0 and hence 1 = j Ej Tj because = P implies = P n and so j = k P n (k, j ) i P n (i, j ) > 0
k

for some n by irreducibility of P .