University of Toronto Department of Mechanical and Industrial Engineering MIE263: Stochastic Operations Research (Winter 2012)

Homework #6 Solutions Problem 1: Recall the Miniopoly game from HW 5. Use the currect Markov chain as provided in HW5 solutions to answer the following: Which property is the most valuable to own, that is which property do players end their turn on must often? Solution From HW5 solutions we have the following transition matrix:

( which allows us to calculate the steady state probabilities using that is [ [ ]= ] [ which gives us ]

Therefore, the most valuable property is the blue one. Problem 2: Sheldon and Raj are playing Rock-Paper-Scissors-Lizard-Spock. When they begin the score is 0-0. To make the game more interesting, if the score ever becomes tied, it is instantly reset to 0-0. Starting from any score the probability that Sheldon gets the next point is 2/5. The probability that Sheldon and Raj tie the next point is 1/5. Suppose the game stops when one players score reaches 2. a) Draw an appropriate Markov chain that describes the game. b) Identify all transient and recurrent states. c) Find P(Raj wins) d) What is the expected number of rounds that Sheldon and Raj will play until one of them wins?

University of Toronto Department of Mechanical and Industrial Engineering MIE263: Stochastic Operations Research (Winter 2012)
Solution a) Denote by (x,y) the score of Sheldon and Raj respectively, a Markov chain that describes the game is
1 1/5 2/5 1/5 2/5 1/5 2/5 1 0,2

2,0

1,0

0,0

0,1

2/5

2/5

2/5

Note that the game ends when either state (0,2) or (2,0) is entered. b) Since we have a finite number of states, a state is recurrent if and only if it is accessible from all the states that are accessible from it, and therefore, states (0,0), (0,1) and (1,0) are transient and states (0,2) and (2,0) are recurrent. c) The probability that Raj wins is the probability that we get absorbed to state (0,2). Setting up the equations we solve for a(1,0), a(0,0), a(0,1) ( ( ) ) ( which yields the following ( ( ( ) ) ) ) ( ( ) ( ) ) ( ( ) ( ) ) ( )

Therefore, the probability of Raj winning is equal to =0.5.

d) This is the expected first passage time to either state (2,0) or (0,2). Setting up the mean first passage time equations we have
( ( ( ) ( ) ( ) )

( ( ) (
)

) ( ) )

) ( )

which yields the following

University of Toronto Department of Mechanical and Industrial Engineering MIE263: Stochastic Operations Research (Winter 2012)
( ( ( ) ) )

Therefore, the expected number of rounds that Sheldon and Raj will play until one of them wins is ( ) .

Problem 3: Consider a Markov chain with 5 states (0,1,2,3,4) and the following transition probabilities:

a) If the chain starts at state 2, what is the expected number of steps until the chain returns back to 2. b) For each state i, compute the expected time to hit 2 for the first time. Solution a) Our Markov chain looks like this:
1 P P P

1-p

1-p

1-p

We are looking for the mean recurrence time. Setting up the mean first passage time equations we have ( ( which gives ) )

University of Toronto Department of Mechanical and Industrial Engineering MIE263: Stochastic Operations Research (Winter 2012)

Therefore we have for the mean recurrence time ( which after simplifying gives (

) )