Markov Analysis

Markov analysis is a technique that deals with the

probabilities of future occurrences by analysing
presently known probabilities.

 Markov analysis has numerous applications in business

including market share analysis, bad debt prediction, future
breakdown of machine, etc.
Markov analysis makes the assumption that the system starts
in an initial state or condition.
Predicting these future states involves knowing the system
likelihood or probability of changing from one state to another.
These probabilities can be collected and placed in a matrix or
table.

 This matrix of transition probabilities shows the likelihood

that the system will change from one time period to next.
This is the Markov process and it enables us to predict future
states or conditions.
States are used to identify all possible conditions of a process
or a system.
For example
a machine can be in one of two states at any point of time. It can be
either functioning correctly or not functioning correctly.
If there are only three grocery stores in a small town, a resident can be
customer of any one of the three at any point in time. Therefore there
are three states corresponding to the three grocery stores.

 In Markov analysis we also assume that the states are both

collectively exhaustive and mutually exclusive.
Our discussion of Markov analysis assumes that there is a
finite number of states for any system.
Collectively exhaustive means that we can list all of the
possible states of a system or process.
Mutually exclusive means that a system can be in one state at
any point in time.

 After the states have been identified the next step is to

determine the probability that the system is in this state.
Such information is then placed into a vector of state
probabilities.
(i) = vector of state probabilities for period i
= (1, 2, 3, ..........., n)

Where n = number of states and 1, 2, 3, ..........., n =

probability of being in state 1, state 2, ....... state n

Three Grocery Stores

There could be a total of 100000 people that shop at the three
grocery stores during any given month. 40000 people may be
shopping at American Food Store. 30000 people may be
shopping at Food Mart and 30000 may be shopping at Atlas
Foods. The probability that a person will be shopping at one of
these three grocery stores is as follows:
State 1: 1 = 40000/100000 = 0.4 = probabilities that a person
will shop at American Food
State 2: 2 = 30000/100000 = 0.3 = probabilities that a person
will shop at Food Mart
State 1: 3 = 30000/100000 = 0.3 = probabilities that a person
will shop at Altas Foods
These probabilities can be placed in the vector of state
probabilities i.e. (0) = (0.4, 0.3, 0.3), which represent the
market shares for these three stores for the first period.

American Food
# 1 : 0.4

Food Mart
#2: 0.3

Atlas Foods
# 3: 0.3

#1: 0.8

0.4x0.8 = 0.32

#2: 0.1

0.4x0.1 = 0.04

#3: 0.1

0.4x0.1 = 0.04

#1: 0.1

0.3x0.1 = 0.03

#2: 0.7

0.3x0.7 = 0.21

#3: 0.2

0.3x0.2 = 0.06

#1: 0.2

0.3x0.2 = 0.06

#2: 0.2

0.3x0.2 = 0.06

#3: 0.6

0.3x0.6 = 0.18

Solution
Tree diagram and the calculations could be used to find the
state probabilities for the next month and the month after that
the tree would soon get very large. Rather than use a tree
diagram it is easier to use a matrix of transition probabilities,
which is a matrix of conditional probabilities of being in a
future state given a current state.
Pij = conditional probability of being in state j in the future
given the current state of i
P = matrix of transition probabilities
P11
P21

P12
P22

.
.

P1n
P2n

Pm1

Pm2

Pmn

P=

Solution
The three grocery stores to determine4 what percentage of the
customers would switch each month.
Put these transitional probabilities into the following matrix:
0.8 0.1 0.1
P = 0.1 0.7 0.2
0.2 0.2 0.6
When current period is 0, calculating the state probabilities for
the next period can be accomplished as (1) = (0) P
Similarly, calculating the state probabilities for the n + 1
period can be accomplished as (n + 1) = (n) P

Solution
When current period is 0, calculating the state probabilities for
the next period can be accomplished as (1) = (0) P
= (0.4 0.3

0.3)

0.8
0.1
0.2

0.1
0.7
0.2

0.1
0.2
0.6

= [(0.4)(0.8) + (0.3)(0.1) + (0.3)(0.2), (0.4)(0.1) + (0.3)(0.7) +

(0.3)(0.2), (0.4)(0.1) + (0.3)(0.2) + (0.3)(0.6)]
=(0.41, 0.31, 0.28)
Market share for American Food and Food Mart has increased
while the market share for Atlas Foods has decreased.

Solution
Will this trend continue in the next period and the one after
that?
Consider two time periods from now: (2) = (1) P
Since we know that (1) = (0) P
So, (2) = (0) P P = (0) P2
In general (n) = (0) Pn

Machine Operator
Paul Tolsky, owner of Tolsky Works, has recorded the
operation of his milling machine for several years. Over the
past two years 80% of the time the milling machine functioned
correctly during the current month if it had functioned
correctly in the preceding month. This also means that only
20% of the time did the machine not function correctly for a
given month when it was functioning correctly during the
preceding month. In addition, it has been observed that 90% of
the time the machine remained incorrectly adjusted for any
given month. Only 10% of the time did the machine operate
correctly in a given month when it did not operate correctly
during the preceding month. In other words, this machine can
correct itself when it has not been functioning correctly in the
past and this happens 10% of the time.

Machine Operator
0.8 0.2
Matrix of transition probabilities for this machine is P =
0.1 0.9
Probabilities of functioning and non functioning (0) = (1, 0)
(1) = (0) P = (1, 0) 0.8 0.2
0.1 0.9
= [(1)(0.8) + (0)(0.1), (1)(0.2) + (0)(0.9) = (0.8, 0.2)
Probability that the machine will be functioning correctly two
months from now.
0.8 0.2
(2) = (1) P = (0.8, 0.2)
0.1 0.9
[(0.8)(0.8) + (0.2)(0.1), (0.8)(0.2) + (0.2)(0.9)] = (0.66, 0.34)

 Repeat the process for 15 periods

Period
State 1
1
1
2
0.8
3
0.66
4
0.562
5
0.4934
6
0.44538
7
0.411766
8
0.388236
9
0.371765
10
0.360235
11
0.352165
12
0.346515
13
0.342560
14
0.339792
15
0.337854

State 2
0
0.2
0.34
0.438
0.5066
0.55462
0.588234
0.611763
0.628234
0.639754
0.647834
0.653484
0.657439
0.660207
0.662145

Machine Operator
The machine starts off functioning correctly in the first period.
In period 5, there is only a 0.4934 probability that the machine
is still functioning correctly and by period 10, this probability
is only 0.360235.
In period 15, the probability that the machine is still
functioning correctly is about 0.34.
Probability that the machine will be functioning correctly at a
future period is decreasing but it is decreasing at a decreasing
rate.

Machine Operator
From table, it appears there will be an equilibrium at
0.333333.
An equilibrium condition exists if the state probabilities do not
change after a large number of periods.
At equilibrium the state probabilities for a future period must
be the same as the state probabilities for the current period.
(next period) = (this period) P or (n+1) = (n) P
At equilibrium (n + 1) = (n) or (n) = (n) P
or = P

Machine Operator
0.8 0.2
(1, 2) = (1, 2)
0.1 0.9
(1, 2) = [(1)(0.8) + (2)(0.1), (1)(0.2) + (2)(0.9)]
1 = 0.81 + 0.12

(1)

2 = 0.21 + 0.92

(2)

Since sum of state probabilities 1 and 2 is 1 i.e.

(3)
1 + 2 = 1
From the above three equations arbitrarily drop any one of the
above two and solve the simultaneous equation, we get

Machine Operator
2 = 0.21 + 0.92
1 + 2 = 1
From the above three equations arbitrarily drop any one of the
above two and solve the simultaneous equation, we get
1 =1/3 = 0.333333

2 = 2/3 = 0.666666

Equilibrium state probability for state 1 is 0.333333 and for

state 2 is 0.66666

Absorbing States and Fundamental Matrix: Account

Receivable Application
Generally it assume that it is possible for the process or system
to go from one state to any other state between any two
periods.
In some cases, if you are in a state you can not go to another
state in the future i.e. when you are in a given state you are
absorbed by it and you will remain in that state.
Any state that has this property is called an absorbing state.
An account receivable system normally places debts or
receivables from its customers into one of several categories
or states depending on how overdue the oldest unpaid bill is.

 The exact categories or states depend on the policy set by each

company. Four typical states or categories for an accounts
receivable application follow:
State 1 (1): paid, all bill
State 2 (2): bad debt, overdue more than three month
State 3 (3): overdue less than one month
State 4 (4): overdue between one and three month
The matrix of transition probabilities for these four states will
reflect the propensity of customers to move among the four
accounts receivable categories from one month to next. The
probability of being in the paid category for any item or bill in
a future month given that a customer is in the paid category for
a purchased item this month is 100% or 1.

 It is impossible for a customers to completely pay for a

product one month and owe money on it in a future month.
Another absorbing state is the bad debt state. If a bill is not
paid in three months we are assuming that the company will
completely write it off and not try to collect it in the future.
Thus once a person is in the bad debts category, that person
will remain in that category forever. For any absorbing state,
the probability that a customer will be in this state in the future
is 1 and the probability that a customer will be in any other
state is 0. For a person in the less than one month category,
there is a 0.6 probability of being in the paid category, a 0
probability of being in the bad debt category, a 0.2 probability
of remaining in the less than one month category and a
probability of 0.2 of being in the one to three month category
in the next month.

 Note that there is a 0 probability of being in the bad debt

category the next month because it is impossible to get from
state 3, less than one month, to state 2, more than three months
overdue, in just one month. For a person in the one to three
month category, there is a 0.4 probability of being in the paid
category, a 0.1 probability of being in the bad debt category, a
0.3 probability of being in the less than one month category
and a 0.2 probability of remaining in the one to three month
category in the next month.
This Month
Paid
Bad Debt
1< Month
1 to 3 Month

Paid
1
0
0.6
0.4

Next Month
Bad Debt 1< Month 1 to 3 Month
0
0
0
1
0
0
0
0.2
0.2
0.1
0.3
0.2

 If we know the fraction of the people in each of the four

categories or states for any given period, we can determine the
fraction of the people in these four states or categories for any
future period. These fraction are placed in a vector of state
probabilities and multiplied times the matrix of transition
probabilities.
In long run everyone will be either in the paid or bad debt
category. This is because the categories are absorbing states.
But how many people or how much money will be in each of
these categories? Knowing the total amount of money that will
be in either the paid or bad debt category will help a company
mange its bad debts and cash flow. This analysis requires the
use of the fundamental matrix.

 To obtain the fundamental matrix, it is necessary to partition

the matrix of transition probabilities P.
I

1
0
0
0
0
1
0
0
0.6 0 0.2 0.2
0.4 0.1 0.3 0.2
A
B

P=

I=

1
0

0
1

A = 0.6 0
0.4 0.1

0=

0
0

B = 0.2
0.3

0
0
0.2
0.2

 I = an identity matrix
0 = a matrix with all 0
Fundamental matrix can be computed as F = (I B)-1
-1
F= 1 0
- 0.2 0.2
0 1
0.3 0.2
F=

F=

a
c

b
d

0.8 -0.2 -1
-0.3 0.8
-1
=

d
-b
ad-bc ad-bc
-c
a
ad-bc ad-bc

F=

0.8
-0.3

-0.2
0.8

-1
=

0.8 -(-0.2)
0.58 0.58
-(-0.3) 0.8
0.58 0.58

1.38
0.52

0.34
1.38

To use the fundamental matrix in computing the amount of bad

debt money that expect in the long run is to multiply the
fundamental matrix F times the matrix A
FA = 1.38 0.38
0.52 1.38

0.6
0.4

0
0.1

0.97 0.03
0.86 0.14

It indicates the probability that an amount in one of the nonabsorbing states will end up in one of the absorbing states.

 Top row of this matrix indicates the probabilities that an

amount in the less than one month category will end up in the
paid and the bad debt category.
The probability that an amount that is less than one month
overdue will be paid is 0.97
The probability that an amount that is less than one month
overdue will end up as a bad debt is 0.03
The second row has similar interpretation for the other non
absorbing state which is the one to three month category.
0.86 is the probability that an amount that is one to three
months overdue will eventually be paid
0.14 is the probability that an amount that is one to three
months overdue will never be paid but will become a bad debt.

 Assume that there is $2000 in the less than one month

category and $5000 in the one to three month category. Then
M = (2000, 5000), where M is the amount of money that is in
each of the non-absorbing states
Amount of money that will end up as being paid and the
amount that will end up as bad debts = MFA
=

(2000, 5000)

0.97
0.86

0.03
0.14

(6240, 760)

Out of the total $7000 ($2000 in the less than one month
category and $5000 in the one to three month category), $6240
will be eventually paid and $760 will end up as bad debt.

 Problem: George Walls, president of Bradley School is

concerned about declining enrolments. Bradley School is a
technical college that specialize in training computer
programmes and computer operators. Over the years, there has
been a lot of competition among Bradley School ,
International Technology and Career Academy. The tree
schools compete in providing education in the areas of
programming, computer operations and basic secretarial skills.
To gain a better understanding of which of these schools is
emerging as a leader, George decided to conduct a survey. His
survey looked at the number of students who transferred from
one school to the other during their academic careers. On the
average, Bradley School was able to retain 65% of those
students it originally enrolled. Twenty percent of the students
originally enrolled transferred to Career Academy and 15%
transferred to International Technology.

 Career Academy had the highest retention rate: 90% of its

students remained at Career Academy for their full academic
programme. George estimated that about half the students who
left Career Academy went to Bradley School and other half
went to International Technology. International Technology
was able to retain 80% of its students after they enrolled. 10%
of the originally enrolled students transferred to Career
Academy and other 10% enrolled in Bradley School.
Currently, Bradley School has 40% of the market. Career
Academy a much newly school has 35% of the market. The
remaining market share 25% consists of students attending
International Technology. George would like to determine the
market share for Bradley for the next year. What are the
equilibrium market shares for Bradley School, International
Technology and Career Academy?

 Solution:
State 1: initial share = 0.4 = Bradley School
State 2: initial share = 0.35 = Career Academy
State 3: initial share = 0.25 = International Technology
Transition matrix values are
Bradley
Bradley
0.65
Career
0.05
International
0.1

Career International
0.2
0.15
0.9
0.05
0.1
0.8

 Solution:
To determine market share for Bradley School, he has to
multiply current market share and transition matrix
probability.
0.65
0.2
0.15
(0.4, 0.35, 0.25)

0.05
0.1

0.9
0.1

0.05
0.8

Market share for Bradley School = (0.4)(0.65) + (0.35)(0.05) +

(0.25)(0.10) = 0.303
Market share for Career Academy = (0.4)(0.2) + (0.35)(0.90) +
(0.25)(0.10) = 0.420
Market share for International Technology = (0.4)(0.15) +
(0.35)(0.05) + (0.25)(0.8) = 0.278

 At equilibrium the future market share is equal to the existing

or current market share times the matrix of transition
probabilities.
Market share for Bradley School = X1
Market share for Career Academy = X2
Market share for International Technology = X3
0.65
0.2
(X1, X2, X3) = (X1, X2, X3)
0.05
0.9
0.1
0.1

X1 = 0.65X1 + 0.05X2 + 0.1X3

X2 = 0.2X1 + 0.9X2 + 0.1X3
X1 = 0.15X1 + 0.05X2 + 0.8X3
X1 + X2 + X3 = 1

0.15
0.05
0.8

 As there are four equations and only three variable so delete

one of the top three equation, then solve by simultaneous
equation procedure to get the equilibrium market share of
Bradley School, Career Academy and International
Technology.
Market share for Bradley School = X1 = 0.158
Market share for Career Academy = X2 = 0.579
Market share for International Technology = X3 = 0.263