Engineering, Construction and Architectural Management

Emerald Article: Markov decision process for sewer rehabilitation REINI WIRAHADIKUSUMAH, DULCY M. ABRAHAM, JUDY CASTELLO

Article information:

To cite this document: REINI WIRAHADIKUSUMAH, DULCY M. ABRAHAM, JUDY CASTELLO, 1999"Markov decision process for sewe rehabilitation", Engineering, Construction and Architectural Management, Vol. 6 Iss: 4 pp. 358 - 370 Permanent link to this document: http://dx.doi.org/10.1108/eb021124 Downloaded on: 19-07-2012 To copy this document: permissions@emeraldinsight.com This document has been downloaded 473 times since 2008. *

Users who downloaded this Article also downloaded: *

ASIM KARIM, HOJJAT ADELI, 1999"A new generation software for construction scheduling and management", Engineering, Construction and Architectural Management, Vol. 6 Iss: 4 pp. 380 - 390 http://dx.doi.org/10.1108/eb021126 SDHABHON BHOKHA, STEPHEN O. OGUNLANA, 1999"Application of artificial neural network to forecast construction duration of buildings at the predesign stage", Engineering, Construction and Architectural Management, Vol. 6 Iss: 2 pp. 133 - 144 http://dx.doi.org/10.1108/eb021106

JASON MATTHEWS, STEVE ROWLINSON, 1999"Partnering: incorporating safety management", Engineering, Construction and Architectur Management, Vol. 6 Iss: 4 pp. 347 - 357 http://dx.doi.org/10.1108/eb021123

Access to this document was granted through an Emerald subscription provided by OXFORD BROOKES UNIVERSITY For Authors: If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service. Information about how to choose which publication to write for and submission guidelines are available for all. Please visit www.emeraldinsight.com/authors for more information. About Emerald www.emeraldinsight.com With over forty years' experience, Emerald Group Publishing is a leading independent publisher of global research with impact in business, society, public policy and education. In total, Emerald publishes over 275 journals and more than 130 book series, as well as an extensive range of online products and services. Emerald is both COUNTER 3 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation.
*Related content and download information correct at time of download.

Engineering, Construction and Architectural Management 1999 6|4, 358-370

Markov decision process for sewer rehabilitation

REINI W I R A H A D I K U S U M A H * , DULCY M . A B R A H A M * & JUDY CASTELLO * School of Civil Engineering, Purdue University, West Lafayette, IN 47907, USA; Asset Management Division, Department of Capital Asset Management, City of Indianapolis-Public Works, Project Management Section, 604 North Sherman Drive, Indianapolis, IN 46201, USA

Abstract Finding the optimal solution to address problems in sewer management systems has always challenged asset managers. An understanding of deterioration mechanisms in sewers can help asset managers in developing prediction models for estimating whether or not sewer collapse is likely. The effective use of deterioration prediction models along with the development and use of life cycle cost analysis (LCCA) can contribute to the goals of reducing construction, operation and maintenance costs in sewer systems. When sewer system maintenance/rehabilitation options are viewed as investment alternatives, it is important, and in some cases, imperative, to make decisions based on life cycle costs instead of relying totally

on initial construction costs. The objective of this paper is to discuss the application of deterioration modelling and life cycle cost principles in sewer system management, and to explore the role of the Markov chain model in decision making regarding sewer rehabilitation. A test case is used to demonstrate the application of the Markov chain decision model for sewer system management. The analysis includes evaluation of this concept using dynamic programming and the policy improvement algorithm. Keywords infrastructure, life cycle cost analysis, Markov chain model, optimization, rehabilitation, sewer deterioration models, sewer management systems, transition probabilities

INTRODUCTION The American Society of Civil Engineers (ASCE) in its 1998 Report Card on America's Infrastructure has given the nation's wastewater systems a grade of D + (ASCE 1998). According to the Environment Protection Agency (EPA), $137 billion is needed to meet the requirements of the Clean Water Act by the year 2012. By that year, the nation will need to increase the number of facilities providing wastewater treatment by 3353 (Grant 1995). Besides building new facilities, maintaining the capacity of the existing water treatment facilities is equally important. Sewer systems form one of the most capital intensive infrastructure systems in the US. A number of the oldest cities have had sewer systems in operation for more than 100 years. While the average age of sewers is 42 years, almost half of the municipalities across the US have sewers with average ages of more than 50 years (Malik et al. 1997). Traditionally, municipalities have addressed the design, construction, maintenance and operation of sewer systems on a crisis-based or reactive approach. This practice results in more frequent sewer failures, difficult and costly rehabilitation, and hence, inefficient use of limited funds (WEFASCE 1994). The cost of sewer failure, i.e. replacement costs, disruptions, adverse publicity and health

and safety problems, could be significantly higher than the cost of rehabilitation and hydraulic upgrading. Two major reasons for the adoption of the crisisbased practice in sewer management have been identified. First, the existing conditions of sewer systems are not readily visible to users. Thus, the actual problems caused by deterioration are not evident until major failures occur. Second, sewer systems are autonomously managed by different municipalities. Hence, collaborative efforts to develop optimal solutions by different municipalities have been scarce. Another issue in sewer management is the fact that the condition of these underground assets is generally not fully documented. While condition assessment is very important in developing a systematic procedure in effective sewer management, most cities lack comprehensive information on sewer condition in their management information systems. Based on a recent survey in over 450 cities and sanitation districts across the US regarding their state-of-practice of sanitary sewers management, most cities do not keep any kind of historical data upon which to base decisions for the future. Only 26% of the surveyed cities have made attempts to predict the future condition of the different sections of the system (Malik et al. 1997). This lack of data on the past and current conditions of sewers has

358

1999 Blackwell Science Ltd

Markov decision process for sewer rehabilitation

359

not only hindered the assessment of existing sewer infrastructure, but it has also affected the development of prediction models and the assessment of the effects of rehabilitation. Finding the optimal solution to address problems in sewer management systems has always challenged asset managers. A systematic approach for the determination of deterioration and obsolescence of sewer systems is necessary to fully gauge the status of this underground system. This involves routine and systematic sewer structural and hydraulic condition assessments, establishment of a standard condition rating system, and developing and updating prediction models for sewer condition. An integrated sewer system management, which includes life cycle cost analysis and the development of prioritization schemes for the selection of rehabilitation projects, is essential for cost-effective asset management. The objective of this paper is to discuss the application of deterioration modelling and life cycle cost principles in sewer system management, and to explore the role of the Markov chain model in decision making regarding sewer rehabilitation. A brief discussion of life cycle cost analysis for highway infrastructure systems and prior efforts in the area of sewer management systems is provided. A P P L I C A T I O N OF LIFE CYCLE COST A N A L Y S I S IN I N F R A S T R U C T U R E MANAGEMENT SYSTEMS The development and use of life cycle cost analysis (LCCA) for public works investments can contribute to the goals of reducing construction, operation and maintenance costs. When sewer system maintenance/ rehabilitation options are viewed as investment alternatives, it is important, and in some cases, imperative, to make decisions based on life cycle costs instead of relying totally on initial construction costs. The use of LCCA for sewer project evaluation has significant potential to reduce capital expenditures. The life cycle cost of an investment is the total cost of owning and operating throughout its life cycle, including maintenance and other associated costs. The employment of life cycle costing means that all significant decisions related to an investment funding have to be evaluated in terms of the life cycle of the investment. In practice, defining and controlling life cycle costs are difficult. The future behaviour of underground sewer lines (i.e. the material, the soil condition and the loading) is uncertain, as are the financial and economic conditions that influence the relationships

between present and future costs. While the analysis is future-oriented, proper judgements supported with experience and engineering expertise provide satisfactory estimates. At this time, the extent to which LCCA is being used in the US suffers from a lack of awareness and sensitivity on the part of the involved parties. A survey performed by Arditi & Messiha (1996) indicated that 60% of the largest municipalities in the US did not use this type of analysis in decision making. Of the other 40% of the municipalities that used LCCA, some had been using it for over 20 years, but the criteria used in selecting projects for its implementation appeared to be arbitrary. Various obstacles in implementing life cycle analysis include the general desires to minimize the initial project cost and meet budgetary restrictions, the lack of formal guidelines for its implementation, the difficulty in estimating future costs and incomes, the failure of owners or managers with short-term responsibility to consider long-term impacts of their decisions, and the lack of data and accepted industry standards for describing the life cycle behaviour of the facility. Although there has been an increasing interest in maintenance management for urban wastewater systems in the recent past, there is a dearth of documentation regarding formal sewer management systems in the US (Abraham et al. 1998). Most of the research in the use of LCCA has been implemented for highway systemsfor pavements and bridges (Butt et al. 1987, 1994; Feighan et al. 1989a,b; Scherer & Glagola 1993). This is partly because the Federal Highway Administration (FHWA) has issued a policy requiring life cycle analysis in evaluating maintenance, rehabilitation and reconstruction alternatives. T o comply with this policy statement, many agencies have developed studies on life cycle costing for highway management systems. O V E R V I E W OF D E T E R I O R A T I O N M O D E L L I N G FOR I N F R A S T R U C T U R E SYSTEMS The effectiveness of an infrastructure management system depends on the availability of an accurate and efficient condition prediction model. Prediction models are necessary because the determination of cost-effective maintenance actions requires information on the current condition and the anticipated future condition. The assessment of current conditions is obtained from inspections, while the future conditions for different maintenance and rehabilitation actions are estimated from the deterioration models.

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

360

Wirahadikusumah,R.et al.

Studies for developing prediction models in highway systems vary in complexity from simple straight-line extrapolation to probability based models. Straight-line extrapolation results in straight-line deterioration curves. However, when sufficient data are available, it is found that the shape of the deterioration curve is generally curvilinear rather than the straight-line. Regression techniques for prediction modelling are valid only if the dependent variables (the variables that affect the independent variable, i.e. the degree of deterioration) can be found. Furthermore, such models are applicable only to specific climatic conditions, materials, construction techniques, etc., for which it has been developed. Butt et al. (1987) discussed the development of pavement prediction model using the Markov process and compared it with the curve-fitting techniques. A probability based Markov model was developed, in order to represent the probabilistic behaviour and rate of deterioration of pavements. The Markov process imposes a rational structure on the deterioration model. This form of predictive model also has the advantage of ensuring that projections beyond the limits of data will continue to have the worsening pattern with age, something that the regression models cannot guarantee. An overview of Markovian modelling for bridge management systems is presented in Scherer & Glagola (1993). One of the most commonly used models in infrastructure deterioration modelling is the Markov chain model. In this model, the primary assumption is that the conditional distribution of a transition from the present state to any future state, given the past state, is independent of any past states and depends only on the present state. Scherer and Glagola's study using bridge data from the state of Virginia, showed that this assumption is reasonably valid for predicting deterioration in bridges. The research concluded that the Markov process is an excellent candidate as the underlying mathematical tool for infrastructure maintenance and management decision support system. An advanced approach using 'latent' variable in modelling pavement deterioration is discussed in BenAkiva & Ramaswamy (1993). The deterioration model is estimated using a latent (not easily measurable or directly observable) dependent variable, whose value is simultaneously estimated from a set of indicators, which are measures of damage on the pavement. The 'condition' of the pavement is affected by many processes, including load-related fatigue, shrinkage due to low temperatures, etc. Because these processes are

unobservable, the actual pavement condition is not directly observable. However, the effects of these processes result in visible surface damage whose extents can be measured. These measured extents of damage then serve as indicators for the latent condition. The relationship between condition data and deterioration models has traditionally been a static one. Lu & Madanat (1994) proposed an updating method that involves a Bayesian statistical approach that uses prior information (i.e. old data or past experience that was used to develop the existing deterioration model) and the sample information (i.e. additional data obtained from facility inspections) into what is called the posterior distribution (i.e. combined old data and additional data to develop the updated deterioration model). An empirical study with bridge deck data indicated that the use of this methodology significantly reduced the uncertainty inherent in condition prediction.

D E C I S I O N M A K I N G FOR SEWER S Y S T E M REHABILITATION Reyna (1993) developed a multiple objective mathematical model using integer-linear programming for the planning of the rehabilitation of combined sewer systems. Multiple objectives included optimizing overall structural performance, hydraulic performance, disruption level, and annual maintenance and claim costs. This integer programming formulation was constrained by the maximum annual construction costs. The model attempted to perform LCCA by estimating annual maintenance and claims unit cost. It was assumed that there was a bilinear relationship between the maintenance and claims unit cost for existing and rehabilitated segments, and their structural condition index and pipe diameter. The proposed model provided the first step towards a holistic approach to sewer management. However, solving integer programming for large networks is computationally expensive. To overcome this limitation, the pipes were ranked using a weighting method that considered the structural, hydraulic, and maintenance and claims costs. Among all the pipes in the network for Reyna's case study, only the worst 25 were selected for the optimization process. Fick et al. (1993) describe the selection procedure of rehabilitation method for the city of San Jose. The procedure was based on a present worth comparison of alternative methods. The criteria for selecting the proper alternatives were constructability, functionality and cost. Pipeline decay curves (defined as the relationship between a pipe's condition and age) for each

1999 Blackwell Science Ltd, Engineering. Construction and Architectural Management 6 | 4 , 358-370

Markov decision process for sewer rehabilitation

361

of the primary pipeline materials were developed based upon an analysis of field and system inventory data, input from city maintenance personnel and estimates of the material service. The development of these decay curves was based on a deterministic approach. The effect of the rehabilitation strategies on the decay curves was determined and modified curves were developed for use in the present-worth analysis. Based on this analysis, the least-cost strategy for the entire system was identified. The model was able to determine the funding needed to maintain the system at its allowable performance level. However, the prioritization of rehabilitation projects under limited budgets was not considered. Burgess (1993) describes the application of a probabilistic model to simulate long-term variation in the structural condition of wastewater collection systems to support sewer system rehabilitation planning. The Markov chain model was employed to develop the prediction model. A heuristic procedure was developed to minimize the life cycle costs of the system, which included costs associated with rehabilitation, water quality degradation, damages resulting from sewer collapse, treatment of clear water (i.e. infiltration), and operation and maintenance. This planning tool supports the development of a pro-active sewer system rehabilitation program by listing the resources (dollar) required for optimal rehabilitation strategy without considering limitation of annual budgets.

D E V E L O P M E N T OF A P R E D I C T I O N M O D E L FOR SEWER C O N D I T I O N A comprehensive study performed by Water Research Center (WRc) (Serpente 1993) concluded that the concept of measuring the 'rate of deterioration' of a sewer is unrealistic, but deterioration is more influenced by random events (such as a storm or an excavation nearby) in a sewer life span. In some cases, severe defects may not lead immediately to collapse. While it is almost impossible to predict when a sewer will collapse, it is feasible to estimate whether a sewer has deteriorated sufficiently for collapse to be likely. In the development of a prediction model for a sewer's future condition, historical records on sewer condition and a knowledge of the mechanisms of sewer deterioration are needed.

related. The extent of structural failures depends on the size of the defect, soil type, interior hydraulic regime, groundwater level and fluctuation, corrosion, method of construction, and loading on a sewer. Hydraulic failures are caused by infiltration and inflow (I/I) problems. Infiltration is water that enters the system from the ground through pipe defects, while inflow is extraneous storm water that enters the system through roof leaders, clean-outs, foundation drains, basement, etc. These I/I problems reduce the hydraulic capacity of sewers, increasing the potential for collapse (WEF-ASCE 1994). Extensive research projects that investigated all aspects of concrete and brick sewer system performance have been performed by the WRc in order to understand the modes of failure for sewers (Serpente 1993). The research included field, laboratory and theoretical studies. Two hundred and fifty cases of sewer collapse were examined and investigations on mechanisms of ground loss and void formation were performed. Investigations were performed on sewers with various life spans, and long-term monitoring was conducted. Based on these studies, it was found that sewer deterioration may involve deterioration of sewer material itself. Joint material and mortar can be eroded, and corrosion in concrete sewers can occur due to the existence of hydrogen sulphide or other chemicals. The state of the surrounding soil is of fundamental importance in assessing the structural condition of a sewer. The main factors that affect the rate of ground loss include sewer defect size, hydraulic conditions (water table, frequency and magnitude of surcharge), and soil properties (cohesive or non-cohesive soil). Severe defects (larger than 10 mm), high water table (above sewer level), frequent and high magnitude of hydraulic surcharge, and certain soil types (silts, silty fine sands, fine sands) also have serious effects on ground loss. The understanding of sewer deterioration mechanisms helps asset managers in developing a prediction model by estimating whether a sewer has deteriorated sufficiently for collapse to be likely. One approach that has been successfully used in other types of infrastructure deterioration modelling is the Markov chain model.

Markov chain model Sewer deterioration mechanisms Sewer collapse involves structural and hydraulic failures. Structural and hydraulic problems are closely Stochastic processes are processes that evolve over time in a probabilistic manner. A stochastic process is defined to be an indexed collection of random variables {Xt}, where the index t runs through a given set

1999 Blackwell Science Ltd, Engineering. Construction and Architectural Management 6 | 4 , 358-370

362

Wirahadikusumah, R. et al.

T. Often T is taken to be the set of non-negative integers, and X, represents a measurable characteristic of interest at time t (Hillier and Lieberman 1995). One special type of stochastic process is called a Markov chain. A stochastic process is a Markov chain if it has the Markovian property: the conditional probability of any future event, given any past event and the present state X, = i, is independent of the past event and depends only upon the present state. This property can be expressed as follows:

For the development of the deterioration models, in order to reduce the complexity of the analysis, the future condition can be assumed to depend only on the present state, and independent of the past condition. It can be further assumed that for all states i and j and all t, P(Xt+ i=it+1\Xl = it) is independent of t. The probability, p i f , that the infrastructure condition is in state i at time t and it will be in a state j at time t + 1 does not change (i.e. it remains stationary) over time (unless rehabilitation is performed, or other external factors change). This stationary assumption is expressed by Equation (2). P(X1+1=j\X, = i)=Pij. (2)

The probability values in the transition matrices can be estimated by analysing the historical records on infrastructure conditions, or by assigning probability values based on expert opinions and appropriate assumptions. If historical data are readily available, statistical relationship between infrastructure condition and age can be estimated (i.e. regression analysis). Using this relationship, a non-linear programming technique can then be applied to determine the probability values by minimizing the sum of the absolute difference between the regression curve and the predicted condition for the corresponding age generated by the Markov chain. T h e objective function of the non-linear programming has the following form: Minimize (4)

where N is the total number of periods, Y(t) is the infrastructure condition at time t on the regression curve, E[X(t,p)] is the expected value of infrastructure condition at time t as predicted by the Markov chain with probability matrix p.

Transition probability values for sewer infrastructure For demonstration purposes, a case study based on a combined sewer rehabilitation project within the city of Indianapolis, IN, USA was used. The city conducted its combined sewer infrastructure assessment project to identify, inspect and provide a priority list for rehabilitation of vulnerable sewer segments to reduce the potential for incidences of failure (Greeley and Hansen 1996). In this project, the large (150 cm or larger) combined sewers were classified into five groups 1-5, where ' 1 ' characterized sewers in optimal structural condition and ' 5 ' represented sewers in critical structural condition. Six alternatives were considered for rehabilitation of the city's large combined sewers: 'do nothing', routine cleaning, shotcrete, cured-in-place pipe, sliplining with fiberglass reinforced pipe liner, open cut excavation and replacement with reinforced concrete pipes. In this system, there are five states of sewers (i.e. condition 1 , . . . , 5) and six options. In this study, the probability values, pij, in P were estimated based on discussions with asset managers in the city of Indianapolis, IN, USA, and were expressed as shown in Equation (5). The values were determined on the basis of prior experience of sewer managers on rehabilitation and maintenance projects of the city's combined sewers. More details about the condition

The term transition is used when the system moves from state i during one period to state./ during the next period. Accordingly, the probabilities, pij, are referred to as the transition probabilities. The transition probabilities are commonly displayed as an m x m matrix called the transition probability matrix P. For instance, if there are five states associated with the five possible conditions of an infrastructure facility (i.e. state 1 corresponding to the best condition and state 5 corresponding to the worst condition). Thus, the transition probability matrix is given by

(3)

The aforementioned transition probability matrix is for one-step (one-period) transition. The n-step transition probability matrix, P(n), of the process that is in state i and will be in state j after n periods (as opposed to in the next period) is computed using ChapmanKolmogorov equation: P(n) = Pn. The n -step transition probability matrix is then obtained by taking the nth power of the one-step transition matrix (Ross 1997).

1999 Blackwell Science Ltd, Engineering. Construction and Architectural Management 6 | 4 , 358-370

Markov decision process for sewer rehabilitation

363

assessment of the combined sewer infrastructure in the city of Indianapolis can be obtained from Abraham et al. (1998). In the following case study, the probabilities that a sewer currently in state 1, will remain in state 1 or deteriorate to state 2 after one transition period are 08 and 02 respectively. For each row of the transition matrix, pij = 1. The value of 1 in the last row of P indicates an 'absorbing' state corresponding to the fact that the sewer condition cannot transit from this state (the worst possible state) unless rehabilitation is performed.

(6)

M A R K O V D E C I S I O N PROCESS Asset managers need to make decisions regarding the selection of optimal rehabilitation action (from several alternatives) for each sewer condition. The action cho sen affects the transition probabilities in the Markov chain model, as well as the immediate and subsequent costs of the system. The decision process for selecting the optimal actions for the respective states when considering immediate and subsequent costs is re ferred to as a Markov decision process. Life cycle cost analysis for a large combined sewer maintenance/rehabilitation problem can be studied us ing a Markov decision process. As the planning period n approaches infinity (long-run), there is a limiting probability (also called steady state probability) that the system will be in state j after a large number of transitions. This probability is independent of the ini tial state i. Due to the long-run behaviour of the Markov chain, the sewer rehabilitation problem can be solved using the assumption that the Markov decision process will be operating indefinitely. This decision process for sewer rehabilitation problem can be summarized as follows: 1. There are five possible states observed after each transition ( i = 1, 2, 3, 4, 5) associated with sewer

(5)

If no rehabilitation is implemented, the condition of sewers is assumed to deteriorate as shown in Fig. 1. This relationship is obtained by analysing the states of sewers after n transitions. The transition probability matrix after 10 transitions, P ( 1 0 ) (computed using the Chapman-Kolmogorov equation), is shown in Equa tion (6). After 10 transitions, the probability of sewers that are initially in state 2, 3, 4 or 5 and will be in state 5 is 1000. The probability of sewers that are initially in state 1 and will be in state 1, 2, 3, 4 and 5 after 10 transitions is 0107, 0043, 0031, 0034 and 0785 respectively. p(10) = (P)10 =

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

364

Wirahadikusumah,R.et al.

Table 1 Possible states of sewers State 1 2 3 4 5 Condition Condition Condition Condition Condition Condition rating 1 rating 2 rating 3 rating 4 rating 5

condition rating 1-5, with 5 being the worst (see Table 1). The index i is used for the initial state, and the index j is for the future state. 2. After each observation, a decision (action) k is chosen from a set of K possible decisions (k = 1, 2, 3, 4, 5, 6). Some of the K decisions may not be relevant (or applicable) for some of the states. Table 2 lists relevant options for each state of sewer. 3. If decision dt = k is made in state i, an immediate cost is incurred that has an expected value Cik. The costs, Cik, in Table 3 are representative estimated unit costs (dollar per metre of sewer pipes) according to a study conducted by the city of Indianapolis, IN. Total cost includes rehabilitation cost, disruption cost due to rehabilitation work, treatment cost of I/I that enters through pipe defects, and damages resulting from (possible) sewer collapse. Disruption cost has to be considered when di = 6 (dig-and-replace with concrete pipe). This cost depends on the location of sewers (rural area versus downtown area, under a busy street, etc.), as well as on the existence of other utility facilities in areas under the influence of sewers. The I/I treatment cost can be estimated using average infiltration/inflow rates (litre/ 5 year period/m) assigned to each state, multiplied by the average wastewater treatment cost (S/litre). The (possible) collapse cost of
Table 2 Decisions and relevant states Relevant to states 1, 2, 3 1, 2, 3 4 4 4 5

sewer pipe in state i can be predicted using a failure probability associated with being in state i, multiplied by the estimated damage costs resulting from sewer collapse. 4. The decision dt k in state i determines what the transition probabilities will be for the next transition from state i. These transition probabilities are denoted by p;j(k) for j=1, 2, 3,4, 5. The parameter k in pij(k) is used to indicate that the appropriate transition probability depends upon the decision k. 5. A specification of the decisions for the respective states (d1, d2, d3, d4, d5) prescribes a policy for the Markov decision process. A policy is a set of decisions (actions) for each possible state. Table 4 lists six relevant policies for large combined sewers. For example, Ra(dlt d2, d3, d4) d5) = (1, 1, 1, 3, 6) describes a policy where decision 1 ('do nothing') is made in states 1, 2 and 3; decision 3 ('shotcrete') is made in state 4; and decision 6 ('dig-and-replace with concrete pipe') is made in state 5. The transition matrix associated with policy Ra is shown in Equation (7). The last row of (7) corresponds to the assumption that sewers which are in state 5 will move to state 1 (with probability of 1) after they are replaced with new concrete pipes.

(7)

The long-run behaviour of Pa is reached after 17 transitions (P ( l 7 ) ), as exhibited by identical entries in each of the five rows. This implies that the probability of being in state j after 17 periods appears to be independent of the initial sewer condition.

Decision k 1 2 3 4 5 6

Action Do nothing Routine cleaning Shotcrete Cured-in-place pipe Reinforced fiberglass sliplining Dig-and-replace with concrete pipe

(8)

6. The objective is to find an optimal policy that minimizes the expected total discounted cost (immediate cost and discounted subsequent costs that result from the future processes).

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

Markov decision process for sewer rehabilitation

365

Table 3 Estimated unit costs in (US$/m) Failure cost (possible) 0 0 328 328 656 656 0 0 0 0

State 1 1 2 2 3 3 4 4 4 5

Decision 1 2 1 2 1 2 3 4 5 6

Rehab cost 0 16 0 16 0 16 656 1558 2231 1148

Disruption cost 0 0 0 0 0 0 0 0 0 656

l/l treatment cost 0 0 164 98 328 230 164 0 0 0

Total cost (Ci) 0 16 492 443 984 902 820 1558 2231 1804

Minimizing the total expected discounted cost For each state i (i = 1, 2, 3, 4, 5) of a Markov decision process operating under policy is the expected total discounted cost when the process starts in state i (beginning the first observed time period) and evolves for n time periods. has two components: Cik, the cost incurred during the first observed time period, and a , the expected total discounted cost of the process evolving over the remaining n 1 time periods.

Pij(k) and Cik to solve the system of five equations.

for i=l,2,

3,4, 5

for all five unknown values of V1(Rn)> V2(Rn),..., V5(Rn). Step 2: policy improvement. Using the current values of the Vi(Rn), find the alternative policy
Rn + 1 such that, for each state i, di(Rn+1) k
th

is

decision

that

minimizes

Cik

for

i=1,2,3,4,5,

(9)

Minimize
k= 1,2,...,6

with This relationship resembles the recursive relationships of probabilistic dynamic programming. The discount factor a, where 0 < a < 1, is equal to 1/(1 + r), where r is the discount rate per 5 year period. As n approaches infinity, Equation (9) converges to

for each state i = 1, 2, 3, 4, 5, and then set di(Rn+1) equal to the minimizing value of k. This procedure defines a new policy 3. Optimality test: the current policy Rn + 1 is optimal if thiis policy is identical to policy Rn. If it is, then the procedure is stopped. Otherwise, reset n = n + 1 and perform another iteration, with R n + 1 . There are three properties of this algorithim: 1. Vi(Rn + 1)Vi(R) for 1,2,....
Table 4 Considered policies Number Policy d1(R) d2(R) d3(R) d4(R)

(10) where Vi(R) can now be interpreted as the expected total discounted cost when the process starts in state i and continues indefinitely. There are five equations and five unknowns, so the simultaneous solution of this system of equations yields Vi(R). The Policy Improvement Algorithm (Hillier & Lieberman 1995) is used to solve Equation (10). The steps in this algoridim are as follows: 1. Initialization: choose an arbitrary initial trial policy R1. Set n = 1 . 2. Iteration n: Step 1: value determination. For policy Rn, use

i=1,2,3,4,5

and n =

d5(R) 6 6 6 6 6 6

1 2 3 4 5 6

Ra Rb
Rc

Rd Re Rf

1 1 1 2 2 2

1 1 1 2 2 2

1 1 1 2 2 2

3 4 5 3 4 5

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

366

Wirahadikusumah, R. et al.

Policy State i 1 2 3 4 5

ft, Decision k 1 1 1 3 6

Transition matrix State 1 2 3 4 5 1 08 0 0 0 1 2 0.2 03 0 06 0 3 0 0v5 01 04 0 4 0 02 05 0 0 5 0 0 04 0 0

Costs ($/m) State i 1 2 3 4 5

Table 5 Trial policy number 1 and associated transition matrix and cost data

Cik
0 492 984 820 1804

2. The algorithm terminates with an optimal policy in a finite number of iterations. 3. T h e algorithm is valid without the assumption that the Markov chain associated with every transition matrix is irreducible.

a Pij(k) Vj(R)] over possible decisions k, for each state i, using the previously calculated values of Vi(R1). The calculation is shown in Table 6.

Applying the Policy Improvement Algorithm for determining sewer rehabilitation policy Iteration No. 1 Choose an arbitrary initial trial policy, Ra = (1, 1, 1, 3, 6), and set n = 1. For illustration purposes, let the discount factor a = 09. The first trial policy, its transition matrix and costs are summarized in Table 5. For policy R1, use Pyi(k) and Cik to solve the system of five equations, Vi(R) = Cik + a j pij(k)Vj(Rl) for i = 1, 2, 3 , 4 , 5,

Table 6 Calculation of Equation (9) using Policy 1 0 0000 p11[k) p11(k) p11(k) p11[k) p11[k) Value of expression State 1 k C 1k p11(k) p12[k) p 1 ( k ) p14(k) p15(k) Value of expres sion 1 2 0 08 02 01 0 0 0 0 0 0 402572 384044

16 09

State 2 k C2k p
2 1

2 2

( k ) p23(k) P 24 ( k ) p25(k)

V1(R1) = 0 + 09 [08V1,( R 1 ) + 02 V 2 ( R 1 )], V2(R1) = 492 + 09[03V2(R1) 02 V4(R1)], + 05 V 3 ( R 1 ) +

1 2

492 0 443 0

03 04

05 04

02 02

0 0

626223 619587

F 3 ( R 1 ) = 984 + 09 [01 V3(R1) + 05 V4(R1) + 04 V 5 ( R 1 )], V4(R1) = 820 + 09[06V2(Rl) V5(R1) = 1804 + + 04 V 3 ( R 1 )],

State 3 k C 3k p31(k) p32(k) p33(k) p34(k)

p35(k)

1 2

984 0 902 0

0 0

01 02

05 04

0-4 04

645521 636689

09[10V1(R1)].

Solving these five simultaneous equations yields V1(R1) =402572, V2(R1) = 626223, V3(Rl) = 645521, V 4 ( R 1 ) = 652548, V5(R1) = 542715. Then, minimize the value of expression [Cik +

State 4 k c4k P41(k)p42(k)p43(k)p44(k)p45(k) 3 820 0 06 08 0 04 0 0 0 0 0 0 0 0 652548 679144 585415

4 1558 02 5 2231 1

State 5 has only one possible decision: k= 6

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

Markov decision process for sewer rehabilitation

367

Table 7 Trial policy number 2 and associated transition matrix and cost data

Policy R2 State i 1 2 3 4 5 Decision k 2 2 2 5 6

Transition matrix State 1 2 3 4 5 1 09 0 0 1 1 2 01 04 0 0 0 3 0 04 02 0 0 4 0 02 0-4 0 0 5 0 0 04 0 0

Costs State i 1 2 3 4 5

cik
16 443 902 2231 1804

Thus, the result of iteration No. 1 is

State i Decision k

1 2

2 2

3 2

4 5

5 6

This policy (2, 2, 2, 5, 6) is not the same as R1 = (1, 1, 1, 3, 6), which means that R1 is not the optimal policy. Hence, start the next iteration with R2 = (2, 2, 2, 5, 6).

Iteration No. 2 Trial n = 2, with policy R2 = (2, 2, 2, 5, 6). The trial policy, its transition matrix and costs for second iteration are summarized in Table 7. Solving iteration No. 2 similarly yields the following optimal policy:

State i Decision k

1 2

2 2

3 2

4 5

5 6

This policy (2,2, 2 , 5 , 6 ) is the same as R2 = (2, 2, 2, 5, 6), confirming that R2 is the optimal policy. This optimal policy recommends that asset managers to do routine cleaning when sewers are in state 1, 2 and 3. Once the sewers are in state 4, the optimal action is to slipline with reinforced fiberglass pipe. Digging and replacing with concrete pipe is the optimal action when sewers are in state 5. If the system will be operating indefinitely, the policy will ensure the minimization of the long-run cost of maintaining the sewer system.

Dynamic programming in conjunction with the Markov chain model The analysis illustrated in the previous section is based

on the long-run (steady state) behaviour of a sewer system. It was assumed that the Markov decision process will operate indefinitely. T h e more general analysis for the finite period Markov decision process can be performed using probabilistic dynamic programming. Dynamic programming is an optimization technique for making a sequence of interrelated decisions. This technique starts with a small portion of the original problem and finds the optimal solution for this smaller problem. It then gradually enlarges the problem, finding the current optimal solution from the preceding one, until the entire problem is solved. T h e reason that this method is attractive is that it provides a quick method of finding an optimal policy when the process goes through a limited number (say n) periods, and it does not require solving a system of simultaneous equations. In this study on combined sewer rehabilitation, a 20 year analysis was performed. The service life of the infrastructure was divided into four stages (n = 0, 1,2, 3, 4), where each stage corresponds to a 5 year period, and used the same scenario of possible states of sewer and decisions (rehabilitation methods) as shown in Tables 1 and 2, respectively. In this example, the value of 09 was assigned for a. The objective of probabilistic dynamic programming was to minimize the total expected discounted cost as stated in Equation (9). Starting at stage 4, the optimal policy that minimized the objective function was found. This process was iterated at each of the preceding stages to optimize the objective function at each stage until the optimization process was completed for the entire problem. The solution for the sewer rehabilitation example using dynamic programming is shown in Table 8. The last columns in each subtable describes the optimal rehabilitation option at every stage (X*i = 1, 2, 3, 4, 5, 6, corresponds to the optimal rehabilitation methods: 'do nothing', 'routine cleaning', 'shotcrete', 'cured-in-place pipe', 'sliplining with reinforced fiberglass liner' and 'dig-and-replace with concrete pipe', respectively). Note that the optimal policy at stage 1 (X*1 = 2, 2, 2, 5, 6 for state 1, 2, 3, 4, 5, respectively) is

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 358-370

368

Wirahadikusumah,R.etal.

n = 4 (year 20) S4

f*4(S4) 0 443 902 820 1804

X*4 1 2 2 3 6

Table 8 Solution of dynamic programming

1 2 3 4 5

n= 3 (year 15)

x3 s3
1 2 3 4 5

f3(S3,X3) =C i +a { Pi..f*4(S4)} 1 7972 116535 208415 2 5627 107480 200951 138419 195702 223097 180446 3 4 5 6

f*3(S3)
56-27 107480 200951 138419 180446

X*3
2 2 2 3 6

n= 2 (year 10)

x2 s2
1 2 3 4 5

f2(S2,X2) = Ci+a{ Pi f*3(S3)} 1 23398 193576 243760 2 15871 180242 241186 212403 252572 228161 185510 3 4 5 6 f*2(S2) X*2 15871 180242 241186 212403 185510 2 2 2 3 6

n=1 (year 5) 1 2 3 4 5

f1(S1,X1) = Ci+a{P i f*2(S2)} 1 43871 244644 282497 2 30718 2342-38 276885 266179 318058 237381 3 4 5 6

f*1(S1) X*1
2 2 2 5 6

30718 234238 276885 237381 194730 194730

n= 0 (year 0)

f0(S0,X0) = C i +[ PiS1)} So 1 1 2 3 4 5 64280 279784 300269 2 47603 271024 295623 3 4 5 6

f*0(S0) X*o
2 2 2 5 6

47603 271024 295623 250743 308188 3666-54 250743 208092 208092

also the optimal policy for the infinite period prob lem, as described in the previous section. Again, the optimization process at stage 0 gives the same solu tion.

This procedure can be extended to 25, 30, 35, etc., year analysis periods. The analyses for longer periods using the dynamic programming technique yield optimal policies, that are identical to the ones

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 | 4 , 58-370

Markov decision process for sewer rehabilitation

369

Table 9 Long-term planning for sewer rehabilitation (total length = 77 224 m) Year 0 5 10 15 20 25 30 35 40 45 50

Sewer network condition (%) Good 6190 8381 8283 7762 7453 7363 7370 7393 7406 7409 74-09 Fair 100 659 1102 1269 1284 1259 1240 1233 1232 1234 1234 Moderate 900 220 308 502 608 635 631 622 618 616 617 Poor 1950 380 220 343 455 500 506 500 495 494 493 Severe 860 360 088 123 201 243 254 252 249 247 247 Rehab cost (x10 6 US$) 421 109 58 82 108 119 121 120 119 119 119

obtained using the policy improvement algorithm. This validates the initial assumption that the dynamic programming method provides good approximations of the optimal policy for the infinite period analysis. If the decision maker applies the suggested optimal rehabilitation strategy (i.e. X*1 = 2, 2, 2, 5, 6 for states 1 , 2 , 3 , 4 , 5 , respectively) at each stage, the expected condition of the sewer network at the beginning of each 5 year period can be estimated by multiplying the initial condition with the corresponding n -step transition matrix. Suppose the current condition of a large combined sewer network (total length of 77 224 m) within a municipality is as follows: 61 0% in good condition (state 1), 10% in fair condition (state 2) 9 0% in moderate condition (state 3), 19 5% in poor condition (state 4), and 8 6% in severe condition (state 5). The expected condition of this network and the 5-yearly budgets needed are shown in Table 9. The figures in Table 9 indicate that the condition of sewer network starts to level off at year 20. The long-term condition of the entire sewer network includes 74 1% in good condition, 12 3% in fair condition, 6 2% in moderate condition, 5% in poor condition, and 2 4% in severe condition. The rehabilitation budget needed every 5 year period at steady state is about $ l l 9 million.

the optimal time of application are determined. The advantages of the proposed method are as follows: (1) it allows us to analyse sewer sections individually, and (2) it provides an optimization procedure that can efficiently find the optimal solution. The aforementioned probabilistic dynamic programming model is unconstrained (i.e. rehabilitation options are suggested assuming budgets are not limited). If the aggregate cost over all sewer segments within a network is higher than the available budget, then a prioritization scheme needs to be developed. Further analysis must be performed to determine which sections should be maintained or rehabilitated within the given budget to acquire a network optimal solution. Several prioritization criteria can be used: the most important sections (located under busy streets, etc.), the worst sections, or the oldest sections, etc. Based on these criteria, sewer sections are ranked in a descending order. The available budget is then allocated to the sections by selecting one section at a time from the ranked section list. The search for section selection is stopped when the available budget is completely used. Sewer sections in the prioritized list that will not be rehabilitated within the present period, will deteriorate further and the effect of deferred maintenance/rehabilitation on the total cost of sewer rehabilitation can also be estimated. The model described in this paper can be modified to account for the assumption that older sewers would deteriorate faster. The 'zoning' method (Butt et al. 1987) can be applied by using a different transition matrix for each zone. By dividing the life of sewers into zones, there is an added advantage that the computation of the complex objective function shown in Equation (4) becomes reasonably manageable.

SUMMARY
The proposed sewer management model is based on the life cycle approach and incorporates information about the current condition of sewers and also predicts their future condition. By taking into account the future condition as a consequence of the present rehabilitation action, the best alternative as well as

1999 Blackwell Science Ltd, Engineering, Construction and Architectural Management 6 [ 4 , 358-370

370

Wirahadikusumah, R.et al

A major challenge facing the developers of this m o d e l is t h e assessment a n d evaluation of disruption costs. In m a n y cases, it would b e necessary to project h o w t h e future condition of 'unrehabilitated' sewers in critical condition can affect c o m m e r c e a n d the c o m m u n i t y . T h e capability of the m o d e l to assist in decision making can b e e n h a n c e d by testing its sensitivity t o varying d i s c o u n t rates a n d inflation rates. O t h e r areas of i m p r o v e m e n t w o u l d b e the inclusion of an e x p a n d e d list of rehabilitation options, t h e developm e n t of a selection p r o c e d u r e for short-listing candidate options for extensive analysis, a n d a detailed analysis of considerations (such as social, political, etc.) what would prevent the application of 'optimal' rehabilitation m e t h o d s .

ACKNOWLEDGEMENTS
T h i s research was funded by grant C M S - 9 8 0 0 2 0 4 from t h e National Science F o u n d a t i o n , w h o s e support is gratefully acknowledged. Any o p i n i o n s , findings, conclusions, or r e c o m m e n d a t i o n s expressed in this study are those of t h e writers a n d d o n o t necessarily reflect the views of t h e National Science F o u n d a t i o n .

REFERENCES
Abraham, D.M., Wirahadikusumah, R., Short, T.J. & Shahbahrami, S. (1998) Optimization modeling for sewer network management. Journal of Construction Engineering and Management, American Society of Civil Engineer, 124(5), 402-410. Arditi, D.A. & Messiha, H.M. (1996) Life-cycle costing in municipal construction projects. Journal of Infrastructure Systems, American Society of Civil Engineer, 2(1), 5-14. American Society of Civil Engineers (1998) 1998 Report Card for America's Infrastructure, Washington DC. Ben-Akiva, M. & Ramaswamy, R. (1993) Approach for predicting latent infrastructure facility deterioration. Transportation Science, 27(2), 174-193. Burgess, E.H. (1993) Planning model for sewer system rehabilitation. In: Proceedings of the International Conference of Pipeline Division, pp. 3 0 - 3 8 . American Society of Civil Engineer, San Antonio, TX. Butt, A.A., Shahin, M.Y., Feighan, K.J. & Carpenter, S.H. (1987) Pavement performance prediction model using the Markov process. Transportation Research Record 1123, Transportation Research Board-National Research Council, pp. 1 2 19.

Butt, A.A., Shahin, M.Y., Carpenter, S.H. & Carnahan, J.V. (1994) Application of Markov process to pavement management systems at netzvork level. Proceedings of The Third International Conference on Managing Pavements, Transportation Research Board-National Research Council, pp. 159-172. Feighan, K.J., Shahin, M.Y., Sinha, K.C. & White, T . D . (1989a) A prioritization scheme for the micro Paver pavement management system. Transportation Research Record 1215, Transportation Research Board-National Research Council, pp. 89-100. Feighan, K.J., Shahin, M.Y., Sinha, K.C. & White, T.D. (1989b) A sensitivity analysis of the application of dynamic programming'to pavement management systems. Transportation Research Record 1215, Transportation Research Board-National Research Council, pp. 101-114. Fick, B., Johnson, M. & Johnson, J. (1993) Predictive model aids in development of systermvide rehabilitation program. In: Proceedings of the International Conference of Pipeline Division, pp. 616-627. American Society of Civil Engineer, San Antonio, TX. Grant, A.A. (1995) Civil infrastructure systems: the big picture. Journal of Infrastructure Systems, American Society of Civil Engineer, 2(2), 7 8 - 8 1 . Greeley & Hansen (1996) Combined sewer infrastructure assessmentFinal report, City of Indianapolis, Department of Capital Asset Management. Hillier, S.F. & Lieberman, G.J. (1995) Introduction to Operations Research, 6th edn. McGraw-Hill, New York. Lu, Y. & Madanat, S. (1994) Bayesian updating of infrastructure deterioration models. Transportation Research Record 1442, Transportation Research Board-National Research Council, pp. 110-114. Malik, O., Pumphrey, N . D . & Roberts, F.L. (1997) Sanitary sewers: state-of-the-practice. In: Proceedings of the Conference on Infrastructure Condition Assessment: Art, Science, and Practice, pp. 297-306. American Society of Civil Engineer, Minneapolis. Reyna, S.M. (1993) Optimal planning of sewer systems rehabilitation. PhD dissertation, School of Civil Engineering, Purdue University, West Lafayette, IN. Ross, S.M. (1997) Introduction to Probability Models, 6th edn. Academic Press, San Diego. Scherer, W.T. & Glagola, D.M. (1993) An overview of Markovian models for bridge management systems. In: Proceedings of the ASCE Conference on Infrastructure Planning and Management, Denver, pp. 382-386. Serpente, R.F. (1993) Understanding the models of failure for sewers. In: Proceedings of the International Conference on Pipeline Division, pp. 8 6 - 1 0 0 . American Society of Civil Engineers, San Antonio, T X . WEF-ASCE (1994) Existing sewer evaluation and rehabilitation, 2nd edn. Water Environment Federation, Alexandria, USA.

1999 BlackweN Science Ltd, Engineering, Construction and Architectural Management 6|4,358-370