Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Evaluating the Quantitative Effects of Workflow Systems Based on Real Cases

Michiko Oba Software Division, Hitachi Ltd. (mie_oba@soft.hitachi.co.jp) Sen'ichi Onoda, Norihisa Komoda Department of Information Systems Engineering Faculty of Engineering, Osaka University (onoda@ise.eng.osaka-u.ac.jp, komoda@ise.eng.osaka-u.ac.jp) Abstract
The effects of introducing a workflow system include improved work efficiency, reduced processing time (that is, turnaround time), standardized business processes, and made processing paperless. However, no concrete methodology for quantitatively evaluating these effects has been established. In this paper, by analyzing some examples of introducing workflow systems, we extracted the factors that influence the effects, and we used quantification theory type I to create a mathematical model for predicting the rate of reduction of processing time as an effect of introducing a workflow system. As a result, we concluded that (1) the process improvement and (2) business process classification before the introduction of the workflow system as the factors that influence the effects. As a predictive model, multiple correlation coefficient R is 0.932, showing that the two factors have a considerable influence. position in a company and managers of related departments must understand the need to improve company-wide efficiency beyond the boundaries of individual departments. To enhance the return on the investment when employing BPR, a business should introduce a workflow system first in a business process in which they can expect the highest effectiveness. The potential effects of workflow systems are as follows
[3]:

Reducing processing time (i.e. turnaround time) Standardizing business processes Promoting a paperless environment Improving the management efficiency of business process

Facilitating system changes Once a workflow system is introduced, in the rate of reduction processing time can be quantitatively measured by comparing the processing time before and after its introduction. However, we need to predict the rate of reduction of processing time before the systems introduction to evaluate the return on investment in BPR beforehand. The examination of evaluation items for workflow systems has been proceeding for some time [4]. What is needed now is a method of determining the quantitative effects of these evaluation items. Currently, there is no established methodology for determining these quantitative effects before introducing the workflow system. To cope with this lack of established methodology, this paper will suggest a method of predicting the rate of reduction of the processing time according to the following: The prediction of the process improvement that can be inferred by analogy before the introduction of the workflow system The original type of business process before the introduction of the workflow system (paper-based 1

1. Introduction
Maintaining and promoting the constant growth of a company often demands a radical redesigning of business processes, as well as constant improvement of activities within the company. Such redesigning is called Business Process Reengineering (or BPR). Workflow methodologies are attracting attention as a promising method of implementing BPR, and many workflow management systems are becoming prevalent on the market. What is referred to here as a workflow is best defined as a flow of work consisting of several processing steps (simply called "processes" hereafter) to attain a goal. The unit of processing to be performed according to the flow is called a work item. Work items in electronic form are transmitted on a network to the computer of the operator who performs processing on these work items that arrive at the computer. The workflow management system defines, manages and controls the workflows [1][2]. The aim of introducing a workflow is to achieve BPR. To implement BPR, those in the highest management

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

processing performed manually, computer system processing using host computers, or a combination of both) Both of the above were obtained by analyzing the situations where Hitachis Groupmax Workflow management product [5] was introduced.

Figure 1 Example of a business process definition

2. Factors influencing the effects of introduction

2.1 Workflow management system Workflow products have the following five main features: A development feature for defining the flow of jobs (business processes) and creating form screens. An execution management feature for controlling workflows. A management feature for defining users and roles. A monitoring feature for monitoring the progress status of jobs and managing log information. A job reexamination feature for checking defined flows for validity through simulation and analyzing jobs based on log information. The workflow engine handles tasks along a predefined job flow called a business process. The definition of the business process for Groupmax, one of the workflow products, is created by connecting processing nodes (representing job processing) and control nodes (representing control methods) with the use of arrows. Control nodes are related to the control of various processes including the parallel feeding of a single document to more than one person, conditional branches, acting for a person in charge when he or she is away, and returning forms when there is a problem. Figure 1 shows a definition example for a business process related to the purchase of material using the above-mentioned features. In Figure 1, if a created form for purchase is rejected in the approval process by the requesting department, it is returned to the applicant for the reentry process. If it is approved by that department and then by the accounting department in its approval process, it is forwarded to the ordering process by the purchase department. The same form is duplicated and sent to the two departments for parallel processing, to the requesting department for the acceptance process and to the accounting department for the confirmation process. When the parallel processing is completed, the entire job is completed. The workflow allows parallel processing for jobs that was traditionally processed serially, so it is effective as a means of improving job efficiency. 2.2 TAT reduction rate This paper uses an estimate of the reduction in processing time as an evaluation item that greatly influences the effects of system introduction. This item shows the effect of workflows including the business process improvement for reducing the total processing time (the turnaround time) from start to finish of a work item in a business process. The evaluation of this effect uses the TAT (Turn-Around Time) reduction rate Red that is defined as follows. It is assumed that the TAT reduction rate (Red) indicates how much workflow introduction improves business processing time as compared to before system introduction. The following shows the definition of Red, where Org indicates the processing time (in days) before implementation and Imp indicates the processing time (in days) after implementation. Red (%) = Org - Imp 100 Org

2.3 Candidate of influence factors Predicting the TAT reduction rate (Red) requires determining the factors that have an influence on it. There are three possible factors that influence the reduction in processing time by the workflow system: Process improvement factor of a workflow by reducing the number of processes and by making processes parallel Business process classification before system implementation (paper-based processing performed manually or computer system processing) Business process size These items appear to influence the TAT reduction rate.

2
0-7695-0493-0/00 $10.00 (c) 2000 IEEE 2

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

(1) Process improvement factor

The process improvement factor is evaluated based on the rate of reduction in the number of processes before and after the business process improvement. The following shows the definition of the process reduction rate P, where Porg indicates the number of processes before system implementation and Pimp indicates the number of processes after implementation. P(%) = Porg - Pimp 100 Porg

3. Examples of real cases

Twenty-five real cases were selected in which Groupmax Workflow was introduced for the prediction expression determination and prediction expression verification of the TAT reduction rate (Red). Table 1 lists the evaluation results of the influential factors in each case. The turnaround time (TAT) after the introduction is extracted quantitatively using logs. The number of processes after the introduction is extracted from the workflow definition data. The factors before the introduction are the results of investigations by the system engineer collected from anecdotes, and therefore contain inaccuracies. Workflows are classified by target jobs into three categories: ad hoc workflows, administrative workflows, and production workflows [6]. In our sample data, the two cases of insurance examinations and six cases of purchasing including material purchasing are classified as production workflows. The four cases of traveling expense reimbursement and application filing including settlement jobs are classified as administrative workflows. The twenty-five cases from the sample data are used as follows: Data for estimation expression determination Twenty cases (case Nos. 1 to 20), including two cases of insurance examination, three cases of traveling expense reimbursement, five cases of purchasing, and ten cases of application filing account for more than three fourths of the total cases and are used as the data for the estimation expression determination. Data for estimation expression verification Five cases (cases No. 21 to 25) including one case of traveling expense reimbursement, one case of purchasing, and two cases of application filing are used as the data for the estimation expression verification. The breakdown of the business processes evaluated is as follows: Three cases of reimbursement of travel expenses Five cases of materials and other purchasing Two cases of examining insurance Ten cases of filing applications including the approval process The breakdown of sample data for the prediction is as follows: (1) Process improvement factor: three cases for Great, three cases for Moderate and 14cases for Small. (2) Business process crassification before the implementation: 14 cases of Paper, three cases of 3
0-7695-0493-0/00 $10.00 (c) 2000 IEEE 3

In this case, the following determinations were made based on the graph distribution of the process reduction rate (P) and TAT reduction rate (Red): If the process reduction rate P is 50% or greater, the process improvement factor is determined as Great; 30 to 49% is determined as Moderate; and 29% or less is determined as Small. The system engineer can also determine the process improvement factor based on additional criteria such as the amount of parallel processing and the amount of automatic calculation in the definition of the business process.

(2) Business process classification before system implementation

The business processes are classified into three types before system implementation: Paper-based: Paper-based manually processing performed

Computer: Processing by a computer system using host computers Mixed: A combination of paper-based manual processing and computer system processing

(3) Business process size

Business process size is determined based on the number of processes before process improvement. Based on the graph distributions of the number of processes (Porg) and TAT reduction rate (Red) before the job improvement, the job scale is determined as follows: If the number of processes (Porg) before the job improvement is 15 or more, the scale is determined as Large; 10 to 14 is determined as Medium; and 9 or less is determined as Small. The system engineer should comprehensively determine the business process size, considering the number of related departments, how complicated the business process is, and the nature of the business processes.

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Computer, and three cases of Mixed. (3) Business process size: Three large-scale cases, eight Medium-scale cases, and nine Small-scale cases The sample data is well balanced, covering all categories of the influence factors. Likewise, well-balanced data has been selected as sample data for verification. The evaluation of the sample data indicates a noticeable effect on the TAT reduction rate of 75% to 85% when the process improvement factor is Great. For the business process for filing applications whose crassification before system implementation is Paper, the TAT reduction rate is 50% or greater even when the process improvement factor is Small. On the other hand, we find no correlation between the business process size and the TAT reduction rate. Therefore, we analyzed the cases when the business process size is included in the influence factors of the TAT reduction rate and when it is not. Table 1 Evaluation results of real cases Case Process Processing Business TAT No. improvement Type process reduction factor size rate
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Great Great Small Moderate Small Great Small Small Small Small Small Moderate Small Small Small Small Small Moderate Small Small Moderate Great Small Small Small Paper-based Paper-based Computer Paper-based Paper-based Mixed Mixed Computer Mixed Paper-based Paper-based Paper-based Computer Paper-based Paper-based Paper-based Paper-based Paper-based Paper-based Paper-based Paper-based Mixed Mixed Paper-based Paper-based Small Small Medium Small Large Small Medium Medium Small Large Medium Medium Medium Small Large Small Small Small Medium Medium Medium Large Medium Small Large 80 85 40 50 50 75 40 38 36 60 50 50 33 50 64 60 50 66 57 50 56 67 40 57 50

4. Analysis using quantification theory type I

4.1 How to predict the TAT reduction rate As discussed in Section 2.3, the factors that influence the TAT reduction rate are based on quantitative data. Quantification theory type I [7] is an efficient method for analyzing multivariate data for which variables are quantitative data. We use quantification theory type I to predict the TAT reduction rate. We associate the terms for quantification theory type I with the influence factors as follows. We defined the three influence factors of the TAT reduction rate (that is, the process improvement factor, the business process type before system implementation, and the business process size) as items. Here, item of quantification theory type I called FACTOR. The definition of FACTOR (=item) j is: j=1: Process improvement factor j=2: Business implementation process classification before

j=3: Business process size Category k for FACTOR j is: j=1 (k=1: Small, k=2: Moderate, k=3: Great) j=2 (k=1: Paper-based, k=2: Mixed, k=3: Computer) j=3 (k=1: Small-scale, k=2: Medium-scale, k=3: Largescale) For the model of quantification theory type I, if the number of categories for FACTOR j is Kj, then the i of TAT reduction rate Red (yi) for case i is predictor y given as follows.
i = y

b
j = 1 k =1

Kj

jk i ( jk )

(4.1)

Here,i(jk) is defined as follows:

( jk ) =

1 IFACTOR j of case i belongs to category k 0 Otherwise

bjk is called a category score: the quantity given to category k of FACTOR j. For quantification theory type I, we determined the value of bjk that minimizes the sum of squares of the deviation i . of yi from y

i =1

( yi

b
j = 1 k =1

Ki

jk ( jk ))

Minimum

n indicates the total number of cases. Determining bjk 4

0-7695-0493-0/00 $10.00 (c) 2000 IEEE 4

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

involves solving M simultaneous equations, where M is equal to:

(

K
j =1

+ 1 (number of items - 1))

extremely small compared to other influence factors and indicates a very low contribution rate to the TAT reduction rate. As a result, it was concluded that the business process size has no positive influence on the TAT reduction rate, and removed it from the influence factors. 4.3 Analyzing the effects using quantification theory type I Table 3 shows the results of analyzing the evaluation data in Table 1 by quantification theory type I for the two influence factors (process improvement factor and business process type before system implementation). Expression 4.3 shows the prediction expression of the TAT reduction rate (Red).
= - 4.8x11 - 3.5x12 + 25.8x13 y

Here, at least eight cases are required. 4.2 Analyzing effects using quantification when business process size is included in the influence factors Table 2 shows the results of analyzing the evaluation data in Table 1 by quantification theory type I for the three influence factors (process improvement factor, business process classification before system implementation, and business process size). Expression 4.2 shows the prediction expression of the TAT reduction rate (Red).
= - 5.1 x 11 - 2.9 x 12 + 26.7 x 13 y

+ 4.7x21 -9.3x22 - 12.4x23 + 54.2

(4.3)

+ 4.6 x 21 - 8.6 x 22 - 12.9 x 23 + - 0.2.x 31 -1.1 x 32 + 4.5 x 33 + 54.2 Here, xjk is defined as follows.
1 FACTOR j of the case belongs to category k x( jk ) = 0 Otherwise

(4.2)

Table 3 Analysis of TAT reduction rate by quantification theory type I (2 FACTORS)

FACTOR Category Category score -4.8 -3.5 25.8 4.7 -9.3 -12.4 Range Multiple correlation coefficient 0.906

Table 2 Analysis of TAT reduction rate by quantification theory type I (3 FACTORS)

FACTOR Category Category score -5.1 -2.9 26.7 4.6 -8.6 -12.9 -0.2 -1.1 4.5 2.6 0.363 Range Multiple correlation coefficient 0.917

Process Improvement factor Business process Type before Implementa tion Business Process size

1. Small 2. Moderate 3. Great 1. Paper-based 2. Mixed 3. Computer 1. Small-scale 2. Mediumscale 3. Large-scale

31.8

Process Improvement Factor Business process Type before Implementation

1. Small 2. Moderate 3. Great 1. Paperbased 2. Mixed 3. Computer

30.6

17.1

0.816

17.5

0.832

Figure 2 shows a graph indicating the relationship between the effect of Expression 4.3 predicted in the quantification theory type I and each FACTOR, with the z-axis representing the effect and the x-axes and y-axes representing the process improvement level and the job execution format before improvement, respectively. Table 4 shows the residual between the predicted result in the quantification method of the quantification theory type I above and the TAT reduction rate (Red) for the verification data (Nos. 21 to 25) listed in Table 1. Also shown in Table 5 is the residual between the predicted result and TAT reduction rate (Red) with data for estimation (Nos. 1 to 20) listed in Table 1.

Table 2 indicates that the multiple correlation coefficient of the business process size is 0.363 which is very small as compared with other influence factors. The category score has also lost its linearity. We can use range Rj as a scale to measure the contribution rate of each influence factor or item (=FACTOR) to the TAT reduction rate.
Range R j of item j = max (b jk ) - min (b jk ) k k

The range of the business process size is 2.6, which is 5

0-7695-0493-0/00 $10.00 (c) 2000 IEEE 5

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Figure 2 Relationship between the prediction effect and the FACTOR

Table 5 Residual reduction rate

Case No.

of

prediction

and

TAT

Red
80 85 40 50 50 75 40 38 36 60 50 50 33 50 64 60 50 66 57 50

Predictor
84.7 84.7 37.0 55.4 54.1 70.7 40.1 37.0 40.1 54.1 54.1 53.4 37.0 54.1 54.1 54.1 54.1 55.4 54.1 54.1

Residual
-4.7 0.3 3.0 -5.4 -4.1 4.3 -0.1 1 -4.1 5.9 -4.1 -5.4 -4.0 -4.1 9.9 5.9 -4.1 10.6 2.9 -4.1

T A T r ed u ctio n r at e
10 0

8 4.7 7 0.7
50

6 7.6

5 5.4 5 4.1 4 0.1

G rea t M ode rate Sm all P a per ba sed

4 1.4 3 8.3
@M ixe d

3 7.0
C om p ute r

P r oc es s im p ro v em en t

B u sin e ss p ro ce ss ty p e be fo re sy st em im p lem e nt atio n

Table 4 Residual between the prediction value (with data for verification) and the TAT reduction rate
Case No. 21 22 23 24 25

Red
60 67 40 57 50

Predictor
53.6 70.7 40.1 54.1 54.1

Residual
6.4 -3.7 -0.1 2.9 -4.1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Where each error term e independently follows normal distribution N(0,2). This model is mathematically equal to the multiple regression model for a dummy variable for external criterion result y; therefore, the linear estimation and test theories can be applied. To determine whether a given FACTOR (FACTOR j) contributes to the external criterion significantly, assume the model excluding FACTOR j. model: y i =

4.4 Considering a predictive expression For the predictive model of Expression 4.3, the multiple correlation coefficient R is 0.932, and the contribution rate (R2), the square of R, is 87.0%. This shows that the two influence factors have a considerable influence on the TAT reduction rate. In general, the closer the multiple correlation coefficient gets to 1, the better the prediction results will be. Generally a practical model must have a multiple correlation coefficient of at least 0.85; therefore, the predictive model of Expression 4.3 provides sufficient accuracy. Since this model uses two FACTORS, it requires at least six cases. Thus, the 20 cases shown above are sufficient in quantity. The influence degree for FACTORS can be determined by conducting an F test for the cases used this time. First, let the following be assumed as a probability model for estimation: model: y i =

j =1 k =1 j j

Kj

b jk i ( jk ) + e i (i=1,2,,n)

We can let the illegal workflows for the residual of the model and model be Sand S respectively when a multiple regression formula is applied to the two models. A test can be conducted using statistic F0.

F0 =

( S S) /( K j ' - 1) S /( n

K
j =1

+ 2 - 1)

j = 1 k =1

Kj

b jk i ( jk ) + e i (i=1,2,,n)

As compared with the limit value in the F distribution, F0 is greater both in the process improvement factor and the job execution format before improvement, indicating that the influence degree is high. Both the "process improvement factor" and the "business process classification before system 6

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

implementation" have linear category scores and large ranges. This shows that both factors greatly influence the TAT reduction rate. For the "process improvement factor", the best results can be expected when the process improvement factor is Great, while there is little difference in effect when the process improvement factor is Small or Moderate. For the "business process classification before system implementation", applying workflows to paper-based processing has the greatest influence. The multiple correlation coefficient is 0.906 for the process improvement factor, and 0.826 for the business process classification before the implementation. This shows that both factors have a great influence on the TAT reduction rate. From the range and multiple correlation coefficient of these influence factors, we find that the process improvement factor has a greater contribution rate than the business process classification before implementation. The margin of error for the prediction is 5.072. Evaluation of the residual for verification sample data (Nos. 21 to 25) listed in Table 4 shows that only case 21 exceeds the margin of error for the prediction. We can assume that case 21 contains parallel processing, and that the processing time was reduced due to the parallel processing at the time of workflow creation. Excluding case 21, we can say that the estimation model is effective because appropriate predictors are obtained within the margin of error for the prediction. On the other hand, the residual for the estimation sample data (Nos. 1 to 20) listed in Table 5 used for estimation expression determination indicates that cases 4, 10, 12, 15, 16, and 18 exceed the margin of error for the prediction, but that cases 4, 10, 12, and 16 are almost equal to the margin of error. Excluding cases 4, 10, 12, 15, 16, and 18, the appropriate predictors are obtained within the margin of error for the prediction. Cases 10 and 18 contained parallel processes. We assumed this shows the effect of handling parallel processes in a workflow on the reduction of the time required for processing. Especially, we found a noticeable effect in case 18 that contains two portions of parallel processes. Noticeably introducing the workflow had a remarkable effect in case 15 in which tasks required the approval of higher-ranking personnel by going through many related departments in dispersed locations. Applying the workflow to heretofore paper-based processing remarkably reduced the processing time. Likewise, in case 16 we found that applying the workflow to paper-based filing processes with many related departments greatly reduced the processing time.

5. Conclusion
By analyzing sample cases of introducing workflow systems in real organizations, we extracted the factors that influence the effects, and we used quantification theory type I to create a mathematical model for predicting the rate of reduction of processing time, from among the effects of implementing the workflow system. As a result, the prediction model suggested is practical. This means that before introducing a workflow system, quantitative effects of that system based on quantitative data can be predicted.

References
[1] G. Klinker, M. Linster, and G. Yost, Cooperative
system for workgroups, IEEE Expert, Vol. 10, no. 3, 1995, pp. 39-40

[2] T. M. Koulopoulos, The Workflow Imperative:

Building Real World Business Solutions, Van Nostrand Reinhold, New York, NY,U.S.A., 1995

[3] P. Lawrence, Workflow Handbook 1997, John Wiley

& Sons, Chichester, U.K., 1997

[4]

J. Becker, C. Uthmann., M. Muhlen, and M. Rosemann, Identifying the Workflow Potential of Business Processes, in Proceedings of the 32nd Hawaii International Conference on Systems Sciences (HICSS-32) on CD-ROM (1999.1.5-8, Wailea, Maui, Hawaii, U.S.A) IDG Communications, Tokyo, Japan, 1999.(in Japanese)

[5] N. Shishido, A. Sugiyama, Groupmax Workflow,

[6] D. Georgakopoulos and M. Hornick : An Overview

of Workflow Management: From Process Modeling to Workflow Automation Infrastructure, Distributed and Parallel Database, 3, pp.119-153 (1995)

[7] Hayashi, C., On the prediction of phenomena from

qualitative data from the mathematics-statistical point of view, Annals of the Institute of Statistical Mathematics, 3, pp. 69-98, 1952

7
0-7695-0493-0/00 $10.00 (c) 2000 IEEE 7