Integrated approach for predicting final performance by connecting multiple micromodels

Sivanantham M sivanantham.m@sonata-software.com
Sonata Software Limited, www.sonata-software.com 1/ 4, APS Trust Building, Bull Temple Road, N.R. Colony, Bangalore - 19

Presented at 2nd International Colloquium on High Maturity Best Practices (HMBP 2011) 26-Aug-2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

2 of 10

1. Abstract
In dedicated Offshore Development centre, the number of defects likely to be uncovered during System Acceptance Test Phase and the amount of effort required for fixing these defects is one of the top considerations for planning and monitoring. The Project Manager is interested in progressively predicting these final performance measures based on what performance is achieved after completion of each phase so that necessary course corrections can be done to ensure that the project is on track to meet the end objectives. Based on the analysis of historical data, an integrated model was developed with each micro-model predicting the outcome of one sub-process/activity which will feed in as an input for predicting the outcome of the next sub-process/activity and eventually predicting the final performance i.e. System Acceptance Test Defect Density and the support effort. This helps the project manager in deciding the number of resources and effort to be allocated for bug fixing so that rest of resources can be used for Feature Development work. Extensive use of these prediction models by the practitioners in delivery team has demonstrated the predictability of project performance and met the high maturity requirements, leading to Sonata Software achieving SEI CMMI v1.2 Level 5.

2. Problem Statement
Prediction is very difficult, especially if it's about the future. - Niels Bohr The top challenge for any software project is the ability to set the right commitments towards budget, schedule and quality and successfully meet the same. Setting of SMART project objectives; tracking relevant metrics to see the projects progress; predicting overall project performance and taking corrective actions become easier said than done. The difficulty in making the realistic commitments during the preliminary stages of the project arises as many inputs are not known and clear at the beginning of the project where as final performance will depend on the decisions taken and the actual performance observed in each of the sub-processes. Absence of effective decision support tool which can help in adaptive decision making based on the contextual performance observed in a particular phase of the project could be risky. The project should be able to predict what will happen in the next phase based on previous phase performance and decide the appropriate corrective actions for the subsequent phase in-order to have adequate confidence in meeting the project objectives eventually.

3. Solution Approach
Every problem has in it the seeds of its own solution. If you don't have any problems, you don't get any seeds. - Norman Vincent Peale An approach was devised to build integrated prediction model by developing connected micro models as shown in Figure 1 below. Each micro model will Sonata Software Limited HMBP 2011 August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

3 of 10

predict the outcome of a sub-process based on controllable factors and other input measures and eventually the final performance metric will be predicted.

Figure 1: Sonatas Approach for Integrated Prediction Model Development

Sonata Software Limited

HMBP 2011

August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

4 of 10

4. How the integrated prediction model was developed

If you find a good solution and become attached to it, the solution may become your next problem. - Robert Anthony Based on the data type and available data from Process Performance Baseline, it was decided to use the appropriate modelling techniques for building each Micro model to eventually predict the System Acceptance Test Defect Density and Support Effort as shown in Figure 2 in the next page. The Micro model 1 for predicting Code Defect Injection Rate was developed using Dummy Variable Regression Analysis and the decision to perform FS Walkthrough with Business or not is used as a Dummy Variable along other input variables such as Requirement Analysis Delivery Rate and Code Creation Delivery Rate. The Micro model 2 for predicting Code Review Defect Density was developed using Multiple Regression Analysis and Code Defect Injection Rate (which is the out put of the previous Micro Model) and Review Effort are used as inputs for prediction. The Micro model 3 for predicting System Acceptance Test Defect Density was developed using Multiple Regression Analysis and Code Defect Injection Rate & Code Review Defect Density (which are the out put of previous Micro Models) are used as input for prediction. The Micro model 4 for predicting Support Effort was developed using a Simple Regression Analysis and System Acceptance Test Defect Density (which is the out put of previous Micro Model) is used as an input for prediction. All regression models are verified and qualified against regression assumptions and PPM guidelines established in QMS which stipulates criteria such as Adj R Square should >=60%, P value against each of the Xs (should be <0.05), VIF (Variance Inflation factor) value should be < 10 etc. Integrated Model is developed in Excel and each Micro-model was developed in one sheet and connected to the next sheet so that output from one model is fed in as an input in the next sheet. In all Regression models, the prediction interval is computed so that the Project Manager can know the likely range estimate for chosen confidence level in addition to the point estimate. Additionally, using the Regression equations derived, these models can also be run as Simulation Model and the Project Manager can choose how he/she wants use the model.

Sonata Software Limited

HMBP 2011

August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

5 of 10

Figure 2: Conceptual View of Integrated Prediction Model

5. Using integrated model for prediction for feature release

Economists give their predictions to a digit after the decimal point to show that they have a sense of humour Unknown The integrated model is used in a particular enhancement release with a size of 100 FP for predicting the quality of delivery and support effort needed during System Acceptance Testing and the same is illustrated below. Prediction was made during planning stage. The screenshots below depicts usage of model for this release. It was decided to perform the FS Walkthrough with Business and Planned values for Requirement Analysis and Code Creation are entered and this is used a Regression Model in Sonata Software Limited HMBP 2011 August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

6 of 10

this scenario. Alternatively, the Project Manager could use the Basis prediction as PPB Values and run as a Simulation Model if required.

Inputs to the Prediction Model (Code Injection Rate & Code Review Defect Density)
Inputs Value 100 Yes Coding EffortEntered Value Entered Values 0 Person-day/FP 15 0.67 45 0.31 Person-days Person-day/FP Person-days Defects 0.07 3 95 Planning Person-day/FP Person-days % Unit of Measure FP Binary Value

Size Functional Specification Walkthrough with Business Basis for Coding Injected Defects Prediction Basis for Coding Detected Defects Prediction Requirements Analysis Delivery Rate Distribution Planned or Actual Effort for Requirements Analysis Coding Delivery Rate Distribution Planned or Actual Effort for Coding Coding Defect Injection Rate Distribution Code Review Delivery Rate Distribution Planned or Actual Effort for Code Review Confidence Level Phase at which prediction is done

Outputs from the Prediction Model (Code Injection Rate & Code Review Defect Density)
Output Coding Injected Defects Value Predicted Value Upper Prediction Limit Lower Prediction Limit 40 56 24 Unit of Measure Number of Defects Number of Defects Number of Defects Coding Defect Injection Rate Value Unit of Measure 0.40 Defects/FP 0.56 Defects/FP 0.24 Defects/FP

Coding Detected Defects Value Predicted Value Upper Prediction Limit Lower Prediction Limit 25 92 Not Applicable Unit of Measure Number of Defects Number of Defects Number of Defects

Coding Defect Density Value 0.25 0.92 Not Applicable Unit of Measure Defects/FP Defects/FP Defects/FP

Sonata Software Limited

HMBP 2011

August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

7 of 10

Inputs to the Prediction Model (System Acceptance Testing Defect Density & Support Effort)

Inputs Size Basis for Prediction Coding Defect Injection Rate Coding Defect Density Actual Number of defects detected in Coding Confidence Level Phase at which prediction is done Number defects Injected in Coding Number defects detected in Coding Coding Defect Injection Rate Coding Defect Density Value 100 Predicted Number of Coding Defects to be detected 0.31 0.21 Number of Defects 95 Planning Computed Inputs 40 25 0.40 0.25 Number of Defects Number of Defects Defects/FP Defects/FP % Unit of Measure FP

Outputs from the Prediction Model (System Acceptance Testing Defect Density & Support Effort)

Predicted Value Upper Prediction Limit Lower Prediction Limit

Predicted Value Upper Prediction Limit Lower Prediction Limit

Output SAT Defects Value Unit of Measure Number of Defects 18 Number of Defects 25 Number of Defects 11 SAT Effort Person-days 31.1 48.7 Person-days Person-days 13.5

SAT Defect Density Value Unit of Measure 0.18 Defects/FP 0.25 Defects/FP

0.11 Defects/FP SAT Delivery Rate 0.31 0.49 0.14 Person day/FP Person day/FP Person day/FP

It was found that the SAT Defect Density and Support Effort are not acceptable and hence Project Manager decided to increase code review effort from 3 days to 7 days and use of Code Review tool Sonar to avoid any standards and compliance Sonata Software Limited HMBP 2011 August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

8 of 10

related defects leaking to SAT. With these increased review effort and corrective actions, the revised prediction results are as below.

Revised Outputs from the Prediction Model (System Acceptance Testing Defect Density & Support Effort) after course corrections

Predicted Value Upper Prediction Limit Lower Prediction Limit

Value 13 20

Output SAT Defects Unit of Measure Number of Defects Number of Defects Number of Defects

SAT Defect Density Value Unit of Measure 0.13 Defects/FP 0.20 Defects/FP

7 SAT Effort Person-days

0.07 Defects/FP SAT Delivery Rate 0.24 0.41 0.06 Person day/FP Person day/FP Person day/FP

Predicted Value Upper Prediction Limit Lower Prediction Limit

23.7 41.2 6.2

Person-days Person-days

6. Validation of Integrated Predicted Model

The logic of validation allows us to move between the two limits of dogmatism and scepticism. -Paul Ricoeur The prediction models are made available in Sonata QMS and used by the project teams. Actual data from completed project releases, along with the prediction results from these models, is used to validate the prediction accuracy and precision. The prediction precision (%actual data falling within prediction limits) is around 90% and the average prediction error (% difference between actual value compared to point estimate dived by point) is around 20% observed over 10 releases. This is reasonably acceptable though there is scope for further improvement.

Prediction Precision & Error

0.3 0.25 SAT Defect Density (Defects/FP) 0.2 0.15 0.1 0.05 0 1 2 3 4 5 6 7 8 9 10 HMBP 2011 August 26, 2011 Actual Value Predicted Value UPL LPL

Sonata Software Limited

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

9 of 10

7. Conclusion
Failure is success if we learn from it. - Malcolm Forbes The Integrated Prediction Models are used in during planning stage and subsequently after completion of each phase to assess the feasibility of meeting of project objectives and it supports in deciding whether any decisions needs to change or planning parameters need to be adjusted to meet the end objectives if any risks are foreseen. The feedback loop designed to use the output performance from one sub-process as an input to next sub-process in integrated model helps in taking right decisions and actions during the course of the project there by avoiding late surprises. This provides adequate confidence to project teams on what is the certainty of meeting the end objectives. While the prediction precision from the models is good, there is a need for considering additional predictor variables for improving the accuracy of prediction results. We are planning to seek additional data for predictor variables such as Percentage of Reuse, Sonar Quality Index etc. for enhancing these prediction models.

Sonata Software Limited

HMBP 2011

August 26, 2011

INTEGRATED APPROACH FOR PREDICTING FINAL PERFORMANCE BY CONNECTING MULTIPLE MICRO-MODELS

10 of 10

References
A Tutorial for Building CMMI Process Performance Models by Robert Stoddard and Dave Zubrow, April 26, 2010, Approaches to Process Performance Modelling: A Summary from the SEI Series of Workshops on CMMI High Maturity Measurement and Analysis, Robert W. Stoddard, II and Dennis R. Goldenson, January 2010 - TECHNICAL REPORT CMU/SEI-2009-TR-021 Use and Organizational Impact of Process Performance Modelling in CMMI High Maturity Organizations, Dennis R. Goldenson James McCurley and Robert W. Stoddard, II A Practitioner View of CMMI Process Performance Models by Robert Stoddard and Rusty Young, March 20, 2008 Guideline for Process Performance Models, Sonata QMS

Authors Biography
Sivanantham M works as a Senior Manager- Quality at Sonata Software Ltd and leads the delivery excellence program for Microsoft Delivery Unit. In Sonatas journey towards CMMI v1.2 Level 5, he played a key role in establishing high maturity foundation for the company and making the practitioners understand the nuances involved in implementing Sub-process Control, Prediction Models and Innovation Initiatives through mentoring and internal assessments. He has more than 15 years of experience covering various areas such as Establishing Management Systems, Process Engineering, Quality consulting & training, Process automation, Metrics based improvements, Audits and Assessments, Project Management and Business Development. He holds an M.Tech. in Quality, Reliability and Operations Research from Indian Statistical Institute, Kolkata and Bachelors degree in Mechanical Engineering from Madurai Kamaraj University.

Acronyms and abbreviations

CMMI Capability Maturity Model Integration FS Functional Specification SAT System Acceptance Test LPL Lower Prediction Limit UPL Upper Prediction Limit

Sonata Software Limited

HMBP 2011

August 26, 2011