Software Reliability Modelling

Engin Deveci

Boazii University

Software Reliability Modelling

Computer Engineering

Agenda
What is Software Reliability? Software Failure Mechanisms
Hardware vs Software

Measuring Software Reliability Software Reliability Models Statistical Testing Conclusion

Boazii University

Software Reliability Modelling

Computer Engineering

What is Software Reliability

the probability of failure-free software operation for a specified period of time in a specified environment Probability of the product working correctly over a given period of time. Informally denotes a products trustworthiness or dependability.
Boazii University
Computer Engineering

Software Reliability Modelling

What is Software Reliability

Software Reliability is an important attribute of software quality, together with
functionality, usability, performance, serviceability, capability, installability, maintainability, documentation.
Software Reliability Modelling
Computer Engineering

Boazii University

What is Software Reliability

Software Reliability Modeling Prediction Analysis Reliability Measurement Defect Classification Trend Analysis Field Data Analysis Software Metrics Software Testing and Reliability Fault-Tolerance Fault Trees Software Reliability Simulation Software Reliability Tools From Handbook of Software Reliability Engineering, Edited by Michael R. Lyu, 1996
Boazii University

Software Reliability Modelling

Computer Engineering

What is Software Reliability

Software Reliability is hard to achieve, because the complexity of software tends to be high. While the complexity of software is inversely related to software reliability, it is directly related to other important factors in software quality, especially functionality, capability.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Failure Mechanisms

Failure cause: Software defects are mainly design defects. Wear-out: Software does not have energy related wearout phase. Errors can occur without warning. Repairable system concept: Periodic restarts can help fix software problems. Time dependency and life cycle: Software reliability is not a function of operational time. Environmental factors: Do not affect Software reliability, except it might affect program inputs. Reliability prediction: Software reliability can not be predicted from any physical basis, since it depends completely on human factors in design.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Failure Mechanisms

Redundancy: Can not improve Software reliability if identical software components are used. Interfaces: Software interfaces are purely conceptual other than visual. Failure rate motivators: Usually not predictable from analyses of separate statements. Built with standard components: Well-understood and extensively-tested standard parts will help improve maintainability and reliability. But in software industry, we have not observed this trend. Code reuse has been around for some time, but to a very limited extent. Strictly speaking there are no standard parts for software, except some standardized logic structures.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Failure Mechanisms

Boazii University

Software Reliability Modelling

Computer Engineering

Software Failure Mechanisms

Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Dont define what you wont collect.. Dont collect what you wont analyse.. Dont analyse what you wont use..

Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Measuring software reliability remains a difficult problem because we don't have a good understanding of the nature of software Even the most obvious product metrics such as software size have not uniform definition.

Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Current practices of software reliability measurement can be divided into four categories:
Product metrics Project management metrics Process metrics Fault and failure metrics

Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Different categories of software products have different reliability requirements:
level of reliability required for a software product should be specified in the SRS document.

A good reliability measure should be observer independent,

so that different people can agree on the reliability.
Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

LOC, KLOC, SLOC, KSLOC McCabe's Complexity Metric Test coverage metrics ISO-9000 Quality Management Standards MTBF
Once a failure occurs, the next failure is expected after 100 hours of clock time (not running time).
Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Failure Classes
Transient:
Transient failures occur only for certain inputs.

Permanent:
Permanent failures occur for all input values.

Recoverable:
When recoverable failures occur the system recovers with or without operator intervention.

Unrecoverable:
the system may have to be restarted.

Cosmetic:
May cause minor irritations. Do not lead to incorrect results.
Eg. mouse button has to be clicked twice instead of once to invoke a GUI function.
Boazii University

Software Reliability Modelling

Computer Engineering

Measuring Software Reliability

Errors do not cause failures at the same frequency and severity.
measuring latent errors alone not enough

The failure rate is observer-dependent No simple relationship observed between system reliability and the number of latent software defects. Removing errors from parts of software which are rarely used makes little difference to the perceived reliability. removing 60% defects from least used parts would lead to only about 3% improvement to product reliability. Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program. The perceived reliability depends to a large extent upon how the product is used. In technical terms on its operation profile.

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Software reliability models have emerged as people try to understand the characteristics of how and why software fails, and try to quantify software reliability Over 200 models have been developed since the early 1970s, but how to quantify software reliability still remains largely unsolved There is no single model that can be used in all situations. No model is complete or even representative.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Most software models contain the following parts:
assumptions, factors, a mathematical function
relates the reliability with the factors. is usually higher order exponential or logarithmic.

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Software modeling techniques can be divided into two subcategories:
prediction modeling estimation modeling.

Both kinds of modeling techniques are based on observing and accumulating failure data and analyzing with statistical inference.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

ISSUES DATA REFERENCE PREDICTION MODELS Uses historical data ESTIMATION MODELS Uses data from the current software development effort

WHEN USED IN DEVELOPMENT CYCLE

Usually made prior to development or test phases; can be used as early as concept phase

Usually made later in life cycle(after some data have been collected); not typically used in concept or development phases

TIME FRAME

Predict reliability at some Estimate reliability at future time either present or some future time
Software Reliability Modelling
Computer Engineering

Boazii University

Software Reliability Models

There are two main types of uncertainty which render any reliability measurement inaccurate: Type 1 uncertainty:
our lack of knowledge about how the system will be used, i.e.
its operational profile

Type 2 uncertainty:
reflects our lack of knowledge about the effect of fault removal.
When we fix a fault we are not sure if the corrections are complete and successful and no other faults are introduced Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Step Function Model
The simplest reliability growth model:
a step function model

The basic assumption:

reliability increases by a constant amount each time an error is detected and repaired.

Assumes:
all errors contribute equally to reliability growth highly unrealistic:
we already know that different errors contribute differently to reliability growth.
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Jelinski and Moranda Model
Realizes each time an error is repaired reliability does not increase by a constant amount. Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Littlewood and Veralls Model
Assumes different fault have different sizes, thereby contributing unequally to failures. Allows for negative reliability growth Large sized faults tends to be detected and fixed earlier As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Variations exists
LNHPP (Littlewood non homogeneous Poisson process) model

Goel Okumoto (G-O) Imperfect debugging model

GONHPP

Musa Okumoto (M-O) Logarithmic Poisson Execution Time model

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Applicability of models:
There is no universally applicable reliability growth model. Reliability growth is not independent of application. Fit observed data to several growth models.
Take the one that best fits the data.

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Observed failure intensity can be computed in a straightforward manner from the tables of failure time or grouped data (e.g. Musa et al. 1987).

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Example: (136 failures total): Failure Times (CPU seconds): 3, 33, 146, 227, 342, 351, 353,444, 556, 571, 709, 759, 836 ..., 88682. Data are grouped into sets of 5 and the observed intensity, cumulative failure distribution and mean failure times are computed, tabulated and plotted.
Boazii University
Computer Engineering

Software Reliability Modelling

Software Reliability Models

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Two common models are the "basic execution time model and the "logarithmic Poisson execution time model" (e.g. Musa et al. 1987).
Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Basic Execution Time Model
Failure intensity () with debugging time :

where 0 is the initial intensity and 0 is the total expected number of failures (faults).
Boazii University
Computer Engineering

Software Reliability Modelling

Software Reliability Models

where () is the mean number of failures experienced by time .

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

In this case the LogarithmicPoisson Model fits somewhat better than the Basic Execution Time Model. In some other projects BE model fits better than LP model.

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

Additional expected number of failures, , that must be experienced to reach a failure intensity objective

where P is the present failure intensity, and F is the failure intensity objective. The additional execution time, , required to reach the failure intensity objective is

Boazii University

Software Reliability Modelling

Computer Engineering

Software Reliability Models

After fitting a model describing the failure process we can estimate its parameters, and the quantities such as the total number of faults in the code, future failure intensity and additional time required to achieve a failure intensity objective.

Boazii University

Software Reliability Modelling

Computer Engineering

Statistical Testing
The objective is to determine reliability rather than discover errors. Uses data different from defect testing.

Boazii University

Software Reliability Modelling

Computer Engineering

Statistical Testing
Different users have different operational profile:
i.e. they use the system in different ways formally, operational profile:
probability distribution of input

Divide the input data into a number of input classes:

e.g. create, edit, print, file operations, etc.

Assign a probability value to each input class:

a probability for an input value from that class to be selected.
Boazii University

Software Reliability Modelling

Computer Engineering

Statistical Testing
Determine the operational profile of the software:
This can be determined by analyzing the usage pattern.

Manually select or automatically generate a set of test data:

corresponding to the operational profile.

Apply test cases to the program:

record execution time between each failure it may not be appropriate to use raw execution time

After a statistically significant number of failures have been observed:

reliability can be computed.

Boazii University

Software Reliability Modelling

Computer Engineering

Statistical Testing
Relies on using large test data set. Assumes that only a small percentage of test inputs:
likely to cause system failure.

It is straight forward to generate tests corresponding to the most common inputs:

but a statistically significant percentage of unlikely inputs should also be included.

Creating these may be difficult:

especially if test generators are used.
Boazii University

Software Reliability Modelling

Computer Engineering

Conclusions
Software reliability is a key part in software quality Software reliability improvement is hard There are no generic models. Measurement is very important for finding the correct model. Statistical testing should be used but it is not easy again Software Reliability Modelling is not as simple as described here.
Boazii University

Software Reliability Modelling

Computer Engineering

Thank you.

Q&A

Boazii University

Software Reliability Modelling

Computer Engineering