Test Metrics

Test metrics accomplish in analyzing the current level of maturity in testing and give a projection
on how to go about testing activities by allowing us to set goals and predict future trends.

Metrics
"Metrics are a system of parameters or ways of quantitative and periodic assessment of a process
that is to be measured, along with the procedures to carry out such measurement and the
procedures for the interpretation of the assessment in the light of previous or comparable
assessments. Metrics are usually specialized by the subject area, in which case they are valid
only within a certain domain and cannot be directly benchmarked or interpreted outside it."

Metrics are measurements. It is as simple as that. We use them all the time in our everyday lives.
Entangling them in wordy definitions is just intended to make them seem more mysterious and
technical than they really are.

So what sorts of things do we measure in our daily lives and how do we use them? Shopping for
food is a good place to start. At the meat counter, there is a choice of cuts of different kinds of
meat, all at different prices. If we just look at the total price, we may be misled. A nice round
steak might cost $10.00 while a round roast might cost $8.00 even though it weighs the same as
the steak. So to get the best value for our money we tend to look at the price per unit weight.
This is a microcosm of the field of metrics.

There are two basis types of metrics. The first type is the elemental or basic measurement such as
weight, length, time, volume, and in this example, cost. The second type is derived, normally
from the elemental measurements. At the meat counter, the derived metric is dollars/weight
(VIZ. $7.49/kg). This is called a normalized metric.

Generally speaking, normalized metrics are the most useful because they allow us to make
comparisons between things that are different. Some other examples are miles/gallon,
dollars/gallon, dollars/share, dollars/hr, and dollars/square foot to give but a few.

We also see metrics in sports. In hockey its shots on goal and plus/minus ratio. In baseball its
batting average and errors per game. All of these numbers are provided in newspapers and sports
magazines and if they disappeared there would be a great uproar among fans.

“When you can measure what you are speaking about and express it in numbers, you know
something about it; but when you cannot measure it, when you cannot express it in numbers,
your knowledge is of the meager and unsatisfactory kind.” - Lord Kelvin

Now Lord Kelvin wasn't right about everything he spoke about. He predicted that heavier than
air flight was impossible. But about metrics, he was dead right.
We as shoppers apply this principle whenever we go to the market. If a cut of meat is marked
$10.00 but has no weight assigned, we are likely to look for something else. The same would
apply if the weight were given but no price. This is just plain old ordinary common sense. Yet
we may go though our professional lives without using metrics to guide us in our work.
Maintaining a "meager and unsatisfactory" knowledge about the way you earn your living is
probably not the best approach.

Elements
The following ISM3 table suggests the elements that must be known for a metric to be fully
defined. This table is criticized sometimes as it omits controls against bias.

Element Description
Metric Name of the metric
Metric Description Description of what is measured
Measurement Procedure How is the metric measured
Measurement Frequency How often is the measurement taken
Thresholds Estimation How are the thresholds calculated
Current Thresholds Current range of values considered normal for the metric
Target Value Best possible value of the metric
Units Units of measurement

Metrics are used in business model, CMMI, ISM3, Balanced scorecard and knowledge
management. These measurements or metrics can be used to track trends, productivity, resources
and much more. Typically, the metrics tracked are key performance indicators, also known as
KPIs. For example, you would use metrics to better understand how a company is performing
compared to other companies within its industry.

Most methodologies define hierarchies to guide organizations in achieving their strategic or

tactical goals. An example can be:

•
o Objectives
 Goals
 Critical success factor (CSFs)
 Key Performance Indicators (KPIs or Metrics).

The intention is to identify future-state objectives, relate them to specific goals that can be
achieved through critical success factors or performance drivers which are then monitored and
measured by key performance indicators. Through this hierarchy, organizations can define and
communicate relationships between metrics and how they contribute to the achievement of
organizational goals and objectives.

Metrics are important in IT Service Management including ITIL; the intention is to measure the
effectiveness of the various processes at delivering services to customers. Some suggest that data
from different organizations can be gathered together, against an agreed set of metrics, to form a
benchmark, which would allow organizations to evaluate their performance against industry
sectors to establish, objectively, how well they are performing.

Types of Test Metrics

• Process Metrics
• Product Metrics

• Project Metrics

Process Metrics

MANY ORGANIZATIONS TAKE measurements or metrics because they have the capability
to measure, rather than determining why they need the information. Unfortunately, measurement
for the sake of a number or statistic rarely makes a process better, faster, or cheaper. A poor
measurement can hurt a process if incorrect decisions are based on the result of that
measurement. People at all levels of organizations continue to take measurements hoping that
they will shed light on the best way to provide a product or service. Though fraught with good
intentions, these poorly contrived measurements add to the confusion of what should and should
not be measured.

Metrics Process Model

Until a year ago, many of the communications and information metrics of Air Force Space
Command (AFSC) were taken because they had been collected for years, and people thought
those metrics must have a purpose.

At that time, many metrics were not being used to make a decision based on fact, but fulfilled a
headquarters' requirement to report on information by a certain date every month. After a fairly
extensive study, the AFSC Senior Communicator (SC) changed the format and collection of
many of these metrics, while deleting the requirement for many that had little value.

Like many discoveries, the process for metrics collection and analysis in this directorate was the
result of a change in leadership. Communications metrics at AFSC seemed to provide good
information, since senior leaders did not complain about content or format of the 30 metrics
collected at the headquarters level. Haphazard metrics collection continued until a number of
new senior leaders asked why these metrics were being collected and if they were the right
measurements for their organizations. These questions sparked a complete review of the metrics
collection, analysis, and reporting process.

Applying the Model

Establish and Validate Organizational Goals:

With that definition in mind, the majority of authors studied noted that the first step in good
metrics collection is understanding the goal. Rather than ask what should be measured, ask what
is important to the organization and its customers.

Many organizations have trouble with this; however, the Communications and Information
Directorate at AFSC did a thorough review of its customers' requirements and understood what
was important to the organization's success. The SC directorate validated its organizational goals
and objectives with its customers, suppliers, and senior managers, when it published its strategic
plan. Re-validated semiannually, this eight-page document outlines the direction the unit is
expected to take in the next few years. Notably missing from the organization's strategic plan
was a link of metrics to measure the progress of these goals. In re-validating this strategic plan,
using metrics as a tool to measure these goals, many people noted that the goals were too general
because they could not be measured. These goals and objectives were revaluated, ensuring that
each objective had an associated measurement to ensure progress.

Management Issues

Although these goals are important to every organization, it can be difficult to focus on defining
clear, measurable goals, based on what is important to customers. Senior management can be
skeptical about the value of spending time defining such goals. The Communications and
Information Directorate at AFSC understood the need for such goals but proceeded cautiously,
defining those goals that were most easily quantified first.

Measures of a system's up-time rates and availability were clear targets with measurable rates
and long data histories. Once these goals were proven to provide useful decision points, senior
leaders were willing to define other goals of interest to the organization and ultimately to the
customer. Each organization must decide how many goals it needs to effectively manage its
resources and meet its customers' requirements. Through trial and error, the organization found
that its customer requirements could be encapsulated into about 10 measurable goals and 40
more specific subgoals called objectives. The goals provided a broad-based definition for what
was important to the organization, while the objectives specified actions necessary to meet
customer requirements. Each objective was written so as to be clearly measurable, and at least
one associated metric was created for each objective to provide decision-making information to
senior management.

Every organization will have a different approach to establishing goals based on customer
requirements, but regardless of the approach, it is important that these goals are measured and
quantified in terms that senior management can understand and fully support.

The Communications and Information Directorate had a strong data collection program, but the
analysis and use of this information was limited. Although the intent of these metrics was to
measure an important or problem area, the number of metrics continued to grow, while the
analysis was almost nonexistent.

A plan was created to validate the purpose of each metric. Rather than modify existing metrics,
the metrics program needed an overhaul. Many of the cost, schedule, and performance metrics
were relevant because they directly measured the mission. However, the metrics process to
collect and analyze this information required updating. We defined an overall metrics philosophy
as an adjunct to the strategic plan and noted that each new metric had to have basic information
associated with it, making it useful to the right people at the right time. Figure 2 is a form we
used to collect this information in a single, neat package so everyone from collectors to decision
makers could understand their purpose in collecting, reporting, and making decisions based on
this metric. Although simple, this broad overview causes people in the organization to think
before creating or approving a metric. It also marks the conditions under which the metric will
remain useful. This makes the process easier for semiannual review of the metrics, because the
criteria are spelled out and metrics that have outlived their usefulness are

Metric Title Brief Description

Link to Goals/Objectives
Who collects data
Who reports data
reported
Who analyzes data
Lowest acceptable values
At what point will you stop
collecting this metric
Decision(s) based on analysis Who makes decision(s)
How is data collected How often is data collected
How and to whom is data How often is data reported
How is data to be analyzed
(formulas and factors)
Highest acceptable numerical Expected values
values

Review Metrics Plan

Review Metrics Plan

In creating this metric plan, we noted that there may be other factors that we had not considered
when defining each metric. To review the data objectively, we surveyed our data collectors and
senior leaders to see if they understood why we collected each metric. The results were
enlightening and helped to create a usable metrics program. In this survey, we asked questions
about the metrics' perceived usefulness, its ability to aid in decision making, goal of the metric,
tolerances set for the highest and lowest acceptable values, and timeliness of feedback based on
analysis of the data. We could have interviewed people instead of taking a survey but believed
anonymous answers would be more honest. We distributed one survey for each metric to each of
three groups.

One survey was given to data collectors and another to senior managers assigned to make
decisions on each metric. The third set of surveys was distributed to the metric owners who
designed the metric and are assigned to analyze the data. These surveys were used as the control
group because those who designed the metric should understand why the metric is important in
making decisions.
Through this survey, we obtained raw data on specific problems and accolades associated with
each metric. Although we addressed specific problems, our primary reason for the analysis was
to assess the overall usefulness of our metrics program. This analysis, though useful to every
level, was of greater use to senior managers who make final decisions on the types of metrics to
be collected in the organization.

The first trend analysis showed that one-third of the metrics were not useful in making decisions.
Some had no reason for being collected; others had outlived their usefulness. Also noteworthy—
few people received timely feedback from those who were assigned to analyze the data. All of
these factors led to a metrics program that provided a lot of data but little useful information.
Before implementing this metrics plan, many believed that their metrics were the "right"
measurement. Changes were made to existing metrics to streamline and standardize collection
processes, and a number of metrics were deleted. After the new metrics passed a trial period,
senior managers were confident that the new metrics provided information necessary in making
decisions.

Implementing a Metrics Plan

A plan is proven only when it is implemented. Senior managers realized this, and after careful
planning, proceeded to provide policy and process clarification to those collecting and analyzing
data.

Policy and Process Issues

Gathering and quantifying information initially takes considerable effort but eventually becomes
a regular facet of the organization. Although a metrics plan can detail how to collect data, only
people can collect and analyze the right data. In gathering metric information, AFSC had to
overcome many logistical concerns not only in getting the data but also in ensuring that the data
was consistent among the nine communications agencies for which this organization compiled
information. They began by clearly defining the requirements in a policy letter signed by the SC.
Information to be collected and suspend dates for collection were defined in this policy, which
each of the nine communications organizations were required to follow.
Once this policy was signed, the task of ensuring consistent, measurable data had just begun.
Though the organization felt that its policy and direction was clear, it took three months for all
data collection agents to consistently collect the information requested. After a series of
clarifications and minor changes in the collection process, a consistent process to collect metric
data was defined and published. Although different for each organization, it can be assumed that
even with the best intentions, consistent data collection is an iterative process requiring
modifications until all data collectors use the same processes and methods. Although automation
can help in this consistency, it is ultimately up to the people who define the metrics to clearly
articulate the process for data collection.

Metric Utilization by Management

Even if a metrics plan were perfectly implemented, it still would be incomplete unless the correct
level of management makes decisions based on the metrics. It has been well-documented that
management buy-in and commitment are necessary before a metrics process can work.

AFSC ensured that its senior management understood the implications of the metric analysis
through monthly metric meetings with senior managers, midlevel managers, and people who
collect and analyze the data. This type of high visibility is crucial for a successful metrics
program. Without definite due-dates and justification for information collection and analysis,
senior managers likely would not make metrics a priority. Everyone who collects, analyzes, or
makes decisions based on metrics data must be aware of the process, due-dates, and most
important, that the metrics are being used to make corporate decisions. When all parties involved
understand the importance of these metrics, they are likely to make an effort to collect accurate
data, analyze it, and ensure reporting is done quickly to aid in the decision-making process.

Reviews

To be effective, even the most perfect plan needs consistent review. The first review of the
metrics plan for this organization shook up the way we used metrics to make decisions.

After the initial review, there was a large turnover in senior leaders, changing some of the
primary goals and focuses of the organization. There was another review at the semiannual point,
and although the changes were much more subtle, metrics were again changed to reflect the
criteria needed for solid decision making within the organization. This continues to be an
iterative process, and the senior leadership of the AFSC SC's office is committed to continuing
this process.
 Product Metrics

Product metrics are for describing characteristics of product such as it’s size, complexity,
features, and performance. Several common product metrics are mean time to failure, defect
density, customer problem, and customer satisfaction metrics.

1. Mean time to failure metric is, to put it plainly, the average time the product runs before
experiencing a crash, which is important for systems like air traffic control that are required to
have no more than a few seconds of down time in a full year.

2. Defect density metric refers to number of imperfections per:

o Lines of code

o Function definitions

o Lines on input screens

3. Customer problem metric is a measure of problems customers have encountered with the
product over the total usage of the product. This metric takes into account that multiple instances
of the product can be used at the same time, which effectively multiplies the length of time the
product has been in operation by the number of product licenses.

4. Customer satisfaction metric is generally a survey asking customers to rate their satisfaction
with the product and/or it’s features on a five-point scale.

Product Metrics Strategy

To begin working towards the metric objectives, the following is the strategy:

S1. Produce a set of well-defined metrics on all RC types.

Use the existing literature to collect a candidate set of metric definitions and interpretations;

Extend and modify the candidate set to create a pool of metric definitions.

S2. Provide interpretations that relate the metrics to quality or reusability.

Use the technique defined here in to refine the interpretation and select the final metric set.
S3. Provide for each metric a mechanism for collection, including

identification of automatically computable metrics; processes for data collection and data
validation for non-automatically computable metrics; and

a discussion of the difficulty and reliability of collection.

S4. Discuss the potential benefits and liabilities of use of the metrics in terms of their
interpretation, possible reliability, and the possible difficulty of collection.

Project Metrics

Unlike software process metrics that are used for strategic purposes, software project metrics are
tactical. That is , project metricsand the indicators derived from them are used by a project
manager and a software team to adapt project workflow and technical activities.

The first application of project metrics on most software projects occurs during estimation.
Metrics collected from past projects are used as a basis from which effort and time estimates are
made for current software work. As a project proceeds, measures of effort and calendar time
expended are compared to original estimates. The project manager uses these data to monitor and
control progress.

As technical work commences, other project metrics begin to have significance. Production rates
represented in terms of models created, review hours, function points, and delivered source lines
are measured. In addition, erors uncovered during each software engineering task are tracked. As
the software evolves from requirements into design, technical metrics are collected to assess
design quality and to provide indicators that will influence the approach taken to code generation
and testing.

The intent of project metrics is twofold. First, these metrics are used to minimize the developing
schedule by making the adjustments necessary to avoid delays and mitigate potential problems
and risks. Second, project metrics are used to assess product quality on an ongoin basis and
when, necessary, modify the technical approach to improve quality.

As quality improves, defects are minimized, and as the defect count goes down, the amount of
rework required during the project is also reduced. This leads to a reduction in overall project
cost.

Selection of Project Metrics

One of the crucial elements of the project charter in the define phase of a Six Sigma project is the
selection of project metrics. Project metrics selected should reflect the voice of the customer
(customer needs), as well as ensure that the internal metrics selected by the organization are
achieved. Metrics selected should be simple and straightforward and meaningful. Metrics
selected should create a common language among diverse team members.

When drafting metrics for a particular project one should consider how the metrics are connected
and related to key business metrics. Typically there is no one metric that fits all the requirements
for a particular situation.

Developing Project Metrics

The most common approach used by teams is to understand the problem statement, brainstorm
metrics, and finally decide what metrics can help them achieve better performance. The team
then reviews these metrics with executive management to ensure that they are in synergy with
the overall strategy of the business, and an iterative approach may be utilized.

Care should be exercised in determining what is measured. Metrics should be based on what, in
fact, needs to be measured to improve the process, rather than what fits the current measurement
system. Metrics need to be scrutinized from the value they add in understanding a process.

Balanced Scorecard Approach To Metrics

Many Six Sigma professionals advocate the use of a Balanced Scorecard type of approach for the
selection of project metrics as a method for ensuring that the project meets both customer and
business needs. The Balanced Scorecard approach includes both financial and non-financial
metrics, as well as lagging and leading measures across the four areas or perspectives: Financial,
Customer, Internal Processes, and Employee Learning and Growth. Lagging measures are those
that are measured at the end of an event, while leading measures are measures that help as
achieve the objectives and are measured upstream of the event.

Most Balanced Scorecard metrics are based on brainstorming, however the approach of
brainstorming can have limited success in establishing sound metrics that have a good balance
between lagging and leading measures.

Typical brainstormed Balanced Scorecard metrics utilized in Six Sigma projects can be
summarized in the below. The primary issue in utilizing a scorecard is that it might not reflect
the actual strategies applied by the team for achieving breakthrough in their project.

Financial Customer

• Inventory Levels • Customer Satisfaction

• Cost Per Unit • On Time Delivery
 • Hidden Factory • Final Product Quality
• Activity Based Costing
• Cost Of Poor Quality • Safety Communications

• Overall Project Savings

Internal Business Processes Employee Learning and Growth

• Defects, Inspection Data, DPMO, Sigma • Six Sigma Tool Utilization

Level • Quality of Training
• Rolled Throughput Yield • Meeting Effectiveness
• Supplier Quality • Lessons Learned
• Cycle Time • Total Trained in Six Sigma
• Volume Shipped • Project Schedule Versus Actual Date
• Number of Projects Completed
• Rework Hours
• Total Savings To Date

The Four Perspectives

Instead of utilizing the Balanced Scorecard approach described above, the teams can employ a
more effective method by answering the questions in the figure given below. This approach
helps team members understand the objectives of the project from each of the four perspectives.

Once the team has brainstormed for each of the four perspectives, the various objectives that
must be met by the project will be more clear. These objectives can then me mapped in a strategy
map cutting across all the perspectives and projects of the organization. The arrows help in
understanding the cause and effect linkages in the strategy. The next illustration shows an
example of a strategy map applied.

Methods for Identifying Test Metrics

Characteristics of Effective Test Metrics

Ideally, identifying test metrics takes place at the beginning of the project, so incorporation into
the appropriate activities is easy. The test metrics you wish to collect need to be:

·quantifiable,

·easy to collect,

·simple,

·meaningful,

·non-threatening.

Quantifiable Measurements

To ensure consistent comparison of findings, the method of measurement needs to be standard,

concise, and quantifiable. For example, to determine the density of defects, you need to identify
what metrics provide this information and a standard of measurement. For example, the test
metric to gather is the number of defects and the method of measurement is lines of code (loc),
(i.e., “x” number of defects per “y” loc).

Definitions must be clear and concise. For example, the definition of defect must state what
constitutes a defect and the definition of lines of code must state the number of lines of code to
be used as the standard of measure, (e.g., 1000). The definitions must also provide any other
information necessary to ensure consistency, (e.g., if the lines of code are commented or not
commented).

Easy to Collect

The information collection process must not take too much of the collector’s time, or the
information will not be collected. The amount of test metrics gathered from any one group needs
to be kept at a minimum, collecting only that which is most useful. Whenever possible, automate
the data collection process.

Simple Information

The information collected should be simple to gather. If it is hard for the collector to determine
what to measure or report, the information is likely to be inaccurate.

Meaningful Purpose
The information gathered must have a specific purpose, (or purposes). For example, the
information will be used to determine the number of defects and time used for each testing
phase, in order to determine the most cost effective ways to minimize errors.

The information to collect must be understandable and viewed as relevant to the collector, or the
information will not be collected. For example, to make the information in the previous example
relevant, explain that the findings will highlight the testing methods that work and methods that
don’t work, so that employee effort is focused on productive activities.

Non-Threatening Use

Avoid using test metrics for employee evaluation purposes. Collection of information that is
perceived as a threat to the employee’s job status is frequently reported inaccurately or
incompletely.

Methods for Identifying Test Metrics

Start the process of identifying test metrics by listing the problems to be solved and objectives
first. Then determine the items to measure and the standards of measurement to use, to achieve
the objectives.

Various methods can be used to complete the test metrics identification process, (e.g.,
brainstorming, use of a committee composed of representatives from management and the
groups that will help with the collection process).

Methods for Gathering Test Metrics

Effective Methods for Gathering Test Metrics

The timing for gathering test metrics depends on the test metrics identified. For example, if
metrics are gathered on the results of design inspections, the gathering occurs during the
structural design stage. If metrics are gathered on defect analysis of system test results, the
gathering occurs during development.

Exercise caution when gathering test metrics data. To ensure the information provided is
accurate and consistent, the definitions of the information required must be clear and understood
by the collector. Also, the method for collecting the information must ensure that the applicable
metrics information is gathered. Effective ways to gather test metrics include:

·train collectors,

·use reliable collection mechanisms,

·provide feedback to collectors.

Train Collectors

To obtain the desired information from the collectors, you must ensure that the test metric
definitions and reporting requirements are understood. One way to accomplish this is to provide
a short training course for applicable personnel, to explain the information needed and how the
information will be used. Review the test metric definitions and reporting requirements
thoroughly, and use realistic sample exercises to reinforce and ensure understanding. Cite
examples where these kinds of metrics have been useful and illustrate the benefits.

Use Reliable Collection Mechanisms

To be useful, test metrics need to be comprehensive. To obtain comprehensive information, the

reporting procedure must be reliable. Whenever possible, use automated methods for gathering
the information. When relying on collectors, ensure reporting is easy and a matter of routine.
Ideally, the reporting should be made a part of established reporting mechanisms, (e.g., time
reporting, additional information reported on a fault report). When information can not be
collected through automation or existing reporting mechanisms, use other methods that prompt
the collector, (e.g., forms).

Provide Feedback

Once collectors are trained and mechanisms are established for reporting the requested test
metrics, it is important to monitor what is being reported. Initially, review at least five reported
test metrics for each collector, then meet with the collector to substantiate the findings and
provide any additional clarification, as necessary. Also, take the opportunity to thank the
collectors for their time and effort.

Documenting the Test Metrics Collection Mechanism

A template and sample illustrate how to document the collection mechanism. Add this
information to the requirements defined during the test metrics identification process.

Methods for Analyzing Test Metrics

Conduct test metrics analysis using several steps, which include:

?sorting data into categories,

?performing calculations,
?analyzing to determine trends.

Sort Data into Categories

Sort the data gathered into the applicable categories based on the objectives defined at the
beginning of the test metrics evaluation process. For example, if the objective is to determine the
most frequent source of defects, the data is sorted by source of defect.

Perform Calculations

Perform calculations, such as percentages and averages, to compare the types of data gathered.

For example, to determine the percentage for each source of defect category, calculate the total
based on each defect source and calculate the total for all defects. Then, divide the number of
defects for each source of defect type by the total number of defects, to determine the category
with the highest percentage of occurrence.

Frequently, additional categories need to be calculated. For example, to determine the cost to fix
each source of defect category, calculate the percentage of effort required to fix the defects for
each source of defect category. To do this, calculate the total number of hours to fix the defects
for each source of defect category, and the total number of hours to fix all defects, then divide
the number of hours for each source of defect category by the total number of hours to fix all
defects.

Analyze to Determine Trends

Review the calculated findings to determine trends. For example, the majority of the defects
were coding defects, but the most costly to fix were the functional specification defects.

Additional calculation may also be necessary. For example, the determination of the average
number of hours to fix each source of defect category may allow better comparison of the cost of
defects.

Evaluate the causes of the findings. For example, further discussion with the programmers
reveals that it takes longer to fix functional specification defects, because the analysis and design
processes must occur again.

The analysis process may also reveal additional factors that need to be collected to improve the
reliability of the findings. For example, to clarify the causes of the coding errors, the
identification of the programs where each defect occurred, as well as the complexity of the
programs, may help. When determining complexity, established metrics, such as McCabe's
Complexity Measure and Function Point Estimating are useful.

Whenever possible, use automated means, such as a database, to accumulate and analyze test
metrics.
 When to Complete the Analysis

Depending on the objective, the analysis process can be done on a continual basis or at the end of
the data gathering period. When the objective is improvement of the current project, the analysis
process should be done on a continual basis during the data collection. When the objective is
improvement of a future project, the analysis process can be done at the end of the data gathering
period.

Continuous analysis can be comparative or cumulative. For example, monitoring system

readiness by calculating and comparing the number of defects each day, to determine if there is a
downward trend, illustrates comparative analysis. In other situations, the analysis needs to be
done on a cumulative basis. For example, when determining the most frequent and costly types
of defects, a continuous analysis of information as it is accumulated helps with early
identification of areas requiring corrective action.

An example of analysis at the end of the data gathering period is the collection of work hours
used for testing, for the purpose of estimating future testing efforts. For greater reliability,
accumulate and analyze data from as many projects as possible.

How to apply TestMetrics

Translate the results of the test metrics analysis into action that improves project practices.
Ideally, focus on preventative, rather than corrective actions. To transform the test metrics
analysis findings into action, complete the following steps:

·summarize the findings,

·recommend action,

·justify the recommendation,

·communicate the results,

·take action.

Summarize Findings

Summarize the test metrics collected, the calculations completed, trends noted, research plan,
research findings, and the problem(s)/cause(s) identified. To determine the causes of the
problems identified, use techniques such as brainstorming and cause and effect diagramming.
Use a template to help summarize findings.
Summary of Test Metrics Findings Template

Summary of Test Metrics Findings Sample

 Recommend Change to Improve Process

Make recommendation(s) for correcting or preventing any identified problems. Document the
recommendation, again using the Summary of Test Metrics Findings Template.

Justify the Recommendation

When justifying a recommendation, identify applicable organizational or industry accepted rules

of thumb, note any assumptions, and document the calculations, which based on the rules of
thumb and the assumptions, justify the recommendation. Use graphs and cost benefit analysis, as
applicable, to support the recommendation. Identify any other benefits of the recommended
action. For further supporting information on the justification process, refer to Practical Software
Metrics for Project Management and Process Improvement, by Robert B. Grady.

Communicate the Results

After preparing the summary of test metrics findings and any appropriate justification,
communicate the information to members of management that have approval authority.

Take Action

After obtaining approval, follow through to ensure action is taken. Also, depending on the action,
a small scale implementation of the recommendation may be advisable to validate effectiveness,
before implementation throughout the organization. Remember to share information on test
metric successes with staff who helped with the collection process.

Depending on the project approach, the focus of the test metrics, and the maturity of the
organization’s test metrics data, test metrics are applied during various stages of a project. For
example, if test metrics are available on the length of testing activities, from a number of
previous projects, test metrics are applied during the project planning stage to plan and staff test
activities. If system test defect analysis information is gathered to improve the current project,
then the metrics are applied during the development stage.

Test metrics are used commonly for a number of activities.

The Business Value of Test Metrics

Measurements are factual data that enable us to glean vital information for rational decision
making. When it comes to testing software, we collect a quite a few measurements that allow us
to primarily evaluate the “quality” of software and help make the go/no-go release decision.

Today it is fairly commonplace to collect metrics related to defects, coverage, risk, estimation,
variance, test cycles, density and test cases that are sharply focused on evaluating the quality of
software to help in making good release decisions. In addition to these some metrics like
escapes, productivity, ROI allow us to evaluate aspects related to the test process and also some
initial measure of business value.
As the testing function/community matures, the test function needs to be aligned to the business
and demonstrate contribution of the overall business. This implies assessing activities that go
beyond release quality and process, like those related to profitability, business scalability etc. In
this tutorial the focus is on understanding test related measurements from this larger context to
evolve a metrics system based on customer, business, organization, division and product.

This tutorial will highlight measurements from this larger context of business to help you go
beyond the typical realm of test measurements to enable you and your team to deliver a larger
positive contribution to your business.