metagroup.

com • 800-945-META [6382]

June 2004

Best Practices for Application Quality

Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

For Compuware

A META Group White Paper

 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

Contents

Problem Introduction: Testing Automation and Process Support .................. 2

The Benefits of Value-Based and Risk-Based Testing ..................................... 3
Starting With Requirements................................................................................ 4
Evolving a Prioritized Testing Approach ........................................................... 4
Reporting and Process Adaptability ...............................................................................................5
Centralized Management ...............................................................................................................5
Change Management and Project Governance.............................................................................5
Organizational Strategy..................................................................................................................6
Bottom Line .......................................................................................................... 7

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
1
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

Problem Introduction:
Testing Automation and Process Support
Although automated testing tools can enable significant improvements over
manual testing, software deployments that lack coordination with best-practice
processes and have poor organizational support often fail. Testers use tools
inconsistently in those circumstances, and there is little coordination with key
business requirements to effectively structure functional testing and support
corporate imperatives. Poor coordination with other aspects of the life cycle —
including project portfolio and change management — also results in gaps.
Decisions about project and code management are not prioritized appropriately,
and changes remain untested, opening up users to greater vulnerability. This
means that testing tools become ineffective for solving quality assurance/quality
control (QA/QC) problems, which minimizes or even negates the benefit of
bringing in testing tools. The return on investment for automated tools comes only
with appropriate, consistent use and evolves with effective reuse of testing
artifacts, over time.

In addition, limited testing resources and tight time frames for project execution
are driving many companies to better target allocation of effort to focus on areas
of the application that offer the greatest value or pose the greatest risk of failure.
Coordination with requirements and functional specifications and the setting up of
a framework for weighting decisions about allocation of testing effort enable
significant return on investment as well. This supports more efficient usage of
testing staff by focusing on the areas that are likely to prove most problematic
(rather than having to test entire applications with insufficient resources).

Poor process coordination for testing can be caused by lack of support for key
functional business requirements (leading to project irrelevance), reduced code
quality and increased complexity/workarounds, and unwise choices about
resource allocation. The result can be an ineffective approach that greatly
increases the cost to deploy, run, and manage applications

These increased costs can appear in multiple ways, including:

• Meager coordination with business goals and desired functionality

• Business irrelevance, leading to project failure
• Costly impact on clients for critical, customer-facing applications, which can
severely affect company revenue and reputation
• Increased staff costs to repair defects and related, significantly increased
software costs for impacted systems (these costs can be 10x-100x or more,
depending on how late in the life cycle these software defects are discovered)

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
2
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

This white paper will focus specifically on the way in which improvements to
testing processes, earlier coordination with requirements management, and
adoption of appropriate automated tools can improve efficiency, increase business
adaptability, and cut costs.

The Benefits of Value-Based and Risk-Based Testing

Significant challenges for the IT organization include pressures of regulatory
compliance, competitive sourcing from external providers (both offshore and
domestic outsourcers), highly constrained time frames for execution (due to
competitive market and internal business pressures), and the risk and prohibitive
costs of failure in these environments. In this context, organizations are seeking to
leverage the concept of value-based and risk-based testing. The concept behind
this principle is simple: Organizations should focus their limited resources on
those areas that will provide the most value to the business as well as those areas
that would pose the greatest risk to the system if they were to fail. Putting the
academic concept of risk-based testing into practice involves significant cultural
and process change upfront. Human barriers to process change usually pose the
greatest obstacle to successful implementation, since people change their
behavior with great reluctance (both professionally and personally).

However, the exercise of going through a rigorous assessment of prioritization

criteria and applying these findings to requirements in coordination with testing
enables organizations to establish a coherent framework for decision-making and
metrics. Such a framework benefits the organization in myriad ways.

Better and more frequent communication between the business and the IT
organization to help establish such a framework upfront creates better quality
information that is more current about business requirements and technology
challenges for implementation. The testing process is often orphaned — isolated
into a last-minute effort when applications are already past deadline and
overbudget. It is helpful to establish a process early on that incorporates testing as
a joint initiative across key groups within the organization, shifting cultural
perception and impacting project quality positively. Requirements must be more
thoroughly vetted to make assessments and also to generate test scripts. The
business must prioritize decision making about key functions to determine which
are of the greatest importance, thereby honing corporate focus. This type of effort
also facilitates iterative development, which benefits the organization with a more
adaptive approach to building targeted functionality faster for the business.

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
3
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

Such a framework enables more effective metrics that tie back to key business
imperatives. These metrics are useful for evaluating project progress for the areas
of highest impact and for assessing project success overall as well as the
capabilities of staff participating in the project (whether internal or outsourced).
Testing organizations must be able to show what they did and did not do, and
why. They need a strong process to guide them through this and to measure
effectiveness against what they are and are not testing. If this process is
repeatable, it enables organizations to create baselines and measure progress
historically, in order to improve on successes and avoid failures. This provides a
consistent and measurable approach for internal, outsourced, and vendor-
provided purchased application implementations.

Organizations can also manage and control testing costs more effectively through
prioritization of resources in this way. QA and QC groups tend to be highly
constrained, with restricted availability. Testing can be — and frequently is
perceived to be — the final “bottleneck” to deployment and project completion.
With this approach, limited testing resources are not overburdened with tasks that
provide negligible return for the effort and are able to execute the quality process
more efficiently (facilitating faster deployment). In addition, other “hard” assets
such as testing labs required for testing can be prioritized and appropriately
allocated.

Starting With Requirements

META Group research indicates that 60%-70% of project failures are tied directly
to poor requirements gathering, management, and analysis. To establish a risk-
based approach to testing in the context of value to the organization, everything
must be driven by the effort of defining and understanding business requirements.
For organizations with low maturity levels regarding requirements, this type of
testing approach can drive process improvement by demanding early coordination
and focus in conjunction with testing and other staff (e.g., architecture,
infrastructure, operations). Even use of simple prioritization ratings such as “high,”
“medium,” and “low” enables organizations to begin to establishing a weighted
approach for execution.

Evolving a Prioritized Testing Approach

Once the requirements are established, organizations align the test requirements
with business and technical drivers to determine the risk of failure. When testing
teams start to decompose the requirements into scripts, they have a context for
where to begin — first through consideration of risk factors and risk weights, and
then by breaking that down into testing decision matrixes. For instance, for a low-
priority, low-risk business requirement, the weighting factors enable testing staff to

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
4
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

make better choices by testing elsewhere. In this way, risks and tests are
assigned based on business decisions and technology context, not merely on
subjective QA decision making, political influence, timing, or other pressures.

Testing staff then builds test cases to determine the likelihood of failure. This
enables users to prioritize which test cases they want to execute first — the
impact of the risk of failure, tied to the likelihood of failure, plus expected business
value. If the organization only has a few weeks to complete a project, it is then
able to focus on the most important business drivers, which have been clearly
identified during the first phase. This facilitates the development, testing, and
execution phases of the project. This concept can also be applied to load testing,
enabling better definition of risks associated with specific loads and facilitating
decisions on what needs to be load and stress-tested.

Reporting and Process Adaptability

Being able to make this information accessible through reports to management on
expected risks, metrics, changing priorities, and risks across differing initiatives is
also important. Global 2000 organizations should evaluate and adopt automated
products that facilitate communication across the organization through relevant
“canned” reports that are easily customizable to address local needs and
requirements. Similarly, in establishing a process approach, adaptability to the
existing culture is vital for facilitating adoption.

Centralized Management
Once organizations have identified the test cases, it is useful to be able to build
test suites in a rapid manner based on available test windows. A centralized
library allows users to share test procedures across the organization — the
system automates access to test cases to facilitate availability and execution.
Often, users create a generic test suite without linkage to the existing manual or
automated tests. Facilitated access to test artifacts enables easier execution and
coordination with emerging change requests, projects, and programs. Appropriate
naming conventions are critical for enabling access to relevant information from
existing projects for potential reuse. (Of course, these approaches must be tested
again in the target environment.) Building these libraries is time-consuming
initially, but ultimately is beneficial.

Change Management and Project Governance

Incorporating a similar, prioritized approach to change and project management
and coordinating with testing and requirements is also a potential benefit to this
shift in approach. As organizations move forward on projects, decision making
about change requests will require discipline and rigor, or projects will burgeon out

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
5
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

of scope and out of control, thus violating cost and scheduling targets. In addition,
changes must be vetted through the testing process to maintain code that is of
good quality and stable. Applying prioritization to decision making and
incorporating an approach that encompasses the smallest unit of change as well
as the broader vision about the overall portfolio is helpful. This enables
organizations to better evaluate impact and target resource allocation
appropriately to benefit the business.

Generally speaking, individuals tend to focus on what is in front of them, whether

they are reading the words in a white paper, creating a requirement, writing a test
script, requesting a product enhancement, or evaluating an overall project.
Enabling a more encompassing vision and providing quantitative data to feed
qualitative project and overall project portfolio analysis provides a further evolution
of this process; this approach establishes even greater value to the organization.
With limited resources across the board and mounting competitive and time
pressures, appropriate decision making on allocation of resources for projects and
programs and even low-level change requests becomes a core business driver
and competitive advantage.

Organizational Strategy
In addition to using automated tools, organizations should set up separate testing
organizations to facilitate the creation of effective processes and appropriate
testing resources. Developers are not in a position to effectively test the code that
they have created beyond unit testing, because they lack incentive (since projects
are usually late, developers are disinclined to test effectively), expertise, and
objectivity.

We see organizations using three different approaches to testing: centralized

testing, decentralized testing, and a hybrid combination of the two (where a core,
centralized group provides testing resources, methodology, and support to
localized testing representatives that are in the lines of business). Although
organizations should opt for the testing organizational framework that best maps
to their culture, the hybrid approach enables more adaptive use of resources by
enabling a combination of both local and distributed resources. In addition, the
local testing representatives are in close communication with the business and
project deployment staff, and therefore are in a strong position to advise the
centralized group on functional and testing decision trees and matrixes.

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
6
 Best Practices for Application Quality Management:
Improve Efficiency, Increase Business Relevance, and Cut Costs

Bottom Line
Organizations should adopt coordinated approaches for testing processes that
leverage requirements for targeting and prioritizing testing resources and enable
support of and alignment with evolving business and technical needs.

Few companies can afford to allocate testing resources indiscriminately. Risk-

based and value-based testing enable organizations to appropriately prioritize and
target testing staff where they will be most effective. This can save time and
money, and enables organizations to be more quickly responsive to business
imperatives by delivering projects sooner.

For the longer term, companies should consider applying these principles of
prioritization across the project and program portfolio. Coordination with
appropriate governance capabilities ultimately allows organizations to focus the
overall resource pools on the areas of greatest payback to the organization.

Global 2000 organizations should establish rigorous, unified processes across

requirements, change management, and testing initiatives. They should apply
these processes consistently to establish targeted, risk-based and value-based
testing, thereby enabling a more adaptive business approach.

Melinda-Carol Ballou is a Senior Research Analyst with Integration &

Development Strategies, a META Group advisory service. For additional
information on this topic or other META Group offerings, contact
info@metagroup.com.

208 Harbor Drive • Stamford, CT 06902 • (203) 973-6700 • Fax (203) 359-8066 • metagroup.com
Copyright © 2004 META Group, Inc. All rights reserved.
7
About META Group
Return On IntelligenceSM
META Group is a leading provider of information technology research, advisory services, and
strategic consulting. Delivering objective and actionable guidance, META Group's experienced
analysts and consultants are trusted advisors to IT and business executives around the world. Our
unique collaborative models and dedicated customer service help clients be more efficient,
effective, and timely in their use of IT to achieve their business goals. Visit metagroup.com for
more details on our high-value approach.