Using Metrics To Drive Quality Improvements and Value - Company Wide

"Using Metrics to Drive Quality Improvements and Value -Company Wide"
Bill Woodworth
Director, IBM Software Test & Development Excellence IBM Corporation billwood@us.ibm.com
Presented at IEEE ICSM Conference 13 September 2004
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Improvements & Value
Agenda:
Background IBM is a Software Company
Driving & Leading Company Wide SQE Initiatives Why = Objectives What = Communities of Testers and of Developers How = Focuses and Success Factors
Using Metrics to Drive Improvements Value Critical Role of and Types of Metrics How Used More Partnership across the Industry
2 Using Metrics to Drive Quality Sept 2004 -- IEEE
Copyright 2004 IBM Corporation
IBM is a Software Company

In fact, IBM is a HUGE SOFTWARE Company
a) Our products span the entire range: firmware / microcode, operating systems, middleware, systems management, applications, imbedded, etc. b) Software revenue = $14.3 B in 2003; 1/3 of total company profits c) World's largest infrastructure (middleware) software provider d) Second largest software business in the world e) #1 or 2 in market share in all our software markets (per IT analysts) f) High double digit thousands of software engineers g) Thousands of software products
And QUALITY Matters Big Time to Our Customers
IBM's systems are generally NOT desktops where you can afford
to crash, burn and re-boot BUT are business-critical systems that MUST NEVER crash Holistic Quality is key to our Success as a Software provider
Using Metrics to Drive Quality Sept 2004 -- IEEE
Our Focus on Holistic Quality

Quality is key to IBMs success as a software provider
Total Customer Satisfaction in our products and solutions Continual improvement = each release better than its predecessor Quality + Function + Speed to Market + Cost
where each aspect plays an important role in delivering value to our customers
Function
We apply a quality management system balancing:

Cost
Software Quality focus must be Holistic: More than reducing defects Quality Speed To Market Ensuring Correctness Increasing Availability, Reliability, Integrity, Installability, Testability Improving Problem Determination, Usability, Interoperability,
Portability, Efficiency, .... Reducing and Preventing Problems
Meeting customer expectations Disciplined risk management

Our Quality System Before 1998

Each unit working to improve Quality of its products Each release expected to be better than its predecessor
BUT limited company wide sharing on:

Best processes and practices Lessons learned Networking and helping each other
Since then, related complexities began really increasing:

Economic need for more effective & efficient operations Quality expectations of customers in the industry Complexity of technologies and combinations in solutions New initiatives in OnDemand and Autonomic Computing Development and Test process pressures from all the above
Need and Opportunity for Software Quality Engineering Improvements in ALL Units Company Wide SQE Focus
Agenda:
Driving & Leading Company Wide SQE Initiatives
In 1998:
Created a company wide
Created Testing Process Focus

IBM Testers Community
Spans all product labs, global services, research
In 2003:
Created a company wide
Added Development Process Focus

S/W Developers Community
Modeled on success & operations of ITCL
Each Community has Same Basic Vision & Purpose: Vision = Teaming to Become BEST OF BREED Purpose = Delivering VALUE to Each Business Unit
Company Wide SQE Initiatives Focus
What we did & are doing = Key Focus Areas

People and Community Processes, Practices and Tools Metrics and Driving for Improvements
How we were successful = Keys to Success

8
Recognized need to improve Recognized need to help each other Unit Leaders = active and empowered Value demonstrated to all players Walk before you run mentality
Highly Successful SQE Communities
Drive and Provide the Means for:

career & skill focus training & recognition
Developing Resources
needs based workgroups best technology/source
Developing People
self assessment using best resources remembering lessons learned
Driving Improvements
Delivering Value
Community
Operational Infrastructure
nurturing communications website, T.R.s, DBs newsletters; meetings
Sharing
impassioned unit leaders change agents
Leading & Deciding
in-person meetings building relationships contacts
Networking
Units Becoming Best of Breed

2 Keys within Each Unit Enabling Unit Self-Improvement
1. ITCL & SDCL Unit Leaders Aggressive Change Agents 2. Meaningful, disciplined and supported application of the IBM Test and Development Self Assessments to affect the most needed improvements
UNIT
Current position Issues & needs
UNIT
Measurably Improved
ITCL & SDCL
Resources
Test & Dev Test & Dev Self Self Assess't Assess't
IBM Rational Products
Rational Unified Process
10
TESTING: From Ideas to Implementation.

_1998_ _1999_ _2000_ _2001_ _2002_ _2003_ _2004_
Team Building & Networking Developing ITCL Resources Developing IBM Test Solutions.. Deploying IBM Test Solutions
Deploying ITCL Resources
Driving Test Improvements. Shifting to Rational
Yielded 100s of Improvement Actions and $10,000,000s of Benefits Each Year

Agenda:
PROCESS METRICS
Critical Role in Improving S/W Development
Quality Process Focus & Cost of Quality
Type of Key Metrics

Delivered Quality Metrics In Process Metrics
How Used..
Business Decision Making Process Driving Improvements Causal Analysis Process Self Assessments Scorecards / Dashboards Used to Demonstrate Progress and Value Gaining Executive Buy In
Delivered Quality Metrics

Customer Satisfaction Customer Reported Problems Customer Reported Defects Release-to-Release Crit Sit Trend Availability and Outages Warranty / Service Costs Cost of handling customer reported problems/defects, sometimes normalized by revenue or dev expense Basic comparisons which demonstrates a given release is better than its predecessor Trend of problems requiring intervention outside the normal support process Number of problems which customers report Reported survey results
Number of problems which customers report which result in code fixes
Number of system failures, usually normalized by usage months

Timing = monthly, quarterly, annually Optional grouping by product maturity

In Process Metrics
Design / Code Complete
Functional Test
System Test Coverage Causal Analysis Development Self-Assessment Test Self-Assessment Process to aid test teams in evaluating their units strengths / weaknesses improvement actions Process to aid development teams in evaluating their units strengths / weaknesses improvement actions
Means to understand progress vs plan schedule, size, functionality design / code peer reviews performed unit test completed
Means to understand progress vs plan schedule (attempts & completions) problems / defects (valid & total) outage durations defect arrival rates and density test effectiveness
Means to understand how much function or code has been tested Analyzing defects / problems = lessons learned & remembered
PROCESS METRICS
Critical Role in Improving S/W Development
Quality Process Focus & Cost of Quality
Type of Key Metrics

Delivered Quality Metrics In Process Metrics
How Used..
Business Decision Making Process Driving Improvements Causal Analysis Process Self Assessments Scorecards / Dashboards Used to Demonstrate Progress and Value Gaining Executive Buy In
Integrated Product Development (IPD)

A management system designed to optimize the development and the delivery of successful quality products and offerings. Fact Based Data to Manage Integrated Portfolio Management Team Your Business (IPMT)
Market Planning
Information Customer Feedback Competitor Information Technology Trends Current Product Portfolio
Resource Allocation
Perform Understand Perform Market the Portfolio Marketplace Segmentat'n Analysis Develop Align & Market Optimize Strategy Business & Plan Plans
Project Management Data Segment Performance Data

Manage Market Segment and Assess Performance
Candidate Projects
Project Management Data Performance Data
Structured Product Development

17
Product Objectives Business Strategy Customer Buying Behavior R&D Roadmap
Concept
Plan
Devel Qualify Launch
Life Cycle
Satisfied Customers $$$$
Project Development Teams

Integrated Product Development (IPD)

Event Driven Formal 'Decision Check Points' (DCPs)
Strategy Offering Investment
Execution
Concept
Plan
Develop
Qualify
Launch
Lifecycle
IPMT and PDT Contract
Concept Plan DCP DCP

Availability DCP
EOL DCP
Closed Loop Development Process

Improving Emphasis
Reducing escapes
TEST
Planning Design / Development Execution
Customer
Reducing escapes
Focus
Development
Process Ease of Use Availability Reliability Learning/ Changing
CUSTOMERS
Experiences fewer problems / defects Improved solution availability
Learning/Changing
Learning a Must
Key = Learning & Remembering From Mistakes
Disciplined & focused causal analysis . Orthogonal Defect Classification (ODC) .
19
Used to Demonstrate Value & Progress
Metrics used to understand an aspect of the business

Not metrics for metrics sake Means to understand: WHERE are we?
Are we making PROGRESS? Improvements in Process / Practices Its all about trends
Progress in what.
Customer satisfaction Customer support Reducing customer found defects and problems Benefits of process / practice improvements made. ..
What is managements business / operational focus?

Align metrics to develop applicable understanding
Using Metrics to Drive Quality Sept 2004 -- IEEE 20
Gaining Executive Buy-In to Drive Improvements

3 APPROACHES (in addition to base line management responsibility) 1. Tops Down =
Corporate Guidance on Quality of Offerings
2. Peer Pressure ++ = Semi-annual Improving S/W Quality Executive Reviews 3. Partnering = Teaming Line Executives with Executives of Software Communities
Function Cost
Sp eed
ty ali u
Teaming Line Executives with Community Executives
Line execs responsible for quality of their offerings
Holistic quality plus operational effectiveness and efficiencies Must inspect what they expect
Active support for Process Self Assessments
Set Expectations == Improving is not only OK, but is expected Communities provide resources (single sourced & anticipatorily) Unit leaders have opportunity to demonstrate leadership
(i.e. look good) and drive improvements from business wise
SQE Community Execs with company wide leadership team

Create IBM wide communication fabric Develop needed resources once
SO ALL UNITS, who need them, can use them
TEAMING TO BECOME BEST OF BREED

Agenda:
More Partnership Across the Industry
In spite of industry wide efforts in using metrics to

drive quality, difficult problems still exist (e.g.): More meaningful and easily collected customer impact metrics Tools to more easily process multi-language legacy products Automated test technologies (auto generation, etc.) Approaches on how to turn academically sound metrics
into successful real world usage
The whole industry should work together to solve

these problems maybe you can help!
24
We must ALWAYS IMPROVE !!!
The world is depending on software more and more and significantly more software is being produced for everything from pacemakers to cars to handhelds to desktops to servers.
Everything industry is doing

today must significantly improve to meet the challenge.
Industry needs more practical solutions for some real

problems!

Using Metrics To Drive Quality Improvements and Value - Company Wide

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Using Metrics To Drive Quality Improvements and Value - Company Wide

Transféré par

Droits d'auteur :

Formats disponibles

"Using Metrics to Drive Quality Improvements and Value -Company Wide"

Presented at IEEE ICSM Conference 13 September 2004

IBM S/W Test & Development Excellence

Using Metrics to Drive Quality Improvements & Value

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

IBM is a Software Company

And QUALITY Matters Big Time to Our Customers

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Our Focus on Holistic Quality

We apply a quality management system balancing:

Meeting customer expectations Disciplined risk management

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Our Quality System Before 1998

BUT limited company wide sharing on:

Since then, related complexities began really increasing:

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Using Metrics to Drive Quality Improvements & Value

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Driving & Leading Company Wide SQE Initiatives

Created a company wide

Created Testing Process Focus

Spans all product labs, global services, research

Created a company wide

Added Development Process Focus

Modeled on success & operations of ITCL

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Company Wide SQE Initiatives Focus

What we did & are doing = Key Focus Areas

How we were successful = Keys to Success

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Highly Successful SQE Communities

Drive and Provide the Means for:

self assessment using best resources remembering lessons learned

nurturing communications website, T.R.s, DBs newsletters; meetings

impassioned unit leaders change agents

Leading & Deciding

in-person meetings building relationships contacts

Using Metrics to Drive Quality Sept 2004 -- IEEE

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Units Becoming Best of Breed

ITCL & SDCL

IBM Rational Products

Rational Unified Process

Using Metrics to Drive Quality Sept 2004 -- IEEE

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

TESTING: From Ideas to Implementation.

Deploying ITCL Resources

Driving Test Improvements. Shifting to Rational

Yielded 100s of Improvement Actions and $10,000,000s of Benefits Each Year

Copyright 2004 IBM Corporation

IBM S/W Test & Development Excellence

Using Metrics to Drive Quality Improvements & Value

Copyright 2004 IBM Corporation