SOFTWARE QUALITY Why do programs have bugs?

ASSURANCE
John Abbott College z Involvement makes us blind
z Expectations mask reality

Automated z
z
Interactions are unpredictable
Testing takes too much time
z Testing is repetitive and tedious
Testing
M. E. Kabay, PhD, CISSP
Director of Education, NCSA
President, JINBU Corp

Copyright © 1997 JINBU Corp. All rights reserved

QA 9 - 1 QA 9 - 2

Quality Assurance Depends

The Cost of Software Quality
on Testing
z At least 60% of your development budget z Critical examination
z is used to test only 20-25% z Doing everything feasible to find errors
z of your application features z Errors are deviations from specification

QA 9 - 3 QA 9 - 4

The System Development

To Achieve Software Quality
Life Cycle (SDLC)
Define Requirements
We must conduct a
critical examination Design Specifications
of a system's
quality every time Code Software

we implement a
change Fix Defects

QA 9 - 5 QA 9 - 6
 Why QA? Why automated testing?

z Errors in mission-critical systems can be real- z SDLC known as "waterfall"

world and business disasters z Software backlog running 2 years and more
z E.g., software error in chemical-plant in z JAD/RAD more popular than ever
Netherlands
z Therefore user I/F changes constantly
– allowed operator to enter codes for wrong
chemical substances
– exploded on contact in vat
– destroyed plant
– killed two people

QA 9 - 7 QA 9 - 8

Current Testing Methods are Consequences of Manual

Inadequate Testing Methods
z Manual input z Quality is not emphasized during SDLC
z Unstructured z Time pressures always squeeze testing
z Slow z Testing never catches all the bugs
z Depend on testers' awareness and attention
z Leave no audit trail
z Poor or no statistics
z Manual demonstration of errors

QA 9 - 9 QA 9 - 10

Automated Testing
z Capture/Playback
– record macros showing mouse
mouvements and alphanumeric input
– typically no editing language
z Structured Automated Testing
– tool creates structured, editable script
– can use databases as source of input
– intelligent handling of errors
Limitations of
Capture/Playback
z Merely automate manual procedures
z Difficult to maintain as application changes
z Cannot build regression database
z Must wait until application is ready
z No mechanism for detecting errors
z No mechanism for reporting results

QA 9 - 11 QA 9 - 12
 Good Applications Are Easily
Structured Testing
Maintained and Enhanced
z Structured development z Modular design
z System documentation z Documentation
z Metric: ease of reliably changing application z Segregation of data from procedures
z QA must learn from general programming z Re-usability
experience

QA 9 - 13 QA 9 - 14

Structured Automated Benefits of Structured

Testing Automated Testing
z Define test plan z Consistent, reproducible testing
z Document logic z Increased test coverage
z Generate test procedures z Easier maintenance
z Apply test procedures z Fully documented testing
z Evaluate results z Higher-quality software

QA 9 - 15 QA 9 - 16

Case Study: Case Study:

COGNOS / Ottawa COGNOS / Ottawa
z 15 days for testing z "Our goal was to improve the level of testing
z 6 people @ $300/day while at the same time reducing the time and
manpower for each release."
z 3 test phases per product release
– Doug Clement, COGNOS
z 3000 manual tests per phase
z 12.5% test coverage
z $81,000 per release @ 12.5%
z $648,000 per release @ 100%

QA 9 - 17 QA 9 - 18
 Case Study:
AutoTester at COGNOS
HRL CANADA / Ottawa
z 5 days elapsed time z Jacques Joanisse, manager
z 6 people z Built regression database
z 3 test phases z 80% of the application features were tested
z 24,000 tests/phase z Redeployed 60-80% of test group back into
z $27,000/phase using AutoTester development

QA 9 - 19 QA 9 - 20

Case Study:
New York Life / Toronto
z Business case showed that up-front cost of
implementing automated QA process will be
repaid in the first year
z Cost-reduction: lower staffing, shorter cycle
z Cost-avoidance: identifying more errors
Sample ROI
z Take $1,000,000 SW development budget
z Manual testing costs 60% = $600,000
z Manual testing: ~20% of application features
z Automated testing: test 90-95% of features
z Automated testing costs 24% = $260,000
z Real savings: $300,000 including cost of
testing tool

QA 9 - 21 QA 9 - 22

Structured Automated PC-based S.A.T. are Platform

Testing
z Emulates trained human operator
z Single tool for multiple platforms
z Supports corporate standards
z Simplifies training and support
Independent
z Works on PCs
z Can test PC programs
z Can test link to hosts using emulators
z Handles online, real-time systems
z Can deal with batch and hard-copy report
systems by examining results on screen
z Has been used for ATMs, POS, telephone
switches

QA 9 - 23 QA 9 - 24
 Solution to the Software
AutoTester Plus Components
Quality Gap
z Script Station z Structured development is key to effective
– Uses screen image and screen definitions maintenance
– Fully generate all scripts for application z Tie test cases to application specs
tests z Don’t write programs to test
z Test Station application
– Menu-driven application z Test-procedures stored in database
– Uses generated scripts to guide users
– Users create, manage and execute test
cases
– Apply meaningful and comprehensive
tests

QA 9 - 25 QA 9 - 26

Independent Analysis Independent Analysis

Gartner Group:
z "AutoTester from AutoTester, Inc. of Dallas
reduces testing effort and improves the
quality of resulting systems."
z "AutoTester can effectively provide a facility
for regression testing most interactive
applications at a fraction of the cost and
effort required to do so manually."
Software Quality Engineering:
z "AutoTester is unique because it can test
both PC-based applications as well as
applications which run on mini or mainframe
computers.
z Variable files support in AutoTester allows
test cases to be imported from external PC or
remote sources, and permits single scripts to
exercise unlimited permutations, thus
conserving script development and
maintenance effort."

QA 9 - 27 QA 9 - 28

Automated Testing is Testing Spans the

Suitable for Many Types of Organization
Quality Assurance Tests
z Way of working
z Unit tests z Not gimmick
z Integration tests z Permeates development team
z Regression tests z Management support required
z Stress tests z Involves users
z Plan for pilot project before choosing tool

QA 9 - 29 QA 9 - 30
 Demonstration Disks
z Load the demo disk onto a workstation
z Follow the instructions to proceede through
the demonstration.
Homework
z Readings: Chapters 7, 11, and 13 from your
textbook
z Answer all the review questions distributed
by the instructor
– Avoid copying the textbook blindly -- you
will not remember as much as if you think
about the answers yourself
– Use simple language; usually a few words
or sentences will be ample
z Submit your work by 09:00 tomorrow
morning.
z Because of the short time available, do not be
late in submitting your review answer

QA 9 - 31 QA 9 - 32