Chapter 6 notes:

Purposes, activities , processes , and context of Software Testing.

Purpose: to ensure that the software systems would work as expected when they are used by
their target customers and users.
o Demonstrations of proper behavior /providing evidence of quality (terms of s/w QA)
/meeting certain quality goals is primary purpose of testing.
o to detect and fix problems
Activities: to show that s/w fulfills the expectations we demonstrate their operations through
some dry-runs or controlled experimentations through program executions before the
products are released/delivered TESTING.
Processes: focus testing in development process to achieve defect free manufacturing process.
Major activities and the generic testing process:
Test Planning and preparation: which set the goals for testing, select an overall testing strategy
and prepare specific test cases and the general test procedure.
Test execution: and related activities. Which also include related observation and measurement
of product behavior.
Analysis and follow up: includes result checking and analysis to determine if a failure has been
observed. If so follow up activities are initiated and monitored to ensure the removal of the
faults / causes that led to the failures.
Refer to fig 6.1 text book.
Test run the execution of a specific test case, or a sub-division of the overall test execution
sequence for some systems that require continuous operation.
Effective test execution is handling the problems of the failed test runs so that it wont block the
executions of the other test runs especially in systems which require continuous operations.
If the test run fails , the data collected during execution or other measurements can be used to
locate and fix the underlying faults that led to the failure.
Analysis reports provide assessments of current status wrt progress,effort,defect, and product
quality so decisions of when to stop testing can be made based on the facts .
Sub-activities in test planning and preparation:
Goal setting: quality views and attributes have been decided by the overall quality engineering
process. Specific testing goals ,such as reliability or coverage goals, to be used as exit criteria.
Test case preparation: includes constructing new test cases or generating them automatically,
selecting from existing ones for legacy products and organizing them in some systematic ways
for easy execution and management.
Test procedure preparation: for systematic testing on large scale for today s/w products and
systems, a formal procedure is more of a necessity. It can be defined and followed to ensure
effective test execution, problem handling and resolution and overall test process management.
Other QA alternatives to complement testing:
Defect prevention reduces defect injections during s/w dev, resulting in fewer faults to be
detected and removed during testing, thus reducing testing efforts and expenditure.
Formal verification to verify correctness of some core functions in a product instead of
applying exhaustive testing to them.
Fault tolerance and failure containment strategies might be appropriate for critical systems.
Test result analysis and follow up activities have to be carried out in parallel to testing and
should not stop to ensure discovered problems are all resolved and long term improvement
initiatives are planned and carried out.
Functional VS Structural testing :
There are different levels of testing corresponding to different views of the code and different levels of
abstraction, as follows:
Detailed level:
o Individual program elements- individual statements, decisions, data items can be
tested.
o Depending on the programming language used, this unit may correspond to a function,
a procedure, a subroutine or a method.
o Components- collection of smaller units that together accomplish something or form an
object
Intermediate level:
o Various program elements or program components may be treated as interconnected
group and tested.
o Done at component, sub-system, or system levels with the help of some models to
capture the interconnection and other relations among different elements or
components.
Abstract level:
o Whole s/w systems can be treated as black box.
o Focus on the functions or input-output relations
Tendency is that at higher levels of abstraction, functional testing is used and at lower levels of
abstraction, structural testing is used.
Functional / black box testing(BBT):
Verifies the correct handling of the external functions provided by the s/w, through the
observation of the program external behavior during execution.
Ad-hoc testing- simplest form of BBT start running the s/w and make observations in
the hope that it is easy to distinguish between expected and unexpected behavior.
o Unexpected behavior like, crashes are easy to detect and after multiple
executions to rule out hardware problems, the info is passed to responsible
party to get it fixed.
Specification Checklists- common form of BBT- lists the external functions that are
supposed to be present, as well as some info about the expected behavior or input-
output pairing.
o Input/output-any action, artifact, or result produced by running program either
at beginning /end or at any time during program execution.
Formalized and systematic BBT can be based on some formal models which are derived
from system requirement or functional specifications.
BBT can follow generic testing process described above.
o Planning stage: identify external functions to test, and derive input conditions to
test these functions.
o Ex: compiler where the input is the source code to be complied and output is
executable code.
o Part of expected behavior is system termination- compiler must produce output
within given time.
o Another expected behavior is when illegal program is inputted then the output is
not generated.
o Collection of programs to be complied constitutes the test suite or collection of
test cases which consists or legal and illegal programs to cover the expected
spectrum of input.
Focus is to observe the external behavior, to ensure orderly execution of all the test
cases, and to record execution information for analysis and follow up activities.
If observed behavior patterns cannot be identified as failures, info needs to be recorded
for further analysis.
Oracle testing problem- once execution result is obtained, analysis is carried out to
compare the specific behavior and the output with expected ones to determine if
expected behavior or failure occurred.
Once failures are detected follow up activities are initiated to detect faults and remove
them. Info recorded during test execution is used to recreate the failure scenarios,
diagnose problems and causes and fix them.
Emphasis is on reducing chances of encountering functional problems by target
customers.
Structural/ white box testing (WBT):
Verifies the correct implementation of internal units, such as program statements, data
structures, blocks, etc. and relations among them.
This is done through test execution by observing the program behavior related to these
specific units.
Connection between execution behavior and internal units needs to be done and its
done with various s/w tools.
Statement coverage testing- simplest form of WBT- helps trace the program executions
with help of various debugging tools and debuggers.
o Tester can see if a specific statement has been executed and if the result or
behavior is as expected.
Adv: problem once detected is also located.
Dis adv: omission or design problems are not easily detected as only what is present in
code is tested.
Tester should be familiar with code under testing to trace through the executions and
also to perform related activities. This dual role makes defect fixing easy.
Formalized and systematic WBT can be based on formal models which are derived from
system implementation details.
WBT can also follow generic testing process but due to extensive amount of
implementation knowledge required, WBT is limited to small scale.
For small products not much formal testing process is needed to plan and execute the
test cases and follow up on execution results.
Stopping criterion: internal quality measures, defect levels, once planned coverage has
been achieved by exercising all the statements and all paths.
Comparing BBT and WBT see text book: pg . 77
Coverage based VS Usage based testing : when to stop testing?
Without formal assessment the decision to stop testing is made in 2 general forms:
o Resource based criteria: where decision is made on resource consumptions.
Stop when you run out of time
Stop when you run out of money.
This is used when product schedule or cost is dominant concerns but it is
not good for quality of product.
o Activity based criteria: stop when you complete the planned test activities.
Based on formal analyses and assessments:
o On global level , exit criteria determined by product release which is determined
by the reliability assessment-involves achieving quality goals and project goals in
over s/w dev process to meet the customers expectations.
o In reliability criterion, u set the reliability goal in the quality planning activity
during the product planning and requirement analysis and later on compares the
reliability assessment based on testing data to see if this pre-set goal has been
reached. If yes, product can be released else continue testing.
o Implications of using this criteria For stopping testing is that Reliability testing
should be as close to what actual users would expect which requires testing right
before product release. Resembles what actual usages by customers. This
resulted in usage based testing
o For testing internal components the use of reliability criteria is not relevant as
users are not in direct contact with it. So alternative exit criteria is needed.
o Rule of thumb: products should not be released unless every component has
been tested.
o Implication for using coverage as stopping criterion is that everything covered is
defect free with respect to this specific coverage aspect as all defects found
would have been fixed before product release.
o Specific quality goal can be translated into specific coverage goal.

Usage based testing (UBST) and operational profiles(Ops)
Actual customer usage of software products can be viewed as a form of usage based testing
and problems if encountered can be reported to software vendors and integrated fixes can
be constructed and delivered.
o This is very expensive due to large no of s/w installations
o Frequent fixes can damage the s/w vendors reputation n business
Beta test helps s/w vendors through controlled release to beta customers who help
improve their s/w quality.
Testing environment resembles the actual operational environment for the s/w product in
the field and the overall testing sequence as represented by the orderly execution of
specific test cases in a test suite, resembles the usage scenarios, sequences, and patterns of
actual software usage by the customers.
For practical implementation of such a testing strategy, actual usage info needs to be
captured in various models, commonly referred to as operational profiles or Ops.
Different OPs are associated with different testing techniques for UBST.2 models are:
o Flat Ops or Musa OPs: which is present commonly used operations in a list, a
histogram, or a tree structure, together with the associated occurrence probabilities.
Simple in model construction and usage.
o Markov chain based usage model/Markov OPs: present the commonly used
operational units in markovs chain, where the state transition probabilities are
history independent. Complete operations can be constructed by linking various
states together following the state transitions, and the probability for whole path is
the product of its individual transition probabilities. Markov models based on state
transitions can capture navigation patterns than flat OPs, but are more expensive to
maintain and use.
UBST is generally applicable to final stage of testing, typically referred to as acceptance
testing right before product release, so stopping testing is = to releasing the product.
Coverage and Coverage based Testing(CBT)
There are various forms of test coverage as stopping criteria:
o Completing various checklists is the simplest form.
For BBT- checklist of major functions based on product specifications
WBT checklist of all product components or all statements
o Testing can be performed until all the items on the checklist have been checked
off.
For systematic testing, formal models are used:
o Formally defined partitions can be used as the basis for various testing
techniques which are similar to checklists but ensure:
Mutual exclusion of checklist items to avoid unnecessary repetition.
Complete coverage defined accordingly
o A specialized type of partitions, input domain partitions into sub-domains, can
also be used to test these sub-domains and related boundary conditions.
o Various programming or functional states can be defined and linked together to
form finite state machines(FSMs) to model the system as the basis for various
testing techniques to ensure state coverage and coverage of related state
transitions and execution sequences.
o This FSM can also be extended to analyse and cover the execution paths and
data dependencies through various testing techniques.

Read rest from text book