Evaluation and Testing

Gihan Seneviratne
April 2013

Verification and Validation

Software testing is part of a broader group of activities called verification
and validation that are involved in software quality assurance
Verification (Are the algorithms coded correctly?)
The set of activities that ensure that software correctly implements a specific
function or algorithm

Validation (Does it meet user requirements?)

The set of activities that ensure that the software that has been built is
traceable to customer requirements

Testing???
Testing is an activity whose goal is to
determine if the implementation is correct.

Testing Phases

Determine Test activities

Select appropriate test data
Determine the expected results
Conduct the tests
Collect test results
Compare test results with expected results.

Design of Test activities

Begins with an analysis of:
The functional specifications of the system.
How the system will be used (referred to as use
cases).

Will result black box testing

Black Box testing

ignores the internal structure
test activities will be based on
the expected external behavior
of the system.

A Test case
Test cases are defined by:
A statement of case objectives.
test data set pay attention to boundaries
expected results

carry out concurrently with system

development. Sometimes you may have to
refine test cases during implementation

High Level: Generic Test Process

Test Planning &

Preparation

Test Execution

NO

Analysis &
Follow-up

Goals
met?

YES

EXIT

Test plan
describes the test cases

Test Planning & Preparation

(important but not always fully performed)

Two important items in a test plan include:

Testing objectives & Targets in terms of:

Reliability
Performance
Customer satisfaction
etc.

General Test Strategy which includes:

What & how much to test

Techniques of testing
Tools needed
Measurements and Analysis (details of)
People & Skills
Schedule

Dont Forget: Test Risks and Contingencies

Common to most of your projects
Poor project management, lack of time, half done etc.

Sample: Recording Test Case & Test suites

Test Case number (use some scheme as your text book shows )
Test Case author
A general description of the test purpose
Pre-condition: includes test results of other pre-req modules
Test inputs
Expected outputs (if any)
Post-condition

Test Case history:

Test execution date
Test execution person
Test execution result (s) : Pass Fail
If failed : Failure information
: fix status

Adequacy of test cases ???

can be measured based on the concept of
coverage
Functional testing (specification-based testing
or black box)
Structural testing (implementation-based or
white box)

Document templates

Project test plan

Component test plan
Integration test plan
Use case test plan
System test plan

Relationships among test plans

Important test plan items

Features not tested
Test suspension criteria and resumption
requirements
Risks and contingencies

Component test plan

Purpose is to define an overall strategy and
specific test cases used to test a certain
component
One for each significant component that
warrants isolated testing

Component test plan sections

Objectives for the class

Guided inspection requirements
Building and retaining test suites
Functional test cases
Structural test cases
State-based test cases
Interaction test cases

Use case test plan

Purpose is to describe system level tests to be
derived from a single use case
Incorporated into system and integration test
plans
Use case levels
High-level
End-to-end system
Functional sub-use cases

Use case test plan

Functionality use cases; modify the data
maintained by the system in some way
Report use cases; access information in the
system, summarized it, and formatted it for
presentation to the user

Integration test plan

Important in iterative development
environments
Specific sets of functionality delivered in
stages as the product slowly emerges
Use cases may not be build up yet
Use the component test plan

System test plan

Summarizes the individual use case test
plans
Provides information on additional testing
needed
System test plan
Use case number

Test case number

Reason for selecting

Coverage
indicates which items have been touched
by test cases
Code coverage
Post-condition coverage
Model-element coverage

Coverage metrics stated in terms of the

product being tested rather than the
process we are using to test it

Software Faults and Failures

Why Does Software Fail?

Wrong requirement: not what the customer

wants
Missing requirement
Requirement impossible to implement
Faulty design
Faulty code
Improperly implemented design

Software Faults and Failures

Types of Faults
Algorithmic fault
Computation and precision fault
a formulas implementation is wrong

Documentation fault
Documentation doesnt match what program does

Capacity or boundary faults

Systems performance not acceptable when certain limits are reached

Timing or coordination faults

Performance faults
System does not perform at the speed prescribed

Standard and procedure faults

Software Faults and Failures

Orthogonal Defect Classification
Fault Type

Meaning

Function

Fault that affects capability, end-user interface, product

interface with hardware architecture, or global data
structure

Interface

Fault in interacting with other component or drivers via

calls, macros, control, blocks or parameter lists

Checking

Fault in program logic that fails to validate data and

values properly before they are used

Assignment

Fault in data structure or code block initialization

Timing/serialization

Fault in timing of shared and real-time resources

Build/package/merge

Fault that occurs because of problems in repositories

management changes, or version control

Documentation

Fault that affects publications and maintenance notes

Algorithm

Fault involving efficiency or correctness of algorithm or

data structure but not design

Testing Issues
Factors Affecting the Choice of Test Philosophy

The number of possible logical paths

The nature of the input data
The amount of computation involved
The complexity of algorithms

The Purpose of Evaluation

the measurement of results against
established objectives
Revisit your first chapter and get the list of
objectives

Objectives:
A Prerequisite for Evaluation
Before any system can be properly evaluated , its
important to have a clearly established set of
measurable objectives

Continued
Dont wait until the end to determine how it will be
evaluated
starts in the planning stage , you break down the
problem into measurable goals and objectives then
after implementing the program , you measure the
results against goals
A highly user friendly system... How to evaluate?

Facts gathering
Surveys and Questionnaires
Used to collect information from a large group of respondents.

Interviews (including focus groups)

Used to collect information from a small key set of respondents.

Experiments
Used to determine the best design features from many options.

Field studies
Results are more generalizable since they occur in real settings.

Copyright 2006 John Wiley and Sons, Inc.

Common mistakes
Standalone development but claim its
network/ web enabled (e.g. MS Access)
Single user system but claims its multi-user
enabled
MS based development but claims platform
independent
IP 127.0.0.1 but web-based

Contd

Browser compatibility of web-based systems

Display compatibilities (std vs wide)
Backup and DR
Parallel runs, run with historical data or hand
made data
HCI issues
Installation dependencies (works only in your
pc)

 Thank you!

Q&A