Software Testing Techniques

How to find bugs in the software?

Objectives To understand the role of testing in ensuring software quality To discuss issues relevant to software testing To describe the different techniques used for testing software

 To understand the role of testing in ensuring software quality To discuss issues relevant to software testing To describe the different techniques used for testing software To design tests that systematically uncover different classes of errors and to do so with a minimum amount of time and effort About Testing ? Testing is a set of activities that can be planned in advanced and conducted systematically. Testing begins at the module level. Testing conducted by the developer and an independent test group. Testing and debugging are different activities. 2

Objectives

Validation & Verification

Validation Are we building the product right? Ensure software meets customers needs Verification Are we building the right product? Ensure software meet specification (error-free)

What is S/w Testing ?

Software Testing is a process of executing a program with the intent of finding an error. A good test case is one that has a high probability of finding an as yet undiscovered error A successful test is one that uncovers an as yet undiscovered error

Software Testing Strategies

Organizing software testing: To be done by the developers and an independent testing group in tandem. A software testing strategy: Viewed as a spiral. S R D C U I V

Software engineering Requirements Design Code

Unit test Integration test Validation test

System Test ST To develop spiral in to test spiral out

The V-Shaped Model

V-Model evolved from waterfall Model.
Each phase must be completed before the next phase begins. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape. Testing is emphasized in this model more than in the waterfall model. It is a structured approach to testing.

Brings high quality into the development of our products.

Steps in the V-Shaped Model

Quality is guaranteed at each project stage.

Stage Containment
This term is used to identify problems existing in the product being developed before proceeding to the following stage.

More errors than defects.

Cost and effort for fixing problems is minimized.
Defect Origin Analyze Design
Detailed Design Component Test Assembly Test

Prod Test

Deploy

Defect Discovered

Analyze

Design

Detailed Design Component Test Assembly Test

Prod Test

Deploy

With Stage Containment

Without Stage Containment

More defects than errors. Fixes become more expensive and difficult.

Defect Origin

Analyze

Design

Detailed Design Component Test

Assembly Test

Prod Test

Deploy

Defect Discovered

Analyze

Design

Detailed Design Component Test Assembly Test

Prod Test

Deploy

Without Stage Containment

Defect Origin

Analyze

Design

Detailed Design

Component Test

Assembly Test

Prod Test

Deploy

Defect Discovered

Analyze

Design

Detailed Design Component Test Assembly Test

Prod Test

Deploy

Worst Case!

Entry and Exit Criteria Entry Criteria The Term Entry Criteria refers tot the input conditions required from the previous processes to support a specific stage. Exit Criteria The term Exit Criteria refers to the output conditions required by a specific process to determine its thoroughness and correct completion.

The Exit Criteria for one stage can constitute part of the Entry Criteria for the following stage.

The Five Key Deliverables of Test Planning

Deliverable
Test Approach Test Scenarios Test Conditions and Expected Results Test Cycle Control Sheet

Description
Explains the objectives and scope of the test; Documents entry/exit criteria and key dates Provides high-level descriptions of functionality to be tested Describes all items and results that must be covered to fulfill each Test Scenarios Groups test scripts into logical categories (or cycles); documents when and by whom each cycle will be executed. Provides step-bystep instructions and detailed results for a test executor to follow during test execution

Test Scripts

Unit testing The most micro scale of Testing A unit = smallest testable software component Objects and methods Procedures / functions Performed by Programmer A tester can help. Requires detailed knowledge of the internal program design and code. The units are tested in isolation. Ensures the component is working according to the detailed design/build specifications of the module. Not to be confused with debugging. Also known as component, module, or program testing.

Integration Testing
Testing of more than one (tested) unit together to determine if they function correctly. Focus on interfaces Communication between units It is done using the integration test design prepared during the architecture design phase. Helps assembling incrementally a whole system, ensuring the correct flow of data from the first through the final component. Done by developers/designers and testers in collaboration Also called Interface Testing or Assembly Testing.

System testing Testing the system as a wholeBlack-box type testing that is based on overall requirements specifications; covers all combined parts of a system. Ensures that system meets all functional and business requirements. Focus Verifying that specifications are met Validating that the system can be used for the intended purpose The system test design is derived from the system design documents and is used in this phase. It can involve a number of specialized types of tests to check performance, stress, documentation etc. Sometimes testing is automated using testing tools. Done by Independent testing group

Acceptance testing

To determine whether a system satisfies its

acceptance criteria and business requirements or not. Similar to System testing in that the whole system is checked, but the important difference is the change in focus. Done by real business users. It enables the customer to determine whether to accept the system or not. Also called as Beta Testing, Application Testing or End User Testing. Approach Should be performed in real operating environment . Customer should be able to perform any test based on their business processes. Final Customer sign-off.

Benefits of V-Model
Faults are prevented and it stops fault multiplication. Avoids the downward flow of defect. Lower defect Resolution cost due to earlier detection.

Improved quality and reliability.

Reduction in the amount of Re-work. Improved Risk Management Validation and Verification at each level of stage containment Allows testers to be active in the project early in the projects lifecycle. They develop critical knowledge about the system.

Software Testing Strategies

Unit testing: uses white box testing, exercises all paths max error detection. Integration testing: dual problem of verification and program construction. Black box testing methods are used here. Validation testing: that software meets all functional, behavioural and performance requirements. System testing: all elements mesh properly. Overall system function/performance is achieved.

Software Testing Strategies

Criteria for completion of testing: Using statistical modelling and software reliability theory, models of software failures uncovered during the testing as a function of execution time can be developed. Logarithmic Poisson execution-time model: f(t) = (1/p) ln[(lopt+1)] f(t) = cumulative number of failures that are expected to occur once the software has been tested for a execution time = t. lo = initial software failure intensity (failures per unit time) at the beginning of testing p = exponential reduction in failure intensity as errors are uncovered and repairs are made.

Software Testing Strategies

Instantaneous failure intensity can be derived as l(t) = lo / (lopt +1) taking the derivative of f(t).

Failures per test hour

Predicted failure intensity l(t)

Execution time t

Software Testing Strategies

Unit testing: Integration testing: Top-down Bottom-up Validation testing: Test Criteria alpha and beta testing System Testing: Recovery Security stress Performance Big Bang , Smoke testing, Sandwich testing Debugging

Testing Terminology
Failure
Incorrect or unexpected output Symptom of at fault

Fault
Invalid execution state Symptom of an error May or may not produce failure

Error
Defect or anomaly in source code Commonly referred as bug May or may not produce fault

21

Why We Need Software Testing?

Reveal faults/failures/errors Locate faults/failures/errors Show system correctness Improved confidence that system performs as specified (verification) Improved confidence that system performs as desired (validation) Indicator of system reliability and system quality

22

Importance of Testing
Critical Element of Software Quality Assurance Well-planned and thorough testing are important in reducing the high cost of software failure Can take up to 40% of development effort In systems that require high reliability, testing can be 3 to 5 times of all the other steps combined
23

Testing Principles
All test should be traceable to customer requirements. Tests should be planned before testing begins. Testing should begin with individual components and move towards to integrated cluster of components. The most effective testing should be conducted by an independent third party.

24

Pre-requisites For Good Testing

Adequate staff Adequate support facilities (tools) Adequate time Who Should Test The Software? Developer (individual units) Independent test group (ITG) removes conflict of interest reports to SQA team
25

Successful Testing
Uncover errors in the software Demonstrates that the software functions appear to be working according to specification Data collected as testing is conducted provide a good indication of software reliability and software quality

Testing Guidelines
Testing a systems capabilities is more important than testing its components Testing old capabilities is more important than testing new capabilities Testing typical situations is more important than testing boundary value cases
26

Test Cases
Test case : unit of testing activity Test cases have 3 parts : Goal
Aspect of the system being tested

Input and system state

Data provided to the system under stated condition

Expected behavior
The output or action the system should take according to its requirements

27

Test Cases (Cont)

Test cases are derived during all phase of development cycle Determine the expected result before running the test case Record test cases and its results Allocate

time

money

people

resources
28

Type Of Test Cases

Test cases are derived for
Valid and expected input Invalid and unexpected input Test if the system does less than specified requirement Test if the system does more than specified requirement

29

White Box Testing

Derived from knowledge of programs structure & implementation Structural testing - analyse code & use knowledge of the structure of a component to derive test data Logical paths are tested by providing test cases that exercise specific sets of conditions and/or loops
30

White Box Testing (Continued)

Thorough white box testing would lead to 100 percent correct programs Exhaustive testing are impractical - too many tests! A limited number of logical paths can be selected and exercised Important data structures can be probed for validity
31

White Box Test Cases

Guarantee that all independent paths have been exercised at least once Exercise all logical decisions on their true and false sides Execute all loops at their boundaries and within their operational bounds Exercise internal data structures to ensure their validity
32

White Box Testing Techniques

Basis path testing
Flow graph notation Cyclomatic complexity Derived test cases Graph metrics

Control structure testing

Condition testing Data Flow testing Loop testing

33

Flow Graph Notation

Edge
1 2,3 2 6 3 6 7
9 11 10
R2

Node

4,5
R1

4 8 5

R3

Region 9

10
R4

11

34

Cyclomatic Complexity
Provide quantitative measure for program logical complexity. Defined number of independent path
Any path that introduce one ser of processing statements or new condition

Eg : Path 1 : 1-11 Path 2 : 1-2-3-4-5-10-1-11 Path 3 : 1-2-3-6-8-9-10-1-11 Path 4 : 1-2-3-6-7-9-10-1-11

35

How Is Cyclomatic Complexity Computed?

1. Number of regions

The flow graph has 4 regions

E : Number of flow graph edges N : Number of flow graph nodes
V(G) = 11 edges 9 nodes + 2 = 4

2. V(G) = E N + 2

3. V(G) = P + 1
P : Number of predicate nodes
V(G) = 3 predicate nodes + 1 = 4

36

Deriving Test Cases

1. Draw flow graph from design/code as foundation.
1 2 3 10 12 13 8 9
37
i=1; 2 total.input = total.valid=0; 1 sum=0; do while value[i] <> -999 and total.input<100 3 4 increment total.input by 1; if value[i] >= minimum AND value[i] <= maximum 6 5 then increment total.valid by 1; sum = sum + value[i] 7 else skip end if 8 increment i by 1 9 End do If total.valid > 0 10 11 then average = sum / total valid; 12 else average = -999; 13 End if

4 11 5

6
7

Deriving Test Cases (cont)

2. Determine cyclomatic complexity
V(G) = 6 regions V(G) = 17 edges 13 nodes + 2 = 6 V(G) = 5 predicates nodes + 1 = 6 Path 1 = 1-2-10-11-13 Path 2 = 1-2-10-12-13 . Path 1 test case :
value(k) = valid input where k < i defined below value(i) = -999 where 2 <= i <= 100 Expected result : Correct average based on k value and proper totals
38

3.

Determine a basis set of linearly independent graph

4.

Prepare test cases

Discussion on White Box Testing

Advantages
Find errors on code level Typically based on a very systematic approach, covering the complete internal module structure

Constraints
Does not find missing or additional functionality Does not really check the interface Difficult for large and complex module

39

Black Box Testing

Derived from program specification Functional testing of a component of a system Examine behaviour through inputs & the corresponding outputs Input is properly accepted, output is correctly produced Disregard internal logical structure
40

Black Box Testing (Continued)

Attempts to find the following errors: Incorrect or missing functions Interface errors Errors in data structures or external database access Performance errors Initialisation and termination errors
41

Black Box Testing Techniques

Graph Based Testing Methods Equivalence Partitioning Boundary Value Analysis Comparison Testing Orthogonal Array Testing

42

Equivalence Partitioning
Divide input domain into classes of data Based on an evaluation of equivalence classes for an input condition Guidelines to define equivalence classes
Range input : One valid and two invalid equivalence Specific value : One valid and two invalid equivalence A member of a set : One valid and one invalid equivalence Boolean : One valid and one invalid equivalence

43

Boundary Value Analysis

Complement equivalence partitioning Test both sides of each boundary Look at output boundaries for test cases Test min, min-1, max, max+1, typical values Example : 1 <= x <=100
Valid : 1, 2, 99, 100 Invalid : 0 and 101

44