System Testing Methodology

QMITST01 / Version 1.0 Effective: June 15, 2005

Purpose

The purpose of system Testing methodology is

• To define how system testing projects are executed and
• To serve as a basis for different phases in a system testing project

Scope
During project planning, this methodology shall be used to define the activities and work
products for each phase.

Only the Testing specific activities are detailed in this methodology. For all other activities like
project management, configuration management, etc. the project shall follow the processes
defined as part of Qualify.

Each system testing project which is taken up may go through all the phases of the Software
Testing Life Cycle which are detailed in this methodology, or may go through only certain
phases of Testing life cycle. For instance, a project may encompass only test execution based
on the customer’s requirements. The phases that will be followed during the course of the
project will have to be detailed in the Software Project Plan (SPP) –Testing of the project. The
applicable phases of execution, the associated work products, verification and validation
criteria for each of the relevant phases shall be at par with this methodology.

The system testing methodology offers the flexibility of entry and exit through any of the
defined phases. The user can use this flexibility by following the Entry Input Task Validation
Output Exit (EITVOX) criteria defined for each phase.

Definitions & Acronyms

Definitions

Keyword Definition
Bug Tracking Bug Tracking process covers the defect lifecycle from identification of
a defect to its resolution / closure.
Functional Testing Process to determine that the features / functionality of the
application / product is as per the requirements
Load testing Testing the application behavior under varying acceptable loads
Performance Testing conducted to evaluate the compliance of a system or
Testing component with specified performance requirements
Regression Testing • Process to ensure that the earlier applications/ products still
work with the new changes.
• Tests ensure that changes do not introduce unintended
behavior or additional errors.
Smoke testing Smoke testing to ensure the build version is ready for undertaking
testing
System Testing Process to determine that the system functions as intended or
documented.
Acronyms

Acronym Explanation
STC System Test Cases
STP System Test Plan

Overview
The system Testing process starts with analyzing/ Understanding of Business/ Functional
requirements.
System testing Objectives are identified specified and prioritized.
(Depending on the client requirements System test may include different tests like Functional
Test, Regression Test and Load-performance Test).
The System Test plan is created.
Test Cases are prepared to validate if requirements are met.
System testing Environment is identified and setup is done to proceed with Test Execution.
Tests are executed to ensure that the application meets the System requirements.
Test Automation follows the manual Test Execution as per Customer Requirement
Defects identified during the system Testing are entered in to Defect tracking process.
Test Result Reports are generated after completion of Testing.

The System Testing project comprises of the following phases:

1. Analyze Requirements
2. Create System Test Plan
3. Create System Test Cases
4. Ensure System Testing Environment
5. Execute System Test Cases
6. Review System Test Results.
7. Prepare Test Report

References
QEdge Testing Methodology
Functional Testing Methodology
Regression Testing Methodology
Load-Performance Testing Methodology

Tailoring Guidelines
It is recommended that this methodology be followed during the course of the project. If any
tailoring is to be made due to the customer’s requirements, then the tailoring shall be
documented along with justification in the Project Tailoring Record (PTR). (Ref… Tailoring
Guidelines)

Requirements Analysis
This phase consists of requirements from BRS/ SRS and Design documents to understand the
application functional areas and prioritizing test objectives

Entry Requirements Specification and Design Specifications are available

 Inputs • Requirements Specification (BRS/ SRS)
• Design Specification
• User Guide
• Installation Guide
Tasks • Study and review the system Requirements document
• Understand application/product architecture in Design
Specification document
• Specify Test Objectives
• Define Strategy/ Approach
• Specify Acceptance Criteria and completion criteria for each test
objective.
• Prioritize the Test Objectives as High and Medium
• High - Most important tests: must be executed
• Medium - Second-level objectives: should be executed after high-
priority tests
• Identify scope of Automation.
• Identify Test Automation Tools as required
Verification / • Review of Test Objectives
Validation • Review of Scope.
Outputs • Identification of critical areas
• Prioritized test objectives
Exit • The test objectives are prioritized
• Approval of Scope

Create System Test Plan

This phase consists of creation of System Test Plan after understanding the requirements. The
plan basically consists of Test Criteria, Scope, Environment, Estimations, Risks, Defect
Tracking, Build Management (Refer… Guidelines for Build and Release Management, Guidelines
for Metrics Definition Standard, Guidelines for Defect Management)

Entry The test objectives are prioritized and the Scope is approved
Inputs • Requirements Specification (BRS/ SRS)
• Design Specification
• User Guide
• Installation Guide
 Tasks • Plan System Test* According to the required tests specified by the
Client.
• Identify all the features to be covered in each kind of test.
• Estimate the time to prepare Test Cases
• Estimate the time to execute the Test Cases.
• Identify Risks.
• Analyze and document effort estimation
• Identify HW/SW requirements.
• Identify the resources requirements.
• Assign responsibilities to the resources.
• Plan to Track and control Effort estimations
• Plan Build Management/ tracking
• Plan Defect Tracking
•
Verification / System test plan review
Validation
Outputs System test plan
Exit Completion of the System Test Plan and sending for approval

* Refer to the Procedure Documents for different types of testing

Create System Test Cases

This phase consist of preparation of Test Cases following different techniques EP, BVA, Cause
Effect, etc. Test Cases are prepared based on the identified Functional scenarios which will
be useful for Test Automation. (Refer… Guidelines for Software Testing)

Entry Approved System Test Plan

Inputs • Design document
• Use cases/ System requirements
• Design Specification
Tasks • Design System Test Cases
• Identify Test Scenarios for Automation
• Identify and Prepare cases/ Scenarios for Automated Testing Scripts
• Create Test Cases from Use Cases
Verification / • Test Cases review
Validation • Automated Test Scripts review
Outputs • System Test Cases
• System Test Scripts
Exit • Completion of designing of System Test Cases
• Completion of Automated Test Scripts
• Send Test Cases and Test Scripts for approval

Ensure System Testing Environment

This phase consist of Environment setup to ensure readiness for execution. (Refer… Guidelines
for Test Data Management)

Entry The test objectives should be prioritized and the scope is approved
 Inputs Prioritized test objectives
Tasks • Ensure the required test environment for System testing is setup as
identified in STP.
<It is recommended that this setup is separate from the
development environment, which is setup during construction
phase>
• Execute the smoke test
<When a build is received, a smoke test is run to ascertain if
the build is stable and it can be considered for further testing.
Smoke testing can be done for testing the stability of any
interim build.>
Verification / Test Environment review
Validation
Outputs System testing environment
Exit • The required test environment for System testing is setup
• The smoke test is executed successfully

Execute System Test Cases

This phase consists of execution of Test Cases and entering the identified defects in to Defect
Tracker.

Entry Approved System Test cases and System Test Scripts

Inputs • System Test Cases
• System Test Scripts
• System Test Plan
Tasks • Execute System Test Cases manually
• Record all System Test runs in the System Test Report.
• Enter “P” (Pass) in the observed behavior column if Actual behavior
matches Expected behavior; otherwise enter “F” (Fail).
• Record observed behavior for every run and repeat the same until
all the defects are closed.
• Record all defects for tracking
• Maintain System Test Reports as a quality record in the project
folder.
• Report and Close Defects using Defect Form/ Defect tracking Tool
• Execute Automated Test Scripts as required.
• Fill in Test Evaluation form
Verification / Test Evaluation using test evaluation form
Validation
Outputs • Defect Log and Defect Summary Report
• Automated Test Reports/ Scripts
Exit • Defect Log and Defect Summary Report is prepared
• Automated Test Reports/ Scripts

Review System Test Results

This phase consists of review of test results.
 Entry • Completion of the execution of the system test cases
• Defect Log is prepared
Inputs Defect log

Task • Review of the system Test results. Reviewer may take some sample
test cases and conduct testing to verify the correctness.
• The reviewer shall verify the following:
• All the relevant test cases are executed
• The choice of test cases covers the main scenarios
• The testing is as per the corresponding test plan
• Testing shall be signed off and all the review comments are
closed.
Validation Test Report evaluation and Review
Output • Test evaluation reports
• Defect Log
• Defect summary report
Exit • Completion of the execution of the System test cases
• Defect Log and Defect summary is prepared

Preparation of Test Report

This phase consists of preparation of Test report. It contains summary information such as
defect data, test evaluation summary.

Entry Defect Log and Defect Summary Report is prepared and Test Reports/
Scripts are Automated
Inputs • Test Evaluation Forms
• Defect Summary Reports
Task • Calculate Test Completion time
• Prepare Metrics
• Analyze Defect data
Validation Test Report evaluation and Review
Output System Test Report
Exit Test Report Completion