Software Testing Lifecycle Practice-

Test Execution and Defect Management

Yujuan Dou

Information Before Class

Document from Joe To answer the questions, it will be helpful for final exam

Microsoft Word Document

Microsoft Word Document

Identifier
HW_TC&TRPT_Group Number _yymmdd

Agenda
Test Execution Defect Management

Test Execution

TEST Execution
This phase involves actual test execution. The testers must execute the documented test cases and capture the results in a test report. The test leader should divide the test cases among the different testers and consolidate the results of testing. In case the application under test is expected to have multiple builds then a mechanism for identifying a correct build for testing should be decided.

Levels of Test Execution

Unit Testing Integration Testing System Testing Acceptance Testing

Test Log
Each Test Log should be made up of a series of entries that present an audit trail for various aspects of the test execution including, but not limited to, the following:
the date and time stamp of when the event occurred a description (usually brief) of the event logged some indication of the observed status additional contextual information where relevant additional details relating to any anomalous or erroneous condition detected

Test Execution
Test Environment Test Data Test Tools

Test Execution
A Test Execution Run Plan could also be prepared depending upon the following criteria:
Impacted Functionality Unstable functionality Customer Priority

TC priority

Execution Order

Test Execution Run Plan

Complexity

Execution rate

Functionality Dependency

TC Interdependency

Execution Order

RUP-Test Execution
Setup test environment to known state Set execution tool options Schedule Test Suite execution Execute Test Suite Evaluate execution of Test Suite Recover from halted tests Inspect the Test Logs for completeness and accuracy Restore test environment to known state Maintain traceability relationships Evaluate and verify your results

DEFECT MANAGEMENT

What are Defects

is a failure to comply with a requirement

Defect Management
This process goes in parallel with the test execution phase. Defect reporting should be clear and concise and should make it easy for the developer to reproduce the failure scenario.

Defect Entry & Severity Classification: Typically, the bugs are logged into a defect database with the following information:
Synopsis - One line description of the bug. Bug environments - Info about OS, Application server, browser, Database, builds used with all the version information where the bug occurs are specified. Other environment on which the bug does not occur are also mentioned Detailed description with exact error message or the observed behavior

Defect Information
Exact steps to reproduce the bug Expected behavior Regression information if the bug was not occurring on any of the recent builds. This information is often used to determine whether the bug is to be fixed or not. Severity of the bug Priority of the bug

Defect Severity
Severity 0: Customers production use of the solution is stopped or so severely impacted that the customer cannot reasonably continue work. There is no workaround or the provided workaround is not acceptable to the customer because of its business impact. Severity 1: Implementation is operational but its functionality is seriously affected. If a workaround has been provided, the loss of functionality can only be sustained for a short time. Or there is a problem preventing roll-out / go-live / implementation.

Defect Severity
Severity 2: Implementation is operational but a problem has been identified and a specific portion of the system either provides incorrect results or is not operating as documented. A workaround is available and is acceptable to the customer. Severity 3: Customer has a question on the product or a request for a product modification or enhancement.

Severity in Bugzilla
This field describes the impact of a bug. Blocker Blocks development and/or testing work Critical crashes, loss of data, severe memory leak Major major loss of function Minor minor loss of function, or other problem where easy workaround is present Trivial cosmetic problem like misspelled words or misaligned text Enhancement Request for enhancement

Priority
This field describes the importance and order in which a bug should be fixed. This field is utilized by the programmers/engineers to prioritize their work to be done. The available priorities range from P1 (most important) to P5 (least important.)

Bug Tracking Tool

A bug tracking tool must be used for tracking the bugs. The tool must at minimum have the following features:
Bug logging and automatic updating into the defect database Automatic generation of bug id which is used for all further tracking of the bug Connecting to the appropriate release based on the product and the release version id

Bug Tracking Tool

Promotion and demotion of bugs Notifications to the respective groups or members whenever there are promotions or demotions. Querying the database for bugs based on a criterion. E.g. On Originators Name or Assignees name, based on a build in which fix is available, based on states, list of open bugs, test bugs and closed bugs for a given release.

Decision regarding Bug Fixing

The decision as to whether the bug needs to be fixed for the current release or whether it can be postponed to a future release is taken by the Bug Assessment team which comprises of the SQA team, development team and concerned third party groups. The meeting is convened on a daily basis during the testing cycle. The list of currently open bugs forms the agenda for the meeting. The main purpose of the meeting is as follows:

Review of all the currently open product defects and determine the appropriate vehicle for delivery. Only the bugs that are needed to be fixed for the current release will be connected to this release for fixing. The typical criteria considered while deciding this are: Severity of the bug Whether it is a regression bug or not Criticality of the bug from the point of view of the endCustomer Verification of duplicate bugs, and to connect them accordingly through the bug tracking tool.

Defect Management
Start Is Bug Valid? No Reject the bug and update thru Bug tracker tool No

Yes Is this bug to be fixed for the current release?

Yes Connect this bug to the current release

Connect this bug to the appropriate release

Assign the bug to developer and send mail

Yes

Are there any more open bugs?

No End

Defect Management
Fixing of bugs and subsequent testing
The developer who is assigned the bug, fixes it, and adds the fix description through the bug tracking tool and also the build in which the fix would be available. The bug is then promoted to the Test state, and assigned to the QE manager for testing the fix. A new build containing a list of bug fixes is then released to SQA. The SQA team then verifies the fix by testing it.

Defect Management
If the bug still exists, then it is demoted to Open/Re-open state If the bug is fixed, then it is promoted to Closed state SQA also performs the regression testing to ensure that the fixes have no side-effects.

Create bug report step by step

Step 1:Login bugzilla with your account

Step2: Click the link New at the bottom of the page

Defect Tracking Process

Defect Tracking Process

APPENDIX A: State Definition

ParentState Submitted Assigned Opened Pending Build Validation Closed Deferred Duplicate Definition Openingstateforadefectwhenthedefectparentiscreated. Defecthasbeentriagedandassignedtoadevelopmentteamlead. Defecthasbeenassignedtoanownerforresolution. DefecthasbeenfixedandisawaitingaBaseline&Buildinthe appropriateUCMProjectIntegrationstream. Defecthasbeenresolved,deliveredandbuilt andisavailabletobe deployedtoatestenvironmentfortestingbyappropriatetest resource. DefectiscompletedandValidated asClosed. Defectresolutionispostponedandwillbeacteduponlater. DefectalreadyexistsinClearQuest.

APPENDIX B: Origin Category

OriginCategory Configuration Development Environment OutofScope ProductionOnly Requirements Requirements/ Interface Specifications TechnicalChange TestIssue ToBeDetermined VendorSoftware Itemnotfunctioningasdefined. Issueoccurredduetothetestenvironment. Itemcouldnotreasonablybeconsideredpartofspecificationsortest scope. Itemfoundinproductionhowevercouldnotberecreatedinthetest environment. Notincludedorerrorinbusinessrequirements Analysisdidnotincorporateup/downstreamimpacts. Notincludedorerrorinfunctionalspecifications/usecases. Changeimprovessystemprocessinghowevercannotbedetectedasa problemthroughtesting. Notincludedintestscopeorplanordeterminedtobenotabug (NAB). Defectunderevaluation.Interimcategory. Vendor/3rd Party software Definition Tablechangeenablessystemtoperformtospecification.

Guideline for Lab

Guideline for Lab Operation.doc