Building the Test Environment

ASHISH VATS

07-Apr-2011

Agenda
 Test Tools  Tool Development & Acquisition  Tools Usage  Testers Competency  Questions from CSTE  Discussion

Page 2

Test Tools
Testing:
is the process of evaluating system or system component by manual or automated means to verify that it satisfies specified requirement or identity differences between expected and actual results.

Tools:
is an entity that interfaces between two or more domains; that facilitates more effective action of one domain upon the other.

Testing Tool:
It is difficult to perform testing economically without the aid of automated tools. Tools are selected to support the test methodology and thus their use should be mandatory and not optional. A tool is a vehicle for performing a test process. A tool is a resource to the tester, but by itself is insufficient to conduct testing

Page 3

Tool Development & Acquisition

 Area of Difficulties for new Tool
1. 2. 3. 4. Organizational Obstacles Problem arising from the tools Obstacles in the computer environment Others
Does not meet the recognized needs of the organization Is difficult to use Requires excessive computer resources Lacks adequate documentation

Page 4

Tool Development & Acquisition (Cont.)

 Role of Test Management
Identify tool objective Approve the Acquisition Plan Define selection criteria Make final selection of Tool/Source

 Roles of Test Manager

Identifying candidate tools Applying the selection criteria Preparing a ranked list of tools or sources Conducting any detailed evaluations or conference room pilots

Page 5

Tool Development & Acquisition (Cont.)

 Distribution of responsibilities reduces the chances of selecting a tool that:
Does not meet the recognized needs of the organization Is difficult to use Requires excessive computer resources Lacks adequate documentation

Page 6

Tool Development & Acquisition (Cont.)

Sequence of Events to Select Testing Tools:
1. 2. Employ a methodical process of identifying candidate tools and selecting among these based on established criteria. Determine the assignment and training of a Tool Manager who can make minor modifications to both the computer environment and the tool.

Recommended Event Sequence

1. 2. Informal procurement (by purchase order) Formal procurement (by a request for proposal-RFP)

Page 7

Tool Development & Acquisition (Cont.)

Steps:
Event 1: Goals Event 2: Tool Objective Event 3: Procure Tool Event 4: Evaluation Plan Event 5: Implementation Plan Event 6: Training Plan Event 7: Tool Received Event 8: Acceptance Test Event 9: Orientation Event 10: Modifications Event 11: Training Event 12: Use in the Operating Environment Event 13: Evaluation Report Event 14: Determine if the Goals are Met

Page 8

Tool Development & Acquisition (Cont.)

Event 1: Goals
Must be in a format that permits later determination that they have been met or not Identify responsibilities(the role that project development may have and coordination with other activities) It may include a not-to-exceed budget and a desired completion date Typical goal statements are:
   Reduce the average test time by one-fifth. Achieve complete interchange ability of test data sets. Adhere to an established standard for documentation format.

Page 9

Tool Development & Acquisition (Cont.)

Event 2: Tool Objectives Acquisition Activities for Informal Procurement:
Event 2: A1 Acquisition Plan:
1. Budgets and schedules for subsequent steps in the tool introduction. 2. Justification of resource requirements in light of expected benefits. 3. Contributions to the introduction expected from other organizations (e.g. the tool itself, modification patches, or training materials). 4. Assignment of responsibility for subsequent events within the IT organization, particularly the identification of the Test Manager. 5. Minimum tool documentation requirements.

Event 2: A2 Selection Criteria:

1. 2. 3. 4. 5. 6. 7. Accomplishment of specified tool objectives Ease of use Ease of installation Minimum processing time Compatibility with other tools Low purchase or lease cost Documentation, Training, and Support availability

Page 10

Tool Development & Acquisition (Cont.)

Event 2: Tool Objectives Acquisition Activities for Informal Procurement:
Event 2: A3 Identify Candidate Tools
1. The first list contains all tools meeting the functional requirements without considering the constraints 2. The second list contains tools that meet both the functional requirements and the constraints.

Event 2: A4 User Review of Candidates

1. Will this tool handle the present file format? 2. Are tool commands consistent with those of the editor? 3. How much training will be required?

Event 2: A5 Score Candidates

1. 2. 3. 4. 5. Information obtained from a vendor's literature. Demonstration of the tool. The user's review. Observation in a working environment. Comments of prior users.

Event 2: A6 Select Tool

Page 11

Tool Development & Acquisition (Cont.)

Event 2: Tool Objectives Acquisition Activities for Formal Procurement:
Event 2: B1 Acquisition Plan:
1. A1 Acquisition Plan + . 2. The constraints on the procurement process. 3. The detailed responsibilities for all procurement documents (statement of work, technical and administrative provisions in the request for proposal (RFP), etc.).

Event 2: B2 Technical Requirements Document:

1. Informal description of the tool requirements and the constraints under which the tool has to operate 2. Utilize much of the material from the acquisition plan but should add enough detail to support a meaningful review by the tool user.

Event 2: B3 User Review of Requirements Event 2: B4 RFP Generation

1. Specification 2. Statement of Work 3. Proposal Evaluation Criteria and Format Requirements

Page 12

Tool Development & Acquisition (Cont.)

Event 2: Tool Objectives Acquisition Activities for Formal Procurement:
Event 2: B5 Solicitation of Proposals
1. Carried out by Administrative and purchasing personnel 2. Names of Potential Bidders are made known to procurement officer 3. When responses are received, they are screened for compliance with major legal provisions of the RFP

Event 2: B6 Technical Evaluation should be Consistent

1. Evaluate each proposals received in response to the RFP. 2. Those deemed to be in the competitive range" are assigned point scores. 3. These point scores are used together with cost and schedule factors that are being separately evaluated by administrative personnel.

Event 2: B7 Source Selection

1. Source is selected on the basis of combined cost, schedule, and technical factors

Page 13

Tool Development & Acquisition (Cont.)

Event 3: Procure Tool
Determine whether the cost of the selected tool is within the approved budget Considers the adequacy of licensing and other contractual provisions and compliance with the "fine print" associated with all the organizations procurements. Identifies the vendor's responsibility for furnishing the source program, meeting specific test and performance requirements, and tool maintenance.

Event 4: Evaluation Plan

Based on the Event 1: Goals & Event 2: Tool Objectives Milestones for installation Dates Performance levels for the initial operational capability and for subsequent enhancements Identify the reports and supporting data that address expected improvements in throughput, response time, or turnaround time Assign responsibility for tests, reports, and other actions A topical outline of the evaluation report
Page 14

Tool Development & Acquisition (Cont.)

Event 5: Implementation Plan
Describes the responsibilities and tasks for the implementation of the tool, and the training that will be required

Event 6: Training Plan

First consider the training inherently provided with the tool Standard training aids supplied by the vendor for Internet and inhouse training such as audio or videocassettes and lectures Number of personnel to receive formal training and adequacy of inhouse facilities should also be specified in the plan

Event 7: Tool Received

From Procuring Organization to the Tool Manager or System Programmer

Page 15

Tool Development & Acquisition (Cont.)

Event 8: Acceptance Test
Performed by Tool Manager and Test Staff Performed on as received state Formal Acceptance Report is issued

Event 9: Orientation
Orientation meeting is organized by Test Management Communicate as directly as possible the objectives of the tool use, such as increased throughput or improved legibility of listings. Highlights of the evaluation plan should also be presented, and any changes in duties associated with the introduction of the tool should be described

Page 16

Tool Development & Acquisition (Cont.)

Event 10: Modification
Performed by Systems programmer and Tool Manager Modifications of the following is performed
The Tool Documentation Operating system

Event 11: Training

Joint responsibility of Tool Manager and tool user Motivate user to use the tool Opportunity to talk about the problems of the tool

Event 12: Use in the Operating Environment

First use must involve most qualified test personnel First use must not be on Tight Schedule Project

Page 17

Tool Development & Acquisition (Cont.)

Event 13: Evaluation Report
Tool manger prepare Evaluation Report based on Evaluation Plan It includes:
User comments and observations of the systems programmer. Whether the general goals and tool objectives were met. Observations on the installation and use of the tool. Cooperation received from the vendor in installation or training. Any other "lessons learned. Tool and host computer modifications. A section of comments useful to future users of the tool.

Event 14: Determine if the goals are MET

Funding management receives the evaluation report and determines whether the goals established in Event 4: Evaluation Plan have been met. Attainment of technical objectives. Adherence to budget and other resource constraints. Timeliness of the effort. Cooperation from other departments. Recommendations for future tool acquisitions.
Page 18

Tools Usage
Most commonly used tools can be grouped into these eight areas:
1. 2. 3. 4. 5. 6. 7. 8. Automated Regression Testing Tools- QTP, WinRunner, SilkTest, vTest, Selenium Defect Management Tools- Bugtrack, Bugzilla, Defect Tracker, Defect Manager, Siebel Performance/Load Testing Tools- WebLOAD, FunkLoad, Pylot, Load Runner Manual Tools- Check List, Decision Tables, Test Scripts Traceability Tools- Requirement Traceability Code Coverage- BullseyeCoverage, IBM Rational Pure Coverage, Clover, Jtest, Ncover, DevPartner Test Case Management Tools- Quality Centre, TFS(Test Manager) Common tools that are applicable to testing- Word Processing, Spread Sheets, Excel,
Computer Graphics

Page 19

Tools Usage
Guidelines for effective and efficient usage of testing tool
Guideline 1: Testers should not be permitted to use tools for which they have not received formal training.

Guideline 2: The use of test tools should be incorporated into test processes so that the use of tools is mandatory, not optional.

Guideline 3: Testers should have access to an individual in their organization, or the organization that developed the tool, to answer questions or provide guidance on using the tool.

Page 20

Testers Competency
  Direct responsibility of the individual and the organization Individual has the primary responsibility to ensure that his/her competencies are adequate and current. Test competency is based on two criteria.
1. Skill sets
for example, skills in writing test plans and using specific test tools CSTE

2.

Performance Skills
for example, how those skills are applied to real-world test situations Certified Manager of Software Testing Certified Innovator of Software Testing

Page 21

Testers Competency
Measuring the Competency of Software Testers

Page 22

Mock Questions
    Describe automated capture/playback tools and list the benefits of using them. Briefly explain the difference between automated & manual tools. Give 2 examples each. What 3 tools would you purchase for your company in testing and justify why would you want them? You are to introduce automated testing tools. List any four automated test tools with vendor name and why did you choose them List five skills a competent tester should have. 1) Test process knowledge; 2) Excellent written and oral communication skills; 3) Analytical ability; 4) Knowledge of test tools; and 5) Understanding of defect management.  Describe the use of a traceability tool, Code Coverage Tool  Tester should know A) Test planning b) Automation tool c) Defect tracking tool d) Programming language

Ans. Defect Tracking tool  What is one of the most powerful quality control tools? Ans. CHECKLIST
Page 23

Discussion

Page 24