CHAPTER 1: FUNDAMENTALS OF TESTING

DEFINTION OF SOFTWARE TESTING: The process consisting of all life cycle activities, both static and dynamic, concerned with Planning, Preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. TESTING PRINCIPLE: Testing is context dependent Exhaustive testing is impossible Early testing Defect clustering Pesticide Paradox Testing shows presence of a defects Absence of errors fallacy

CAUSES OF SOFTWARE DEFECTS Errors:- A human action the produces an incorrect results Defects (bug, faults):- A flaw in a component or system that can cause the component or the system to fail to perform its required function. A defect encountered during the execution, may cause a failure to component or system. Failure: Deviation of the component or system from its expected delivery, service or result.

DO OUR MISTAKES MATTER? Defects in software , systems or documents may results in failure , but not all do cause failures.

Classification: GE Confidential

It is not just defects that give rise to failure. Failures can caused by : Environmental conditions for eg. Radiations burst Human error in interacting with the software, for eg. Wrong input entered or output being misinterpreted. Malicious damage: someone deliberately trying to cause a failure in a system. When we think about what might go wrong we have to consider defects and failures arising from: Errors in specification, design and implementation of the software and system Errors in use of the system Environmental conditions Intentional damage Potential consequences of earlier errors, intentional damage, defects and failures. What is the Cost of defects? The cost of finding and fixing defects rises considerably across the life cycle If an error is made and consequent defect is detected in the requirements at the Specification stage , then it is relatively cheap to find and fix and then specification can be corrected and re-issued. If the defects detected in the design stage then the design can be corrected and re-issued with relatively little expenses. If the defect is introduced in the requirement , specification and it is not detected Until accepatance testing or even once the system has been implemented then it will be much more expensive to fix. Testing and Quality Testing can give confidence in the quality of software if it finds few or no defects Testing helps us to measure the quality of the software in terms of the number of defects found , the tests run and the system covered by the tests. Quality : The degree to which a component , system or process meets specified requirements or user or customer needs and expectations. Validation: Is the right Specification? Verification : Is the system correct to specification?

Classification: GE Confidential

How much testing is enough? ( Test Principle Exhaustive testing impossible)

Instead of exhaustive testing, we use risks and priorities to focus testing efforts. Pressures on a project include time and budget as well as pressure to deliver technical solution that meets customer needs. Customer and project manager will want to spend an amount on testing the produces Return on Investments for them. Return on Investments Preventing failures after releases that are costly. By assessing and managing risk is are of the important activities. How much testing is enough is according to level of risk, technical and business Risks related to product and project Detect Defects: Help us understand the risks associated with putting the software into operational . Fixing the defects improves the quality of the products. Identifying the defects has another benefits to improve the development process and make fewer mistakes in future work.

When can we meet our test objective? (Test principle Early Testing)
Finding the defects Gaining confidence in and providing the information about level of quality. Preventing defects. Benefits of early testing Early test design and review activites finds defects early on when they are cheap to find and fix. Fousing on defects can helps us plan our tests --- (Testing Principle Defect clustering) Main focus of reviews and other static tests is to carry out testing as early as possible finding and fixing defects are more cheaply and preventing defects from appearing at later stages of this project. These activites helps us find out about defects earlier and identify potential clusters.
Classification: GE Confidential

Debugging : The process of finding , analyzing and removing the causes of failures in software.

TEST PLANNING Determine the scope and risks and identify the objectives of testing Determine the test approach (techniques, test items, coverage,testware) . Implement the test policy and the test strategy. Determine the required resources Schedule test analysis and design tasks,test implementation ,execution and evaluation Determine the exit criteria

TEST CONTROL Measure and analyze the results of reviews and testing . Monitor and document progress,test coverage and exit criteria. Provide information on testing. Initiate corrective actions Make decisions. TEST ANALYSIS AND DESIGN Review the test basis. Identify test conditions based on analysis of test items, their specifications. Design the tests Evaluate testability of the requirements and system. Design the test environment set-up and identify any required infrastructures and tools

TEST IMPLEMENTATION AND EXECUTION IMPLEMENTATION: Develop and prioritize our test cases. Create the test suites from the test cases for efficient test execution. EXECUTION: Execute the test suites and individual test cases. Log the outcome of test execution and record the identities and version, test tools and testwares. Compare the actual results with expected results Repeat the activities as a result of action taken for each discrepancy. EVALUATING EXIT CRITERIA AND REPORTING Check test logs against the exit criteria specified in the test planning

Classification: GE Confidential

Assess if more tests are needed or if the exit criteria specified should be changed. Write a test summary report for stakeholders.

TEST CLOSURE ACTIVITIES Check which planned deliverables with actually delivered and ensure all incident reports have been resolved through defects repair or deferral. Finalize and archive testware,such as scripts, test environment and infrastructure. Hand over testware to the maintenance team. Evaluate the testing and analyze the lessons learned for future projects. PSYCHOLOGY OF TESTING. We need to be careful when we are reviewing and when we are testing. Communicate findings on the product in a neutral, fact focused without criticizing the person who created it. Explain that by knowing about this now we can work round it or fix it so the delivered the system is better for the customer. Start with collaboration rather than battles. Remind everyone common goal of better quality system.

Classification: GE Confidential

CHAPTER 2: TESTING THROUGHOUT THE SOFTWARE LIFE CYCLE

In every development life cycle , a part of testing is performed on VERIFICATION Testing and part is focused on VALIDATION Testing. VERIFICATION: To determine whether it meets the requirements. Is the deliverable built according to the specification? VALIDATION: To determine whether it meets the user needs ---Is the deliverable Fit for purpose?.

V-MODEL Water fall model was one of the earliest models to be designed . It has the natural timeline where the tasks are executed in a sequential fashion. Draw backs of this model is difficult to get feedback passed backwards up the waterfalls and there are difficulties if we need to carry out numerous iterations for a particular phase. The V-Model was developed to address the problems experienced using the traditional Waterfall approach. The V-Model provides guidance that testing needs to begin as early as possible in the life cycle. The type V-Model uses four test levels. Component testing Integration testing System testing Acceptance testing

Iterative life cycles

A common feature of iterative approaches is that the delivery is divided into Increments or builds with each increments adding a new functionality.
Classification: GE Confidential

 Intial increment will contain infrastructure required to support the build functionali\ty The increment produced by a iteration may be tested at several level as part of its development.

Testing within a life cyle model

In summary, whichever life cycle model is being used, there are several Characteristics of good testing: For every department activity there is a corresponding testing activity. Each test level has test objectives specific to that level. The analysis and design of tests for a given test level should begin during the corresponding development activity Testers should be involved in reviewing documents as soon as drafts are are available in the development cycle.

Test Levels Component Testing:

Also known as unit, module and program testing, that are separately testable. Component testing may include testing of functionality and specific non-functional Characteristics such as resource-behavior (e.g. memory leaks), performance or Robustess testing as well as structural testing. One approach in component testing, used in Extreme Programming (XP), is to Prepare and automate test cases before coding. This is called a test-first approach or test-driven development.

Integration Testing:
Integration testing tests interfaces between components, interactions to different parts of system such as an operating system, file system and hardware or interfaces between systems. There may be more than one level of integration testing and it may carried out on test objects of varying size. Component integration testing tests the interaction between software components and after component testing System integration testing tests the interaction between the different systems and may be done after system testing. Big-Bang Integration testing one extreme is that all component or system are integrated simultaneously, after

Classification: GE Confidential

which everything is tested as a whole. Big Bang testing has the advantage that everything is finished before integration testing starts. Disadvantage is time consuming and difficult to trace the cause of failures. Different approach of integration Top-down approach Bottom- up approach Functional incremental.

System Testing:
System testing is concerned with the behavior of the whole system/product as defined by the scope of a development project or product. System testing should investigate both Functional and non-functional requirements of the system. System testing requires a controlled test environment with the regard to amongst Others things, control of software versions, testware and the test data.

Acceptance Testing:
The goal of acceptance testing is to establish confidence in the system. Acceptance testing is focuses on validation type of testing, determine whether the system is fit for purpose. Finding defects should not be the main focus in acceptance testing. Acceptance testing may occur at more than just a single level. A Commercial off the Shelf (COTS) software product may be acceptance tested when it is installed or integrated Acceptance testing of the usability of a components may be done during the component testing. Acceptance testing of a new functional enhancement may come before system testing. Different types of Acceptance testing Operational Acceptance test (testing of backup/restore, disaster recovery) Compliance Acceptance test (testing is performed against the regulations, such as legal or Safety regulations). Contract Acceptance test(performed against a contracts acceptance criteria for producing custom-developed software).

Classification: GE Confidential

Two stages of Acceptance tests. Alpha testing: Tests take place at the developers site. Beta testing: Tests take place at the customers site (under real world working conditions).

Test Types:
Testing of function: Functional testing considers the specified behavior and is often as referred as Black- box testing. Function testing can based upon ISO 9216, be done focusing on suitability Interoperability, security, accuracy and compliance Testing functionality can be done from two perspectives: Requirements based testing uses a specification of the functional requirement for the system as the basis for desiging tests. Business- process- based testing uses knowledge of the business processes, which describes the scenarios involved in day to- day business use of the system. Use cases are a very useful basis for test cases from business perspective. Testing of software product characteristics (Non-functional testing) Non-functional testing includes of performance testing, load testing, stress testing Usability testing , maintainability testing, reliability testing and portability testing. The ISO 9216 standard defines Six quality characteristics and the subdivision Reliability: sub-characteristics maturity (robustness), fault-tolerance, Recoverability and compliance Usability: understandability, learnability, operability, attractiveness and compliance. Efficiency: Time behavior, resource utilization and compliance. Maintainability: analyzability, changeability, stability, testability and compliance. Portability: adaptability, installability, co-existence, replaceability and compliance. Testing software structure/architecture (structural testing) Structural testing is often referred as white-box or glass-box because we are interested in what is happening inside the box. Structural testing is most often used as a way of measuring the thoroughness
Classification: GE Confidential

of testing through the coverage of a set of structural elements or coverage items. Testing related to changes: Confirmation testing (re-testing): When a test fails and we determine that the cause of failure is software defect, the defect is reported and we can expect a new version of the software that has had the defect fixed. We will need to execute the test again to confirm that the defect has indeed been fixed. This is known as Confirmation Testing. Regression testing Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software as a result of the changes made. It is performed when the software or its environment is changed. Maintenance Testing: Modification of a software product after delivery to correct defects, to improve performance or other attributes or to adapt the product to the modified environment . Impact analysis and regression testing: Usually maintenance testing will consist two parts: Testing the changes Regression tests to show that the rest of the system has not been affected by the maintenance work. Impact analysis: The assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements.

Triggers for maintenance testing:It is triggered by Modifications, Migration, or retirement of the system. Modifications: Include to planned enhancement changes, corrective and emergency changes and changes of environment(planned O/S or database upgrades, or patches to newly exposed discovered vulnerabilities of O/S).

Classification: GE Confidential

Migration: Operational testing of the new environment, as well as the changed software. Retirement of the system: Testing of data migration or archiving, if long data-retention periods are recovered. Planned Modification: Types of Planned Modification: Perfective modification: - by supplying new functions or enhancing performance. Adaptive modification :- adapting software to environmental changes such as new hardware, new systems software or new legislations. Corrective Modification :- deferrable corrections of defects. On average, planned modification represents over 90% of maintenance work on systems.

CHAPTER 3: STATIC TECHNIQUES

During static testing, software work products are examined manually, or with a Set of tools, but not executed. Types of Defects can detect during static testing: Deviations from standards Missing requirements Design defects Non-maintainable code Inconsistent interface specifications

The use of static testing and various advantages of reviews on software products: Static testing can start early in the life cycle, early feedback on quality Issues can be established. By detecting defects at an early stage, rework of costs are most often relatively low and relatively cheap improvement of software quality Rework effort substantially reduced, development products figure likely to increase. The evaluation by a team has the additional advantage that there exchage information between the participants. Static tests contribute to an increased awareness of quality issues. Phases of a formal review:

Classification: GE Confidential

Planning Kick-off Preparation Review meeting Rework Follow-up. Planning: In this phase the entry check is carried out to ensure that the revie wers time is not wasted on a document that is not ready for review. Within reviews the following focuses can be identified: Focus on higher-level documents, e.g. the design comply to the requirements Focus on standards, e.g. internal consistency, clarity, naming conventions, templates. Focus on related documents at the same level, e.g. interfaces between software functions. Focus on usage, e.g. for testability or maintainability. Kick-off: The goal of this meeting is to get everybody on the same wavelength regarding the document under review and to commit to the time that will spent time on checking the document. The relationships between the documents under review and the other documents(sources) are explained. Role assignments, checking rate , the pages to be checked, process changes are discussed in this meeting. Preparation: The individual participants identify defects, questions and comments, according to their understanding of the document and role . Using checklists in this phase can make reviews more effective and efficient. A critical success factor for a thorough preparation is the number of pages checked per hour. By collecting data and measuring the review process, company-specific criteria for checking rate and document size can be set. Review meeting: This meeting typically consists of following elements: Logging phase: The focus is on logging as many as defects possible within a certain timeframe. Discussion phase: Participants can bring forward their comments and reasoning. Moderator ensure that all discussed items have an outcome by the end of this meeting. Decision phase. : A decision on the document under review has to be made by participants, based on formal exit criteria.

Types of review:
Classification: GE Confidential

 Walkthrough Inspection Technical review Walkthrough: A step- by-step presentation by the author of a document in order to gather information and to establish the common understanding of its contents. A Walkthrough is especially useful for higher-level documents such as requirements, specifications and architectural documents. Key characteristics of walkthrough: The meeting is led by authors, often a separate scribe is present Scenarios and dry runs may be used to validate the content Separate pre-meeting preparations for reviewers is optional Technical review: A Peer group discussions activity forced on achieving Consensus on the technical approach to be taken. There is a little or no focus on defect indentification on basis if referenced documents, intended readership and rules. Key characteristics: It is a documented defect-detection process that involves peers and technical Experts It is often performed as peer review without management participation. Led by trained moderator but possibly also by a technical experts A Separate preparation is carried out during which product is examined and defects are found formal characteristics such as the use of checklists & logging list or issue log optional Inspection: is the most formal review type. The document under inspection Is prepared and checked thoroughly by the reviewers before the meeting, Comparing the work product with its sources and other referenced documents And using rules and checklists. Key Characteristics: Led by a trained moderator It uses defined roles during the process It involves peers to examine the product. Rules and checklists are used during the preparation phase. A separate preparation carried out during which the product is examined and the defects found. The defects found are documented in a logging lists or issue log A formal follow-up is carried out by the moderator applying exit criteria

Classification: GE Confidential

 Optionally, a causal analysis step is introduced to address process improvements issues and learn from the defects found Metrics are gathered and analyzed to optimize the process.

Statics Analysis Tools:

Statics analysis tools are typically used by developers before and sometimes during component and integration testing and by designers during software modeling. Statics analysis is an examination of requirenments, design or code without actual executing the software artifacts being examined. Statics analysis is ideally performed before the types of formal review. Static analysis is unrelated to dynamic properties of the requirements, design and code, such as test coverage. The goal of static analysis is to find defects, whether or not they may cause failures. As with reviews, static analysis finds defects rather than failure. Coding standards: Coding standard consists of a set of programming rules The three main causes for this: The number of rules in a coding standard is large that nobody can remember them all. Some context-sensitive rules that demands review of several files are very hard to check by human being. If people spend tine checking coding standards in reviews, that will distract them from other defects. Code metrics: Structural attributes of code:- comment frequency, depth of nesting , Cyclomatic number and number of lines of code. Complexity metrics: Identify high risk, complex areas. Code Structure: There are several aspects of code structures: Control flow structure Data flow structure Data structure Control flow analysis can also be used indentify unreachable (dead) code. The summary the value of static analysis is especially for: Early detection of defects prior to test execution Early warning about suspicious aspects of the code, design or requirements Identification of defects not easily found in dynamic testing Improved maintainability of code and design engineers work according to documented standards and rules. Prevention of defects, provided that engineers are willing to learn from their
Classification: GE Confidential

errors and continuous improvement is practiced.

CHAPTER 4: TEST DESIGN TECHNIQUES:

Test conditions, Test cases and procedures( or scripts ) are prepared according to the test documentation standard(IEEE 829) Test Conditions: An items or event of the component or system is verified One or more test cases. Test design technique: A procedure used to derive and or select test cases. Traceability: The ability to identify related items in documentation and Software , such as requirements with associated tests. IEEE 829 TEST DESIGN SPECIFICATION TEMPLATE:Test design identifier Features to be tested Approach refinements Features pass/fail criteria Test identification. Test Case Specification: A document specifying a set test cases, inputs, test action, expected results and execution preconditions for a test item. Test Oracle: A source to determine expected results with the actual results of the software under test. IEEE 829 STANDARDS: TEST CASE SPECIFICATION TEMPLATE: Test case specification identifier Test item Input specification Output specification Environmental needs Special procedural requirements Inter-case dependencies.

Classification: GE Confidential

TEST IMPLEMENTATION:- Specifying test procedures or test scripts : Test Procedure specification: A document specifying the sequences of action for the execution of a test. Also known as a test script or manual test script. Test script: Commonly used to refer to a test procedure specification, especially an automated one. IEEE 829 STANDARDS: TEST PROCEDURE SPECIFICATION: Test Procedure Identifier Purpose Special requirements Procedure steps. SPECIFICATION BASED TESTING TECHNIQUE (BLACK BOX) A procedure to derive and / or select a test based on analysis of the Specification, either functional or non-functional, of a system without reference to its internal structure. STRUCTURE BASED TESTING TECHNIQUE (WHITE-BOX) A procedure to derive and/or select a test cases based on internal structure of a component or system. EXPERIENCE BASED TESTING TECHNIQUE : Peoples knowledge, skills and background are prime contributor to the test Conditions and test cases. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUE: Equivalence partionining Boundary Value Analysis Decision table State Transisition Equivalence - Partitioning: A black box testing technique in which test case designed To execute from the representative of equivalence partition. In principle designed to Cover each partition atleast once. Boundary Value Analysis: An Input or output value which is on the edge of the equivalence partitioning or smallest incremental distance on either side of the edge. BS-7925 -2 standard for software component testing.

Classification: GE Confidential

Decision table: are more focused on business logic and business rules and good way To deal with combination of things. This technique referred as cause effect table

State Transition testing: In which test case are designed to execute the valid And invalid state transition, where some aspect of the system can be described in what is called a finite state machine. A finite state machine is often known as state diagram. Four basic parts of the state transition model: The The The The states that the software may occupy; transitions from one state to another; events that cause a transition; actions that result from a transition.

Use case testing: In which the test case are designed to execute user sceneriors Actor- a user of the system. Each use case describes the interactions the actor has with the system in order to achieve the specific task. Error guessing is a technique that should always be used as a complement to Other more formal techniques. Exploratory testing: a test design technique where the tester actively controls the design of the tests as those test are performed and uses information gained while testing to design new and better tests.

Classification: GE Confidential

Classification: GE Confidential

Classification: GE Confidential