Software testing is a process of verifying and validating that a software application or program meets the business and technical

requirements that guided its design and development, and works as expected. Software testing also identifies important defects, flaws, or errors in the application code that must be fixed. In simple words, software testing is an activity to check whether the actual results match the expected results and to ensure that the software system is defect free. Objective: Software testing helps to make sure that it meets all the requirement it was supposed to meet. It will bring out all the errors, if any, while using the software. Software testing helps to understand that the software that is being tested is a complete success Software testing helps to give a quality certification that the software can be used by the client immediately. It ensures quality of the product Testing can done in two ways:
1. 2.

Positive software testing Negative software testing

Positive software testing is the testing by giving the expected data to know whether it works well with expected data, where as negative software testing is testing by giving what is unexpected to know how the software reacts to such a data. Negative software testing is also known as illegal testing since the testing is using a abnormal data. Positive software testing is the usual testing done to check the functionality of the software. Both these testing helps us in finding errors. Negative software testing is always giving us a positive view about software testing itself. It helps us to know the software behaves as it is required to behave. The aim of a software tester should be to make the software fail. He should never think like a software maker
1

Quality Assurance (QA) is the process that focuses upon each and every role and their responsibilities in the development process and checks whether they are being accomplished as per the guidelines provided by the quality standards or not. It concentrates on process of the product and is done throughout life cycle. It is defect prevention method. Quality Control (QC) is the process which usually includes inspections and audit operations to segregate bad from good. It concentrates on specific products and is done after the product is built. It is defect-detection and correction method

TESTING TYPES: Unit Testing: In computer programming, unit testing is a procedure used to validate that individual units of source code are working properly. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a method; which may belong to a base/super class, abstract class or derived/child class. Unit testing is typically done by developers and not by Software testers or endusers.

Benefits: The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits.
2

Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e. regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified and fixed. Readily-available unit tests make it easy for the programmer to check whether a piece of code is still working properly. Good unit test design produces test cases that cover all paths through the unit with attention paid to loop conditions. In continuous unit testing environments, through the inherent practice of sustained maintenance, unit tests will continue to accurately reflect the intended use of the executable and code in the face of any change. Depending upon established development practices and unit test coverage, up-to-the-second accuracy can be maintained. Limitations: Testing, in general, cannot be expected to catch every error in the program. The same is true for unit testing. By definition, it only tests the functionality of the units themselves. Therefore, it may not catch integration errors, performance problems, or other system-wide issues. Unit testing is more effective if it is used in conjunction with other software testing activities. Like all forms of software testing, unit tests can only show the presence of errors; it cannot show the absence of errors. Six Rules of Unit Testing 1. 2. 3. 4. Write the test first Never write a test that succeeds the first time Start with the null case, or something that doesn't work Don't be afraid of doing something trivial to make the test work 5. Loose coupling and testability go hand in hand 6. Use mock objects
3

Static Testing: The Verification activities fall into the category of Static Testing. During static testing, you have a checklist to check whether the work you are doing is going as per the set standards of the organization. These standards can be for Coding, Integrating and Deployment. Review's, Inspection's and Walkthrough's are static testing methodologies. Static testing is a form of software testing where the software isn't actually used. This is in contrast to Dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code or and manually reading of the code or document to find errors. This type of testing can be used by the developer who wrote the code, in isolation. Code reviews, inspections and walkthroughs are also used. From the black box testing point of view, static testing involves review of requirements or specifications. This is done with an eye toward completeness or appropriateness for the task at hand. This is the verification portion of Verification and Validation. Bugs discovered at this stage of development are less expensive to fix than later in the development cycle. Dynamic Testing: Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies. Dynamic testing or (dynamic analysis) is a term used in software engineering to describe the testing of the dynamic behavior of code. That is, dynamic analysis refers to the
4

examination of the physical response from the system to variables that are not constant and change with time. In dynamic testing the software must actually be compiled and run; this is in contrast to static testing. Dynamic testing is the validation portion of Verification and Validation. Some of dynamic testing methodologies include: 1. 2. 3. 4. Unit Testing Integration Testing System Testing Acceptance Testing

Black Box Testing: Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or nonfunctional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure. This method of test design is applicable to all levels of software testing: unit, integration, functional testing, system and acceptance. The higher the level, and hence the bigger and more complex the box, the more one is forced to use black box testing to simplify. While this method can uncover unimplemented parts of the specification, one cannot be sure that all existent paths are tested. Testing strategies/techniques

black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function data outside of the specified input range should be tested to check the robustness of the program

boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output the number zero should be tested when numerical data is to be input stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems crash testing should be performed to see what it takes to bring the system down test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing. finite state machine models can be used as a guide to design functional tests According to Beizer the following is a general order by which tests should be designed:

1. Clean tests against requirements. 2. 3. 4. 5. 6. Additional structural tests for branch coverage, as needed. Additional tests for data-flow coverage as needed. Domain tests not covered by the above. Special techniques as appropriate--syntax, loop, state, etc. Any dirty tests not covered by the above.

Black Box Testing Strategy Black Box Testing is not a type of testing; it instead is a testing strategy, which does not need any knowledge of internal design or code etc. As the name "black box" suggests, no knowledge of internal logic or code structure is required. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software
6

application. Black box testing is sometimes also called as "Opaque Testing", "Functional/Behavioral Testing" and "Closed Box Testing". The base of the Black box testing strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the system. Now a days, it is becoming common to route the Testing work to a third party as the developer of the system knows too much of the internal logic and coding of the system, which makes it unfit to test the application by the developer. In order to implement Black Box Testing Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action. Various testing types that fall under the Black Box Testing strategy are: functional testing, stress testing, recovery testing, volume testing, User Acceptance Testing (also known as UAT), system testing, Sanity or Smoke testing, load testing, Usability testing, Exploratory testing, ad-hoc testing, alpha testing,beta testing etc. These testing types are again divided in two groups: a) Testing in which user plays a role of tester and b) User is not required. Testing method where user is not required: Functional Testing: In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected.

Stress Testing: The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand. Load Testing: The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the website/application fails or at what point its performance degrades. Ad-hoc Testing: This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing. Exploratory Testing: This testing is similar to the ad-hoc testing and is done in order to learn/explore the application. Usability Testing: This testing is also called as Testing for User-Friendliness. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user. Smoke Testing:

This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level. Recovery Testing: Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.

Volume Testing: Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system. Testing where user plays a role/user is required: User Acceptance Testing: In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to. Alpha Testing: In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers. Beta Testing:
9

In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers. Advantages of Black Box Testing

more effective on larger units of code than glass box testing tester needs no knowledge of implementation, including specific programming languages tester and programmer are independent of each other tests are done from a user's point of view will help to expose any ambiguities or inconsistencies in the specifications test cases can be designed as soon as the specifications are complete

Disadvantages of Black Box Testing

only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever without clear and concise specifications, test cases are hard to design there may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried may leave many program paths untested cannot be directed toward specific segments of code which may be very complex (and therefore more error prone) most testing related research has been directed toward glass box testing

White Box testing:

10

White box testing is a security testing method that can be used to validate whether code implementation follows intended design, to validate implemented security functionality, and to uncover exploitable vulnerabilities. The general outline of the white box testing process is as follows: 1. Perform risk analysis to guide the whole testing process. 2. Develop a test strategy that defines what testing activities are needed to accomplish testing goals. 3. Develop a detailed test plan that organizes the subsequent testing process. 4. Prepare the test environment for test execution. 5. Execute test cases and communicate results. 6. Prepare a report. In addition to the general activities described above, the process diagram introduces review cycles, reporting mechanisms, deliverables, and responsibilities. Some of the artifacts relevant to white box testing include source code, a risk analysis report, security specification/requirements documentation, design documentation, and quality assurance related documentation.

Source code is the most important artifact needed to perform white box testing. Without access to the code, white box testing cannot be performed, since it is based on testing software knowing how the system is implemented. Architectural and design risk analysis should be the guiding force behind all white box testing related activities, including test planning, test case creation, test data selection, test technique selection, and test exit criteria selection. If a risk analysis was not completed for the system, this should be the first activity performed as part of white box testing. The following section discusses risk analysis.
11

Design documentation is essential to improve program understanding and to develop effective test cases that validate design decisions and assumptions. Security specifications or requirements are a must, to understand and validate the security functionality of the software under test. Security testers should have access to quality assurance documentation to understand the quality of the software with respect to its intended functionality. Quality assurance documentation should include a test strategy, test plans, and defect reports. Load and performance tests are important in understanding the constraints placed on the system and the behavior of the system under stress. Any artifact relevant to program understanding should be available to white box testers.

Risk Analysis Security is always relative to the information and services being protected, the skills and resources of adversaries, and the costs of potential assurance remedies; security is an exercise in risk management. The object of risk analysis is to determine specific vulnerabilities and threats that exist for the software and assess their impact. White box testing should use a risk-based approach, grounded in both the systems implementation and the attackers mindset. White box testing should be based on architecture and designlevel risk analysis. This content area will discuss how to use the results of risk analysis for white box testing, while the Architectural Risk Analysis content area discusses risk analysis in detail. Risk analysis should be the guiding force behind all white box testing related activities. The following paragraphs briefly
12

introduce how the risk analysis results are used in white box testing. The subsequent sections discuss the activities in detail. The risk analysis report, in combination with a functional decomposition of the application into major components, processes, data stores, and data communication flows, mapped against the environments across which the software will be deployed, allows for a desktop review of threats and potential vulnerabilities. The risk analysis report should help identify

the threats present in each tier (or components) the kind of vulnerabilities that might exist in each component the business impact (consequence and cost of failure of software) of risks< the probability (likelihood) of the risks being realized existing and recommended countermeasures to mitigate identified risks

Use the above information from the risk analysis report to

develop a test strategy: Exhaustive testing is seldom costeffective and often not possible in finite time. Planned testing is therefore selective, and this selection should be based on risks to the system. The priority (or ranking) of risks from the risk analysis should be the guiding rule for the focus of testing, simply because highly vulnerable areas should be tested thoroughly. The test strategy captures all the decisions, priorities, activities, and focus of testing based on the consequence of failure of software. The following section discusses test strategy in detail. For detailed research on risk-based test planning. develop test cases: While a test strategy targets the overall test activities based on risks to the system, a test case can target specific concerns or risks based on the threats, vulnerabilities, and assumptions uncovered during the analysis. For example, tests can be developed to validate

13

controls (or safeguards) put in place to mitigate a certain risk. determine test coverage: The higher the consequence of failure of certain areas (or components), the higher the test coverage should be in those areas. Risk-based testing allows for justifying the rigor of testing in a particular area. For example, a specific component or functionality may have high exposure to untrusted inputs, hence warranting extra testing attention.

Test Strategy The first step in planning white box testing is to develop a test strategy based on risk analysis. The purpose of a test strategy is to clarify the major activities involved, key decisions made, and challenges faced in the testing effort. This includes identifying testing scope, testing techniques, coverage metrics, test environment, and test staff skill requirements. The test strategy must account for the fact that time and budget constraints prohibit testing every component of a software system and should balance test effectiveness with test efficiency based on risks to the system. The level of effectiveness necessary depends on the use of software and its consequence of failure. The higher the cost of failure for software, the more sophisticated and rigorous a testing approach must be to ensure effectiveness. Risk analysis provides the right context and information to derive a test strategy. Test strategy is essentially a management activity. A test manager (or similar role) is responsible for developing and managing a test strategy. The members of the development and testing team help the test manager develop the test strategy. Test Plan The test plan should manifest the test strategy. The main purpose of having a test plan is to organize the subsequent
14

testing process. It includes test areas covered, test technique implementation, test case and data selection, test results validation, test cycles, and entry and exit criteria based on coverage metrics. In general, the test plan should incorporate both a high-level outline of which areas are to be tested and what methodologies are to be used and a general description of test cases, including prerequisites, setup, execution, and a description of what to look for in the test results. The high-level outline is useful for administration, planning, and reporting, while the more detailed descriptions are meant to make the test process go smoothly. While not all testers like using test plans, they provide a number of benefits:

Test plans provide a written record of what is to be done. Test plans allow project stakeholders to sign off on the intended testing effort. This helps ensure that the stakeholders agree with and will continue to support the test effort. Test plans provide a way to measure progress. This allows testers to determine whether they are on schedule, and also provides a concise way to report progress to the stakeholders. Due to time and budget constraints, it is often impossible to test all components of a software system. A test plan allows the analyst to succinctly record what the testing priorities are. Test plans provide excellent documentation for testing subsequent releasesthey can be used to develop regression test suites and/or provide guidance to develop new tests.

A test manager (or similar role) is responsible for developing and managing a test plan. The development managers are also part of test plan development, since the schedules in the test plan are closely tied to that of the development schedules
15

Test Automation Test automation provides automated support for the process of managing and executing tests, especially for repeating past tests. All the tests developed for the system should be collected into a test suite. Whenever the system changes, the suite of tests that correspond to the changes or those that represent a set of regression tests can be run again to see if the software behaves as expected. Test drivers or suite drivers support executing test suites. Test drivers basically help in setup, execution, assertion, and teardown for each of the tests. In addition to driving test execution, test automation requires some automated mechanisms to generate test inputs and validate test results. The nature of test data generation and test results validation largely depends on the software under test and on the testing intentions of particular tests. In addition to test automation development, stubs or scaffolding development is also required. Test scaffolding provides some of the infrastructure needed in order to test efficiently. White box testing mostly requires some software development to support executing particular tests. This software establishes an environment around the test, including states and values for data structures, runtime error injection, and acts as stubs for some external components. Much of what scaffolding is for depends on the software under test. However, as a best practice, it is preferable to separate the test data inputs from the code that delivers them, typically by putting inputs in one or more separate data files. This simplifies test maintenance and allows for reuse of test code. The members of the testing team are responsible for test automation and supporting software development. Typically, a member of the test team is dedicated to the development effort. Test Environment
16

Testing requires the existence of a test environment. Establishing and managing a proper test environment is critical to the efficiency and effectiveness of testing. For simple application programs, the test environment generally consists of a single computer, but for enterprise-level software systems, the test environment is much more complex, and the software is usually closely coupled to the environment. For security testing, it is often necessary for the tester to have more control over the environment than in many other testing activities. This is because the tester must be able to examine and manipulate software/environment interactions at a greater level of detail, in search of weaknesses that could be exploited by an attacker. The tester must also be able to control these interactions. The test environment should be isolated, especially if, for example, a test technique produces potentially destructive results during test that might invalidate the results of any concurrent test or other activity. Testing malware (malicious software) can also be dangerous without strict isolation. The test manager is responsible for coordinating test environment preparation. Depending on the type of environment required, the members of the development, testing, and build management teams are involved in test environment preparation. Test Execution Test execution involves running test cases developed for the system and reporting test results. The first step in test execution is generally to validate the infrastructure needed for running tests in the first place. This infrastructure primarily encompasses the test environment and test automation, including stubs that might be needed to run individual components, synthetic data used for testing or populating databases that the software needs to run, and other applications that interact with the software. The issues being sought are those that will prevent the software under test from being executed or else cause it to fail for reasons not related to faults in the software itself.
17

The members of the test team are responsible for test execution and reporting. Testing Techniques Data-Flow Analysis: Data-flow analysis can be used to increase program understanding and to develop test cases based on data flow within the program. The data-flow testing technique is based on investigating the ways values are associated with variables and the ways that these associations affect the execution of the program. Data-flow analysis focuses on occurrences of variables, following paths from the definition (or initialization) of a variable to its uses. The variable values may be used for computing values for defining other variables or used as predicate variables to decide whether a predicate is true for traversing a specific execution path. A data-flow analysis for an entire program involving all variables and traversing all usage paths requires immense computational resources; however, this technique can be applied for select variables. The simplest approach is to validate the usage of select sets of variables by executing a path that starts with definition and ends at uses of the definition. The path and the usage of the data can help in identifying suspicious code blocks and in developing test cases to validate the runtime behavior of the software. For example, for a chosen data definition-to-use path, with wellcrafted test data, testing can uncover time-of-check-to-time-ofuse (TOCTTOU) flaws. The Security Testing section in [Howard 02] explains the data mutation technique, which deals with perturbing environment data. The same technique can be applied to internal data as well, with the help of data-flow analysis.

Code-Based Fault Injection:

18

The fault injection technique perturbs program states by injecting software source code to force changes into the state of the program as it executes. Instrumentation is the process of non-intrusively inserting code into the software that is being analyzed and then compiling and executing the modified (or instrumented) software. Assertions are added to the code to raise a flag when a violation condition is encountered. This form of testing measures how software behaves when it is forced into anomalous circumstances. Basically this technique forces nonnormative behavior of the software, and the resulting understanding can help determine whether a program has vulnerabilities that can lead to security violations. This technique can be used to force error conditions to exercise the error handling code, change execution paths, input unexpected (or abnormal) data, change return values, etc. In [Thompson 02], runtime fault injection is explained and advocated over code-based fault injection methods. One of the drawbacks of code based methods listed in the book is the lack of access to source code. However, in this content area, the assumptions are that source code is available and that the testers have the knowledge and expertise to understand the code for security implications. Abuse Cases: Abuse cases help security testers view the software under test in the same light as attackers do. Abuse cases capture the nonnormative behavior of the system. While in [McGraw 04c] abuse cases are described more as a design analysis technique than as a white box testing technique, the same technique can be used to develop innovative and effective test cases mirroring the way attackers would view the system. With access to the source code, a tester is in a better position to quickly see where the weak spots are compared to an outside attacker. The abuse case can also be applied to interactions between components within the system to capture abnormal behavior, should a component misbehave.
19

The technique can also be used to validate design decisions and assumptions. The simplest, most practical method for creating abuse cases is usually through a process of informed brainstorming, involving security, reliability, and subject matter expertise. Known attack patterns form a rich source for developing abuse cases. Trust Boundaries Mapping: Defining zones of varying trust in an application helps identify vulnerable areas of communication and possible attack paths for security violations. Certain components of a system have trust relationships (sometimes implicit, sometime explicit) with other parts of the system. Some of these trust relationships offer trust elevation possibilitiesthat is, these components can escalate trust privileges of a user when data or control flow cross internal boundaries from a region of less trust to a region of more trust [Hoglund 04]. For systems that have n-tier architecture or that rely on several third-party components, the potential for missing trust validation checks is high, so drawing trust boundaries becomes critical for such systems. Drawing clear boundaries of trust on component interactions and identifying data validation points (or chokepoints, as described in [Howard 02]) helps in validating those chokepoints and testing some of the design assumptions behind trust relationships. Combining trust zone mapping with data-flow analysis helps identify data that move from one trust zone to another and whether data checkpoints are sufficient to prevent trust elevation possibilities. This insight can be used to create effective test cases Code Coverage Analysis: Code coverage is an important type of test effectiveness measurement. Code coverage is a way of determining which code statements or paths have been exercised during testing. With respect to testing, coverage analysis helps in identifying
20

areas of code not exercised by a set of test cases. Alternatively, coverage analysis can also help in identifying redundant test cases that do not increase coverage. During ad hoc testing (testing performed without adhering to any specific test approach or process), coverage analysis can greatly reduce the time to determine the code paths exercised and thus improve understanding of code behavior. There are various measures for coverage, such as path coverage, path testing, statement coverage, multiple condition coverage, and function coverage. When planning to use coverage analysis, establish the coverage measure and the minimum percentage of coverage required. Many tools are available for code coverage analysis. It is important to note that coverage analysis should be used to measure test coverage and should not be used to create tests. After performing coverage analysis, if certain code paths or statements were found to be not covered by the tests, the questions to ask are whether the code path should be covered and why the tests missed those paths. A risk-based approach should be employed to decide whether additional tests are required. Covering all the code paths or statements does not guarantee that the software does not have faults; however, the missed code paths or statements should definitely be inspected. One obvious risk is that unexercised code will include Trojan horse functionality, whereby seemingly innocuous code can carry out an attack. Less obvious (but more pervasive) is the risk that unexercised code has serious bugs that can be leveraged into a successful attack. Classes of Tests Creating security tests other than ones that directly map to security specifications is challenging, especially tests that intend to exercise the non-normative behavior of the system. When creating such tests, it is helpful to view the software under test from multiple angles, including the data the system is handling, the environment the system will be operating in, the users of the software (including software components), the options available
21

to configure the system, and the error handling behavior of the system. There is an obvious interaction and overlap between the different views; however, treating each one with specific focus provides a unique perspective that is very helpful in developing effective tests. Data: All input data should be untrusted until proven otherwise, and all data must be validated as it crosses the boundary between trusted and untrusted environments [Howard 02]. Data sensitivity/criticality plays a big role in data-based testing; however, this does not imply that other data can be ignored non-sensitive data could allow a hacker to control a system. When creating tests, it is important to test and observe the validity of data at different points in the software. Tests based on data and data flow should explore incorrectly formed data and stressing the size of the data. The section Attacking with Data Mutation in [Howard 02] describes different properties of data and how to mutate data based on given properties. To understand different attack patterns relevant to program input, refer to chapter six, Crafting (Malicious) Input, in [Hoglund 04]. Tests should validate data from all channels, including web inputs, databases, networks, file systems, and environment variables. Risk analysis should guide the selection of tests and the data set to be exercised.

Environment: Software can only be considered secure if it behaves securely under all operating environments. The environment includes other systems, users, hardware, resources, networks, etc. A
22

common cause of software field failure is miscommunication between the software and its environment [Whittaker 02]. Understanding the environment in which the software operates, and the interactions between the software and its environment, helps in uncovering vulnerable areas of the system. Understanding dependency on external resources (memory, network bandwidth, databases, etc.) helps in exploring the behavior of the software under different stress conditions. Another common source of input to programs is environment variables. If the environment variables can be manipulated, then they can have security implications. In general, analyzing entities outside the direct control of the system provides good insights in developing tests to ensure the robustness of the software under test, given the dependencies. Component Interfaces: Applications usually communicate with other software systems. Within an application, components interface with each other to provide services and exchange data. Common causes of failure at interfaces are misunderstanding of data usage, data lengths, data validation, assumptions, trust relationships, etc. Understanding the interfaces exposed by components is essential in exposing security bugs hidden in the interactions between components. The need for such understanding and testing becomes paramount when third-party software is used or when the source code is not available for a particular component. Another important benefit of understanding component interfaces is validation of principles of compartmentalization. The basic idea behind compartmentalization is to minimize the amount of damage that can be done to a system by breaking up the system into as few units as possible while still isolating code that has security privileges. Test cases can be developed to validate compartmentalization and to explore failure behavior of
23

components in the event of security violations and how the failure affects other components. Configuration: In many cases, software comes with various parameters set by default, possibly with no regard for security. Often, functional testing is performed only with the default settings, thus leaving sections of code related to non-default settings untested. Two main concerns with configuration parameters with respect to security are storing sensitive data in configuration files and configuration parameters changing the flow of execution paths. For example, user privileges, user roles, or user passwords are stored in the configuration files, which could be manipulated to elevate privilege, change roles, or access the system as a valid user. Configuration settings that change the path of execution could exercise vulnerable code sections that were not developed with security in mind. The change of flow also applies to cases where the settings are changed from one security level to another, where the code sections are developed with security in mind. For example, changing an endpoint from requiring authentication to not requiring authentication means the endpoint can be accessed by everyone. When a system has multiple configurable options, testing all combinations of configuration can be time consuming; however, with access to source code, a risk-based approach can help in selecting combinations that have higher probability in exposing security violations. In addition, coverage analysis should aid in determining gaps in test coverage of code paths. Error handling: The most neglected code paths during the testing process are error handling routines. Error handling in this paper includes exception handling, error recovery, and fault tolerance routines. Functionality tests are normally geared towards validating
24

requirements, which generally do not describe negative (or error) scenarios. Even when negative functional tests are created, they dont test for non-normative behavior or extreme error conditions, which can have security implications. For example, functional stress testing is not performed with an objective to break the system to expose security vulnerability. Validating the error handling behavior of the system is critical during security testing, especially subjecting the system to unusual and unexpected error conditions. Unusual errors are those that have a low probability of occurrence during normal usage. Unexpected errors are those that are not explicitly specified in the design specification, and the developers did not think of handling the error. For example, a system call may throw an unable to load library error, which may not be explicitly listed in the design documentation as an error to be handled. All aspects of error handling should be verified and validated, including error propagation, error observability, and error recovery. Error propagation is how the errors are propagated through the call chain. Error observability is how the error is identified and what parameters are passed as error messages. Error recovery is getting back to a state conforming to specifications. For example, return codes for errors may not be checked, leading to uninitialized variables and garbage data in buffers; if the memory is manipulated before causing a failure, the uninitialized memory may contain attacker-supplied data. Another common mistake to look for is when sensitive information is included as part of the error messages.

Tools: Source Code Analysis:

25

Source code analysis is the process of checking source code for coding problems based on a fixed set of patterns or rules that might indicate possible security vulnerabilities. Static analysis tools scan the source code and automatically detect errors that typically pass through compilers and become latent problems. The strength of static analysis depends on the fixed set of patterns or rules used by the tool; static analysis does not find all security issues. The output of the static analysis tool still requires human evaluation. For white box testing, the results of the source code analysis provide a useful insight into the security posture of the application and aid in test case development. White box testing should verify that the potential vulnerabilities uncovered by the static tool do not lead to security violations. Some static analysis tools provide data-flow and control-flow analysis support, which are useful during test case development. Program Understanding Tools: In general, white box testers should have access to the same tools, documentation, and environment as the developers and functional testers on the project do. In addition, tools that aid in program understanding, such as software visualization, code navigation, debugging, and disassembly tools, greatly enhance productivity during testing. Coverage Analysis: Code coverage tools measure how thoroughly tests exercise programs. There are many different coverage measures, including statement coverage, branch coverage, and multiplecondition coverage. The coverage tool would modify the code to record the statements executed and which expressions evaluate which way (the true case or the false case of the expression). Modification is done either to the source code or to the executable the compiler generates. There are several commercial
26

and freeware coverage tools available. Coverage tool selection should be based on the type of coverage measure selected, the language of the source code, the size of the program, and the ease of integration into the build process. Profiling: Profiling allows testers to learn where the software under test is spending most of its time and the sequence of function calls as well. This information can show which pieces of the software are slower than expected and which functions are called more often than expected. From a security testing perspective, the knowledge of performance bottlenecks help uncover vulnerable areas that are not apparent during static analysis. The call graph produced by the profiling tool is helpful in program understanding. Certain profiling tools also detect memory leaks and memory access errors (potential sources of security violations). In general, the functional testing team or the development team should have access to the profiling tool, and the security testers should use the same tool to understand the dynamic behavior of the software under test. Benefits There are many benefits to white box testing, including the following:

Analyzing source code and developing tests based on the implementation details enables testers to find programming errors quickly. For example, a white box tester looking at the implementation can quickly uncover a way, say, through an error handling mechanism, to expose secret data processed by a component. Finding such vulnerabilities through black box testing require comparatively more effort than found through white box testing. This increases the productivity of testing effort.
27

Executing some (hard to set up) black box tests as white box tests reduces complexity in test setup and execution. For example, to drive a specific input into a component, buried inside the software, may require elaborate setup for black box testing but may be done more directly through white box testing by isolating the component and testing it on its own. This reduces the overall cost (in terms of time and effort) required to perform such tests. Validating design decisions and assumptions quickly through white box testing increases effectiveness. The design specification may outline a secure design, but the implementation may not exactly capture the design intent. For example, a design might outline the use of protected channels for data transfer between two components, but the implementation may be using an unprotected method for temporary storage before the data transfer. This increases the productivity of testing effort. Finding unintended features can be quicker during white box testing. Security testing is not just about finding vulnerabilities in the intended functionality of the software but also about examining unintended functionality introduced during implementation. Having access to the source code improves understanding and uncovering the additional unintended behavior of the software. For example, a component may have additional functions exposed to support interactions with some other component, but the same functions may be used to expose protected data from a third component. Depending on the nature of the unintended functionality, it may require a lot more effort to uncover such problems through black box testing alone.

Requirements Testing Requirements seem to be ephemeral. They flit in and out of projects, they are capricious, intractable, unpredictable and
28

sometimes invisible. When gathering requirements we are searching for all of the criteria for a system's success. We throw out a net and try to capture all these criteria. Using Blitzing, Rapid Application Development (RAD), Joint Application Development (JAD), Quality Function Deployment (QFD), interviewing, apprenticing, data analysis and many other techniques, we try to snare all of the requirements in our net. Abstract We accept that testing the software is an integral part of building a system. However, if the software is based on inaccurate requirements, then despite well written code, the software will be unsatisfactory. The newspapers are full of stories about catastrophic software failures. What the stories don't say is that most of the defects can be traced back to wrong, missing, vague or incomplete requirements. We have learnt the lesson of testing software. Now we have to learn to implement a system of testing the requirements before building a software solution.

The Quality gateway As soon as we have a single requirement in our net we can start testing. The aim is to trap requirements-related defects as early as they can be identified. We prevent incorrect requirements from being incorporated in the design and implementation where they will be more difficult and expensive to find and correct. To pass through the quality gateway and be included in the requirements specification, a requirement must pass a number of tests. These tests are concerned with ensuring that the requirements are accurate, and do not cause problems by being unsuitable for the design and implementation stages later in the project.
29

Make the requirements measurable Christopher Alexander describes setting up a quality measure for each requirement. "The idea is for each requirement to have a quality measure that makes it possible to divide all solutions to the requirement into two classes: those for which we agree that they fit the requirement and those for which we agree that they do not fit the requirement." In other words, if we specify a quality measure for a requirement, we mean that any solution that meets this measure will be acceptable. Of course it is also true to say that any solution that does not meet the measure will not be acceptable. The quality measures will be used to test the new system against the requirements. The remainder of this paper describes how to arrive at a quality measure that is acceptable to all the stakeholders. Quantifiable Requirements Consider a requirement that says "The system must respond quickly to customer enquiries". First we need to find a property of this requirement that provides us with a scale for measurement within the context. Let's say that we agree that we will measure the response using minutes. To find the quality measure we ask: "under what circumstances would the system fail to meet this requirement?" The stakeholders review the context of the system and decide that they would consider it a failure if a customer has to wait longer than three minutes for a response to his enquiry. Thus "three minutes" becomes the quality measure for this requirement.

30

Any solution to the requirement is tested against the quality measure. If the solution makes a customer wait for longer than three minutes then it does not fit the requirement. So far so good: we have defined a quantifiable quality measure. But specifying the quality measure is not always so straightforward. What about requirements that do not have an obvious scale?

Non-quantifiable Requirements Suppose a requirement is "The automated interfaces of the system must be easy to learn". There is no obvious measurement scale for "easy to learn". However if we investigate the meaning of the requirement within the particular context, we can set communicable limits for measuring the requirement. Again we can make use of the question: "What is considered a failure to meet this requirement?" Perhaps the stakeholders agree that there will often be novice users, and the stakeholders want novices to be productive within half an hour. We can define the quality measure to say "a novice user must be able to learn to successfully complete a customer order transaction within 30 minutes of first using the system". This becomes a quality measure provided a group of experts within this context is able to test whether the solution does or does not meet the requirement. An attempt to define the quality measure for a requirement helps to rationalise fuzzy requirements. Something like "the system must provide good value" is an example of a requirement that everyone would agree with, but each person has his own meaning. By investigating the scale that must be used to measure "good value" we identify the diverse meanings. Sometimes by causing the stakeholders to think about the requirement we can define an agreed quality measure. In other cases we discover that there is no agreement on a quality
31

measure. Then we substitute this vague requirement with several requirements, each with its own quality measure. Coherency and Consistency When a poet writes a poem he intends that it should trigger off rich and diverse visions for everyone who reads it. The requirements engineer has the opposite intention: he would like each requirement to be understood in the same way by every person who reads it. In practice many requirements specifications are more like poetry, and are open to any interpretation that seems reasonable to the reader. This subjectivity means that many systems are built to satisfy the wrong interpretation of the requirement. The obvious solution to this problem is to specify the requirement in such that it is understood in only one way. For example, in a requirements specification that I assessed, I found the term "viewer" in many parts of the specification. My analysis identified six different meanings for the term, depending on the context of its use. This kind of requirements defect always causes problems during design and/or implementation. If you are lucky, a developer will realise that there is inconsistency, but will have to re-investigate the requirement. This almost always causes a ripple effect that extends to other parts of the product. If you are not lucky, the designer will choose the meaning that makes most sense to him and implement that one. Any stakeholder who does not agree with that meaning then considers that the system does not meet the requirement. Completeness We want to be sure that the requirements specification contains all the requirements that are known about. While we know that there will be evolutionary changes and additions, we would like to restrict those changes to new requirements, and not have to play "catch-up" with requirements that we should have known about in the first place. Thus we want to avoid omitting
32

requirements just because we did not think of asking the right questions. If we have set a context for our project, then we can test whether the context is accurate. We can also test whether we have considered all the likely requirements within that context. The context defines the problem that we are trying to solve. The context includes all the requirements that we must eventually meet: it contains anything that we have to build, or anything we have to change. Naturally if our software is going to change the way people do their jobs, then those jobs must be within the context of study. The most common defect is to limit the context to the part of the system that will be eventually automated. The result of this restricted view is that nobody correctly understands the organisation's culture and way of working. Consequently there is misfit between the eventual computer system and the rest of the business system and the people that it is intended to help. Relavence When we cast out the requirements gathering net and encourage people to tell us all their requirements, we take a risk. Along with all the requirements that are relevant to our context we are likely to pick up impostors. These irrelevant requirements are often the result of a stakeholder not understanding the goals of the project. In this case people, especially if they have had bad experiences with another system, are prone to include requirements "just in case we need it". Another reason for irrelevancy is personal bias. If a stakeholder is particularly interested or affected by a subject then he might think of it as a requirement even if it is irrelevant to this system. Requirement or Solution? When one of your stakeholders tells you he wants a graphic user interface and a mouse, he is presenting you with a solution not a requirement. He has seen other systems with graphic user interfaces, and he wants what he considers to be the most up-todate solution. Or perhaps he thinks that designing the system is
33

part of his role. Or maybe he has a real requirement that he has mentally solved by use of a graphic interface. When solutions are mistaken for requirements then the real requirement is often missed. Also the eventual solution is not as good as it could be because the designer is not free to consider all possible ways of meeting the requirements. Stakeholder Value There are two factors that affect the value that stakeholders place on a requirement. The grumpiness that is caused by bad performance, and the happiness that is caused by good performance. Failure to provide a perfect solution to some requirements will produce mild annoyance. Failure to meet other requirements will cause the whole system to be a failure. If we understand the value that the stakeholders put on each requirement, we can use that information to determine design priorities. Traceability We want to be able to prove that the system that we build meets each one of the specified requirements. We need to identify each requirement so that we can trace its progress through detailed analysis, design and eventual implementation. Each stage of system development shapes, repartitions and organises the requirements to bring them closer to the form of the new system. To insure against loss or corruption, we need to be able to map the original requirements to the solution for testing purposes Regression testing Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired, stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes.
34

Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged.

Background Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but often a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature. Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a 'test suite' contains software tools that allow the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool, such as BuildBot. Regression testing is an integral part of the extreme programming software development method. In this method, design documents are replaced by extensive, repeatable, and
35

automated testing of the entire software package at every stage in the software development cycle. Regression Test Generation Effective regression tests generate sufficient code execution coverage to exercise all meaningful code branches. Therefore, software testing is a combinatorial problem. Every boolean decision statement requires at least two tests: one with an outcome of "true" and one with an outcome of "false". As a result, for every line of code written, programmers often need 3 to 5 lines of test code. Traditionally, regression testing has been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of developer testing. Although developers have always written test cases as part of the development cycle, these test cases have generally been either functional tests or unit tests that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases. When regression test generation is supported by a sustainable process for ensuring that test case failures are reviewed daily and addressed immediately, the end result is a regression suite that evolves with the application, and becomes more robust and more intelligent each day. If such a process is not implemented and ingrained into the team's workflow, the application may evolve out of sync with the generated test suite- increasing false positives and reducing the effectiveness of the test suite. Types of Regression Local - changes introduce new bugs. Unmasked - changes unmask previously existing bugs. Remote - Changing one part breaks another part of the program. For example, Module A writes to a database.
36

Module B reads from the database. If changes to what Module A writes to the database break Module B, it is remote regression. There's another way to classify regression.

New feature regression - changes to code that is new to release 1.1 break other code that is new to release 1.1. Existing feature regression - changes to code that is new to release 1.1 break code that existed in release 1.0. Mitigating Regression Risk Complete test suite repetition Regression test automation (GUI, API, CLI) Partial test repetition based on traceability and analysis of technical and business risks Customer or user testing o Beta - early release to both potential and current customers o Pilot - deploy to a subset of users o Parallel - users use both old and new systems simultaneously Use larger releases. Testing new functions often covers existing functions. The more new features in a release, the more "accidental" regression testing. Emergency patches - these patches are released immediately, and will be included in future maintenance releases.

Uses of Regression Testing Regression testing can be used not only for testing the correctness of a program, but it is also often used to track the quality of its output.

37

For instance in the design of a compiler, regression testing should track the code size, simulation time and compilation time of the test suite cases. Strategies and Factors Some strategies and factors to consider during regression testing include the following:

Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause. Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs. Write a regression test for each bug fixed. If two or more tests are similar, determine which is less effective and get rid of it. Identify tests that the program consistently passes and archive them. Focus on functional issues, not those related to design. Make changes (small and large) to data and find any resulting corruption. Trace the effects of the changes on program memory

Error Handling Testing Error handling refers to the anticipation, detection, and resolution of programming, application, and communications errors. Specialized programs, called error handlers, are available for some applications. The best programs of this type forestall errors if possible, recover from them when they occur without terminating the application, or (if all else fails) gracefully terminate an affected application and save the error information to a log file. In programming, a development error is one that can be prevented. Such an error can occur in syntax or logic. Syntax errors, which are typographical mistakes or improper use of
38

special characters, are handled by rigorous proofreading. Logic errors, also called bugs, occur when executed code does not produce the expected or desired result. Logic errors are best handled by meticulous program debugging. This can be an ongoing process that involves, in addition to the traditional debugging routine, beta testing prior to official release and customer feedback after official release. A run-time error takes place during the execution of a program, and usually happens because of adverse system parameters or invalid input data. An example is the lack of sufficient memory to run an application or a memory conflict with another program. On the Internet, run-time errors can result from electrical noise, various forms of malware or an exceptionally heavy demand on a server. Run-time errors can be resolved, or their impact minimized, by the use of error handler programs, by vigilance on the part of network and server administrators, and by reasonable security countermeasures on the part of Internet users. Usage

It determines the ability of applications system to process the incorrect transactions properly Errors encompass all unexpected conditions. In some system approx. 50% of programming effort will be devoted to handling error condition.

Objective

Determine Application system recognizes all expected error conditions. Determine Accountability of processing errors has been assigned and procedures provide a high probability that errors will be properly corrected. Determine During correction process reasonable control is maintained over errors.

39

How to Use? A group of knowledgeable people is required to anticipate what can go wrong in the application system. It is needed that all the application knowledgeable people assemble to integrate their knowledge of user area, auditing and error tracking. Then logical test error conditions should be created based on this assimilated information.

When to Use? Throughout SDLC. Impact from errors should be identified and should be corrected to reduce the errors to acceptable level. Used to assist in error management process of system development and maintenance. Manual Testing Manual testing is the oldest and most rigorous type of software testing. Manual testing requires a tester to perform manual test operations on the test software without the help of Test automation. Manual testing is a laborious activity that requires the tester to possess a certain set of qualities; to be patient, observant, speculative, creative, innovative, open-minded, resourceful, unopinionated, and skillful. Overview As a tester, it is always advisable to use manual white box testing and black-box testing techniques on the test software. Manual testing helps discover and record any software bugs or discrepencies related to the functionality of the product. Manual testing can be replaced by test automation. It is possible to record and playback manual steps and write automated test script(s) using Test automation tools. Although, test automation tools will only help execute test scripts written primarily for
40

executing a particular specification and functionality. Test automation tools lack the ability of decision-making and recording any unscripted discrepancies during program execution. It is recommended that one should perform manual testing of the entire product at least a couple of times before actually deciding to automate the more mundane activities of the product. Manual testing helps discover defects related to the usability testing and GUI testing area. While performing manual tests the software application can be validated whether it meets the various standards defined for effective and efficient usage and accessibility. For example, the standard location of the OK button on a screen is on the left and of CANCEL button on the right. During manual testing you might discover that on some screen, it is not. This is a new defect related to the usability of the screen. In addition, there could be many cases where the GUI is not displayed correctly and the basic functionality of the program is correct. Such bugs are not detectable using test automation tools. Repetitive manual testing can be difficult to perform on large software applications or applications having very large dataset coverage. This drawback is compensated for by using manual black-box testing techniques including equivalence partitioning and boundary value analysis. Using which, the vast dataset specifications can be divided and converted into a more manageable and achievable set of test suites

Steps for Manual Testing A manual tester would typically perform the following steps for manual testing: 1. Understand the functionality of program 2. Prepare a test environment
41

3. Execute test case(s) manually 4. Verify the actual result 1. 2. 3. 4. Record the result as Pass or Fail Make a summary report of the Pass and Fail test cases Publish the report Record any new defects uncovered during the test case execution

There is no complete substitute for manual testing. Manual testing is crucial for testing software applications more thoroughly. Test automation has become a necessity mainly due to shorter deadlines for performing test activities, such as regression testing, performance testing, and load testing.

Usage: It involves testing of all the functions performed by the people while preparing the data and using these data from automated system. Objective:

Verify manual support documents and procedures are correct. Determine Manual support responsibility is correct. Determine Manual support people are adequately trained. Determine Manual support and automated segment are properly interfaced. How to use Process evaluated in all segments of SDLC. Execution of the can be done in conjunction with normal system testing. Instead of preparing, execution and entering actual test transactions the clerical and supervisory personnel can use the results of processing from application system. To test people it requires testing the interface between the people and application system.
42

When to use Verification that manual systems function properly should be conducted throughout the SDLC. Should not be done at later stages of SDLC. Best done at installation stage so that the clerical people do not get used to the actual system just before system goes to production. Example Provide input personnel with the type of information they would normally receive from their customers and then have them transcribe that information and enter it in the computer. Users can be provided a series of test conditions and then asked to respond to those conditions. Conducted in this manner, manual support testing is like an examination in which the users are asked to obtain the answer from the procedures and manuals available to them.

Intersystem Testing There is software , some built in the OS and many after market that can do in system "house cleaning " and testing. One example is PC Alert. It shows up as a small open eye in task bar and when We move mouse over it, it reads out my CPU temp. If I right click it and go to the main page, I can test the inter system of every function of my computer as well as all the external systems as well. An inter system test, for example, could be a memory map, showing what is in memory at that moment. Another is a test of my fans, speed, efficiency etc. It basically is the way you find out if the inter system is working with out any problem and is a big help when you do have a problem in finding a fix for it.

43

Objective

Determine Proper parameters and data are correctly passed between the applications. Documentation for involved system is correct and accurate. Ensure Proper timing and coordination of functions exists between the application system.

How to use

Operations of multiple systems are tested. Multiple systems are run from one another to check that they are acceptable and processed properly.

When to use

When there is change in parameters in application system. The parameters, which are erroneous then risk associated to such parameters, would decide the extent of testing and type of testing. Intersystem parameters would be checked / verified after the change or new application is placed in the production.

Disadvantages

Time consuming and tedious if test automation not done. Cost may be expensive if system is run several times iteratively.

44

Control Testing

Usage: Control is a management tool to ensure that processing is performed in accordance to what management desire or intents of management. Objective: Accurate and complete data. Authorized transactions. Maintenance of adequate audit trail of information. Efficient, effective and economical process. Process meeting the needs of the user. How to use To test controls risks must be identified. Testers should have negative approach i.e. should determine or anticipate what can go wrong in the application system. Develop risk matrix, which identifies the risks, controls; segment within application system in which control resides. When to use Should be tested with other system tests. Example File reconciliation procedures work. Manual controls in place.

Parallel Testing As a test engineer, you may have explored ways to enhance test system performance in the past though parallel testing. However, the latest off-the-shelf test management software tools simplify parallel test system implementation. These tools increase test throughput and drive down test system costs.

45

Introduction In general, parallel testing involves testing multiple products or subcomponents simultaneously. A parallel test station typically shares a set of test equipment across multiple test sockets, but, in some cases, it may have a separate set of hardware for each unit under test (UUT). The majority of nonparallel test systems test only one product or subcomponent at a time, leaving expensive test hardware idle more than 50 percent of the test time. Thus, with parallel testing, you can increase the throughput of manufacturing test systems without spending a lot of money to duplicate and fan out additional test systems.

Common Parallel Process Modules When testing the proper assembly or functionality of a UUT, there are a variety of tasks to perform in any test system. These tasks include a mix of model or family-specific tests as well as many procedures that have nothing to do with actually testing the UUT. A process model separates the system-level tasks from the UUT-specific tests to significantly reduce development efforts and increase code reuse. Some of the tasks that a process model handles are tracking the UUT identification number, initializing instruments, launching test executions, collecting test results, creating test reports, and logging test results to a database. TestStand provides two process modules, the parallel process model and the batch process model, to facilitate the general test flow of parallel testing based on your UUT test requirements.
46

You can use a parallel process model to test multiple independent test sockets. With this model, you can start and stop testing on any UUT at any time. For example, you might have five test sockets for performing radio board tests. Using the parallel process model, you can load a new board into an open socket while the other sockets test other boards.

Volume Testing Volume Testing belongs to the group of non-functional tests, which are often misunderstood and/or used interchangeably. Volume testing refers to testing a software application for a certain data volume. This volume can in generic terms be the database size or it could also be the size of an interface file that
47

is the subject of volume testing. For example, if you want to volume test your application with a specific database size, you will explode your database to that size and then test the application's performance on it. Another example could be when there is a requirement for your application to interact with an interface file (could be any file such as .dat, .xml); this interaction could be reading and/or writing on to/from the file. You will create a sample file of the size you want and then test the application's functionality with that file to check performance What is Volume Testing? At first glance it might appear that volume testing should be fairly high up on the V model, especially if we take the above definition. Certainly it comes under non-functional requirements testing and Performance testing. Volume testing can (and should) be used in component testing. This would be closely correlated with the code, so might at this level be connected to Dynamic Analysis. Volume testing will also be undertaken (normally) as part of the User Acceptance test. Stress testing is closely related, as it seeks to find out how the software will behave beyond its specified limits. Why to Use Volume Testing? 1. To find faults 2. Give credible information about the state of the component, on which business decisions can be taken. As regards faults that should be found through volume testing are those, where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger volumes of data, than that of small retailer software. A fault which is only manifested on a
48

table with a million records will be of no concern to the retail software, but will be picked up by the bank testers. Credible information about how the software will behave is essential. During the dot com boom many websites went live without knowing, what the effect would be if the back end database grew exponentially. Of course many suffered crashes as a result. Why test at the component level? This is because we can then see how the code behaves, and confirm that the component will not be a bottle neck and slow down the whole system. (Alternatively, use too many system resources.) For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records, is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level. Some methodologies such as the RUP suggest early testing. Therefore in the early phases, (inception and elaboration), volume testing might take place to confirm that the central core architecture is the one to proceed with. Who to Use Volume Testing? As volume testing can be introduced almost anywhere, from component testing to user acceptance testing, the range of people who can undertake it is similarly large. Developers through to customers and end-users can do it. The testing may be outsourced to a testing lab, which specialises in performance testing.
49

How to Use Volume Testing? Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it. The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on. The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by adding rules directly on to the database, with SQL. As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes. Stress Testing Stress testing deals with the quality of the application in the environment. The idea is to create an environment more demanding of the application than the application would experience under normal work loads. This is the hardest and most complex category of testing to accomplish and it requires a joint effort from all teams. A test environment is established with many testing stations. At each station, a script is exercising the system. These scripts are usually based on the regression suite. More and more stations are added, all simultaneous hammering on the system, until the system breaks. The system is repaired and the stress test is
50

repeated until a level of stress is reached that is higher than expected to be present at a customer site. Race conditions and memory leaks are often found under stress testing. A race condition is a conflict between at least two tests. Each test works correctly when done in isolation. When the two tests are run in parallel, one or both of the tests fail. This is usually due to an incorrectly managed lock. A memory leak happens when a test leaves allocated memory behind and does not correctly return the memory to the memory allocation scheme. The test seems to run correctly, but after being exercised several times, available memory is reduced until the system fails. Performance Testing Overview System performance is generally assessed in terms of response time and throughput rates under differing processing and configuration conditions. To attack the performance problems, there are several questions should be asked first: How much application logic should be remotely executed? How much updating should be done to the database server over the network from the client workstation? How much data should be sent to each in each transaction? According to Hamilton, the performance problems are most often the result of the client or server being configured inappropriately. The best strategy for improving client-sever performance is a three-step process. First, execute controlled performance tests that collect the data about volume, stress, and loading tests. Second, analyze the collected data. Third, examine and tune the database queries and, if necessary, provide
51

temporary data storage on the client while the application is executing. Definition Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Qualitative attributes such as reliability, scalability and interoperability may also be evaluated. Performance testing is often done in conjunction with stress testing. Performance testing can verify that a system meets the specifications claimed by its manufacturer or vendor. The process can compare two or more devices or programs in terms of parameters such as speed, data transfer rate, bandwidth, throughput, efficiency or reliability. Performance testing can also be used as a diagnostic aid in locating communications bottlenecks. Often a system will work much better if a problem is resolved at a single point or in a single component. For example, even the fastest computer will function poorly on today's Web if the connection occurs at only 40 to 50 Kbps (kilobits per second). Slow data transfer rate may be inherent in hardware but can also result from software-related problems, such as:

Too many applications running at the same time A corrupted file in a Web browser A security exploit Heavy-handed antivirus software The existence of active malware on the hard disk.

Effective performance testing can quickly identify the nature or location of a software-related performance problem.
52

Method It is important that the users, tasks and environment used for the test are representative of the intended context of use. Select the most important tasks and user groups to be tested (e.g. the most frequent or the most critical). Select users who are representative of each user group. 3-5 users are sufficient to identify problems. 8 or more users of each type are required for reliable measures. Produce a task scenario and input data and write instructions for the user (tell the user what to achieve, not how to do it). Plan sessions allowing time for giving instructions, running the test, answering a questionnaire, and a post-test interview. Invite developers to observe the sessions if possible. An alternative is to videotape the sessions, and show developers edited clips of the usability problems. Two administrators are normally required to share the activities of instructing and interviewing the user, operating video equipment (if used), noting problems, and speaking to any observers. If possible use one room for testing, linked by video to another room for observation. If usability measures are required, observe the user without making any comments. If measures are not required, prompt the user to explain their interpretation of the contents of each screen and their reason for making choices. Output Produce a list of usability problems, categorised by importance (use post-it-notes to sort the problems), and an overview of the types of problems encountered. Arrange a meeting with the project manager and developer to discuss whether and how each problem can be fixed.

53

If measures have been taken, summarise the results of the satisfaction questionnaire, task time and effectiveness (accuracy and completeness) measures. If a full report is required, the Common Industry Format provides a good structure.

Benefits Major usability problems are identified that may not be revealed by less formal testing, including problems related to the specific skills and expectations of the users. Measures can be obtained for the users' effectiveness, efficiency and satisfaction. Summary Performance testing is a rigorous usability evaluation of a working system under realistic conditions to identify usability problems and to compare measures such as success rate, task time and user satisfaction with requirements. Agile Testing

Agile software development is a conceptual framework for software engineering that promotes development iterations throughout the life-cycle of the project. There are many agile development methods; most minimize risk by developing software in short amounts of time. Software developed during one unit of time is referred to as an iteration, which may last from one to four weeks. Each iteration is an entire software project: including planning, requirements analysis, design, coding, testing, and documentation. An iteration may not add enough functionality to warrant releasing the product to market but the goal is to have an available release
54

(without bugs) at the end of each iteration. At the end of each iteration, the team re-evaluates project priorities. Agile methods emphasize face-to-face communication over written documents. Most agile teams are located in a single open office sometimes referred to as a bullpen. At a minimum, this includes programmers and their "customers" (customers define the product; they may be product managers, business analysts, or the clients). The office may include testers, interaction designers, technical writers, and managers. Agile methods also emphasize working software as the primary measure of progress. Combined with the preference for face-toface communication, agile methods produce very little written documentation relative to other methods. This has resulted in criticism of agile methods as being undisciplined

History Agile software development is a conceptual framework for software engineering that promotes development iterations throughout the life-cycle of the project. There are many agile development methods; most minimize risk by developing software in short amounts of time. Software developed during one unit of time is referred to as an iteration, which may last from one to four weeks. Each iteration is an entire software project: including planning, requirements analysis, design, coding, testing, and documentation. An iteration may not add enough functionality to warrant releasing the product to market but the goal is to have an available release (without bugs) at the end of each iteration. At the end of each iteration, the team re-evaluates project priorities. Agile methods emphasize face-to-face communication over written documents. Most agile teams are located in a single open office sometimes referred to as a bullpen. At a minimum, this
55

includes programmers and their "customers" (customers define the product; they may be product managers, business analysts, or the clients). The office may include testers, interaction designers, technical writers, and managers. Agile methods also emphasize working software as the primary measure of progress. Combined with the preference for face-toface communication, agile methods produce very little written documentation relative to other methods. This has resulted in criticism of agile methods as being undisciplined. The modern definition of agile software development evolved in the mid 1990s as part of a reaction against "heavyweight" methods, as typified by a heavily regulated, regimented, micromanaged use of the waterfall model of development. The processes originating from this use of the waterfall model were seen as bureaucratic, slow, demeaning, and inconsistent with the ways that software developers actually perform effective work. A case can be made that agile and iterative development methods are a return to development practice seen early in the history of software development. Initially, agile methods were called "lightweight methods." In 2001, prominent members of the community met at Snowbird, Utah, and adopted the name "agile methods." Later, some of these people formed The Agile Alliance, a non-profit organization that promotes agile development. Methodologies similar to Agile created prior to 2000include Scrum (1986), Crystal Clear, Extreme Programming (1996), Adaptive Software Development, Feature Driven Development, and DSDM (1995). Extreme Programming (usually abbreviated as "XP") was created by Kent Beck in 1996 as a way to rescue the struggling Chrysler Comprehensive Compensation (C3) project. While that project was eventually canceled, the methodology was refined by Ron Jeffries' full-time XP coaching, public discussion on Ward Cunningham's Portland Pattern Repository wiki and
56

further work by Beck, including a book in 1999. Elements of Extreme Programming appear to be based on Scrum and Ward Cunningham's Episodes pattern language. Principles Agile methods are a family of development processes, not a single approach to software development. In 2001, 17 prominent figures in the field of agile development (then called "lightweight methodologies") came together at the Snowbird ski resort in Utah to discuss ways of creating software in a lighter, faster, more people-centric way. They created the Agile Manifesto, widely regarded as the canonical definition of agile development, and accompanying agile principles. Some of the principles behind the Agile Manifesto are

Customer satisfaction by rapid, continuous delivery of useful software Working software is delivered frequently (weeks rather than months) Working software is the principal measure of progress Even late changes in requirements are welcomed Close, daily cooperation between business people and developers Face-to-face conversation is the best form of communication Projects are built around motivated individuals, who should be trusted Continuous attention to technical excellence and good design Simplicity Self-organizing teams Regular adaptation to changing circumstances

The publishing of the manifesto spawned a movement in the software industry known as agile software development.

57

In 2005, Alistair Cockburn and Jim Highsmith gathered another group of people management experts, this time and wrote an addendum, known as the PM Declaration of Interdependence.

Comparison with other methods Agile methods are sometimes characterized as being at the opposite end of the spectrum from "plan-driven" or "disciplined" methodologies. This distinction is misleading, as it implies that agile methods are "unplanned" or "undisciplined". A more accurate distinction is to say that methods exist on a continuum from "adaptive" to "predictive". Agile methods exist on the "adaptive" side of this continuum. Adaptive methods focus on adapting quickly to changing realities. When the needs of a project change, an adaptive team changes as well. An adaptive team will have difficulty describing exactly what will happen in the future. The further away a date is, the more vague an adaptive method will be about what will happen on that date. An adaptive team can report exactly what tasks are being done next week, but only which features are planned for next month. When asked about a release six months from now, an adaptive team may only be able to report the mission statement for the release, or a statement of expected value vs. cost. Predictive methods, in contrast, focus on planning the future in detail. A predictive team can report exactly what features and tasks are planned for the entire length of the development process. Predictive teams have difficulty changing direction. The plan is typically optimized for the original destination and changing direction can cause completed work to be thrown away and done over differently. Predictive teams will often institute a change control board to ensure that only the most valuable changes are considered.

58

Agile methods have much in common with the "Rapid Application Development" techniques from the 1980/90s as espoused by James Martin and others. Suitability of agile methods Although agile methods differ in their practices, they share a number of common characteristics, including iterative development, and a focus on interaction, communication, and the reduction of resource-intensive intermediate artifacts. The suitability of agile methods in general can be examined from multiple perspectives. From a product perspective, agile methods are more suitable when requirements are emergent and rapidly changing; they are less suitable for systems that have high criticality, reliability and safety requirements, though there is no complete consensus on this point. From an organizational perspective, the suitability can be assessed by examining three key dimensions of an organization: culture, people, and communication. In relation to these areas a number of key success factors have been identified (Cohen et al., 2004):

The culture of the organization must be supportive of negotiation People must be trusted Fewer staff, with higher levels of competency Organizations must live with the decisions developers make Organizations need to have an environment that facilitates rapid communication between team members

Agile data One of the most challenging parts of an agile project is being agile with data. Typically this is where projects hit legacy systems and legacy requirements. Many times working with data systems requires lengthy requests to teams of specialists who are not used to the speed of an agile project and insist on
59

exact and complete specifications. Typically the database world will be at odds with agile development. The agile framework seeks as much as possible to remove these bottlenecks with techniques such as generative data models making change fast. Models for data serve another purpose, often a change of one table column can be a critical issue requiring months to rebuild all the dependent applications. An agile approach would try to encapsulate data dependencies to go fast and allow change. But ultimately relational data issues will be important for agile projects and are a common blockage point. As such agile projects are best suited where projects don't contain big legacy databases. It still isn't the end of the world because if you can build your data dependencies to be agile regardless of legacy systems you will start to prove the merit of the approach as all other systems go through tedious changes to catch up with data changes while in the protected agile data system the change would be trivial. Agile Methods Some of the well-known agile software development methods:

Agile Modeling Agile Unified Process (AUP) Agile Data Daily kickoff and review of goals short release cycles Responsive Development Generalism - Use of generic skill sets which are common across the team not reliance on specific skill sets which are scarce Test Driven Development (TDD) Feature Driven Development (FDD) Behavior Driven Development (BDD) Essential Unified Process (EssUP)

60

Other approaches:

Software Development Rhythms Agile Documentation ICONIX Process Microsoft Solutions Framework (MSF) Agile Data Method Database refactoring

61