Hans Schaefer

The Hows and Whys of Integration Testing

Hans Schaefer Software Test Consulting Reigstad N-5281 Valestrandsfossen NORWAY e-mail hans.schaefer@ieee.org

http://www.softwaretesting.no
A shorter version of this article was published in Better Software Magazine ( www.bettersoftware.com ). Software integration, always difficult work, is becoming more problematic because of outsourcing, the inclusion of third-party libraries and components, and the sheer size of todays systems. Integration testing is often poorly planned, poorly estimated, and poorly performed. This article describes the process of integration testing, typical integration defects, and practical approaches to discovering these defects. The most common cause of integration defects is the failure to communicate. Without adequate knowledge, people may make incorrect assumptions about components created by others. A good integration process seeks to minimize those assumptions. A good integration testing process seeks to discover the defects created by these assumptions. What To Test Integration testing is interface testing. Interfaces define how components interact with each other. These components can be units of code, subsystems, systems, and users. Interfaces must be tested because they could have been specified incorrectly, defined incorrectly, or implemented incorrectly. The risk of integration defects increases as the organizational distance between the people involved in the development work increases. In addition, complexity, time pressure, and poor processes can also lead to integration defects. Figure 1 shows what is tested during integration.

QuickTime and a TIFF (LZW) decompressor are needed to see this picture.

Figure 1: What is tested during integration A useful test architecture Aside from the interface between the user and the system, all other interfaces are inside the system and thus are difficult to observe or manipulate. By creating a special test architecture, testers can insert selected data values into interfaces and observe the operations of modules. In addition, special monitors, protocol analyzers, and data capture tools can be used. It is much more convenient for testers if access points are designed and built into the software during development. Figure 2 shows a typical test architecture.

1/8

Hans Schaefer

QuickTime and a TIFF (LZW) decompressor are needed to see this picture.

Figure 2: Actual test architecture What to test In principle, integration testing should be concerned only about interfaces. (If testing was not done well at previous stages, all those undiscovered defects leak into integration testing.) Thus, to design test cases effectively and efficiently, we need a detailed description of the internal interfaces of the system. This information is part of an system architecture or high-level design document. However, in practice, this information is not always available. The following list suggests where to look for the needed interface information: Communication protocols Subsystem definitions Sequence and collaboration diagrams Call-pairs and their argument lists Lists of events to be processed by the system Definition, use, and flow of global data API definitions Files and their formats Interaction Online - batch - online Synchronization of parallel processes and transactions, especially queue handling. Multitasking: Capacity of shared resources (place and speed) Failure handling and recovery processing in components. For example special cases at interfaces (data not available, data in improper format, empty files, network unavailable, etc.) Port assignments and handling Internal security rules The information required consists of both syntax and semanticsthe structure of what is flowing through the interface and what it means. Both the sending and receiving sides of the interface must understand and implement these correctly. Typical Faults This section summarizes the most common integration problems. It is not exhaustive, but provides a good basis for fault-based testing. For various architectures and programming languages, some of these problems may not be possible, as they are prevented by the selected technology. The taxonomy is modified from Beizer(1): 1. 2. Component invocation bugs. The wrong component, a faulty one or a non-existing component is invoked. Problems with parameters. Parameters may be incorrect in their number, order, data type, values, conventions etc. a. Initialization problems. Variables, files, and other objects must be properly initialized. b. Incorrect use of defaults. Defaults are a problem if different developers use them differently. c. Wrong data type, wrong measuring unit is used. 2/8

Hans Schaefer References: Pointers are dangerous, especially over interfaces. Format inconsistency. Whoever has used a file converter knows this problem. Both structure and meaning may be wrong. f. Wrong boundary values are implemented. g. The order of parameters is mixed up. Wrong wiring, wrong ports. 3. Wrong interpretation of names, data, codes, etc., both by the called component and when returned to the caller. 4. Wrong state of a called object when it is invoked. 5. Data flow problems. Data elements used may be not defined, not initialized but used, overwritten at the wrong time, never used, or not cleaned up. 6. Device, external interfaces and timing issues. a. Output media / messages / files are too large, empty, or missing. b. Trouble with device drivers (wrong driver, wrong interpretation, wrong state, initialization) c. Wrong use of communication channels (not connected, disconnect, confusion about who starts communication, ...). d. Conflict in shared resources (speed, memory, bandwidth, DLLs, assumptions). The classic problem is a race condition. e. Speed, capacity and timing problems. 7. Architectural problems. a. Cleaning up. Are reserved resources freed, unnecessary data or files deleted? b. Parallel programs or processes: start orders, restart and recovery. c. Synchronization and queues. 8. Interference from a hostile environment and recovery problems. If programs assume the environment will work correctly, this may not always be true. 9. Platform related problems. A component may have inadvertently been implemented with platform dependencies. 10. Wrong configuration (of some components). IN THE FOLLOWING ARE DETAILS ABOUT THESE BUG TYPES. 1. Component invocation bugs. The wrong component or a non-existing component is called. This typically happens either because of a spelling error in the components name or a miscommunication about the components name or its functionality. The wrong component may have a similar name, or it may be the wrong version. Wrong functionality may be anything, from a small deviation in implemented versus required functionality to completely different implementation. Often, the fault will be found just by just executing the call. But, if there is not much difference in the way the incorrect component behaves (the fault has a small footprint), the tester may need to check many input equivalence classes or even all input values to detect this defect. Problems with parameters. Parameters are passed in messages or invocations between the components. They may be incorrect in their number, order, data type, values, conventions etc. a. Initialization problems. Variables, files, and other objects must be properly initialized. b. Incorrect use of defaults. Defaults are a problem if different developers use them differently. If developers do not communicate well, they may use the defaults they are used to, and these may be wrong. A good test would be to use every expected default value and another value. c. Wrong data type, wrong measuring unit is used. There may be misunderstandings regarding integer and floating point numbers, different machine word lengths, Boolean representation, ASCII versus Unicode, etc. The Ariane 5 rocket crash is a classic example when a 64-bit value was assigned to a 16-bit variable and the overflow was ignored. The loss of a NASA Mars Polar Lander occurred because one group used metric units while another used English units. Static analysis and reviews as well as strongly typed languages can help here. d. References: Pointers are dangerous, especially over interfaces. It must be assured that both sides know what is pointed to. Check that a pointer really points to something. Exception handling must be tested if NULL pointers are not prevented. Static analysis may help. e. Format inconsistency. Whoever has used a file converter knows this problem. Both structure and meaning may be wrong. When the receiver should accept different formats, some formats or elements of it may be misunderstood, not accepted at all, or not processed correctly. All parts of a format definition should thus be included in a test. Syntax testing is a useful method to deal with this problem. f. Wrong boundary values are implemented. It is generally not a problem if the called module accepts 3/8 d. e.

2.

Hans Schaefer a larger domain than the calling one. But if the input domain is smaller than the output domain of the sender, defects can occur. Boundary misunderstandings can be found by complete boundary value testing. g. The order of parameters is mixed up. Wrong wiring, wrong ports. The typical example situation for this is the different National date formats. Day, month and year are written in a different order in many different countries. If different people work on interfacing components, this problem may arise. However, it is not just dates. Any series of data may be implemented in an incorrect order. For the testing, different data values in every field may help. 3. Wrong interpretation of names, data, codes, etc., both by the called component and when returned to the caller. Any type of data or may be misinterpreted. Every type of data should be known by both components and be tested through the interface. 4. Wrong state of a called object when it is invoked. Typically, the called object does not expect to be called at this point in time. It may not be initialized or ready. Input is ignored or misunderstood. Evaluating the return value may give a clue to the origin of the problem. 5. Data flow problems. Data elements used may be not defined, not initialized but used, overwritten at the wrong time, never used, or not cleaned up. 6. Device, external interfaces and timing issues. a. Output media / messages / files are too large, empty, or missing. For any media, the minimum and maximum volume should be specified and tested. Additionally, it is important to test failures and the processes for handling them. b. Trouble with device drivers. The wrong driver may be used, there may be wrong interpretation og commands and data, wrong state, wrong initialization of the channel. c. Wrong use of communication channels (not connected, disconnect, confusion about who starts communication, ...). (not connected, disconnect, confusion about who starts or ends communication, ...). Part of this problem is the one of distributed state machines, where every subsystem implements a part of it. The other part is confusion about identification of channels. The test methods are standard state machine testing, as well just running every communication channel in order to see if it is connected at all and disconnected after use. d. Conflict in shared resources (speed, memory, bandwidth, DLLs, assumptions). The classic problem is a race condition. This is extremely difficult to find by testing, as it works fine most of the time. The issue is to be aware of the possibility. Stress testing generally increases the chance to find such defects. e. Speed, capacity and timing problems. Too fast or slow transmission or transmission at the wrong point of time (unexpected). The classic situation is the manual synchronization of modem speeds we had years ago. If sending speed is too high, the receiver will only get part of the information. If traffic is too slow, there may be timeouts or other kinds of transaction cancellations. Otherwise, signals may be sent at a point of time where a process does not expect them. They may then interrupt the process or go unnoticed. 7. Architectural problems. a. Cleaning up. Are reserved resources freed, unnecessary data or files deleted? Double operations like allocate/de-allocate or reserve/free for any resource are dangerous. This is especially true if the two operations are in different components to be integrated, or if they are the responsibility of different developers during implementation. This problem is best detected through static analysis, not by integration testing. The prototype fault is called resource leak. b. Parallel programs or processes: Start orders, restart and recovery may be messed up. If one program waits for another one to start, and the second one waits for the first one, we have a deadlock. If a program waits too short a time during startup, and then fails because something else is not ready, the whole startup sequence may be aborted or frozen. c. Synchronization and queues. Tests for queues have been described by Beizer (1). Transaction flow testing is another method to detect these defects. The queue regime may be different than expected, or a queue is implemented as a stack or vice versa, or the priorities are wrong. 8. Interference from a hostile environment and recovery problems. If programs assume the environment will work correctly, this may not always be true. 9. Platform related problems. A component may have inadvertently been implemented with platform dependencies. If it should work on a different platform, these dependencies may create problems. There are no effective test methods against this problem. Rather, it should be researched how platform dependencies can be prevented using design and coding standards. Typical ones are defaults, machine word lengths, and channel numbers. 10. Wrong configuration (of some components). ). This should be avoided by configuration management, but testing can check if this always works. . 4/8

Hans Schaefer This list is not complete! There is room for more bugs. END OF THE DETAILED EXPLANATION OF POSSIBLE BUGS. The first step in test execution Every test consists of to parts, the pre-test and the main test. In our case, the pre-test is the integration itself. This means checking the versions of the delivered components and the completeness of the delivery, installing the components to be integrated in the test environment, starting them and checking if they communicate at all. Another part of the task is checking to make sure that the test bed works. This is a main responsibility of the test team. If the test bed does not work, the testing effort delays the project. After the pre-test comes the main test. For the main test we need welldesigned test cases. Test design methods How should we design effective test cases? The answer is: Use standard test methods, but, in addition, use fault based testing, heuristics based on the faults described above and any other expected faults. The main methods are: Data variation testing State-Transition testing Entity life history testing Data flow testing Transaction flow testing Fault based testing using heuristic rules Method 1: Data variation testing In principle, this is easy: The sender should provide a test data point for every equivalence class to be tested, and the receiver should be induced to send back a response from every equivalence class (especially error messages and exceptions). Boundary values, or values derived from domain analysis, should also be included. The difficulty is that we have no immediate access to the interface to be tested: We can only send data to the outside input channels of the aggregate to be tested. Basically we have to transform the data to be sent at the interface into data to be sent in from the outside. This may be difficult or impossible. Another way to test is manipulating the values at the interface itself (using instrumentation) and inserting selected values. Then it can be determined if they are processed correctly at the other side. This tests understanding of data values, but not the complete logic of both parts together. A good principle for robust interfacing is that the receiver should always accept more than the sender may send, it should be liberal or robust, processing correct input correctly, and notifying the sender of incorrect input. Method 2: State-Transition Testing (1,7) This approach applies to programs whose behavior is state dependent. Combine State-Transition diagrams of both components (although this is not always possible since the cross product could be huge). From the (combined) start state(s), traverse all states and all transitions. Test every feasible input (input equivalence class) in every state. For higher reliability, cover combinations of transitions (2,3, up to the number of states - switch coverage). If this is too complicated, check to determine if decoupling is possible. (Consider using queues, buffers, un-interruptable receiving process, or only one state to sense inputs). Currently, there are no effective methods to solve the problem of state explosion when combining state machines. This means this test must necessarily be incomplete if it cannot be totally automated. Method 3: Entity life history testing (4) All entities, whether they are simple data items or complex objects have a life cycle. This life cycle may be influenced by different components. They should be tested to verify that the entity is created, used, modified and destroyed properly. An example is the life of an airline reservation: After creation, it may be changed, cancelled, or used. It may be changed before cancellation or use, even several times. Even after using the reservation, it may be changed, if the flight is cancelled or rescheduled. The testing approach is to list all possible events and make test cases using event permutations and long scenarios. Starting at the creation event and moving to destruction, list all valid sequences of events affecting entitys life. Optional events should be tested at least in three ways: No occurrence, one occurrence, 5/8

Hans Schaefer and several occurrences. Method 4: Data flow testing (3,5) Data flow testing is a test design technique in which test cases are designed to execute definition and use pairs of variables. (9). Definition means writing a value, use means reading it. Data flow testing is easiest if there is a data flow diagram. In principle, data flow should not be tested, but analyzed by a static analysis tool, if one is available. If a tool is not available, then the most important data flows should be tested. Testing all possible data flows requires analysis of them, which may be an impossible task if done manually. Thus, in order to improve testability, the freedom for data to flow between components should be restricted! If data flow diagrams exist, then test design is easy: Test every connection between two boxes in the diagram and even connections back to itself. Test these with all equivalence classes possible for the data involved. But the test should also include failure of the first component, where after the second component tries to perform its function but finds no valid data. This finds wrongly implemented assumptions (that data are available and valid). Here is an example for flight reservation and check-in:

Test case 1: Get reservation, success, and then check in Test case 2: Get reservation, success, and then Get reservation once more Test case 3: Get reservation, unsuccessful, and then try to check in - check that no reservation exists (first component failed) Test cases 4 through n: Do this for all ticket types! (different equivalence classes) Method 5: Transaction flow testing This method can be used if transactions run in parallel and if one transaction creates child transactions, or if transactions wait for others to complete. Example: Baggage handling after check-in. Part of this testing is checking what happens with different orders of events and testing the applications of queues. Test cases can be constructed as follows: Varying numbers of children (0, 1, several, max, too many) Not allowed numbers of children Termination of children (exceptions) Different order of processing children 6/8

Hans Schaefer Termination of parent (what happens to children then?) Varying numbers of parents Event perturbations for parallel events give five tests: Only event 1 Only event 2 Event 1 before 2 Event 2 before 1 Both at the same time Special tests: Nothing at all happens (testing the timeout) Stress test: Repeated event arriving fast. Different time distances between events, different arrival times Testing queues: This is necessary every time any waiting scheme is involved Is it a queue or a stack? Several parallel queues with different (correct?) priorities? Empty queue? Full queue? Overflow? Correct time stamping?

Method 6: Fault based testing using heuristic rules This section lists a few rules special for integration test. They are derived from some of the common faults: The ones, which are findable by an easily described procedure. Make all data fields different, if possible. (In order to find wrong order of parameters) For default-values test with two test cases: The default and some other value. Try to make field values wrong at places they should not be. (Thus, if wrong order occurs, the program will probably protest. Example: Choose a date like 25 August 99, where 25 cannot be a month and 99 cannot be neither day nor month.) Try any values derived from common faults in this article. Choose input such that all outputs will be different from all inputs. This finds problems where output fields are not filled, or input values just copied to output fields instead of creating real outputs. Example: A component should compute the payment date and send it back. Payment should be 30 days in the future. In order to see if fields are not just copied, use a day in December, as all fields (day, month, year) will change in the output. In test sequences, the next test should have different values from the one before in every data position. This may detect that output is not generated but just copied from the case before. Integration strategies Integration strategies for sequential programs are (4): Top-down Bottom up Both ways (Sandwich) Neighborhood (2) Strategies for distributed and parallel programs are: Data flow (following data, from source to use, or all components accessing some data element) Functional groups (integrating functions that belong together, for example functions selectable from one menu) Layer-wise (typical for communication protocols) Skeleton (a general strategy, putting together a skeletal system where everything else is integrated into later). Strategies for object-oriented programs: Integration of inheritance hierarchies: Top-down. Test the super class first, then every subclass with the super class. Retest the working together of existing and new functions. (Consistency). Integration of calls between objects: Server first, then client. A class calling another class is tested after the called class has been tested by itself. 7/8

Hans Schaefer There is a severe problem with stubbing of used classes, but this can at least partially be solved by generator tools. Priorities for integration and how to save on it Integration should be well planned in advance. It is of utmost importance to tell early what should be integrated first, what later. Here are a few rules for what to integrate early: Important modules Unstable system parts Modules that make the test easier After a good component test larger groups of modules can be integrated at a time. Not just one and one module is added. This is a kind of Big-Bang-Test. It makes it more difficult to debug any failures, as it is not known which individual interface or module has been added last. But it saves time because less scaffolding is necessary. Thus, integration testing is dependent on the quality of earlier tests. Summary Integration testing is necessary but difficult. When interfaces are not standardized, integration varies widely. Integration strategy varies with system architecture. The most common problem introducing interface defects is human miscommunication and misunderstanding. In this article, many typical defects are outlines, together with some advice on how to find such defects. Literature and references 1) 2) 3) 4) 5) 6) Boris Beizer, Software Testing Techniques, Van Nostrand Reinhold, 1990. Paul Jrgensen, Examination of Integration Testing Strategies, ICSTEST-E, 2004, www.sqs.es Boris Beizer, Black Box Testing, 1995 Glenford Myers, The Art of Software Testing, 1979 Marc Roper and Ab.Rahim, Software Testing Using Design-based Techniques, Presentation at EuroSTAR 93 Leung, H.K.N.; White, L.J.: A Study of Integration Testing and Regression Testing at the Integration Level. Proceedings of the Conference on Software Maintenance, San Diego, IEEE, 1990, S. 290-301 7) Linnenkugel, U.; Mllerburg, M.: Test Data Selection Criteria for (Software) Integration Testing. Proceedings of the 1st International Conference on Systems Integration, IEEE, 1990, S. 707-717 8) Web site http://www.software-integration.ch (in German) 9) ISTQB glossary of testing terms. See www.istqb.org

8/8