2005-2006 Collard & Company/SQE V2.

2005-2006 Collard & Company/SQE V2.2

The next set of notes and slides shows variations on these general areas

2005-2006 Collard & Company/SQE V2.2

Response Time; measure how long the system takes to complete a task or group of tasks. Throughput; measure how much traffic passes through a point in the system, or how many tasks are completed, within a specified period of time and under a specified load. Availability: record the percentage of uptime for a system or component. Resource Utilization: monitor the levels of utilization of system resources. Capacity: measure the unused capacity of a system, to support additional work. Error Rate: measure the error rates of features or functions when the system is under normal or heavy load. (Fast response and high throughput are irrelevant if the user cant do his or her job.)

2005-2006 Collard & Company/SQE V2.2

Usage-Based: exercise the system in test mode the same way as in live operation, according to its operational profile. Standard Benchmark: use a standard work load rather than a user-specific one, when it is difficult to discern how a system actually will be used. Load Variation: vary the test load during the performance measurement, to reflect the typical pattern of how the real load ebbs and flows over time. Ramp-Up: measure the time needed to initiate a process. (For example, lets say an inactive network device is triggered when there is work for it to do, and it must be able to begin working within 50 milliseconds. The term ramp-up test is also used a different way, referring to the need to gradually ramp-up a test load to full strength, to avoid overwhelming a system during start-up.) Component-Specific: examine the performance and robustness of one system component (or sub-assembly). This can be done as soon as the component is ready, before other components are built and well before the fully integrated system is ready for testing. Calibration Check a system against mandated requirements such as those from a government regulatory agency. 2005-2006 Collard & Company/SQE V2.2 5

Scalability: investigate a systems ability to grow, by increasing the work load per user, or the number of concurrent users, or the size of a database; or the number of devices connected to a network, and so on. Duration or endurance: run the system for days or weeks in continuous operation in the test lab, to detect long-fuse bugs like slow memory leaks. Especially important for 24x7 always-on systems. Hot Spot: Focus a heavy, targeted load on a specific, limited portion of the system, in order to detect if it is a weak point. (We employ this in areas we suspect are vulnerable, like the weak links in a chain.) Spike: utilize intense spikes in the work load, for very short durations, to determine how the system handles abrupt increases in demand. (This tests whether the system can respond to rapid significant changes and re-direct the use of its resources, for example, through load balancing.) Breakpoint: increase the load until the system fails. (Or increase the load as much as feasible within the constraints of the test environment, if the heaviest feasible load is not sufficient to force the system to fail.)

2005-2006 Collard & Company/SQE V2.2

Deadlock testing: Running volume tests to detect if there are resource contentions in the system such as multiple requestors trying to access the same data base and causing the system to lock up the user request due to conflicts in areas like data base record locks. Degraded Mode of Operation: determine whether the system can still provide the (possibly reduced) level of service as expected, when not all the resources are working. (An example of a degraded mode test is to deliberately power down a server, in a server cluster with redundant application servers, and attempt to continue normal operation.)

2005-2006 Collard & Company/SQE V2.2

2005-2006 Collard & Company/SQE V2.2

Once an estimate has been determined for the number of transactions per second, network traffic and users that the application/system can be expected to attract etc., the next step is to build a load profile for these visitors; i.e., what they are doing on the application/system and what characteristics they have that might affect the performance of the application/system under test. These are the characteristics used for Host, Client/server, Web types of applications. However, the same process can be applied to embedded and other types of internal applications/systems. The key differences are in the users. For embedded types of systems the users are the devices and and components that provided the inputs and accept the outputs from the application.

2005-2006 Collard & Company/SQE V2.2

If you have an existing application you can gather numbers from there. If you are building a new application you will either have to use predictive numbers from the marketing team (or other source) or you will have to purchase industry information from one of the many statistical information sources such as keynote.com, statmarket.com, Neilsens netratings.com, etc.

2005-2006 Collard & Company/SQE V2.2

10

If you sample a specific time range and you have activity in many time zones you may skew the results of your samples. Here there is dead time at the beginning and end of each day. A typical work day is not a steady load. There is a ramp-up in the morning a drop off before lunch and a ramp-up to the end of day followed by a fall off. Including all these dead times, plus the dead time between 9PM eastern and 7Am eastern will even further skew the results of the samples.

2005-2006 Collard & Company/SQE V2.2

11

Example of load calculation based on marketing projections

How many web site visits or sessions are expected per month? If the site can expect 100,000 (regular users) each have 2 visits or sessions a month = 200,000 monthly sessions for monthly users. Casual visitors add another 15,000 sessions monthly, for a total of 215,000 sessions per month. How many web site visits or sessions are expected in a typical hour? 215,000 monthly sessions divided by 30 days = 7,167 daily sessions on average. 90% of these sessions occur between 10.00 am and midnight. 7,167 daily sessions multiplied by 0.9 means that 6,450 sessions occur between 10.00 am and midnight, a span of 14 hours. 6,450 sessions divided by 14 hours per day = 461 hourly sessions on average. Lets say 500 sessions or visits per hour to be conservative. How many web site visits or sessions are expected in the peak-demand hour of a typical day? 7,167 daily sessions multiplied by 20% = 1,434 sessions on average during the peak hour of a typical day. Lets say 1,500 sessions or visits in a peak hour, to be conservative. The answer to this question also can be calculated another way. IF are assumptions state that the number of sessions in a peak hour is expected to be 3 times the number in the average hour of the day (including only the hours from 10.00 am to midnight, and ignoring the other ten hours). Using this approach, 500 sessions or visits per hour multiplied by 3 yields an answer of 1,500 sessions in a peak hour. These two answers are consistent. What if they were not? In that case, there would be some incorrect data, conflicting assumptions or a simple arithmetic error in our calculations. Wed have to go back and debug them. The two answers are consistent here because the assumptions are reasonably consistent, though not exactly so.

2005-2006 Collard & Company/SQE V2.2

12

The number of users on the site will vary based on many factors y Type of site y Offerings y Seasonal events y Special events y Etc. Each group/type of user will perform some mix of activities with the application/system. It is the mix of users and their activities that create loads on the system. This same type of profile can be created for an embedded system using devices, components, interrupts, etc. in place of user types.

2005-2006 Collard & Company/SQE V2.2

13

Understanding Concurrence
What is the expected average number of concurrent events? E.g. If 500 sessions occur in a typical hour Divided by 4 (assuming each session lasts for 15 minutes, and assuming the visits are evenly distributed over time during an hour), this means there are 125 concurrent visitors on average. What is the expected peak number of concurrent events in a typical hour? E.g. At any given point in time, each visitor can have only one visit. (Over time, one visitor could have several visits as that person logs off and then on again.) So for this question we can assume that the number of visits and visitors are the same thing. There are 500 sessions in a typical hour and there are 125 overlapping sessions or concurrent visitors on average. This 125 is the answer if the sessions are spread evenly over the hour. But the sessions may not be evenly spread, so 125 is not the only possible answer.

2005-2006 Collard & Company/SQE V2.2

14

If instead all 500 sessions start precisely at the beginning of the hour and terminate at the end of the first 15 minutes (possible but unlikely), then the answer is that there are 500 overlapping sessions during that 15-minute period. In the remaining 45 minutes of the hour when no sessions are active at all, the answer at that time is zero sessions. The peak number of concurrent visitors during this hour, which is 500, is the highest or maximum peak that is ever possible with 500 sessions. The lowest or minimum that the peak number of concurrent visitors could ever be, if there are 500 sessions within the hour, is 125. This minimum peak occurs when all 500 sessions are exactly 15 minutes long and are equally spread across the hour: that is, a quarter of them start every 15 minutes into the hour. The answer to this question depends on how the 500 sessions are distributed during the hour. Wed need to know the possible distribution patterns and the probabilities of the different distribution occurring, in order to calculate the expected (modal value) of the peak. In the absence of better information, lets assume that 312.5 peak concurrent visitors (the average of the lowest and highest possible peaks, 125 and 500) is the most suitable answer. We should try to obtain a better sense of this before proceeding with our testing.

2005-2006 Collard & Company/SQE V2.2

15

Think time is generally considered to be the time taken from when the client receives the last data packet for the GUI/Web page currently being viewed and the moment that the client requests a new Screen/page. Different users/clients take different amounts of time to think about the information that they have just received; some users race from page to page, others contemplate what they have just read, and some go out for lunch. Clients can be categorized as follows: y Aggressive: Client only thinks about their most recent screen/page for a few seconds y Casual: Client typically takes from 15 seconds to several minutes to request a new GUI/Web page y Out to lunch/gone for coffee: Client has not moved on to another application/Web site, but has become preoccupied with another task. Expects to be able to return to the GUI/Web page minutes or even hours later and pick up where he/she left off. The general behavior of the users of the application/system will affect the concurrency. More aggressive uses will have higher concurrency, more casual will lower the concurrency.

2005-2006 Collard & Company/SQE V2.2

16

Unless the various production servers are going to be dedicated to supporting the application (E.g. the web server may be dedicated but the database server may be shared with several Client/Server applications), ensure that the servers in the system test environment are loaded with appropriate background tasks to compensate for the other tasks that they will need to support in the production environment. When designing a load to test the performance of a system consider what additional activities need to be added to the test environment to accurately reflect the performance degradation caused by background noise created by: y Other applications that will be running on the production servers once the application under test moves into production. For instance, performance monitors, intruder detection systems, or log analyzers y Other network traffic that will consume network bandwidth and possibly increase the collision rate of the data packets being transmitted over the LAN and/or Wan (Internet). For instance, company email using the same ISP connection

2005-2006 Collard & Company/SQE V2.2

17

2005-2006 Collard & Company/SQE V2.2

18