LoadUIWeb Pro

by SMARTBEAR

A Guide to Effective Load Testing

Contents
Performance Maturity Assessment............................................ 3
Are your business associates planning your Performance Intervention?........ 3

What Makes a Good Load Test Report..................................... 5

How ready are you for heavy load on your website?......................................... 8 Transactional testing................................................................................... 9 Geographic diversity................................................................................... 9

Secrets of Effective Performance Analysis.............................. 10

Endurance testing..................................................................................... 10 Stress testing............................................................................................ 10 Time to first byte.............................................................................................. 12 Response transfer time.................................................................................... 12 Gaps between requests................................................................................... 12

The Secrets of Proactive Performance Management............. 13 Summary................................................................................. 16

LoadUIWeb Pro
by SMARTBEAR

Performance Maturity Assessment

Are your business associates planning your Performance Intervention?
We have seen it happen in many organizations at some time or another, when the blissfully-unaware Performance Manager finds out that their current approach to performance monitoring is disappointing their organization. Slow response time when web pages are trying to load, broken links and redirects, timeouts, all are perceived by the end-user as lousy performance, but the internal organization does not have a handle on where the bottlenecks are occurring. The Performance Manager is invited into a conference room, and the dialogue often sounds like: Help Desk Manager: Why are we always hitting the same issues? Cant we fix the app to stop doing that? The Network Manager: It is NOT a firewall issue. The Development Manager: Its not the application either! Your Boss: What are you DOING about this? The Performance Manager, without the benefit of solid performance analytics, is forced to simply acquiesce to the charges. Are you in danger of an Intervention? Take our performance quiz to see if you might be headed for an ambush in a conference room. Consider the following point system: 1. Clueless These things are happening on my site, but I have no method to anticipate them, plan for them, or guard against them. My help desk is my alert system 2. Foggy I have some idea what is hitting us, but I have no way of compiling different indicators and statistic into something useful. We

LoadUIWeb Pro
by SMARTBEAR

use end-of-month issue analysis to decide what will be our focus for the next month. 3. Handle I have a basic grip on what we see day-to-day, I have a process I can use to define Root Cause Analysis on the big issues, but I am still reacting to these issues based on business impact. 4. Driver I have a system that is collecting useful indicators automatically, and it is giving me enough data on which to base strategic decisions. 5. Leader My decisions are second-nature, based on the full scope of information I am getting. This information and analysis is at my fingertips in real-time, making it easy to fix issues before my customers or my peers even see degradation in performance.

Rate your current System Performance with a 1 to 5 on the following questions. Question 1: How would you rate your ability to draw meaningful conclusions on your systems performance based on the information presented in the reports you currently use? Question 2: How comfortable are you that your system and its individual components are certified against peak loads, and times of stress on the network? How ready are you for your next peak interval? Question 3: How easy is it for you to quickly assess the root cause of an issue, isolating a problem down to a component or procedure? Question 4: How are improvements driven on your system, are they anticipated, planned and implemented based on metrics, or are they frequently deployed in a particular response to an outage or customer gripe?

LoadUIWeb Pro
by SMARTBEAR

Question 5: How comprehensive a Service Level Agreement are you willing to offer your customers? On what would you base your numbers, and how would you argue them when your customer says you have not achieved them? Based on the described point system, calculate your score to rate your approach to performance management. 25-20 There is no way they could surprise you with an intervention, and they know that even if they tried, you would be able to turn the tables painfully back on them 19-15 You are in reasonable shape, but something could surprise you 14-7 Some positive steps in Performance Management could make this a whole lot easier on you Under 7 Do you have a minute to talk in this conference room? And Performance Managers, if you have had this forwarded to you by someone <gulp>, you might already be headed into a session. Be prepared. In this series, we will discuss each of the five sectors of Performance Management, and will present you with ways you can raise your score, make your application more bullet-proof, and hopefully keep you out of the dreaded conference room intervention.

What Makes a Good Load Test Report

Load testing can be a complex activity, from defining the tests accurately to running them effectively. Most conversations about proper load testing focus on those activities but an often overlooked part of the discussion is how effective your test reports are. Being able

LoadUIWeb Pro
by SMARTBEAR

to accurately interpret the results of your load test may help prevent production problems later. Performance reporting should follow the same basic rules of reporting that we learned in elementary school:

Who-What-When-Where-Why-How A concise opening

statement of theme

And some concrete

argument to back up the theme statement In essence, a performance reporting system should be

How would you rate your ability to draw meaningful conclusions on your systems performance based on the information presented in the test reports you currently use?

able to deliver a high-level overview of how a web site is performing under load, but it should also contain detailed visibility into the internal structure of the site and its infrastructure. Reports should illustrate the context and highlights of performance so clearly that anyone could interpret them without having to be an expert about the data and metrics used to reach these indicators. The system should also have some degree of customizability, allowing the information to be re-contrasted as needed - often the most compelling analysis involves only the interaction of two or three key elements of the data gathered and the report should be able to be re-formatted to showcase them. The timeliness of this information is paramount, as near-instant availability of the metrics and the ability to concentrate on different testing timeframes, will allow the proper context of an issue to be stressed. It is also important that the load test reporting system be able to hold and reproduce test results of multiple passes of a given test, as a method to baseline results over time and thus gauge improvement.

LoadUIWeb Pro
by SMARTBEAR

A good load test report should be able to show: Who: A good reporting system identifies and reports on the Key Performance Indicators (KPIs) of the website youre testing. For example: Fulfillment numbers are not where we would like., Shopping cart abandonment is too high, Customers are complaining of slowness. The reports should focus on KPIs as success criteria. When: A good report should also be able to address a discrete time duration, be it an instant test of a single component, or a duration test of multiple scenarios on a website happening over a weekend or longer period. As stated previously, regular testing of a website might also be compiled as a series of referencable reports, to baseline performance of the site and its infrastructure. Where: There are two where components to consider: a physical version of where that alludes to the infrastructure and hosts of the website, and a virtual where, indicating at what step of the application the issues are being seen.

With a product like SmartBears LoadUIWeb Pro, a distributed

platform of load testing can be deployed using LoadUIWeb Pro Remote Agent Services at multiple locations on your network, inside and outside of your firewall, to contrast load testing results, and physically pinpoint where bottlenecking is occurring.

By setting alerts on parameters such as such as Quality of

Service or Content Validation, analysis of the steps in the application can be graphically represented in the report as errors, allowing for quick identification of logically where in the application the issue is being seen.

LoadUIWeb Pro
by SMARTBEAR

Why: (More correctly Why Not?) Reporting should have some method to plainly identify the reason why the KPI objectives are not being met, by indicating an issue on an individual object or subsystem Hierarchical systems, such as a compiled list of the poor performers or a graphical representation of the performance of each request made to the website can quickly determine a course of action for remediation. SmartBears WebLoadUI offers both a Top 10 list of poor performers, and a waterfall representation of each of these Top 10, for ease of diagnosis. With the proper reporting, the overall status and health of the website and infrastructure is quickly comprehended by all of the stakeholders. Such a concise view can suggest possible areas to address, but more importantly gives the intelligence into the system for the Performance Manager to base their decision. However, the ease of generating a concise report, pre-formatted with a summary section and compilation of the relevant data, is of most concern. Using WebLoadUI, this step has been automated, so that a report is produced at the end of every test run, using the data contained in the tests unique log file.

How ready are you for heavy load on your website?

Load Testing is an important subset of any overall Performance Management strategy. It is a technical investigation done to determine the scalability, speed, and/or stability characteristics of the system under test. Load testing is crucial for assessing if the software solution, including the server equipment on which the solution runs, the infrastructure network where the servers exist, and the web access routes and equipment, will react with the necessary speed when subjected to normal or above-normal amounts of user traffic. In its infancy, load testing entailed events where multiple people accessed an application simultaneously from distributed computers. While this may successfully simulate a stressful event on a system, it lacks the qualities that are crucial to a true evaluation of a system

LoadUIWeb Pro
by SMARTBEAR

under load. Through load test automation using tools like LoadUIWeb Pro, we can create and control populations of virtual users that can simulate valuable aspects of load testing: Transactional testing

How comfortable are you that your system and its individual components are certied against peak loads, and times of stress on the network? How ready are you for your next peak interval?

To fully test an application, merely accessing the landing page of a website is not going to give an adequate assessment of how well the supporting pages of that site are performing. A better test is to simulate a virtual population running a transaction comprised of a series of steps, each of these steps possibly consisting of multiple requests and responses to both your host server as well as third-party sources, should be able to be performed by the virtual user population. Further, as different users to a website may follow differing sets of steps on that site, comprehensive load testing should include running multiple test scenarios concurrently. These can be further varied by using multiple browser styles, transmission bandwidths, individual session IDs, and other variations to the test parameters to most closely simulate a real world experience. Geographic diversity For a website that is meant to be accessed from outside of the firewall, testing that only occurs on the internal infrastructure will overlook some important possible bottlenecks, such as single sign-on mechanisms or other security devices. Even a singular perspective inside of the firewall, such as initiating all virtual users from just one location, will show a more myopic result than testing done from multiple separate locations in the network. It is therefore important to be able to place remote instances of your virtual user population, but be able to configure, control and correlate data from them with one central management console.

LoadUIWeb Pro
by SMARTBEAR

Endurance testing Endurance testing is an assessment lasting many hours or days, and is a way of testing the durability of the solution and its components. These exercises can reveal the occurrence of memory leaks or similar system degradations on the host servers over time. Here, the ability to draw server-side metrics into the test results can be greatly beneficial to the assessment of the systems durability in the production environment. Stress testing Testing in this phase entails pushing the website and its infrastructure well past its anticipated normal loads, as a way to assess the upper limit of system capability, finding the theoretical capacity for a solution. This type of testing should be used to not simply knock over the service, but to look at the elasticity of the solution, and determine if it can recover from being pushed past infrequent stressful loads. In addition to finding that upper limit of performance, stress testing can also be useful for right-sizing your system, and being able to justify future build-out plans and timeframes. Load testing is vital to your Performance Management planning. Having the answers from these kinds of tests allow the tester to really feel the capabilities of their infrastructure, and more importantly, to instinctively recognize the signs of a website undergoing a stressful incident and make intelligent decisions.

Secrets of Effective Performance Analysis

Successful Performance Analysis depends on the ability to identify bottlenecks in the application or system infrastructure. A bottleneck could be caused by any object on a page that is taking longer than other page objects to fully load, or it could be an overloaded segment of the network or a security process that is delaying the browsers requests and responses.

LoadUIWeb Pro
by SMARTBEAR

10

Analysis and tuning is not a one and done type of event, but rather a cyclical process of evaluation and elimination of performance bottlenecks, using iterative load testing of the application. Every pass of a load test could uncover new issues in the system, as the elimination of one larger issue in a previous pass might unmask other issues. The errors found should become more granular with each testing cycle and subsequent remediation, so the results of iterative testing are stronger, faster applications with fewer performance defects. If we know that an application has been tuned previously, new degradations to the system can be easily identified on the latest run of the test. Waterfall charting can be a useful tool for identification of root cause.

How easy is it for you to get to the root cause of an issue, isolating a problem down to a component or procedure?

LoadUIWeb Pro
by SMARTBEAR

11

Through the graphical representation of each request against time, we can easily see the objects causing degradation in the test. The different sub-timers included in the graphed request, such as time to first byte and response transfer time, can give us insight into suggested remediation for the object. Here are some things to look at when performing any kind of load test analysis.

Time to first byte

A long first byte time indicates that the host server is delayed in getting the beginning of the requested information back to the browser. This is most frequently caused by an overloaded host server, or a congested pathway between the server and the browser. Remediation on an object with a long first byte value might include having it served from a different host, moving the serving host to another location in the infrastructure, or possibly optimizing the object through the use of a Content Delivery Network (CDN)

Response transfer time

LoadUIWeb Pro
by SMARTBEAR

12

A long transfer time might indicate that the object itself might be oversized for the application. This could be rectified by allowing the application to call down the information contained in this object through a series of smaller requests, relying on the nature of the browser to perform these requests in parallel.

Gaps between requests

In the waterfall report, it might appear that there is a short length of time between the conclusion of one request and the beginning of processing for the next request. The most frequent reason for this is that the browser requires extra time to establish a connection to the target web server. This can be tuned in the application.

The comparison of a recent waterfall chart to a historical version can give you the necessary detail for your first assessment of root cause. Also, by graphically seeing the components and infrastructure used by the site, you can correlate information he has received from other sources regarding network activity or recent outages, which might also suggest root cause.

The Secrets of Proactive Performance Management

Reactivity to issues is often an expensive and imprecise method for dealing with outages or degradations in service. Often your customers are citing that an issue is chronic, they are demanding to know what concrete steps are being taken to fix the issue, and the technology

LoadUIWeb Pro
by SMARTBEAR

13

team feels pressure to provide a quick fix. Reactivity saps the teams productivity, as triage is no longer based on strategic improvement, but rather on the seat-of-the-pants, If it bleeds, it leads approach to daily issues; the team never seems to get ahead of problems. Possibly, equipment and resources purchased for testing of new solutions are now being repurposed into the production environment, resulting in accounting nightmares. In previous posts, we have discussed the value of stress testing and a comprehensive reporting system in finding and reducing bottlenecks, and in tuning application performance. This testing can have a strategic purpose as well, as we can use it to determine triggering criteria for expansion or upgrade of the system. By proactively testing capacity limits on the system, you can accurately predict the thresholds for your applications.

How are improvements driven on your system, are they anticipated, planned and implemented based on metrics, or are they frequently deployed in a particular response to an outage or customer gripe?
Stress testing allows you to feed what if situations like the following to an application:

What if we experience 300% growth instead of our planned

200% growth?

What if interest in our site goes international? We are adding another 2 steps to the checkout process, what
will it do to our overall session times? All of these situations can be modeled with a robust load testing tool, like SmartBears LoadUIWeb Pro, which allows instances of the testing platform to be deployed on remote servers inside your network or in the cloud for testing of your site from anywhere in the world. By allowing this flexibility, you can test a situation virtually identical to what the customer will experience on their own browsers in production. Further,

LoadUIWeb Pro
by SMARTBEAR

14

by knowing at what stage the new system will outgrow current capacity, infrastructure upgrades can be intelligently scheduled to avoid busy seasons for the application. Of course, load testing can, and should, also be used as an internal quality assurance test when system improvements are made. As we have outlined, load test reporting can provide a baseline if run at regular intervals. By running a short load test directly after a maintenance interval has concluded, we can immediately gauge the integrity of the system as it returns to service, using the user scenarios to fully test the application and assuring that all features are in a ready status for real customer traffic. If a key portion of the application is still down, it is far less impactful to discover it internally than to have a customer open a trouble ticket regarding a failed transaction. Running an internal load test can also be beneficial at this time so we can quantitatively compare the results of the test against the baseline, giving us immediate knowledge of the performance changes we just experienced. There are definitely times when changes on the system must be made on the fly in response to an outage. However, a system that has a level of change control in place has a greater chance of stability over the long term.

How comprehensive a Service Level Agreement are you willing to offer your customers? On what would you base your numbers, and how would you argue them when your customer says you have not achieved them?
The true benchmark of Performance Management success is the ability to set a Service Level Agreement with your customer. An SLA not only defines a contracted level of service quality, but it represents the confidence that both sides have in their solution, and the value they place on partnership. To be willing to offer an SLA, you must be confident enough in your preparations:

LoadUIWeb Pro
by SMARTBEAR

15

 You have sufficiently hardened your website through analysis of

the components and infrastructure. The application has been reviewed down to the object level, and you have optimized your application through redundancy and load balancing. You have the baseline data to prove it as well.

You have performed stress testing on your system, and have

documented the warning signs of a network in a stressed condition. You have shared your findings with your team and coached them on these warning signs, and they know the steps to take when these signs appear. Further, you know when your busy times are on the website, and have eliminated the extraneous activity on the network at those peak times to guard against contention.

Your infrastructure has been right-sized for the traffic you are
expecting to see in the next year. Upgrades to the infrastructure have already been planned out for that time down the road when you foresee you will expand. Further, you have a plan for analysis on the new application code when Development says it is ready for pre-deployment testing, and your Development team knows the standards to which you will be holding their work.

You have baselines of the system performance over time and

you can determine by week, month and quarter your averages for response times, based on data from multiple locations inside your network, and if appropriate, from outside your network as well. What are the key metrics, the issues most important to your customer? Though there will be some of these metrics that will fall outside of Performance Management influence (such as metrics pertaining to Service Desk ticket performance), the data provided from regular load testing can be used as a basis for creating a very workable SLA.

LoadUIWeb Pro
by SMARTBEAR

16

Summary
We have illustrated how load testing can positively impact each of the five sectors of Performance Management, and offered suggestions on with ways you can raise your performance score by making your application stronger. The key take-away offered here is that load testing is not a singular event to be done only once during a software development cycle. Rather, it is a continuous process that is constantly employed to find bottlenecks on our applications and infrastructure. Software applications and the networks they run on are constantly improving and evolving, and we should never lose sight of the value of a regular performance check-up on them to keep them in top form.

LoadUIWeb Pro
by SMARTBEAR

17

About SmartBear Software

More than one million developers, testers and operations professionals use SmartBear tools to ensure the quality and performance of their APIs, desktop, mobile, Web and cloud-based applications. SmartBear products are easy to use and deploy, are affordable and available for trial at the website. Learn more about the companys award-winning tools or join the active user community at http://www.smartbear.com, on Facebook or follow us on Twitter @smartbear and Google+.

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com 2012 by SmartBear Software, Inc. Specifications subject to change. SB-AQt-030513-WEB