This article is provided courtesy of STQE, the software testing and quality engineering magazine.

Tools & Automation

Three Web load testing blunders, and how to avoid them

by Alberto Savoia

TRADE

se c
FROM A WEB TESTING EXPERT
QUICK LOOK
re t
■ Why concurrent users is
s
a misleading metric
■ The impact of user abandonment
■ Accurately analyzing results

L
oad testing has rapidly become one of
the top QA priorities for companies with
mission-critical Web sites. How many
users will your site be able to serve while
still maintaining acceptable response times?
That’s an indispensable piece of information for planning marketing cam-
paigns, estimating IT budgets, and basic delivery of service. And yet practi-
cally all Web site load tests are seriously flawed—because they all seem to
make mistakes that have a huge impact on the accuracy of the test and the
reliability of the results. Let’s look at three of the biggest and most common
Web load testing blunders, and how to avoid them.
54
www.stqemagazine.com STQE May/June 2001
 This article is provided courtesy of STQE, the software testing and quality engineering magazine.
1. Misunderstanding
Concurrent Users
The first blunder centers on the wide-
spread use of the concept of concur-
rent users to quantify loads and de-
sign load tests. I find it amazing that
concurrent users is the prevailing
metric when it comes to describing a
Web load, because the approach is
riddled with obvious problems. So
many problems, in fact, that I could
probably write a book about it—but
fortunately for you, the space con-
straints of this article force me to be
concise.
In that spirit, I’ll focus on the
main problem with concurrent users:
the number of concurrent users
shouldn’t be seen as input for a load
test run, but as the result of a number
of factors. And yet whenever you read
a load testing plan, you probably see
something like this: “The Web site will
be tested with a load of 1,000 concur-
rent users.”
To explain why this is the wrong
way to look at things, let’s do a simple
thought experiment.
Let’s assume that three users—
Alan, Betty, and Chris—visit a finan-
cial Web site to get stock quotes on
three consecutive days, and that each
of them plans to get three different
stock quotes. None of our three users
knows each other, so their actions on
each day are completely independent
and unsynchronized, like those of
most Web users.
On Monday, Alan starts his session
at 12:00, Betty at 12:01, and Chris at
12:02, and each of their sessions con-
sists of four page requests (Home
Page→Quote 1→Quote 2→Quote 3).
Each of our users will, after receiving
each page, spend 10 seconds looking
at it before requesting the next page
(in load testing parlance, this is called
think time). If the Web site takes 5
Receiving a page Reading a page
Alan
Betty
Chris
12:00 12:01 12:02 12:03
Monday
FIGURE 1 Users’ concurrency influenced by page request times
May/June 2001
seconds to respond to each page re-
quest, Alan’s, Betty’s, and Chris’s ses-
sions will not overlap. Alan will have
received and read his four pages be-
fore Betty’s session starts, and Betty
will be done with her session before
Chris starts his (see Figure 1). In this
case, it’s accurate to say that Chris’s,
Betty’s, and Alan’s sessions are not
concurrent.
The number of
concurrent users
shouldn’t be seen as
input for a load test
run, but as the result
of a number of factors.
Let’s now assume that by
lunchtime on Tuesday the site has
slowed down a little bit. Now instead
of taking 5 seconds, each page re-
quest takes 10 seconds, but the time it
takes the user to read it remains con-
stant. Each session will now last 80
seconds instead of 60. Alan’s and Bet-
ty’s sessions will overlap, and so will
Chris’s and Betty’s. In this case we
will have some periods of time with
one user and some periods of time
with two concurrent users.
On Wednesday, the Web site
slows down even more. Now instead
of taking 10 seconds, each page re-
quest takes 30 seconds. Each session
will now last 160 seconds. Alan’s and
Betty’s sessions will overlap, as well
as Chris’s and Betty’s, and for a while
all three sessions will also overlap, re-
sulting in three concurrent users.
Two concurrent users Three concurrent users
12:00 12:01 12:02 12:03
Tuesday
STQE
As Figure 1 illustrates, the num-
ber of concurrent users is not a mea-
sure of load. The load was identical in
all three cases: three users, starting a
minute apart, viewing four pages each
with 10 seconds of think time per
page. The number of concurrent users
was a result: a measure of the Web
site’s ability to handle a specific load.
A slower Web site resulted in more
concurrent users.
When it comes to measuring Web
site scalability, it turns out that the
number of concurrent users is not
even that useful as an output result of
load testing. If the Web site is some-
what slow, the number of concurrent
users increases. If it’s really slow, a
lot of real users will abandon it, thus
reducing the number of concurrent
users. (More on user abandonment in
the next section.) But, on the other
hand, if a Web site is very fast ses-
sions will complete more quickly, also
reducing the number of concurrent
users. You see the problem?
The bottom line is that concur-
rent users is a dangerously mislead-
ing metric that can be misused in so
many ways that it’s practically guar-
anteed to give you questionable re-
sults.
So what should you use in its
place? For describing an input load,
my favorite metric is user sessions
started per hour. This metric offers a
major advantage: the number of user
sessions started per hour is a con-
stant, unaffected by the performance
of the Web site under test. If you’ve
launched a big marketing campaign
and expect to draw a peak of 10,000
user sessions per hour to your Web
site, those users will come to the Web
site and request the first page, regard-
less of whether or not your site can
handle them. Whether they complete
their sessions or not, however, de-
pends on the Web site’s ability to sup-
12:00 12:01 12:02 12:03
ANNIE BISSETT
Wednesday
55
www.stqemagazine.com
 This article is provided courtesy of STQE, the software testing and quality engineering magazine.
port the load with acceptable re-
sponse time—and that’s precisely
what you want to find out with a load
test, isn’t it?
2. Miscalculating
User Abandonment
Let’s move on to another constantly
overlooked load testing factor that
has a huge impact on load testing re-
sults: user abandonment. Have you
ever left a Web site because its pages
were loading too slowly? Unless you
have the patience of a Benedictine
monk, I am sure you have. And since
many people seem to have the atten-
tion span of a chipmunk when
they’re on the Internet, abandoned
sessions are extremely common.
Considering that the magnitude of
user abandonment is going to be
quite high at the levels that are likely
to be used in a load test, and that this
user abandonment will have a signifi-
cant impact on the resulting load, I
find it very surprising that most Web
load tests don’t even attempt to sim-
ulate this ver y common behavior
with any degree of realism.
You should simulate user aban-
donment as realistically as possible.
If you don’t, you’ll be creating a type
of load that will never occur in real
life—and creating bottlenecks that
might never happen with real users.
At the same time, you will be ignor-
ing one of the most important load
testing results: the number of users
that might abandon your Web site
due to poor performance. In other
words, your test might be quite use-
less.
User Abandonment Matrix
% %
ABANDONMENT
0–5
PA G E T Y P E SECONDS
Home Page 0%
Stock Quote 0%
Stock Transaction 0%
Account Information 0%
TABLE 1 Estimated user abandonment rates for different page types
56
www.stqemagazine.com
To explain why this is, let’s per-
form another simple thought experi-
ment.
Let’s assume that you want to test
your Web site at a load of 10,000 con-
current users (just testing—I hope
you’ve banished concurrent users
from your vocabulary)…As I was say-
ing, let’s assume you want to test your
Web site at a load of 10,000 user ses-
sion starts per hour. Let’s also as-
sume that if the home page response
time is less than 5 seconds, no users
will abandon the Web site because of
performance. We’ll say that as home
page response time increases farther
and farther away from 5 seconds, more
and more users will abandon. For ex-
ample: 30% will abandon between 5
and 10 seconds, 45% between 10 and
15 seconds, and so on. Let’s also as-
sume that each complete user session
consists of four pages.
In one scenario, let’s assume that
the Web site under test is able to han-
dle the 10,000 user sessions per hour
with a home page response time con-
sistently below 5 seconds. In this case,
all 10,000 users will complete their
sessions and the Web site will have
served a load of 40,000 pages.
In another scenario, let’s assume
that the Web site is not as scalable.
When it’s confronted with a load level
of 10,000 user session starts per hour,
the response time increases to 15 sec-
onds per page. What happens in this
case is not as straightforward. Initially,
as the performance deteriorates, some
users will start to abandon; but since
this abandonment reduces the load,
the performance will start improving
again. As performance improves, few-
% %
ABANDONMENT ABANDONMENT
5–10 10–15
SECONDS SECONDS
30% 45%
15% 25%
0% 0%
5% 15%
STQE
er additional users will abandon—at
least until the load increases again and
the cycle repeats itself.
It’s clear that in this example not
all sessions will conclude happily; a
number of people will abandon their
session. That’s a very important result,
a critical piece of information that your
load test should help you discover. Af-
ter all, aren’t you doing a load test to
ensure that the Web site can serve a
specific number of users at a specific
performance level? User abandonment
is a very important metric and a clear
indication that the Web site is not able
to operate satisfactorily at that load
level.
Considering how critical aban-
donment is, it’s surprising that most
load tests are designed to use scripts
that simulate abandonment only in ex-
treme cases. (I most commonly hear
60 to 120 seconds timeout. Unfortu-
nately, I don’t know anybody in my
immediate and extended family with
that kind of patience; actually, I don’t
know anybody in my area code with
that kind of patience.)
So, how do you implement real-
istic user abandonment? Here’s a
simple approach you can use to get
started. When you write your load
testing scripts, determine what the ac-
ceptable page response times would
be for each type of page, and what the
likely abandonment rates are going to
be when they are exceeded. Then pro-
gram each simulated user script to
terminate if the response time for a
page exceeds its pre-specified thresh-
old.
Table 1 shows a sample matrix in
which to map out the possibilities.
ABANDONMENT
15–20
SECONDS
75%
45%
15%
35%
May/June 2001
 This article is provided courtesy of STQE, the software testing and quality engineering magazine.
This matrix takes into account the
fact that people expect home pages to
load very quickly, but are more toler-
ant of pages that they assume require
more work for the servers (e.g., com-
pleting a stock transaction).
This kind of table is actually a
great way to get your peers and
management to formally discuss and
document the performance expecta-
tions for your Web site. As we will
see a little later, this type of model
also lets you create much better load
testing reports—since you will be
providing information that’s going
to be significantly more meaningful
and relevant than what you’d be able
to deliver without a simulation of
user abandonment.
Accurately Simulating
Abandonment
Whenever I talk about user abandon-
ment, people agree with its impor-
tance, and with the goal of simulat-
ing it as realistically as possible. But
how can they do that? They have no
idea what the user abandonment be-
havior might be for their Web site,
and what percentages they should
put in their user abandonment ma-
trix. This is a very valid concern
that, fortunately, can be addressed in
several ways, depending on how ac-
curate and realistic you want to be.
If you want to be as accurate
and realistic as possible, you could
set up your Web site to redirect a
percentage of your visitors to a
slower mirror version of the Web
site—one that’s identical to the main
site, except that it’s artificially
slowed down. This is not as compli-
cated as it sounds, and it can be ac-
complished in a number of ways; the
simplest method might be to add to
each page some simple JavaScript
code whose only purpose is to add
an artificial delay of several seconds
before displaying the content of the
page. Let’s walk through a very sim-
ple example of how you might use
this approach.
Assume that you’ve set up your
Web site so that 90% of the traffic is
sent to the regular server, while the
other 10% is routed to a server iden-
tical to the first, but in which the
home page has been artificially
slowed down by, say, 5 seconds. Run
your Web site in this configuration
May/June 2001
for a few hours, or a few days, until
you have enough sessions to make
the results statistically significant. (I
would set a minimum threshold of
1,000 user sessions before drawing
any conclusions.)
After this period of time, using a
log file analyzer on both the regular
server and the slowed-down server,
take a look at what percentage of
Abandonment is not
only an interesting
result in itself; it also
has a major impact on
what parts of your Web
site will get stressed
under real conditions.
sessions requested the home page
and then proceeded no further (i.e.,
all the sessions that abandoned after
the home page—I call them home-
alone pages). If the percentage of
home-alone pages is 6% for the reg-
ular server and, say, 20% for the
slowed-down server, you must con-
clude that, since everything else was
equal, the increased abandonment of
14% had to be caused by impatient
users not putting up with an addi-
tional delay of 5 seconds on the
home page.
This approach does have a
downside: it requires some effort,
and you will have caused some in-
convenience for 10% of your Web
site visitors (and may have lost a few
of them). But for some Web sites the
cost of this type of experimentation
will be easily justified if it leads to a
better understanding of customer
behavior—an understanding which
can then be applied to maximize the
success of the overall Web site.
Frederic Haubrich, the Chief Web
and Technical Officer for the Web
site Hooked on Phonics™
(www.hop.com), for example, regular-
STQE
ly tests new Web site designs and
navigation options on a small per-
centage of users to determine if the
new design increases or decreases
the percentage of sessions that re-
sult in a purchase. New 40KB graph-
ics for a home page might look
great, but will the improved aesthet-
ics compensate for the additional
loading time?
The investment required for this
approach is appropriate for a large,
mission-critical Web site, where just
1% abandonment may mean losing
big annual revenue; but it may be
harder to justify for smaller sites. In
such cases, your best bet is to make
some educated guesses about the
low and high abandonment rates for
the various pages. You can be pretty
certain, for example, that if a home
page takes 30 seconds to load, a lot
of people will not put up with it; so
you can set the abandonment range
with a low (best case) of 20% and a
high (worst case) of, say, 50%. You
can then run one load test with the
best-case numbers and one with the
worst-case numbers to get a rough
idea of what the range of abandon-
ment might be. Your goal is not to
get these percentages exactly right,
but to recognize and document your
users’ expectations and behavior.
This abandonment is not only an
interesting result in itself; it also has
a major impact on what parts of your
Web site will get stressed under real
conditions. If you have a very slow
home page, for example, most real
users will not continue their session
and therefore will not put any load
on the rest of the Web site. In this
case, if you don’t realistically simu-
late home page abandonment, you
will apply an improbable and dispro-
portionate load to the rest of the
Web site. This improbable load
might create an improbable bottle-
neck, and you might get stuck fixing
a virtual problem that might never
have occurred naturally—while ig-
noring the slow home page that is
causing massive abandonment.
It’s important to realize that
even the most primitive abandon-
ment model is a giant leap in realism
when compared to the commonly
used 60- or 120-second timeouts.
And since there’s a lot of money in-
volved, there is no doubt that our
57
www.stqemagazine.com
 This article is provided courtesy of STQE, the software testing and quality engineering magazine.
understanding of Internet user ex-
pectations and behavior is going to
increase dramatically in the next few
years, allowing us to create increas-
ingly accurate load models. (See this
article’s Sticky-Notes for more infor-
mation.)
3. Over-Averaging
Page Response Times
Even if you manage to design, devel-
op, and execute an incredibly realis-
tic load test, you have one last op-
portunity to mess things up, really
mess them up, when you analyze and
report the results. As you might ex-
pect, load tests generate loads of
data, and all this data can be
processed, mangled, diced, and
sliced in a number of ways by using,
misusing, and abusing statistics to
produce reports that can look very
pretty but fail to shed light on what
really matters. (Before you think
that I have something against statis-
tics, let me reassure you that I am a
big fan of this discipline…at least
62.3% of the time. And even though
73.7% of statistics are usually made
up on the spot, the remaining 26.3%
are potentially very useful.)
When it comes to reporting load
testing results, the greatest opportu-
nity for voluntary, or involuntary,
misuse of statistics is related to av-
erage page response time (APRT).
Typically, the main objective of a
load test is to determine the scalabil-
ity of a Web site, an important ques-
tion if you don’t want to lose poten-
tial customers due to performance
problems (i.e., slow-loading pages).
Unfortunately, by averaging page re-
sponse times, you run the risk of
masking serious performance prob-
lems that will impact your users.
Let me show you how, by using
a final thought experiment.
Let’s assume that you run a test
with a load of 10,000 session starts
per hour and you get an APRT for
the home page of 4 seconds. Since
your chart tells you that at less than
5 seconds you will experience negli-
gible abandonment, you are in pretty
good shape, right? Well maybe yes,
maybe no—this single piece of data
ANNIE BISSETT
does not tell you much. You could
have an APRT of 4 seconds, al-
though at that load level the Web site
58
www.stqemagazine.com
performance would be unacceptable
for a large percentage of users.
How? Let’s consider three cases:
1. One way you can get an APRT of
4 seconds is if each home page
was returned in approximately 4
seconds.
2. Another way to get an APRT of 4
seconds is if 5,000 of the home
pages were returned in approxi-
mately 2 seconds and the other
5,000 in approximately 6 sec-
onds.
3. Yet another way of getting an
APRT of 4 seconds is if 9,000
users experience a 1-second re-
sponse time, and 1,000 users
experience a 31-second re-
sponse time.
In the first case, all users should be
happy, since they all experienced a
response time below 5 seconds. In
the second case, the APRT is the
same, but half the users are experi-
encing a 6-second page response
time—a time lag that you know will
cause some abandonment in a real-
world situation. In the last case, even
though the APRT is still the same 4
seconds, you have a thousand users
with a truly unacceptable response
time of 31 seconds, which points to a
potentially serious performance
problem and massive abandonment.
Unfortunately, this danger sign was
well hidden by the averaging
process.
Another way that APRT distorts
otherwise valid load test results is
Page Response Time
20
Average
15 Range (minimum
to maximum)
10
5 median value of 4.8 seconds.”)
2,000 4,000 6,000 8,000
Load Level
(Sessions started per hour)
FIGURE 2 Chart showing average,
minimum, and maximum response times
STQE
when the response times from differ-
ent types of pages are carelessly
combined into a single APRT. Let’s
assume that you run a load test and
you get an APRT of 4 seconds for all
pages. Pretty good, no?
Well, possibly not. The home
page response time may have been
30 seconds (which can easily happen
when Web designers get carried
away with fancy graphics), while all
the other pages loaded in 2 or 3 sec-
onds. In this case the APRT might
look good, but—as you should know
by now—most real users would never
have gotten past that home page.
Here’s a simple way to remem-
ber that averages can be very mis-
leading. The next time you hear an
average number, remember this: I
could put one of your feet in a bucket
of icy cold (0 degrees Celsius) water
and the other one in a bucket of boil-
ing (100 degrees Celsius) water and
tell you that, on average, your feet
are in a nice, warm, cozy 50-degree
bath.
Overcoming the
Problem of Averages
So, how do you get around the prob-
lems associated with the APRT? For-
tunately, there are several ways.
1. First of all, make sure that you
report different APRTs for dif-
ferent types of pages.
2. Augment the APRT number with
other statistical information such
as standard deviation or median;
however, first make sure that you
take into account the statistical
competence of your audience.
How many people will be able to
really understand what your sta-
tistics mean, or how to act on
them? (For example, “At a load of
6,000 user sessions per hour, the
average APRT for the home page
was 4.9 seconds with a standard
deviation of 2.3 seconds and a
3. Show a chart with not only the
10,000
average values, but the minimum
and maximum values as well, as
seen in Figure 2.
4. Forego the average altogether
and report the percentage of
May/June 2001
 This article is provided courtesy of STQE, the software testing and quality engineering magazine.
Distribution of Page Response Times (at 6,000 sessions/hr)
35%
30%
25%
20%
15%
10%
5%
1 2 3 4 5 6
Page Response Time (Seconds)
FIGURE 3 Histogram of page response times
pages returned within a specific,
relevant, time limit (e.g., “At a
load of 6,000 users sessions per
hour, 63% of the product informa-
tion pages were returned in under
5 seconds.”).
5. Present a histogram of the page
response time distribution, as il-
lustrated in Figure 3.
All of these methods enrich the APRT
and will help highlight any perfor-
mance abnormality. But unfortunately,
Make page response
time just one of the
result metrics, and
instead focus on user
satisfaction and
potential abandonment.
they still deal with pretty dry metrics to
which most people cannot relate. Case
in point, this conversation between two
people who speak different languages:
QA project leader: Thirty-seven per-
cent of the users experienced a home
May/June 2001
STQE magazine is produced by STQE Publishing, division
Number of Sessions
7,000 Completed sessions
6,000
5,000
4,000
3,000
2,000
1,000
7 8 9 10 2,000
FIGURE 4 User abandonment rates
page response time greater than five
seconds.
VP of Sales and Marketing: How in-
teresting. But…so what? How do I use
this number? Is it good or bad? What
should it be, ideally?
Using the table and model described
in the previous section, you can turn
those numbers into meaningful infor-
mation, showing not only perfor-
mance problems, but the impact they
might have on the business. My fa-
vorite approach is to make page re-
sponse time just one of the result
metrics, and instead focus on user
satisfaction and potential abandon-
ment. The chart in Figure 4 is an ex-
ample of how you might report the
results, showing how changes in per-
formance will impact the user experi-
ence and, potentially, your business.
This will greatly complement the
over-simplified (and potentially mis-
leading) data based on page re-
sponse time.
Conclusion
Concurrent users, session timeouts,
and average page response time are
three of the most fundamental con-
cepts in load testing—three con-
cepts that are regularly misunder-
stood, misused, and misrepresented,
leading to potentially misleading
load testing results.
When you adopt concurrent
users as a load testing input parame-
ter and fail to account for user aban-
STaQ
E of Software Quality Engineering
Abandoned sessions
4,000 6,000 8,000 10,000
Load Level (Sessions started per hour)
donment, you run the risk of creat-
ing loads that are highly unrealistic
and improbable. As a result, you
may be confronted with bottlenecks
that might never occur under real
circumstances. Risks abound at the
other end of the load testing cycle,
too: improper use of simple aver-
ages in the analysis phase might eas-
ily obscure ver y serious perfor-
mance problems.
Some of the solutions we’ve
looked at here are very simple to im-
plement, while others require sub-
stantially more work. In the end, it’s
going to be up to you as a quality as-
surance professional to determine
how realistic and accurate your load
tests—and their results—have to be
for your particular situation. STQE
Alberto Savoia (alberto.savoia@
keynote.com) is Chief Technologist
of Keynote’s load testing division,
and has also served as founder and
CTO of Velogic, General Manager of
SunTest, and Director of Software
Research at Sun Microsystems Lab-
oratories. His sixteen-year career
has been focused on applying scien-
tific methodology and rigor to soft-
ware testing.
Editors note: This is the second of
Alberto Savoia’s three Web-related
articles for STQE. His first article,
Web Load Test Planning, appeared
in the March/April 2001 issue. His
third article in the series will ap-
ANNIE BISSETT
pear in the July/August 2001 issue.
59
www.stqemagazine.com