A Day in the Life of 3G

During March and April, we spent a day testing the major 3G services
in 13 cities across the United States. Verizon's service showed a
combination of speed and reliability, Sprint's results lent credence to
its 'most dependable' claim, and AT&T's network showed fast upload
speeds in most cities.
Mark Sullivan, PC World Jun 28, 2009 5:00 pm

Only after the thrill of picking out your new

smart phone is over--after you’ve marveled
at all the stylish new gadgetry and features,
signed a service contract and finally laid
your money down--do you finally get a feel
for the speed and reliability of the wireless
service that connects your new phone. That
wireless service--so often an afterthought to
smart phone buyers--is hugely important: it
connects your new phone to all the fun and
useful apps and services that made you lust
after a smart phone in the first place.

The truth is, the 3G wireless networks of today are not perfect (they are,
collectively, a work in progress), and they are not all created equal.

Because independent research on these networks is very hard to come by,

PC World took a single-day, real-world snapshot of the performance of the
three biggest 3G networks in the U.S. – Verizon Wireless, AT&T and Sprint
– using industry-accepted testing technology and techniques. If there’s a
smart phone in your future, we hope to give you some idea of the wireless
service that may be in store for you, beyond the anecdotal information you
hear from other users, on the web and in the media, and aside from the
claims made by the wireless service providers themselves.

Now for an important note before we start reviewing our results. Wireless
signal, by its nature, is extremely variable; that is, many things, such as
obstruction by fixed objects (buildings, trees, etc.), weather, network load,
cell tower locations, and time of day, can affect the quality of the signal.
These factors can cause service from a single wireless service to vary
widely from day to day and from neighborhood to neighborhood. Our
results, taken together, provide a snapshot of the performance of the
largest 3G networks in 13 major markets during March and early April. But
they are by no means exhaustive, and your own connection speeds may
differ from ours.

During March and early April, our testing partner, Novarum Inc., used Ixia
ixChariot testing software to measure network performance from more than
twenty fixed locations in each of the following cities: Baltimore, Boston,
Chicago, Denver, New Orleans, New York City, Orlando (Florida), Phoenix,
Portland (Oregon), San Diego, San Francisco, San Jose, and Seattle. In
all, our testing partner ran 5443 individual tests from 283 testing
locations. At each location, Novarum measured download speed, upload
speed, and reliability for each provider's 3G service.
Testing Results in a Nutshell
In Novarum's tests for us, Verizon Wireless demonstrated a good mix of
speed and reliability. Across more than 20 testing locations in each of the
13 cities we tested, Verizon had an average download speed of 951 kbps.
Verizon demonstrated good reliability, too; the network was available at a
reasonable and uninterrupted speed in 89.8 percent of our tests.

Sprint's 3G network delivered a solid connection in 90.5 percent of our 13-

city tests. Sprint's average download speed of 808 kbps across 13 cities
wasn't flashy (at that speed, a 1MB file downloads in 10 seconds), but
dependability is an important asset. The Sprint network performed
especially well, both in speed and in reliability, in our test cities in the
western part of the United States.

The AT&T network's 13-city average download speed in our tests was 812
kbps. Its average upload speed was 660 kbps. Reliability was an issue in
our experience of the AT&T system: Our testers were able to make a
connection at a reasonable, uninterrupted speed in only 68 percent of their
tests.

Somewhat surprisingly, our testers also found that the "bars of service"
readings on their phones were rarely an accurate predictor of the quality of
the ensuing connection. In most places and with most wireless providers,
the "bars" did little more than indicate whether the phone had access to
some service or to no service.

As Advertised
Do wireless providers deliver the connection speeds they promise for their
3G networks? In our tests, on average, they did. However, the services
promise speeds within a wide range--if they provide a low end to the range
at all--due to the wide variability of network performance from day to day
and from neighborhood to neighborhood. So in practical terms, these
ranges don't represent much of a commitment to consumers.

Verizon says publicly that its wireless customers can expect download
speeds of up to 1.4 mbps and average upload speeds of 500 kbps to 800
kbps from its 3G network. Verizon came reasonably close to those speeds
in our tests, delivering downloads at an average rate of 951 kbps across 13
cities, and uploads at an average rate of 426 kbps across the same cities.
At these speeds, a 1MB file would download from the Web in 8.4 seconds
or upload to the Web in 18.8 seconds. Verizon, which claims to have the
"largest and most reliable" wireless network in the United States, delivered
a reliability number of 89.8 percent across our tests in 13 cities.

Sprint easily delivered on its dependability promises in our tests, with a

reliability rate of 90.5 percent. (The company claims to have "America's
most dependable 3G network.") Sprint promises average download speeds
of 600 kbps to 1.4 mbps, and average upload speeds of 350 kbps to 500
kbps. In our tests, Sprint came through on both counts, with an average
download speed of 808 kbps and an average upload speed of 377 kbps. At
those speeds, a 1MB file would download from the Web in 10.1 seconds, or
upload to the Web in 21.6 seconds.

AT&T claims to be the "nation's fastest 3G network" and promises

download speeds between 700 kbps and 1.7 mbps, and upload speeds
between 500 kbps and 1.2 mbps. While we can't crown AT&T as having the
nation's fastest network, we found that, on average, AT&T delivered upload
and download speeds that fell within their promised ranges in our tests.

Verizon Speeds Fast, Consistent

Verizon Wireless did very well in our speed tests in the central and eastern
parts of the country, especially in the cities of Baltimore, Chicago, Denver,
New Orleans, New York, and Orlando.

Verizon achieved its fastest speeds in New Orleans, with averages of 1425
kbps for downloads and 550 kbps for uploads in our tests. The company's
single fastest connection speed among the 13 cities where we tested was
just over 2.3 mbps (also in New Orleans). Its poorest showing in our results
came in Portland, with download speeds of 622 kbps and up--load speeds
of 410 kbps on average.
Verizon says that since the company formed in 2000, it has invested $50
billion in its wireless network. As of January 2009, the network supported
some 80 million subscribers, the company says.

Sprint Fast in West Coast Tests

Sprint's test results were competitive with those of Verizon Wireless in most
of the cities where we took samples, especially in the West Coast cities of
Portland, San Diego, San Francisco, and Seattle. Sprint's top speeds came
in our Seattle tests, where it clocked an average download speed of 1005
kbps and an average upload speed of 469 kbps. Sprint's single fastest
download speed in our results was a mark of 2.1 mbps in San Francisco.
Sprint's worst average performance was in our New Orleans tests, with a
download speed of 599 kbps and an upload speed of 240 kbps.

Sprint won't divulge precisely how much it has invested in its 3G network,
but it does say that it has poured $18 billion into its wireless and wireline
networks since 2006. The carrier's 3G network, which it launched in 2005,
uses the same 3G wireless protocol used by Verizon Wireless-CDMA,
EvDO Revision A. This may have something to do with the comparable
performances of the two networks.

Sprint says that it has moved well down the 3G road, quickly shifting its
subscribers away from older networks. "The vast majority of our customers
use our 3G network, and the vast majority of the Sprint Mobile Broadband
Network has been upgraded to faster [3G] EvDO, Revision A technology,"
says Sprint spokesperson Stephanie Vinge-Walsh. "Use of 2G continues to
phase out; we expect those few customers still using our 2G network to
upgrade as they upgrade their devices."

But unlike Verizon's, Sprint's wireless subscriber base has been shrinking.
The network connected 49.3 million customers at the end of 2008,
compared to 53.8 million at the end of 2007. Has the decrease in number
of customers reduced the load on Sprint's network and contributed to its
solid performance in our tests? Sprint's vice president of network
development and engineering, Iyad Tarazi, insists that the two things are
unrelated and that traffic on its 3G network has continued to increase
throughout the past year.

Sprint will be the sole provider of the much-hyped Palm Pre smartphone,
which it hopes will quickly increase its subscriber numbers. The iPhone
certainly lit a fire under AT&T's wireless business, and the right killer device
might work for Sprint, too.

AT&T Speed Results

In our 13-city testing, AT&T's 3G network produced download speeds that
averaged 818 kbps, and upload speeds that averaged 549 kbps. The
network clocked some of it best results in our tests in Boston and Chicago.
In Boston, AT&T's 3G network delivered an average download speed of
1259 kbps and an average upload speed of 708 kbps in our 20 test
locations in the city. AT&T's Boston network also proved available at a
reasonable speed in 90 percent of our tests there. In Chicago, the AT&T
network clocked an average download speed of 1148 kbps and an average
upload speed of 712 kbps. In New York City, the company's network
delivered an average download speed of 502 kbps and an average upload
speed of 308 kbps across our 20 test locations there. AT&T delivered
relatively fast upload speeds (549 kbps on average) in the 13 cities we
tested.

"AT&T stands behind its claim of providing the nation's fastest 3G network,
as verified by two independent third-party testing companies," says AT&T
spokeswoman Jenny Bridges. "The third-party companies that measure the
network performance of AT&T and its competitors conduct comprehensive
drive tests throughout the year, spanning more than 1 million road miles in
more than 340 markets and more than 1 million 3G data sessions
combined," Bridges adds.

"Beyond this third-party testing, AT&T conducts millions of its own tests
across the nation to measure performance and maximize the service
quality and reliability of our network. For comparison purposes, we also test
our competitors' networks, as they do ours. We believe this combined data
provides the most thorough, comprehensive view of the network
performance of AT&T and its competitors."

Spending Big on Improvements

Over the past few months, AT&T has announced plans for big
improvements to its 3G networks. The company announced in March that it
would in--vest $17 billion to $18 billion in its networks in 2009, "two-thirds of
which will extend and enhance the company's wireless and wired
broadband networks to provide more coverage, speed and capacity."

More recently, on May 27, AT&T announced plans to increase the speed of
its 3G service by upgrading its networks to the faster High Speed Packet
Access (HSPA) 7.2 technology, utilizing more radio-frequency spectrum,
increasing backhaul capacity, and adding 2100 new cell towers. The
company says that it will begin the upgrade this year and expects to
complete the process in 2011.

Sprint Bolsters 'Most Dependable' Claim

Of course speed isn't everything. The fastest wireless connection in the
world is worth little if service gets interrupted in the middle of a session. In
each city, we measured how often the AT&T, Sprint, and Verizon networks
delivered poor service (similar to or slower than dial-up speed), interrupted
service, or no service at all. If any of these conditions occurred during one
of our tests, we labeled service at that location "faulty." Thus, if 10 out of 20
tests were faulty in a given city, we gave the operator a reliability score of
50 percent. Using that standard, we calculated an average reliability score
for each service from our thousands of individual tests.

Sprint's 3G network delivered high reliability scores, especially in our

testing in Boston, Chicago, Denver, Portland, San Francisco, and Seattle.
Sprint's network proved reliable in 90.5 percent of our numerous tests
across 13 cities. In fact, in Boston, Denver, and Seattle, Sprint's service
earned perfect scores for reliability, proving available at reasonable speeds
in 100 percent of our tests.
Sprint's vice president of network development and engineering, Iyad
Tarazi, wasn't surprised by the reliability results. "We've invested heavily in
our 3G network; we've made it the most critical part of our network," he
says.

Nationwide, Tarazi says, Sprint simply has more base stations on the
ground than its competitors do, which leads to higher reliability. "On the
reliability side, you're seeing something very similar to our own [network
testing] numbers," Tarazi says of our results.

Verizon's EvDO service was available at a reasonable speed in 89.8

percent of our 13-city tests. Verizon's network showed impressive reliability
results in Baltimore, New Orleans, New York, Phoenix, and San Jose. In
New Orleans and New York City, Verizon's 3G service showed perfect
reliability throughout our testing locations in each city.

What Do Bars Say About Your Connection?

If you've ever wondered why your phone was showing five out of five bars
and yet your service was poor, you're not alone. In one of the most
surprising findings of our study, we discovered that the number of bars
showing on our phones was by no means an accurate predictor of our
connection speed or of network service reliability. In most cases the bars
were useful only for telling whether we had some signal or no signal.

Phoenix is the only city we tested where the bars-of-service reading did a
good job of predicting service quality. There we saw a correlation be--tween
bars and service quality in more than 70 percent of our tests, across all
carriers. In the other 12 cities where we tested, bars of service were far
less meaningful. In San Francisco, for example, only 13 percent of our tests
showed any correlation between bars and service quality.
Our test results suggest that the bars-of-service readings tend to be a more
accurate predictor of service quality on networks that have enough capacity
to handle all of the devices connected to them. Overall, the bars on Verizon
devices did the best job of predicting service quality in our testing. We saw
at least some correlation between number of bars and Verizon's network
performance in 11 of 13 cities. Bars were a moderately useful predictor of
Sprint service in our tests in only two cities--San Diego and Phoenix.

In our tests of AT&T's networks in four cities (New York, San Diego, San
Francisco and San Jose), the number of bars was not predictive of the
service quality we saw; in the other nine cities in our study, the number of
bars proved marginally useful in predicting how well our test experience
would turn out.

The Problem of Backhaul

Backhaul refers to a wireless operator's method of moving data back and
forth from the cell tower to the wireline Internet backbone. Bars of service
measure your phone's connection to the cell tower, but that's just the first
part of the trip. Bars offer no clue as to how smoothly and quickly the
backhaul network is connecting with the Internet. That's why your phone
may show the optimum five bars even though your service seems to be
taking forever.

Let's suppose that you have keyed in ‘www.nytimes.com' on your

smartphone. Your phone shows five bars, indicating that it has connected
very quickly to the nearest cell tower. The cell tower then sends your
request over the backhaul network, which connects to the system of fiber-
optic and copper lines constituting the backbone of the Internet. Over this
system, your request reaches the server that contains the New York Times
content. But again, if the backhaul system is slow or overloaded, it creates
a bottleneck in the system, lengthening your response time even though
your device connects to the cell tower quickly.

Understandably, wireless carriers are investing heavily in more-efficient

networks and technology to backhaul wireless data, which is rapidly
increasing in volume. Perhaps the fastest and most reliable backhaul
strategy uses fiber-optic cable to route wireless data back and forth
between the cell tower and the Internet backbone. The carrier leases, buys,
or builds new fiber-optic lines. To reach cell towers that are far away from
existing fiber-optic infrastructure, mobile operators will increasingly use
wireless approaches such as ethernet over microwave technology as a less
expensive alternative to fiber for backhauling traffic.

Mr. Smartypants Shops for a Smartphone

Over the next year, it will become obvious that the iPhone and the
Blackberry aren't the only cool smartphones on the market. New models
such as the Palm Pre and a wave of new Android-based units will emerge
to compete for the many consumers who haven't yet caught the
smartphone bug. Mr. Smartypants is one such consumer; and when he
goes shopping for a phone, he asks a lot of questions that any smart
smartphone shopper should. Some of these questions are about the phone
itself; others relate to the network that will connect the phone to the
Internet.

Smartypants: How many cell towers do you have in town?

Store rep: Umm. I'll have to call the main office to get that.

Smartypants: About how many wireless subscribers does each cell tower
support?

Store rep: Oh, yeah, I, uh...I don't think the company gives out those
numbers.

Smartypants: Is the chipset in this phone optimized for the flavor of 3G

service you're selling-you know, 1xEvDO Rev 0 or Rev A? HSPA or HSDPA
+?

Store rep: Yes! This phone does have a very large screen!

Smartypants: What kind of wireless backhaul does the network use?

Fiber?

Store rep: Well, see here, on the screen, if all five of these little bars are lit
up, that means it's, uh, backhauling well...and those bars are always lit up!

Smartypants: Can this phone be converted to connect to a 4G network

when one becomes available? And by the way, when will you offer 4G
service here in town?

Store rep: Look, buddy--Sprint's right across the street. Go bother them.

It may seem as though Mr. Smartypants was just dipping into his
knowledge base to give the store rep the needle, but all of his questions
focus on issues that can dramatically affect a 3G network's performance.
Wireless companies ought to be ready and willing to provide accurate
answers to these questions. Why? Because their customers have a right to
know what they're buying.
How We Tested and What the Ratings Mean
We decided to test the three major 3G cellular wireless broadband
providers in 13 U.S. cities that we judged to be broadly representative of
the locales where most customers are likely to use these services. In each
city, we randomly chose 20 test locations, evenly distributed over the
metropolitan area. We performed all of our tests inside a parked car.

We created a 1-minute stress test to evaluate the quality and performance

of the wireless service. We tested network delay, upload speed, download
speed, and reliability, as well as the correlation between "bars of service"
and network performance.

We conducted the tests using industry-standard wireless-testing software

(Ixia Chariot) running on a Windows XP SP3 laptop. We tested on a laptop,
rather than on a smartphone, because we needed the laptop's processing
power to run Ixia's rigorous 1-minute tests, and because a laptop can test
the strengths and weaknesses of the network more accurately than a cell
phone can. To connect to each network, we used the latest USB modem
from each vendor: AT&T's USBConnect Option Quicksilver, Sprint's Sierra
Wireless USB 598, and Verizon Wireless's Novatel Wireless USB 727. All
of the client adapters we used came from the respective vendors and were
recommended by the outlets where we purchased them.

Test Definitions
Download speed: the average speed (in kilobits per second) at which we
downloaded random data from a known Internet server during a 1-minute
streaming test.

Upload speed: the average speed (in kilobits per second) at which we
uploaded random data to a known Internet server during a 1-minute
streaming test.

Reliability: the percentage of tests for a given city in which we could detect
a signal, connect at a reasonable speed (faster than dial-up), and sustain
an uninterrupted connection for the duration of a 1-minute streaming test.