Google, Amazon, and Microsoft Swarm China For Network Gear - Wired Enterprise - Wired

Exclusive: Google, Amazon, and Microsoft Swarm China for Network ...
1 of 9
http://www.wired.com/wiredenterprise/2012/03/google-microsoft-net...
Previous post
Next post
468
97
Tweet
152
Share
By Cade Metz
Email Author
March 30, 2012 |
6:36 am |
Categories: Data Centers, Networking, Secret Servers, Servers
Like
Send
432 people like this.
J.R. Rivers once built networking hardware at Google. Now he helps web giants buy their
networking hardware directly from China and Taiwan. Photo: Jon Snyder/Wired
5/10/2012 6:54 AM
2 of 9
Google, Amazon, Microsoft, and Facebook buy more networking hardware than practically
anyone else on earth. After all, these are the giants of the internet. But at the same time, theyre
buying less and less gear from Cisco, HP, Juniper, and the rest of the worlds largest networking
vendors. Its an irony that could lead to a major shift in the worldwide hardware market.
Over the past few years, the giants of the web have changed the way they purchase tens of
thousands of the network switches inside the massive data centers driving their online services,
quietly moving away from U.S.-based sellers to buy cheaper gear in bulk straight from China and
Taiwan. According to J.R. Rivers an ex-Google engineer Google has built its own gear in
tandem with varous Asian manufacturers for several years, and according to James Liao who
spent two years selling hardware for Taiwan-based manufacturer Quanta Facebook, Amazon,
and Microsoft are purchasing at least some of their networking switches from Asian firms as well.
My biggest customers were these big data center [companies], so I know all of them pretty well,
Liao says. They all have different ways of solving their networking problems, but they have all
moved away from big networking companies like Cisco or Juniper or [the Dell-owned] Force10.
The move away from U.S. network equipment stalwarts is one of the best-kept secrets in Silicon
Valley. Some web giants consider their networking hardware strategy a competitive advantage
that must be hidden from rivals. Others just dont want to anger their business partners in the
hardware sector by talking about the shift. But cloud computing is an arms race. The biggest web
companies on earth are competing to see who can deliver their services to the most people in the
shortest amount of time at the lowest cost. And the cheapest arms come straight from Asia.
J.R. Rivers is one of the arms dealers. He runs a company called Cumulus Networks that helps
the giants of the web and other outfits buy their networking hardware directly from
original design manufacturers, or ODMs, in China and Taiwan. And hes worked in this world
for an awfully long time. Hes one of the Google engineers who secretly designed a new breed of
networking switch for the companys data centers, the massive computing facilities that drive its
search engine and the rest of its web services.
Rivers joined Google in October 2005, after five ears as a distinguished engineer at Cisco, the
company that dominated the worldwide market for networking gear. At the time, Google was still
connecting its servers using standard networking switches from the likes of Cisco and Force10
Networks. But these mass-market switches just didnt suit Googles unusually large operation.
When Google looked at their network, they need high-bandwidth connections between their
servers and they wanted to be able to manage things at scale, Rivers says. With the
traditional enterprise networking vendors, they just couldnt get there. The cost was too high,
and the systems were too closed to be manageable on a network of that size.
So Google drew up its own designs working alongside manufacturers in Taiwan and China
and cut the Ciscos and the Force10s out of the equation. The Ciscos and the Force10s build their
gear with many of those same manufacturers. Google removed the middlemen.
The search giant does much the same with its servers, buying custom-built machines straight
from Asia rather than going through traditional sellers such as Dell and HP. Because its web
services were used by such an enormous number of people, Google faced all sorts of data center
problems no one else faced problems of power and space as well as cost and logistics. So it
built all sorts of custom hardware to solve those problems.
5/10/2012 6:54 AM
3 of 9
They all have different ways of solving their networking problems, but they have all moved away
from big networking companies like Cisco or Juniper or Force10
Now, the other giants of the web are running into the same issues, and they too are going straight
to Asia for hardware. Following closely behind are companies that run large internal server
farms, including financial houses and healthcare outfits.
As J.R. Rivers serves this market with Cumulus Networks, James Liao is doing much the same
thing with a second startup called Pica8, offering networking gear that comes straight from the
ODMs. Pica8 is a spinoff of Liaos former employer, Quanta one of the companies that
manufactured Googles original networking switches, according to Rivers.
According to Liao, tens of thousands of switches are already being sold by the Asian ODMs
directly to the likes of Amazon, Facebook, and Microsoft. And that doesnt include the gear
Google has bought over the past seven years. This is just the beginning, Liao says, pointing out
that these buyers operate the biggest data centers on earth. These companies account for only a
part of the $7-billion-a-year Ethernet switch market, but as more and more outfits move their
operations into the proverbial cloud, the influence of these web giants will only grow.
Liao estimates that Amazon, Microsoft, Facebook and others have bought Asian network switches
spanning millions of network ports i.e., connections to servers and he guesses that in 2011,
about 60 percent of these ports provided 10Gigabit Ethernet connections. According to Matthias
Machowinski a directing analyst with Infonetics, a research firm that tracks the networking
market the official market for 10Gigabit Ethernet spanned about 9 million ports in 2011.
J.R. Rivers declines to name the companies hes working with at Cumulus Networks, but he
confirms that some of the big-name web outfits are already buying networking switches from
ODMs in Asia. In all likelihood, these companies are also purchasing switches from other sources
as well. Cisco says it has a significant presence and mindshare in the big-name web market, and
Juniper says it has a relationship with all of the top five web players, pointing out that data
center networks require more gear than just switches. But the market is on the move.
The Future of Web Giant 3.0
We are continuously exploring new infrastructure technologies that may evolve further
efficiencies across our portfolio. We normally have discussions with ODMs and large and small
OEMs to better understand their capabilities and evaluate their products, reads a statement sent
to Wired by a Microsoft spokesperson and attributed to Dileep Bhandarkar, a distinguished
engineer who oversees the data centers driving Microsofts online services. But the statement did
not specifically address the purchase of networking gear.
Amazon did not respond to a request for comment about its hardware practices, and a Google
spokeswoman sent us a one-sentence statement: We work with a variety of vendors to
manufacture the equipment we use in our data centers, she said. These two companies
particularly Google are rather tightlipped about their data center practices.
This supply chain change is nascent. But its the most exciting thing going on in Silicon Valley
right now
Facebook declined to discuss how it purchases networking gear, but in response to the secretive
approach of Amazon and Google, the company has openly discussed some of its other practices,
and it has actually shared its server and data center designs with the rest of the world. It
purchases its servers directly from Quanta and Wistron, another Taiwanese ODM.
Martin Casado the chief technology officier of a third Silicon Valley networking startup, Nicira
confirms that the hardware market is shifting to Asia. Offering a software platform that
virtualizes networking gear in much the same way that VMware virtualized servers, Nicira helps
some of the big web players build their networks. The Nicira platform was designed specifically
for companies along the lines of Google that want to use cheap commodity switches to physically
construct their network but then do all the complex management in software.
If youre building web giant 3.0, you can go to Quanta in Taiwan and buy crates of switches,
he says. This supply chain change is nascent. But its the most exciting thing going on in Silicon
Valley right now.
Google Goes to Asia
5/10/2012 6:54 AM
4 of 9
According to J.R. Rivers, Google began work on its custom-built networking switches in early
2005, before he arrived at the company. In the beginning, River says, Google worked in tandem
with Quanta and other Asian ODMs, but eventually, the web giant took all the engineering work
in house. Basically, Rivers says, the company wasnt happy with the work the ODMs did at the
time. Google engineers would design the switches, and then they would bring the completed
designs to contract manufacturers in Asia, outfits along the lines of Foxconn, the Asian company
that builds Apples iPhones and iPads.
Google has never discussed its practices publicly, but rumors have long indicated that the
company built its networking switches in this way. In 2007, research analyst Andrew Schmitt
noticed that certain manufacturers were producing enormous numbers of chips for 10Gigabit
Ethernet switches but that the switches themselves werent actually turning up on the market. It
didnt make sense to me why someone would be building so much of a given component if there
were no customers that could use it, he says. What I was able to determine is that Google was
purchasing switch chips straight from the comment suppler.
It didnt make sense to me why someone would be building so much of a given component if
there were no customers that could use it
The switches Google was building typically sat at the top of a rack of servers in the data center,
connecting the servers to the rest of the network. As Juniper points out, this is only part of the
networking hardware used in the data center, but its a large part.
Google, Rivers says, is a unique company. It has the wherewithal and the talent to built its own
switches, but other companies may not be up to the task. With Cumulus Networks, J.R. Rivers
and his partner, Nolan Leake, are trying to grease the wheels. [The other web players] are trying
to figure out what the best model is, and thats one of the reasons we started up, Rivers says.
Google is unique in its willingness to build something just because they know it can be done.
Most other people see a risk/reward trade-off. We seek to minimize that risk.
Though Rivers declined to name the ODMs his company is working with, he says that these are
well-known manufacturers in Taiwan and China. Weve been working for the last year on
opening up a supply chain for traditional ODMs who want to sell the hardware on the open
market for whoever wants to buy, he says. For the buyers, there can be some very meaningful
cost savings. Companies like Cisco and Force10 are just buying from these same ODMs and
marking things up. Now, you can go directly to the people who manufacture it.
This has become possible in recent years, Rivers says, because the ODMs have slowly acquired
more and more engineering talent. You can now buy commodity gear from more places.
Networking is opening up much like the transition from mainframes to RISC machines and later
to x86 servers, says Rivers partner, Nolan Leake. Were moving towards a world where
customers have more control over their destiny.
The Arms Dealer
Before spinning Pica8 out of Quanta, James Liao was already selling similar networking switches
to the big web players. Niciras Martin Casado refers to James Liao as the arms dealer in this
networking revolution. Hes the conduit between the rest of the world and Quanta. He knows
this space better than anyone, Casado says. And I love him because he talks like hes part of
organized crime.
From July 2009 to September 2011, Liao was the senior director at Quanta in charge of product
strategy for network switching and data center products. He was based in Silicon Valley, and his
job was to serve the giants of the web. He declines to go into much detail about how these
companies acquire their hardware, but hes unequivocal in saying that the other big companies
Amazon, Microsoft, and Facebook are now following Googles lead in going directly to Asia for
their gear.
5/10/2012 6:54 AM
5 of 9
James Liao. Photo: Courtesy Liao

Networking switches, he says, have become a commodity. They all use the same chips. They
have to same latency. They have the same bandwidth. This is a clear signal that the hardware
platform is commoditizing, he says. You can actually find a lot of [ODM] suppliers that have
the capability to manufacturer and design this kind of platform.
Like Cumulus Network, Liaos new venture, Pica8, brings this low-cost networking hardware to a
much larger market. In the past, one of the problems with buying directly from the ODMs is that
you had build your own software to drive your switches. But Pica8 aims to provide software for
those companies that dont want to build their own. The company has open sourced an early
version of this software known as Picos and it plans to open source a more extensive version
of the platform next month.
We give you the hardware and the software, Liao says. If you take our platform and compare it
to Cisco, the protocol features we provide and the hardware performance are all in the same
range. The only difference is that the price is 40 percent to 60 percent lower.
Though Pica8 spun off of Quanta, Liao says that the company will also sell switches from other
ODMs. But he declined to name them. But he does say Pica8 is selling gear to Japanese telecom
giant NTT and Baidu, the company that dominates the Chinese search engine market.
Matthias Machowinski, of research firm Infonetics, says he is very much aware of this trend,
though he adds that it is extremely hard to track. He says that the big web giants account for only
a part of the overall switch market the number of customers that might choose to go down
this route are very limited. Today, you can count them on one hand, and maybe over the next two
years, two hands might be enough but he also acknowledges that as businesses move their
applications onto services such as Amazon EC2 and Microsoft Azure rather than running stuff
in their own data centers these web giants will account for an even larger part of the switch
market.
Like Server, Like Switch
This shadow networking market is a repeat of what happened in the server world. Years ago,
Google started building its own servers in tandem with the Asian ODMs, and other web giants
followed. These companies are looking to save cost, but theyre also looking to reduce their
power consumption, customizing machines so theyre far more efficient than their mass-market
brethren.
In 2009, Google revealed some server designs it produced several years before. But, as with
networking practices, the company says very little about its server gear. Amazon operates in
much the same way. But Facebook had taken a different approach. Last year, after building its
own data centers and working with various manufacturers to build its own servers, the social
networking giant open sourced these designs to the rest of the world, hoping that others across
the industry can help improve those designs, buy more hardware based on the designs, and
5/10/2012 6:54 AM
6 of 9
ultimately drive down the price of the hardware.

Its kind like buying couches. If you buy one, you go to a retail store. If you buy 10,000 couches,
you go straight to the factory
This Open Compute Project already has several other big-name backers, including Texas-based
cloud computing outfit Rackspace and Japans NTT. And it doesnt stop at data centers and
servers. Last month, Frank Frankovsky the ex-Dell man who oversees hardware design at
Facebook told us that the company is in the midst of building its own storage hardware and
that these designs will be open sourced in early May.
In these cases, Facebook and Amazon and Google and others bypassed original equipment
manufacturers, or OEMs, such as Dell and HP. The servers sold by the likes of HP and Dell are
actually manufactured by those same ODMs in Taiwan and China.
James Liao, of Pica8 and formerly of Quanta, does not work with servers. But he says that its
common knowledge that like Google and Facebook Amazon purchases at least some of its
servers from ODMs in Asia. For servers, Facebook and Amazon are taking almost exactly the
same approach, he says. Amazon also has some very high power designers, but they dont do
the design themselves. They come up with a certain architecture and they tell the ODMs: This is
my vision. These are the goals. And I want help designing the hardware.
Now, Liao says, this same sort of thing is happening with, well, everything. All of the data center
hardware is bought this way, Liao says. You can refer to Facebook as an example, where one of
the big projects inside the Open Compute effort is storage. Even the storage side is being
commoditized. Servers, storage, and networking all of them are going to this way.
Nolan Leake and J.R. Rivers of Cumulus Networks. Photo: Jon Snyder/Wired
Howard Wu the president of greater China for Joyent, an Amazon-like cloud computing outfit
based in San Francisco agrees. If youre a small business and youre going to buy five servers,
youre going to Dell or HP, because of the support services. But if youre a data center operator
5/10/2012 6:54 AM
7 of 9
and youre going to buy 10,000 servers, youre going straight [to the ODMs], he says. Its kind
like buying couches. If you buy one, you go to a retail store. If you buy 10,000 couches, you go
straight to the factory.
That said, Joyent is not yet buying its gear from the ODMs.
We are definitely in talks, but it hasnt actually happened yet, he says. We have other
contractual obligations right now. The market has not completely shifted to Asia. Its moving in
stages. These web companies have many suppliers thats just good for businesses and in
some cases, theyre still buying hardware from the traditional players perhaps because they
still have contracts in place. Facebook, for instance, is still buying some servers from Dell and
HP. And Amazon is still buying custom servers from Rackable, a stateside manufacturer, and
apparently other outfits based here in America.
The hardware supply chain is vast and varied. But its consolidating. Now that they have the
engineering talent, J.R. Rivers says, the ODMs are transforming into OEMs. The market is
maturing to the point where anyone can buy directly from ODMs, he says. You dont have to be
Google.
Update: In response to this story which refers to Pica8 as a spin-off of Quanta Quanta sent
Wired a statement that reads: Pica8 has licensed technology from Quanta and is not a spin
off of Quanta. Quanta has no ownership interest in Pica8 and has never owned Pica8s
shares.
Pages: 1 2 3 View All
Related
You Might Like
Related Links by Contextly
Mavericks Invent Future Internet Where Cisco Is Meaningless

Amazon's Secretive Cloud Carries 1 Percent of the Internet
Vanishing Webpage Links Google to Network Maverick Nicira
Disappearing Net Names Hint at Amazon Data Center Expansion
Super-Secret Google Builds Servers in the Dark
Mystery Men Forge Servers For Giants of Internet
Google Reincarnates Dead Paper Mill as Data Center of Future
Show
More Hacks Shipping Dock Into World-Class Server Lab
Facebook
Windows 8 Release Preview Coming in Early June

HTML5 Offers 'Scoped' CSS for Precision Styling
8 Visionaries on How They Spot the Future
Facebook Opens Up Hardware World With Magic Hinge
Say Hello to Windows Azure, The World's Most Misunderstood Cloud
5/10/2012 6:54 AM
8 of 9
Cade Metz is the editor of Wired Enterprise. Got a NEWS TIP related to this story -- or to
anything else in the world of big tech? Please e-mail him: cade_metz at wired.com.
Read more by Cade Metz
Post Comment | 60 Comments and 454 Reactions | Permalink
Back to top
Tweet
468
Like 432
97
Share
152
Reddit Digg Stumble Upon Email

and 5 others liked this.
Like
Showing 5 of 62 comments
That's comforting because there is no way china is slipping chips into them to spy on us.
They are our friends.
We are talking hardware, not software. Unless it was built right into the bios, that
would be pretty hard to do.
No, gman, it would NOT be hard to do. I work in hardware design and it's
VERY easy...
5/10/2012 6:54 AM
9 of 9
This comment was flagged for review.
Ill be sure to check out that offer Cecil!
M Subscribe by email S RSS
Reactions
5/10/2012 6:54 AM

Google, Amazon, and Microsoft Swarm China For Network Gear - Wired Enterprise - Wired

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Google, Amazon, and Microsoft Swarm China For Network Gear - Wired Enterprise - Wired

Transféré par

Droits d'auteur :

Formats disponibles

Exclusive: Google, Amazon, and Microsoft Swarm China for Network ...

432 people like this.

James Liao. Photo: Courtesy Liao

ultimately drive down the price of the hardware.

Mavericks Invent Future Internet Where Cisco Is Meaningless

Windows 8 Release Preview Coming in Early June

Reddit Digg Stumble Upon Email

This comment was flagged for review.

Ill be sure to check out that offer Cecil!

M Subscribe by email S RSS

Vous aimerez peut-être aussi