NONSENSENUMBERS

Wil Harris

Using authenticity, fairness and

transparency to beat the benchmarks
Customers don't care about bench- So how do you engage with that kind of
marks. Get used to it. tit for tat, lowest denominator advertis-
ing? And if savvy customers don't care
OK, let's qualify that. Customers don't about your benchmarks, what do they
care about your benchmarks. In today's care about?
corporate world, where consumers are
bombarded with hundreds of advertise- The answer is - disengage. Then, reen-
ments a day, Brand X saying they're bet- gage on your own terms and serve your
ter than Brand Y doesn't mean anything customer, not your competitor.
any more. There's no trust. Customers
are saying - we don't believe you. Identify who your competitor is mar-
keting to when they're pitching
At least, the savvy ones are. The down- benchmarks. Then market better.
side of the scenario is that the stupid cus-
tomers - and there are more stupid peo- What is synonymous with better? Authen-
ple in the world than savvy people - often tically. Transparently. Fairly. The follow-
buy into advertising. This means that ing is a starting point for a benchmarking
'headline-grabbing' benchmark adver- and marketing campaign that engages on
tisements - "Brand X is 90% faster than those terms.
Brand Y in Test Z!" - will always still have
an impact.

1
 Problem 1: Communicating the
performance of server products
to an audience of server buyers.
Wil Harris
that they trust. Sure, you're not going to
win every benchmark. But you'll win
most of them, right? And you can at least
keep score of the overall total.

Here's the issue: companies love their Move the goalposts on the benchmarks,
SPEC numbers. SPEC numbers are too - be authentic. Performance, as we
easy to put on a graph. They're also understand it today, isn't about raw per-
easy to manipulate, and anybody can formance. It's about heat, cost, space
make a SPEC say anything. Companies and power, so solicit the savvy server
who use SPEC continue to give credibility community input on designing bench-
to something which should, in a world of marks, then try them out. Sometimes
authenticity and transparency, deserve they'll work, sometimes they won't. At
none. To overcome an opponent playing worst, the people buying the technology
on SPEC, disengage. will understand better the problems trying
to compare it. At best, you'll come up
Then reengage - and do so with insane with a metric that makes sense to a lot of
gusto. Savvy server buyers are their own people and shifts the goalposts away
masters. They believe that they under- from stupid SPEC numbers to real-world
stand, better than anyone, what their own benchmarks suites, designed by server
needs are. They don't trust company- savvy people, to help server savvy peo-
made benchmarks, although SPEC num- ple.
bers are attention grabbing. Do some-
thing that that grabs their attention more Involve your buyers, your savvy
transparently, and more fairly, and leads server community, in your perform-
to a more authentic result. ance metrics and they will buy into
your brand.
In this case, transparency means a blog.
An Intel Server Performance Analysis Disengage from inauthentic, attention
blog.

Give an employee or two

a really simple brief - to
take AMD and Intel chips,
benchmark them, and
publish the results.

And the methodology. And

photos of the rigs. And
videos of the benchmarks
in progress. And thoughts
about problems with
benchmarks. And notes
about hardware quirks.
And details of what they
had for lunch. Solicit input from the grabbing headlines and reengage with
server savvy community in the Com- authentic ones. "Intel wins 20/30 tests of
ments section of the blog, and implement server performance designed by, and
them. Run a high-profile campaign ask- transparently conducted in association
ing for input from these guys. with, savvy server buyers in a community
wide participation exercise in authentic
Don't aim for headline grabbing single benchmarking" isn't quite as snappy as
numbers from single benchmarks. Get "15% in SPEC", but that's just down to
your team to design workloads that accu- the turn of phrase. Authentic beats inau-
rately reflect real-world performance thentic.
situations, and then get them to fairly
benchmark the chips in these situations.
If a Commenter on the blog points out a
discrepancy, or something unfair in the
test setup, go back and re-do the test.
Involve the savvy server community in
the testing, and then you can give them
back results that they are invested in and

2
 Problem 2: Communicate the transparent, are not fair and authentic,
performance of desktop prod- and thus will be found lacking by the cus-
tomer.
ucts to consumers
As for the problem of undefinable per-
Wil Harris

The fundamental problem with desktop formance - don't push them raw num-
processing, from a competitive point of bers, push them dreams, aspirations
view, is that almost all desktop proces- and capabilities.
sors are now fit for general desktop use.
In the high-end world of computer enthu-
It's almost impossible for an average siasts, raw performance has long since
consumer to sit down and really tell been dropped in favour of capability test-
the difference between an AMD and an ing. In other words - assuming it's fast
Intel processor in the things that they enough, what more can it do? Apply this
use their computers for 75% of the metric.
time.
Disengage from the tit for tat and reen-
Here's the second problem: aside from gage on your own terms, better. Trans-
being mostly irrelevant to general usage, lating Intel's superior capabilities into
any performance related benchmark is better-than-benchmark benchmarks will
going to be accused of being biased. do more for the consumer than perform-
ance based benchmarks.
What's the answer? To be more authen-
tic, transparent and fair. The good news Where is Intel superior? Process tech-
is that this is easier in the desktop space nology, for example. Take a $200 proc-
than it is in the server space. Unlike in essor from each company, then measure
servers, where independent testing is how much electricity it uses. Assuming
difficult and expensive, there are hun- one (on 65nm) is better than the other
dreds of independent publications world (on 90nm), tell consumers that whatever
the performance benefits, Intel proces-
sors are better for the environment. Intel
changes socket platform less often than
AMD, having been on LGA775 for ages.
That means easier upgrades and less
wastage, again better for the environ-
ment. The environment is hot right now,
in case you hadn’t noticed. (In more
ways than one, natch).

AMD has fallen into the same trap that

Intel did in the Pentium 4 days - mar-
keting based on a number.

In those days, AMD successfully shifted

wide that already do testing and already the game from the megahurtz race to ca-
have trust with the consumer. Utilise pabilities (think the XD-bit and 64-bit).
those brands to do your talking for you. Now, it's gone back to touting numbers -
so play them at their own game and shift
A good publication will publish all its back to capability benchmarking. It's a
benchmark methodologies and will have more representative view of the con-
evolved them, in conjunction with the sumer experience, it's easily verifiable by
community, to a point where they are al- independent means and it talks to con-
most beyond reproach. Communicating sumers in language that they can under-
performance, then, is simply a case of stand. In other words, its more transpar-
republishing what they say in a manner ent, more fair and more authentic.
that is fair. That means quoting fairly,
providing a balanced view of editorial
coverage, but proving your point over-
whelmingly - assuming that's what the
consensus is. Taking quotes out of con-
text, or making a benchmark look better
than it might be, or using a publication of
ill repute - all of those things, whilst

3
 Problem 3: Communicating to
non-savvy business people the
Intel advantage over AMD.
Wil Harris
much does it cost to equip a server room
with Intel processors compared to AMD?
How much will the next upgrade cost?
That's one way of approaching the
money game. Alternatively, flip that ar-

The problem - these are exactly the kind gument on its head - "Last year, server
of high-powered types that fall foul of savvy firms spent $x million on Intel
sheep-driven marketing. The kind of processors and $x million on AMD proc-
people that will authorise millions of essors. What's the reason?"
pounds of budget because they saw
somewhere that AMD was faster than And the real kicker? Turn their own lack
Intel in a 'benchmarket'. How do youof tech savvy on its head, and laud them
communicate with these kind of people
for it. These people delegate what they
when they are precisely the kind of peo-
don't know, and know what they do know,
ple that are won over by headline- and that’s why they’re paid a lot of
grabbing half-truths? money. So put it in these terms - "This
year, x million spotty kids bought Intel
The answer is simple - be more processors to spend 8 hours a day in
headline-grabbing whilst simultane- World of Warcraft. They don't know your
ously being more authentic. How on business, but they know Intel's proces-
earth do you do that? sors. Harness their geek, so you don't
have to."

Everybody trusts a computer geek, be-

cause, usually, computer geeks are
amongst the most blunt - and authentic -
people around.

Benchmarketing conclusions
All of these things require moving out of
the comfort zone. They all require disen-
gaging from traditional marketing meth-
ods and reengaging in new ones. But
here’s the rub - you don’t win an argu-
ment by tit for tatting back and forth. You
win by moving the goalposts to a point
that your competitor can’t continue to
By playing on their predispositions, that's play the game. It’s a more tricky play, but
how. Somebody reading the Wall Street it’s the ultimate end-game.
Journal is already predisposed to the
Wall Street Journal. They already trust it The bottom line is this - the best way
enough to pay money for it. What the t o b e a t i n a u t h e n t i c , f l a w e d
WSJ says carries weight, more weight benchmarking-based advertising is to
than an advert. So combine the two. employ advertising that is authentic,
What was the last thing Walt Mossberg transparent, fair, and which moves the
said about Intel technology? Probably benchmarking goalposts.
something flattering in relation to the lat-
est Apple notebook. So take that quote, Anything else is just playing the game.
put it on a full page ad, and give it a suit-
able tagline, like 'No Nonsense Numbers.
Beyond Benchmarking.' Anybody read-
ing that ad is going to give it more weight
than anything AMD can put out that For illustrations, check out the seminal
quotes SPEC numbers, because readers Hugh McLeod at gapingvoid.com.
of the WSJ trust the WSJ more than
AMD.

Communicate in language they under-

stand - money. People at the high end
understand money more than they un-
derstand performance, so communicate
in that language, authentically. How