capacity planning for

LAMP
what happens after you’re scalable

MySQL Conf and Expo

April 2007
 John Allspaw

• Engineering Manager (Operations) at

flickr (Yahoo!)
•
•

This is me. I work at Flickr.

 Yay!

• You’re scalable! (or not)

• Now you can simply add hardware as
you need capacity.

• (right ?)
• But:
• How many servers ?
 BUT, um, wait....
• How many databases ?
• How many webservers ?
• How much shared storage ?
• How many network switches ?
• What about caching ?
• How many CPUs in all of these ?
• How much RAM ?
• How many drives in each ?
• WHEN should we order all of these ?
All of these are very legitimate questions. But what is capacity ?
 some stats
• - ~35M photos in squid cache (total)
• - ~2M photos in squid’s RAM
• - ~470M photos, 4 or 5 sizes of each
• - 38k req/sec to memcached (12M
objects)
• - 2 PB raw storage (consumed about
~1.5TB on Sunday)
•
 capacity

Increased growth (usage) means needing capacity.

SCALABILITY (horizontal or vertical) = ability to easily add capacity to accommodate
growth.
We’ll talk about MySQL, squid, and memcached.
 capacity
doesn’t
mean
speed

Stop performance tuning.

Stop tweaking.
Accept what performance you do have now, and make predictions based on that.
Capacity isn’t performance.
 capacity is for business
Bring capacity planning into the product discussion EARLY.
Get buy-in from the $$$ people (and engineering management) that it’s something to
watch.
We’ll talk about MySQL, and Squid, and memcached.
 too much
Buying enough
for now

enough
not

too soon too late

Don’t buy too much equipment just because you’re scared/happy that your site will
explode.
 3 main parts

• - Planning (what ?/why ?/when ?)

• - Deployment (install/config/manage)
• - Measurement (graph the world)

Planning includes realizing what you have right NOW, and predicting what you’ll need later.
Deployment includes making sure you can deploy new capacity easily.
Measurement is ongoing, all the time, save as much data as you can.
 boring queueing theory
• Forced Flow Law:
• X =Vi i x X0
Little’s Law:
N=XxR
Service Demand Law:
Di = Vi x Si = Ui / X0
•
We can use these...but they’re boring. And take a long time. Too long.
Don’t read books with these equations in them to learn about capacity planning
for web operations.
 my theory

• capacity planning math should be

based on real things, not abstract
ones.

I don’t have time for a dissertation on how many MySQL machines we’ll need in an
abstract “future”.
 predicting the future
Can’t predict the future until you know what the past and present are like.
Must find out what you have right now for capacity.

TWO TYPES OF CAPACITY: consumable, and concurrent/peak-driven

 consumable

Disk space, RAM caches, bandwidth (sorta consumable).

(Like candy)
 concurrent usage
concurrent usage:
MySQL, memcached, squid, apache....things that don’t deplete over time.
The trick here is to plan for peaks.
(Like engines)
 considerations:
social applications

• - Have the ‘network effect’

• - Exponential growth
•
•

more users means more content

more content means more connections
more connections means more usage

etc., etc., etc.

 considerations:
social applications
• Event-related growth
• (press, news event, social trends, etc.)

• Examples:

• London bombing, holidays, tsunamis, etc.

•

•
We get 20-40% more uploads on first work day of the year than any previous peak the
previous year.
40-50% more uploads on Sundays than the rest of the week, on average.
 What do you have
NOW ?

• When will your current capacity be

depleted or outgrown ?

Predicting the future is hard, since it’s impossible.

Try to graph usage per resource (cluster) and plot how that changes over time.
 finding ceilings

• MySQL (disk IO ?)
• SQUID (disk IO ? or CPU ?)
• memcached (CPU ? or network ?)

Probably the most important slide.

What is the maximum something that every server can do ?
How close are you to that maximum, and how is it trending ?
 forget benchmarks

• boring
• to use in capacity planning...not usually
worth the time
• not representative of real load

Benchmarks are fine for getting a general idea of capabilities, but not for planning.
Artificial tests give artificial results, and the time is better used with testing for real.
 • test in production
Don’t be afraid, it’s ok. :)
Best approximation to how it will perform in real life, because it’s real life.
This means build into the architecture mechanisms (config flags, load balancing, etc.) with
which you can deploy new hardware easily into (and out of) production.
 what do you expect ?
• define what is acceptable
• examples:
• squid hits should take less than X
milliseconds
• SQL queries less than Y
milliseconds, and also keep up with
replication

These are your internal SLAs, to help you guide capacity.

measurement
 accept the
observer effect

• measurement is a necessity.
• it’s not optional.

speed freaks and tweakers and overclockers out there: suck it up.
measurement and pretty graphs are good.
 http://ganglia.sf.net

- Uses multicast and/or unicast to squirt xml data into an rrdtool frontend.
- Super super easy to make custom graphs
- originally written to handle stats data from HPC clusters
 gmetad

db1 db2 db3

XML over TCP
xml over UDP on 239.2.11.84
(multicast)

www www www

1 2 3

xml over UDP on 239.2.11.83

(multicast)

You have redundant machines in clusters, why not use the redundancy for cluster
stats as well ?
 gmetad

db1 db2 db3

XML over TCP
xml over UDP on 239.2.11.84
(multicast)

www www www

boom!
1 2 3

xml over UDP on 239.2.11.83

(multicast)

One box goes away, then another can be used as a spokesperson for that cluster.
 super simple graphing

• #!/bin/sh
• /usr/bin/iostat -x 4 2 sda | grep -v ^$ | tail -4 > /tmp/
disk-io.tmp
• UTIL=`grep sda /tmp/disk-io.tmp | awk '{print $14}'`
• /usr/bin/gmetric -t uint16 -n disk-util -v$UTIL -u '%'

Basically, if you can produce a number on the command line, then you can spit it into
rrdtool with ganglia.
 memcached

This is one of our memcached boxes. 4GB, 2 instances of 1.5GB each.

2.5% user CPU and 10% system CPU at peak.

This was built within ganglia with the excellent add-on:

http://wtf.ath.cx/screenshots.html
 what if you have graphs
but no raw data ?

• GraphClick
• http://www.arizona-software.ch/
applications/graphclick/en/
•

$8 US. Worth it.

This helpful for MRTG graphs given by ISPs. You have the images, but no raw data.
GraphClick allows you to digitize any image of a graph with units, and spit out tabular data
for use in Excel, etc.
 application usage
• Usage stats are just as important
• as server stats!
• Examples:
• # of user registrations
• # of photos uploaded every hour
Build in custom metrics to measure real-world usage to server-based stats.
Example:
How many users can 1 database machine handle, given W/X/Y/Z selects, inserts, updates,
deletes ?
More on this later.
 not a straight line

Growth is exponential or nonlinear in some way.

another not straight line
 but straight relationships!

This means that you can intuitively tie photos growth to user growth.
Not rocket science, and expected, but knowing how steep that line is...is important.
measurement examples
 queries

A week view of one of our MySQL machines. A Dell PE2950 with 6 disk 15K RPM SCSI
drives,
RAID10, with 16GB of RAM. 2x Quad Core CPUs, Intel Clovertown E5320 @ 1.86GHz.

Never went above 2% CPU, these are IO-bound machines.

Which means it’s a good bet that we are disk IO-bound...so, let’s watch that...
 disk I/O

Watch disk IOwait. Any amount of disk utilization is “ok” if IOwait doesn’t increase
dramatically.

By “ok” we mean:
- no slave lag
- query response time is still acceptable
 What we know now

• we can do at least 1500 qps (peak)

without:
- slave lag
- unacceptable avg response time
- waiting on disk IO

So...how many users were on this database ? 400,000 ?

Then plan that every X of that h/w platform can support (within the architecture it’s in)
400X users. Add a pair of them for every 400k registrations.
 MySQL capacity
1. find ceilings of existing h/w
2. tie app usage to server stats
3. find ceiling:usage ratio
4. do this again:
- regularly (monthly)
- when new features are released
- when new h/w is deployed

Of course this is architecture-specific! A master/multi-slave layout will perform differently

and see different limitations than a partitioned or federated multi-master situation.
 caching maximums

Everyone loves caching. Everyone loves memory. Everyone thinks RAM is the answer to
everything.
They might be right.
 caching ceilings
squid, memcache
• working-set specific:
• - tiny enough to all fit in memory ?
• - some/more/all on disk ?
• - watch LRU churn

Least Recently Used replacement policy chooses to replace the page which has not been
referenced for the longest time.
 churning full caches

• Ceilings at:
• - LRU ref age small enough to affect
hit ratio too much
• - Request rate large enough to affect
disk IO (to 100%)

Caching isn’t helpful if churn is too high.

 squid requests and hits

2 days graphed. Daytime peaks are pretty clear here.

 squid hit ratio

We MISS on purpose for some larger objects. Usually the hit ratio exists somewhere
between 75-80%.
 LRU reference age

Fits with request rates. More requests for more unique objects, more churn.
Low point = 0.15 days = 3.66 hours.
Max point = 0.23 = 5.52 hours.

Must make a sanity check for these with response times.

 hit response times

50ms is still within reasonableness. So, ok!

 What we know now

• we can do at least 620 req/sec (peak)

without:
- LRU affecting hit ratio
- unacceptable avg response time
- waiting too much on diskIO

Squid request rate goes up over time, so week-by-week graphs are made.
This is an example, we know that we can do actually 900-1000 req/sec without
the response time for hits getting above 100ms.
So for every 900 req/sec we expect, we should be adding another squid machine.
 not full caches

• (working set smaller than max size)

• - request rate large enough to bring
network or CPU to 100%

Hard pressed to get memcached to eat up CPU, but squid can.

 deployment
Being able to deploy capacity easily is a necessity.
 Automated Deploy
Tools
•SystemImager/SystemConfigurator
•- http://wiki.systemimager.org
• CVSup:
• - http://www.cvsup.org
• Subcon:
• - http://code.google.com/p/subcon/
•
These are all great OSS projects which can automate the installation, configuration,
deployment, and general management of clusters of machines.
 questions ?

•http://flickr.com/photos/gaspi/62165296/
•http://flickr.com/photos/marksetchell/27964330/
•http://flickr.com/photos/sheeshoo/72709413/
•http://flickr.com/photos/jaxxon/165559708/
•http://flickr.com/photos/bambooly/298632541/
•http://flickr.com/photos/colloidfarl/81564759/
•http://flickr.com/photos/sparktography/75499095/