7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited

http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 1/8
Skip to Content
Search
Search
Blog Team Contact
Home
Problems
We Solve
SQL SERVER
Critical Care

Training
First Aid
Blog
Subscribe
Older Posts
Newer Posts
Solving SQL Server Scalability Problems
by Jeremiah Peschka October 4, 2013
1 comment
Youve been nursing your SQL Server through day after day of increasing load. Stored procedures are taking
longer and longer to run. There are timeouts on the client side. Both customers and management are getting
upset. As youre bemoaning the terrible code that the developers wrote, it hits you: you dont have a code
problem, you have a scalability problem.
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 2/8
A slight scale
problem
The Types of SQL Server Scalability Problems
Hardware
It seems obvious to an outsider, but hardware has to be replaced on a regular basis. Between replacements,
hardware is like code: it requires attention and maintenance to keep it running smoothly. As a DBA, its your job
to pay as much attention to the database hardware as you do to the wait stats, slow queries, and missing indexes.
This doesnt mean you can throw a monitoring package in place and walk away. Understanding how SQL
Server and the underlying hardware respond under your applications workload is important. Once you have a
baseline on SQL Server and the hardware, you can easily tell if a problem is bad code or you need to call up
your friendly hardware salesperson.
Design
Database design is probably not your bailiwick; you might even say youre not a database architect, youre a
database administrator. That may be, but youve also got the keys to the kingdom of database performance.
Through the DMVs you can tell which indexes have lock escalation problems, which files have slow reads, and
even narrow these problems down to specific queries and times of day. Even if you cant tell Boyce-Codd
Normal Form from Backus-Naur Form, you have tools to help you identify problems at a physical level.
You probably want an example if you add several new queries to the application and suddenly there are a lot
more lock escalation attempts on a table, you can safely conclude that at least one of those queries would benefit
from a new index.
Code
I lied, sometimes the scalability problem is a code problem.
DBAs love to blame those pesky developers for causing problems. Theres some truth in that statement
developers introduce changes into an otherwise stable system. But when developers are adding new features and
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 3/8
functionality, they cant always know which indexes will be the best; after all, you didnt have the resources to
provide them with a full copy of the database, right?
Thankfully, you can track down top resource consuming queries very easily. Once youve identified those
queries, you can either tune them or work with the developers to educate them about what they can do better the
next time. All too often, time is the only issue that prevents things from being done correctly developers are
pressed for time to get a feature out, so they dont spend as much time tuning code as theyd like. Help them out
find the bad queries and share the knowledge or just tune them yourself.
Outside of bad T-SQL, there are a number of patterns that can cause problems for database performance. So
bad, in fact, that theyre going in a new section.
No matter what you call it, that
doesnt work.
(Anti)Patterns
If youve thought that something didnt smell quite right in the database, you were probably right. There are a
number of ways to use SQL Server that work for a while. I like to call these scalability anti-patterns. These
anti-patterns work well as long as you can keep throwing more physical resources at the problem.
Anti-patterns can be difficult to identify as an anti-pattern is it just bad code or are you seeing something
worse? Over at ACM Queue theres a guide 20 Obstacles to Scalability that lists patterns that will prevent you
from moving forward. The most common patterns youll find are a lack of caching, serial processing, using the
database as a queue, and full-text search.
Fixing anti-patterns will take a long time. These are architectural decisions that are baked into the entire
application, not just the database. In order to fix these, youre going to need to work with the developers,
research new tools, and figure out how to implement the features
Solving the Problems
Easy Mode: Hardware
Lets be honest, using money to solve your scalability problems isnt a solution. Using money to solve scalability
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 4/8
problems can often be a quick fix. If storage is slow, you can embark on a query tuning project or buy additional
memory (or even SSDs). Although youll eventually need to tune those queries, you can buy yourself some
breathing room by adding new hardware.
Dont let new hardware lull you into a false sense of security. If you buy hardware to temporarily solve a
problem, make sure you also record the performance problems and get time scheduled in the future to put a fix in
place.
Medium Difficulty: Tuning
When you finally decide to tune your application, identify your biggest bottlenecks (if you dont know, we can
teach you how) to make sure that youre tuning in the right place. You cant tune T-SQL to make the network
faster. Once you know what youre trying to fix, identify the queries cause the most pain. Is memory a problem?
Look for queries with large memory grants and tune them. Think about adding indexes.
Database performance tuning is as much about improving code as it is about making the database do less. Better
indexes mean you have to sort less, but so does sorting in the application layer. Maybe you can pre-compute
complex queries even aggregations may be slightly out of date, there may not be a real-time requirement.
The Hard Stuff: Redesign
When hardware and tuning arent enough, its time to roll up your sleeves and redesign parts of the application.
Youll still need to identify bottlenecks, but you arent going to be tuning code.
Got a queue in the database? Figure out how you can move that queue based processing outside of SQL Server
or process the queue in large batches rather than line by line.
Reporting against live data? Find ways to report against readable secondary servers or even report against a time
delayed copy of the data in a data warehouse.
There are ways to redesign any feature for scalability. Tools and techniques have been documented in books,
articles, blog posts, and conference talks. The question is no longer How could you possibly do that? The
question is Which technique are we going to use?
What Should You Do?
For most of you, the answer is simple: find your bottleneck and decide if you can make a simple fix or if you
need to devote more time to the problem. If you need more time, buy a bit of hardware to help you through.
Dont jump on the re-write bandwagon too early, but always keep ideas in the back of your head. And, above
all else, make sure youre solving the right problem with the right solution. Adding more processors wont help if
every query is wrapped in a SERIALIZABLE transaction.
Back to top
1 comment
Tweet 14
2
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 5/8
Leave a comment
1.
Allen McGuire October 4, 2013 | 11:55 am
Being a DBA is synonymous with being a firefighter. Were typically outnumbered by a 10:1 ratio
(developers:DBAs). Developers want to hit releases and milestones, and often forget about coordinating
with the DBA as new code is pushed to production. Its the nature of the beast. In fact I just read this the
other day: http://www.confio.com/logicalread/devops-essentials-for-dba-developer-collaboration.
At my current position, in my production servers Ive gone from four to eight cores, doubled the RAM,
enabled hyperthreading, and put transactional replication in place to offload read overhead. That was the
easy way to address performance issues (or cover up less than optimal procs, database design, etc.), but
the first place I started was basic performance tuning. Create jobs to harvest and analyze performance
metrics (DMVs) so they persist over time. Trace for low-hanging fruit (high Reads, high CPU, long
Duration, etc.). It was important to ask the developers (not database developers, coders) to leave
performance tuning to me its much better use of everybodys time. Communication is huge between the
developers and DBA(s) good communication can really cut down on performance issues because it
allows us to avoid performance issues proactively rather than reactively. The last thing I do is, as you
alluded to, point out the Top X poor performing procs/statements. Work with the developers to find
alternative means/methods to get the data they need.
DBAs are not only firemen, but what we do is largely an art and we have to understand all the tools we
have at our disposal. If we do our job, nobody knows us
Great topic Jeremiah! I think this article sums up the life cycle of a DBA.
Reply
Leave a Reply
Your email address will not be published. Required fields are marked *
Name *
Email *
Website
Yes, I would like to receive Brent Ozar's sublime newsletter
3 Like
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 6/8
Comment
You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym
title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q
cite=""> <strike> <strong>
Post Comment
Notify me of followup comments via e-mail. You can also subscribe without commenting.
Older Posts
Newer Posts
SQL Server Classes
"I would simply say Im much better at my job at the end than I was at the beginning of the course." Jason Kidd,
TX
Join us in Philadelphia this September.
Need Help?
Contact the Brent Ozar
team online or call:
(773) 420-9626
It won't hurt.
We Promise.
SQL Server Training
Performance Troubleshooting
How to Be a Senior DBA
DBA Interview Question and Answer Kit
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 7/8
Developer's Guide to SQL Performance
How to Tune Indexes
Introduction to Hadoop
Introduction to Internals
SQL Server on VMware and SANs
Popular Topics
SQL Server Clusters
Table Partitioning
SQL Server 2012 AlwaysOn
SQL Server Indexing
Performance Tuning
SQL Server on SANs
SQL Server on VMware
SQL Interview Questions
SQL Server Setup Checklist
sp_Blitz: Free Health Check
Archives by Author
Brent Ozar
Jeremiah Peschka
Kendra Little
Doug Lane
Jes Borland
Recent Posts by Date
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
Get SQL news before it actually happens! Enter your email address here
Sign up
7/12/2014 Solving SQL Server Scalability Problems | Brent Ozar Unlimited by Brent Ozar Unlimited
http://www.brentozar.com/archive/2013/10/solving-sql-server-scalability-problems/ 8/8
Home
Problems We Solve
Training
First Aid
Blog
Team
Contact
Brent Ozar Unlimited 2014 All Rights Reserved
Privacy Policy