Building Scale-out Database

Applications
with SQL Azure
Lev Novik
Software Architect
Microsoft Corporation
Agenda
SQL Azure --- what is it for?
A quick recap

Why scale-out?


Building scale-out solutions today

Scaling through databases

Futures
 Microsoft SQL Azure
 Information Platform as a Service

Managed Service Scale on Demand Innovate Faster

ties delivered as a service (Database
y, greater flexibility
and deployment
ility and fault tolerant
Database
as a, Reporting
, Data Sync utility with
…) pay as you grow scaling
Innovate with new data application patterns
Rely on Business-ready SLAs
Build cloud-based data solutions on a familiar relational
Enable multi-tenant solutions
Build on existing developer skills and familiar Transact-S
Manage multiple servers
Provisioning Model
Each account has zero or more servers
Establishes a billing instrument
Account
Each server has one or more databases
Logical concept equal to a master DB
Unit of authentication, geo-location, billing,
reporting
Server Generated DNS-based name

Each database has standard SQL objects

Users, Tables, Views, Indices, etc
Unit of consistency
Database
Scaling database applications
Scale up
Buy large-enough server for the job
But big servers are expensive!
Try to load it as much as you can
But what if the load changes?
Provisioning for peaks is expensive!

Scale-out
Partition data and load across many servers
Small servers are cheap! Scale linearly
Bring computational resources of many to bear
800 little servers is very fast
Load spikes don’t upset us
Load balancing across the entire data center
Scale-out with SQL Azure Today
Elastic Provisioning of Databases
CREATE DATABASE and go
No VMs, no servers


Pay-as-you-go business model

Don’t need it --- DROP it


Zero Physical Administration

Built-in High Availability, patching, maintenance

Database Copy, SQL Azure Data Sync

Ticket Direct
Challenges Today Solution
“Capacity Bursting“
Elast ic scale – dat abase as a
service
Capacit
y

Average
Usage Time
Pay as you grow and shrink
# of Hr’s
Comput
e

Average Usage Easy t o provision and m anage

dat abase
No hardware, no m anual
Prom ot ions, event s, t icket selling dat abase adm inist rat ion
businesses are “ burst s – bound” by required
nat ure
Capacit y const raint s lim it business
agilit y
High cost s of ent ry int o new business
Difficult t o roll out ext ra capacit y
quickly
Idle capacit y “ off-burst s” is cost
prohibit ive
Scale-out for Multi-tenant applications
Put everything into one DB? Too big…
Create a database per tenant? Not bad…
Sharding Pattern: better
Application is already prepared for it!

T1 T2 T3 T4 T5 T1 T2 T3 T4 T5

T6 T7 T8 T9 T10
T6 T7 T8 T9 T10
All my data
T11 T12 T13 T14 T15
is handled by
T11 T12 T13 T14 T15 one DB on one serve
T16 T17 T18 T19 T20
T16 T17 T18 T19 T20
Sharding Pattern
Linear scaling through database
independence
No need for distributed transactions in
common cases App


Application-influenced partitioning
Rather than complete transparency

Local access for most

Connection routing
Query, transaction scoping

Distributed access for some

DIY Sharding: Problems
Provisioning
Growing and shrinking capacity Cove r e d
by
Managing SQL
Upgrading, patching, HA for lots of
databases Azu r e
t od ay
Routing
Where is the directory? Com in g
How to scale it and use it?

u p in SQL 201
Azu r e : 1
Partition Management
Splitting and Merging, without loss of
Fe d e r a t io
availability ns
Distribution of Data
Partitioned
Spread across member
machines
Each piece is on one
Con
machine (+HA) -fig
Most of the data!
Centralized Dat
a1
Dat
a2
Dat
a3
Dat
a4
Dat
a5
re re re re re
Only available in one place f f f f f
Read and write, but not too
much
Replicated
Copied to all member
SQL Azure Federations: Concepts
Root
Federation
Represents the data being sharded
Federation “CustData”
Federation Key (Federation Key:
CustID)
The value that determines the routing of Member: [min, 100)
a piece of data AU AU AU
Atomic Unit PK= PK=25 PK=35
5
All rows with the same federation key Member: [100, 488)
value: always together!
AU AU AU
Federation Member (aka Shard) PK=10 PK=23 PK=36
5 5 5
A physical container for a range of
atomic units Member: [488, max)
AU AU AU
Federation Root PK=55 PK=25 PK=35
The database that houses federation 5 45 65
directory
Creating a Federation
SalesDB
Create a root database
CREATE DATABASE SalesDB Federation
Location of partition map “Orders_Fed”
(Federation Key: CustID)
Houses centralized data Member: [min, max)

Create the federation in root

CREATE FEDERATION Orders_Fed
(RANGE BIGINT)
Specify name, federation key type
Start with integral, guid types
Creates the first member, covering
the entire range
Creating the schema
SalesDB
Products
Federated tables
CREATE TABLE orders (…) FEDERATE ON (customerId)
Federat ion
Federation key must be in all unique indices“ Orders_Fed”
(Federat ion Key: Cust ID)
Part of the primary key Mem ber: [ m in,
Value of federation key will determine the member
ordersm ax)zipcode
Reference tables
CREATE TABLE zipcodes (…)
Absence of FEDERATE ON indicates reference
Centralized tables
Create in root database
Splitting and Merging
SalesDB
Splitting a member Products
When too big or too hot
ALTER FEDERATION Orders_Fed Federat ion
SPLIT (100) “ Orders_Fed”
(Federat ion Key: Cust ID)
Creates two new members Mem ber: [ m in,
Splits (filtered copy) federated data ordersm ax)zipcode
Copies reference data to both
Online! Member: [min, 100)
orders zipcode

Merging members Member: [100, max)

When too small orders zipcode
ALTER FEDERATION Orders_Fed
MERGE (200)
Connecting and Operating
Connect to atomic unit
USE FEDERATION Orders_Fed (56) WITH App
FILTERING=ON
Connection routed to member containing 56
Only data with federation key value 56 is
visible Member: [min, 100)
Plus reference data AU AU AU
Safe: atomic unit can never be split PK= PK=25 PK=56
5 zipcode

Connect to entire federation member

USE FEDERATION Orders_Fed (56) WITH
FILTERING=OFF
Connection routed to member containing 56
All data within the member database is
Schema Distribution
Federation members can have different schemas at a point
in time:
Temporary, while schemas are being upgraded
Temporary, while customer is testing new schema on some shards
Permanently, because shards are different

To alter schema:
Manually
Connect to each federation member (USE FEDERATION Orders(56) WITH
FILTER=OFF)
Alter it (ALTER TABLE Customers …)
Future: schema-distribution service
Connect to root
Manage and apply schemas asynchronously
Federation Split

demo

Sharding in SQL Azure: Beyond v1
Schema Management
Allow multi version schema deployment and management across
federation members.
Fan-out Queries
Allow single query that can process results across large number of
federation members.
Auto Repartitioning
SQL Azure manages the federated databases for you through
splits/merges based on some policy (query response time, db size
etc)
Multi Column Federation Keys
Federate on enterprise_customer_id+account_id
Summary
Scale-out is the way to build cloud solutions


You can build scale-out solutions with SQL Azure today

Provisioning, auto-management, pay-as-you-go are your friends
Sync and Database Copy help
Implement sharding pattern today --- slide into federation support
tomorrow!

Federation support (2011) will make it a lot easier

Partition management
Routing
On-line splitting and merging
 © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be
a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.
Walk-through: setup

CREATE FEDERATION Orders_Fed (RANGE BIGINT)

USE FEDERATION Orders_Fed(0) WITH FILTERING=OFF

CREATE TABLE orders(orderid bigint, odate datetime, customerid bigint,
primary key (orderid, customerid))
FEDERATE ON (customerid)

CREATE UNIQUE INDEX o_idx1 on orders(customerid, odate)

CREATE INDEX o_idx2 on orders(odate)

CREATE TABLE orderdetails(orderdetailid bigint, orderid bigint, partid bigint, customerid bigint,
primary key (orderdetailid, customerid))
FEDERATE ON (customerid)

ALTER TABLE orderdetails add constraint orderdetails_fk1 foreign key(orderid,customerid) references

orders(orderid,customerid)

CREATE TABLE uszipcodes(zipcode nvarchar(128) primary key, state nvarchar(128))

Walk-through: real work

Connect to: InitialCatalog=‘SalesDB’

– get some regular work done (within customer 239)

USE FEDERATION Orders_fed(239) WITH FILTERING=ON

SELECT * FROM Orders JOIN OrderDetails ON …

INSERT INTO Orders (customerid, orderid, odate) VALUES (239, 2, ‘5/7/2010’)

– get some cross-customer work done

USE FEDERATION Orders_fed(0) WITH FILTERING=OFF

DELETE from Orders WHERE odate < ‘1/1/2000’

- Repeat for other members…

-- go back to root
USE FEDERATION ROOT

UPDATE CleanupSchedule set LastCleanupDate = GETSYSTIME()

Walk-through: ups and downs!

--Day#2 business grows!

ALTER FEDERATION Orders_Fed SPLIT AT(1000)

--Day#3 black friday!

ALTER FEDERATION Orders_Fed SPLIT AT(100)

ALTER FEDERATION Orders_Fed SPLIT AT(200,300,400…)

--Day#4 recession hits!

ALTER FEDERATION Orders_Fed MERGE AT(100)

--Day#5 oh boy… double dip.

ALTER FEDERATION Orders_Fed MERGE AT(200,300,400…)
Database Copy
xyz.database.windows.net efg.database.windows.net

Master prod1 Prod2 prod2

DR1 Master
clone
copy

prod2
clone
copy

abc.database.windows.net

prod2
Master Dev1
clone
copy

Sout h Cent ral US North Central US

CREATE
CREATE DATABASE
DATABASE efg.prod2clone
abc.prod2clone
xyz.prod2clone AS
AS COPY
COPY OF
OF xyz.prod2
xyz.prod2
SQL Azure Data Sync – V1 Overview

CTP1
SQL Azure
Database
Sy c
nc n
Sy

Data Sync Service

For SQL Azure Remote Offices
Retail Stores
c Syn
Syn c

Sync

CTP2
nc Sy
Sy nc

On-Premises (Headquarters)
 © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be
a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.