Académique Documents
Professionnel Documents
Culture Documents
Architectures
with MySQL Proxy
Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems
John Loehrer
Data Architect, Gaia Online
Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 1
Agenda
• Proxy concepts
• MySQL Proxy architecture
• Proxy with a single back-end
• Proxy with multiple back-ends
• broken?
• missing feature?
• not flexible?
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
traditional way
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
traditional way
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
traditional way
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
traditional way
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
• Open source way
source modify
code
new
compile source
code
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Solving database problems
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
what can you do with MySQL Proxy
• create new commands
• filter queries (deny specific queries)
• collect statistics on usage
• implement usage quotas
• execute shell commands
• create customized logs
• implement server-side pivot tables
• start/stop a MySQL server remotely
• play movies (seriously!)
• make coffee (really?)
• sharding
• load balancing servers
Basic principles
Basic principles
PROXY
CORE Lua script
connection function
function
hook function
{
Perl ?
PHP?
Why not ...
Javascript?
[whatever]?
Lua
read_auth
context
session
context
session
context
session
context
session
read_auth_result
context
read_query
read_query_result
disconnect_client
Proxy overview
/usr/local/sbin/mysql-proxy \
--proxy-lua-script=/path/name.lua
IMPORTANT!
THE SCRIPT DOES NOT START UNTIL THE FIRST
CLIENT CONNECTION
read_query and read_query_result
client
query
result
function
query
result
function
read_query_result
function MySQL
read_query Proxy only if a query is
query queue
added to the query
query
query
query queue, its result is
result
evaluated by
read_query_result
SERVER
An example: user quotas
• You want to limit user access
• When the user reaches a given amount of data,
further queries should be rejected
• the quota must work across sessions
(code online)
http://forge.mysql.com/tools/tool.php?id=111
User quotas
-- some global variables
-- proxy.global.bandwidth
-- will sum up the used bytes
proxy.global.bandwidth
= proxy.global.bandwidth or {}
a = a or 0
-- corresponds to:
if a == nil if a ~= nil
then then
a = 0 a = a
else else
a = a a = 0
end end
BREAK (a handy Proxy Lua function )
-- returns an error to the client
-- ...
proxy.global.bandwidth[session_user ]
=
proxy.global.bandwidth[session_user]
+ packet:len()
proxy.queries:append(1, packet )
return proxy.PROXY_SEND_QUERY
end
User quotas
-- read_query_result (1)
-- adding row headers to the totalizer
function read_query_result(inj)
local fields = inj.resultset.fields
local rows = inj.resultset.rows
if fields then
for i = 1, #fields do
proxy.global.bandwidth[session_user]
=
proxy.global.bandwidth[session_user]
+
(fields[i] and fields[i].name:len() or 0)
end
User quotas
-- read_query_result (2)
-- adding rows contents to the totalizer
if rows then
for row in rows do
for i = 1, #fields do
proxy.global.bandwidth[session_user]
=
proxy.global.bandwidth[session_user]
+
(row[i] and row[i]:len() or 0)
end
end
end
end
User quotas
-- read_query_result (3)
-- displaying the current bandwidth
Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems
John Loehrer
Data Architect, Gaia Online
Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 36
quick-start guide to mysql muxing
ME
online hangout
for teens
5 million unique users per month
1.3 billion forum entries
How does Gaia use the proxy?
connection multiplexer
turning 7,500 connections ...
into 100 MySQL connections
MySQL Proxy
7,500 connections
100 connections
MySQL
PHP Server
Why MUX?
a little history ...
Gaia’s data center in 2003
our server farm...
as our site grew ...
some scaling issues
added more servers ...
site explodes
diagnose the problem
GAIA, under the hood
typical LAMP stack
Linux
Share-Nothing Architecture
each apache server = 50 connections
Parent
Process
httpd
worker thread
child httpd
worker thread
listener thread
...
50 connections * 150 servers = stampede
close connections each time
setup / tear-down is expensive
connecting to MySQL
read write read
connect challenge auth auth OK
read write read
connect challenge auth auth OK write query read result
75% 19% 6%
connect
Client MySQL Server
connect
Client MySQL Server
Thread
connect
new
Client MySQL Server
Thread
connect
new
accept
threads stack up
MySQL I/O
listen
accept
listen
close
accept
end program
NEW THREAD
listen
close
accept
end program
NEW THREAD
blocking read
listen
close
accept
end program
NEW THREAD
blocking read
data
blocking write
listen
close
accept
end program
NEW THREAD
blocking read
eof data
close thread
talking to multiple databases ...
Forum Page Session DB User DB Forum DB
start session
get userdata
get topic
save session
idle threads
we need a connection pool
ideally ...
Event Queue Thread Pool
Thread
allocate thread
process query
free
release the thread after each query
it’s like a limo ...
... vs. a taxi
decouple threads from connections
Libevent’s callback I/O very sweet!
listen
register callback
event dispatch
client
Proxy
client
MySQL Server
Client Proxy Back-End Pool
MySQL Server
Client Proxy Back-End Pool
com-change-user
Client Proxy Back-End Pool
com-change-user
challenge
Client Proxy Back-End Pool
com-change-user
challenge
authenticate
Client Proxy Back-End Pool
com-change-user
challenge
authenticate
OK
Client Proxy Back-End Pool
com-change-user
challenge
authenticate
OK
free back-end
run the query
don’t be afraid to let go
Client Proxy
query
Client Proxy Back-End Pool
query
allocate
Client Proxy Back-End Pool
process query
Client Proxy Back-End Pool
process query
free back-end
connections scale!
Designing Scalable
Architectures
with MySQL Proxy
Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems
John Loehrer
Data Architect, Gaiaonline
Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 160
MySQL Load Balancer
MySQL software which can route database queries to
the appropriate database in a scale-out environment.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 161
MySQL Query Analyzer
MySQL Enterprise Monitor feature that allows users
to trace, monitor, and analyze MySQL query activity
for specific servers, users, and applications.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 162
MySQL Query Analyzer
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 163
Next Steps
MySQL Proxy Forum
http://forums.mysql.com/list.php?146
http://dev.mysql.com/tech-resources/quickpolls/
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 164
Questions?
Giuseppe Maxia
MySQL Community Team Leader - Sun Microsystems
John Loehrer
Data Architect, Gaiaonline
Jimmy Guerrero
Sr Product Marketing Manager - Sun Microsystems, Database Group
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 165