MySQL Schema

Performance Schema in
MySQL
Debugging performance issue and getting deeper
insight into server’s behavior
Danil Zburivsky
April, 2013
Thursday, April 25, 13

About Pythian
• Recognized Leader:
– Global industry-leader in remote database administration services and consulting
for Oracle, Oracle Applications, MySQL and Microsoft SQL Server
– Work with over 250 multinational companies such as Forbes.com, Fox Sports,
Nordion and Western Union to help manage their complex IT deployments
• Expertise:
– Pythian’s data experts are the elite in their field. We have the highest concentration
of Oracle ACEs on staff—9 including 2 ACE Directors—and 2 Microsoft MVPs.
– Pythian holds 7 Specializations under Oracle Platinum Partner program, including
Oracle Exadata, Oracle GoldenGate & Oracle RAC
• Global Reach & Scalability:
– Around the clock global remote support for DBA and consulting, systems
administration, special projects or emergency response
7 © 2013 Pythian

About Me
– Pythian Consulting Group
– Managing hundreds MySQL
servers for dozens of
customers
– MySQL, Hadoop, NoSQL
– http://www.pythian.com/
news/author/zburivsky/
– @zburivsky
6 © 2013 Pythian

Debugging MySQL Performance

MySQL is a great database, but
instrumentation sucks
Slow query log SHOW ENGINE

INNODB STATUS
OS STATS
SHOW
PROCESSLIST

I think there is a problem with
the database
• Identify timeframe (use trending tools)

• Collect data during incident
• Collect more data
• Collect as much data as you can!
• Try to make sense of it
• Tune settings, SQL, hardware. Did it help?
• I think there is a problem with database...

Tools that may help you
• pt-stalk (http://www.percona.com/doc/percona-
toolkit/2.1/pt-stalk.html)
• innotop (http://code.google.com/p/innotop/)
• Or you write your own stuff

What do “Big Boys” have?
• Oracle: SQL-traces,
kernel events timings,
tons of books on
performance tuning
• SQL Server: SQL

profiler, Data
Management Views,
Extended Events,tons
of books on
performance tuning

Performance Schema in MySQL 5.5

The Idea
• Executing a query is broken down to hundreds of

smaller tasks
• There are background tasks as well
• We want to instrument it all to know where server

is spending time
10

Implementation
• Instrumentation: measuring when event begins

and ends
• Implemented in MySQL code on server end

storage engine level
• Can be enabled/disabled or customized
11

Schema and a Storage Engine
+----------------------------------------------+
| Tables_in_performance_schema |
+----------------------------------------------+
| cond_instances |
| events_waits_current |
| events_waits_history |
| events_waits_history_long |
| events_waits_summary_by_instance |
| events_waits_summary_by_thread_by_event_name |
| events_waits_summary_global_by_event_name |
| file_instances |
| file_summary_by_event_name |
| file_summary_by_instance |
| mutex_instances |
| performance_timers |
| rwlock_instances |
| setup_consumers |
| setup_instruments |
| setup_timers |
| threads |
+----------------------------------------------+
17 rows in set (0.00 sec)
12

+----------------------------------------------+
+----------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| setup_consumers |
| setup_timers |
| threads |
+----------------------------------------------+
12

+----------------------------------------------+
+----------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| setup_consumers |
| setup_timers |
| threads |
+----------------------------------------------+
12

+----------------------------------------------+
+----------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| setup_consumers |
| setup_timers |
| threads |
+----------------------------------------------+
12

+----------------------------------------------+
+----------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| setup_consumers |
| setup_timers |
| threads |
+----------------------------------------------+
12

+----------------------------------------------+
+----------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| setup_consumers |
| setup_timers |
| threads |
+----------------------------------------------+
12

Instruments
+------------------------------------------------------------+
| NAME |
+------------------------------------------------------------+
| wait/synch/mutex/sql/PAGE::lock |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active |
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool |
..... |
| wait/synch/rwlock/sql/LOCK_grant |
| wait/synch/rwlock/sql/LOGGER::LOCK_logger |
......
| wait/synch/cond/sql/COND_thread_count |
| wait/synch/cond/sql/COND_thread_cache |
..... |
| wait/io/file/sql/binlog |
| wait/io/file/sql/binlog_index |
+------------------------------------------------------------+
13

Consumers
SELECT * FROM setup_consumers;
+----------------------------------------------+---------+
| NAME | ENABLED |
+----------------------------------------------+---------+
| events_waits_current | YES |
| events_waits_history | YES |
| events_waits_history_long | YES |
| events_waits_summary_by_thread_by_event_name | YES |
| events_waits_summary_by_event_name | YES |
| events_waits_summary_by_instance | YES |
| file_summary_by_event_name | YES |
| file_summary_by_instance | YES |
+----------------------------------------------+---------+
14

What are top wait events in my
server?
SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT

FROM events_waits_summary_global_by_event_name
ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+---------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+---------------------------------------------+-----------------------+---------------+---------|
| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 |
| wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 |
| wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 |
| wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 |
| wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 |
| wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 |
| wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |
+---------------------------------------------+------------+-------------------+----------------+
15

innodb_buffer_pool_instance=10
SELECT * FROM events_waits_summary_global_by_event_name

ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+-----------------------------------------------+------------+-------------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |
+-----------------------------------------------+------------+-------------------+----------------+
| wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 |
| wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 |
| wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 |
| wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 |
| wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 |
| wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |
+-----------------------------------------------+------------+-------------------+----------------+
buf_pool_mutex is no longer in top 10 wait

events
16

Which files are being accessed the
most?
mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE

FROM file_summary_by_instance
ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;
+------------------------------+--------------------------+---------------------------+
| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |
+------------------------------+--------------------------+---------------------------+
| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 |
| /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 |
| /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 |
| /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 |
| /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 |
| /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 |
| /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 |
| /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 |
| /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 |
| /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |
+------------------------------+--------------------------+---------------------------+
17

Threads table
(performance_schema) > SELECT * FROM threads;
+-----------+----------------+----------------------------------------+
| THREAD_ID | PROCESSLIST_ID | NAME |
+-----------+----------------+----------------------------------------+
| 0 | 0 | thread/sql/main |
| 12 | 0 | thread/innodb/srv_lock_timeout_thread |
| 14 | 0 | thread/innodb/srv_monitor_thread |
| 16 | 0 | thread/innodb/srv_purge_thread |
| 13 | 0 | thread/innodb/srv_error_monitor_thread |
| 2 | 0 | thread/innodb/io_handler_thread |
| 25 | 8 | thread/sql/one_connection |
| 15 | 0 | thread/innodb/srv_master_thread |
| 17 | 0 | thread/sql/signal_handler |
+-----------+----------------+----------------------------------------+
18

Average wait per thread (I)
SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)

FROM events_waits_summary_by_thread_by_event_name e
LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_ID
WHERE event_name LIKE 'wait/synch/mutex/innodb/%'
AND t.NAME='thread/sql/one_connection'
AND AVG_TIMER_WAIT > 0
GROUP BY t.thread_id, e.EVENT_NAME
ORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)
19

Average wait per thread (II)
+-----------+-------------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |
+-----------+-------------------------------------------------------+-------------------------+
| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 |
| 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |
.....
.....
| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 |
| 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 |
| 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 |
| 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |
----------------------------------------------------------------------------------------------
| 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 |
| 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |
.....
.....
| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 |
| 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 |
| 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 |
| 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 |
20

Average wait per thread (III)
innodb_change_buffering = none
+-----------+--------------------------------------------------+-------------------------+
| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |
+-----------+--------------------------------------------------+-------------------------+
.....
.....
| 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 |
| 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 |
-----------------------------------------------------------------------------------------
....
....
| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 |
| 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 |
| 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 |
21

Picoseconds, shmicoseconds!
ps_helper to rescue!
• http://www.markleith.co.uk/ps_helper/
• Great examples of using
performance_schema
• Useful tools for converting time, bytes,

paths into human readable format
22

Example
(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency

FROM top_global_io_consumers_by_bytes_usage;
+-------------------------+------------+-----------+---------------+-------------+-------------+
| event_name | total_read | avg_read | total_written | avg_written | avg_latency |
+-------------------------+------------+-----------+---------------+-------------+-------------+
| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms |
| innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms |
| myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs |
| sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs |
| sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs |
| myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs |
| mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs |
| sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs |
| sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs |
| sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs |
| archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs |
| sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs |
| sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |
+-------------------------+------------+-----------+---------------+-------------+-------------+
23

No free lunch: PFS overhead
• Instrumentation doesn’t come for free

• In CPU-bound workloads overhead can
be ~15-20%
• In IO-bound — 5%-8%
• Significant improvements in 5.6
24

Performance Schema in MySQL 5.6

What’s new in 5.6?
• performance_schema enabled by default!

• Less overhead: 5%-10% for CPU-bound
workload
• Statements, Stages, Actors and Objects
26

Actors: filter event’s by user
(performance_schema) > SELECT * FROM setup_actors;

+------+------+------+
| HOST | USER | ROLE |
+------+------+------+
| % | % | % |
+------+------+------+
1 row in set (0.00 sec)
27

Objects: filter events by database/table
name
(performance_schema) > SELECT * FROM setup_objects;

+-------------+--------------------+-------------+---------+-------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |
+-------------+--------------------+-------------+---------+-------+
| TABLE | mysql | % | NO | NO |
| TABLE | performance_schema | % | NO | NO |
| TABLE | information_schema | % | NO | NO |
| TABLE | % | % | YES | YES |
+-------------+--------------------+-------------+---------+-------+
28

Statements
SELECT * FROM events_statements_summary_by_digest\G
*************************** 4. row ***************************

DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb
DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY
`SUM_TIMER_WAIT` DESC LIMIT ?
COUNT_STAR: 1
SUM_TIMER_WAIT: 511539000
MIN_TIMER_WAIT: 511539000
AVG_TIMER_WAIT: 511539000
MAX_TIMER_WAIT: 511539000
SUM_LOCK_TIME: 72000000
SUM_ERRORS: 0
SUM_WARNINGS: 0
SUM_ROWS_AFFECTED: 0
SUM_ROWS_SENT: 5
SUM_ROWS_EXAMINED: 91
SUM_CREATED_TMP_DISK_TABLES: 0
SUM_CREATED_TMP_TABLES: 0
SUM_SELECT_FULL_JOIN: 0
SUM_SELECT_FULL_RANGE_JOIN: 0
SUM_SELECT_RANGE: 0
SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 1
SUM_SORT_MERGE_PASSES: 0
SUM_SORT_RANGE: 0
SUM_SORT_ROWS: 5
SUM_SORT_SCAN: 1
SUM_NO_INDEX_USED: 1
SUM_NO_GOOD_INDEX_USED: 0
FIRST_SEEN: 2012-10-16 21:50:13
LAST_SEEN: 2012-10-16 21:50:13
29

Summary. Pros.
• Provides insight into internal server

metric
• Flexible
• Relational model allows to build your own
views on top of this data
30

Summary. Cons.
• Steep learning curve: not very well

documented
• Overhead for CPU-bound loads can be

significant
31

Thank you – Q&A
To contact us
sales@pythian.com
1-877-PYTHIAN
To follow us
http://www.pythian.com/blog
http://www.facebook.com/pages/The-Pythian-Group/163902527671
@pythian
http://www.linkedin.com/company/pythian
8 © 2013 Pythian

MySQL Schema

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

MySQL Schema

Transféré par

Droits d'auteur :

Formats disponibles

Performance Schema in

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Slow query log SHOW ENGINE

Thursday, April 25, 13

• Identify timeframe (use trending tools)

Thursday, April 25, 13

• Or you write your own stuﬀ

Thursday, April 25, 13

• SQL Server: SQL

Thursday, April 25, 13

Thursday, April 25, 13

• Executing a query is broken down to hundreds of

• There are background tasks as well

• We want to instrument it all to know where server

Thursday, April 25, 13

• Instrumentation: measuring when event begins

• Implemented in MySQL code on server end

• Can be enabled/disabled or customized

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT

Thursday, April 25, 13

SELECT * FROM events_waits_summary_global_by_event_name

buf_pool_mutex is no longer in top 10 wait

Thursday, April 25, 13

mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE

Thursday, April 25, 13

Thursday, April 25, 13

SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)

Thursday, April 25, 13

Thursday, April 25, 13

Thursday, April 25, 13

• Useful tools for converting time, bytes,

Thursday, April 25, 13

(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency

Thursday, April 25, 13

• Instrumentation doesn’t come for free

Thursday, April 25, 13

Thursday, April 25, 13

• performance_schema enabled by default!

• Statements, Stages, Actors and Objects

Thursday, April 25, 13

(performance_schema) > SELECT * FROM setup_actors;

Thursday, April 25, 13

(performance_schema) > SELECT * FROM setup_objects;

Thursday, April 25, 13

*************************** 4. row ***************************

Thursday, April 25, 13

• Provides insight into internal server

Thursday, April 25, 13

• Steep learning curve: not very well

• Overhead for CPU-bound loads can be

Thursday, April 25, 13

Thursday, April 25, 13

* 4. row *