MySQL High Availability Solutions

Lenz Grimmer <lenz@grimmer.com> < http://lenzg.net/ | Twitter: @lenzgr 2011-01-26 | San Francisco MySQL Meetup | USA

$ whoami

1998

2002

2008

2010

Agenda

High Availability: Concepts & Considerations MySQL Replication DRBD / Pacemaker MySQL Cluster Other HA tools/applications

Single MySQL Server

MySQL Client

SQL (SELECT, INSERT, UPDATE, DELETE) MySQL Storage Engines (InnoDB, MyISAM, PBXT...)

Disk Storage (XFS, ReiserFS, JFS, ext3...)

Why HA?

Something can and will fail Service Maintenance Downtime is expensive Adding HA to an existing system is complex

Elimination of the SPOF

Identify what will fail eventually

Hard disks Fans Application crashes OOM-Situations, Kernel-Panics Network connections, Cables Power supply

Consider what might fail

What is HA Clustering?

Redundancy One system or service goes down others take over IP address takeover, service takeover Ensuring data availability & integrity Not designed for high-performance

High Availability Levels

Rules of High Availability

Prepare for failure Aim to ensure that no important data is lost Keep it simple, stupid (KISS) Complexity is the enemy of reliability Automate it Test your setup frequently!

HA Components

Heartbeat

Checks that services that are monitored, are alive. Can check individual servers, software services, networking etc. Configuration of the services Ensures proper shutdown and startup Allows manual control

HA Monitor

Shared storage / Replication

Split-Brain

Communications failures can lead to separated partitions of the cluster If those partitions each try and take control of the cluster, then it's called a split-brain condition If this happens, then bad things will happen Use Fencing or Moderation/Arbitration to avoid it

Redundancy Using MySQL Replication

MySQL Replication

MySQL Replication

Unidirectional Statement- or row-based (MySQL 5.1) Built into MySQL Easy to use and set up One Master, many Slaves Asynchronous Slaves can lag behind New in MySQL 5.5: Semisync Replication

MySQL Replication (2)

Master maintains Binary logs & index Replication on Slave is single-threaded No automated fail-over No heartbeat, no monitoring New in 5.5: replication heartbeat
http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave

MySQL Replication - Overview

Read & Write

Web/App Server

Write Relay Log

mysqld

I/O Thread

SQL Threa d

Index & Binlogs

Data
Replication mysqld Binlog Data

MySQL Master

MySQL Slave

Statement-based Replication

Pro

Proven (around since MySQL 3.23) Smaller log files Auditing of actual SQL statements No primary key requirement for replicated tables Non-deterministic functions and UDFs LOAD_FILE(), UUID(), CURRENT_USER(), FOUND_ROWS() (but RAND() and NOW() work)

Con

Row-based Replication

Pro

All changes can be replicated Similar technology used by other RDBMSes Fewer locks required for some INSERT, UPDATE or DELETE statements More data to be logged Log file size increases (backup/restore implications) Replicated tables require explicit primary keys Possible different result sets on bulk INSERTs

Con

Replication Topologies
Master > Slave Master > Slaves

Master > Slave > Slaves

Masters > Slave (Multi-Source)

Master < > Master (Multi-Master)

Ring (Multi-Master)

Master-Master Replication

Two nodes are both master and slave to each other Useful for easier failover Not suitable for (write) load-balancing Don't write to both masters simultaneously! Use Sharding or Partitioning instead (e.g. MySQL Proxy)

MySQL Replication as a HA Solution

What happens if the Master fails? What happens if the Slave fails? This doesnt sound like High Availability! Yes! Replication is only part of a HA configuration

Pacemaker (Linux-HA)

Supports 2 or more Nodes (v2) Resource monitoring (Apps and HW) Active fencing mechanism (STONITH) Node failure detection in seconds Supports many applications (incl. MySQL) http://clusterlabs.org/
http://www.clusterlabs.org/wiki/Load_Balanced_MySQL_Replicated_Cluster http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo

Replication & HA

Combined with Pacemaker Virtual IP takeover Slave gets promoted to Master Side benefits: load balancing & backup Can be tricky to fail back No automatic conflict resolution Proper failover needs to be scripted

Redundancy With Disk Replication

Disk Replication

DRBD

Distributed Replicated Block Device RAID-1 over network Synchronous/asynchronous block replication Automatic resynchronisation Application-agnostic Can mask local I/O-Errors Active/passive configuration Dual-primary mode (requires cluster file sytem like GFS or OCFS2) http://drbd.org/

DRBD in Detail

DRBD replicates data blocks between to block devices DRBD can be combined with Linux-HA and other HA solutions MySQL runs normally on primary node MySQL is not active on the secondary node DRBD is Linux only
Active Node

Applications

Virtual IP

Passive Node

DRBD

Redundancy Using Shared Storage

Replication vs. SAN

Data Consistency / Integrity Synchronous vs. asynchronous SAN can become the SPOF Cold caches Split brain-Situations SAN/NAS I/O Overhead

Redundancy with MySQL Cluster

MySQL Cluster

Shared-nothing-Architecture Automatic partitioning Distributed fragments Synchronous replication Fast, automatic fail-over of data nodes Automatic resynchronisation Transparent to MySQL applicationen Supports transactions http://mysql.com/products/database/cluster/

MySQL Cluster

In-memory indexes Not suitable for all query patterns (cross-table JOINs, range scans) No support for foreign keys Not suitable for long running transactions Network latency crucial Can be combined with MySQL replication (RBR)

MySQL Cluster & Replikation

MySQL Cluster

Easy failover from one MySQL node to another Scaling write load using multiple SQL nodes

Asynchronous replication from Cluster to regular MySQL slaves Slaves take read load (InnoDB/MyISAM) Quick setup of new slaves (Cluster Online Backup) Easy failover and fast recovery

http://johanandersson.blogspot.com/2009/05/ha-mysql-write-scaling-using-cluster-to.html

Galera Replication

Patch for InnoDB plus external library Synchronous replication Single- or multi-master Multicast-Replication HA plus load sharing possible Certifikate-based replikation (instead of 2PC)
http://codership.com/products/mysql_galera

MMM

MySQL Master-Master Replication Manager Perl-Script for monitoring, failover and management Master-master replication configuration (one active writable master) Failover by moving virtual IP-Address Also supports read balancing http://mysql-mmm.org/

Continuent Tungsten Replicator

Database-external solution Asynchronous, Master-Slave, Fan-out & Fan-in Java Log-based Events are stored in the Transaction History Log (THL) Modular architekture (Pipelines, Stages)
http://continuent.com/community/tungsten-replicator

Red Hat Cluster Suite

HA and load balancing Supports up to 16 nodes Shared storage Monitors services, file systems and network status Fencing (STONITH) or distributed lock manager Configuration synchronization http://www.redhat.com/cluster_suite/

Solaris Cluster / OpenHA Cluster

Provides failover and scalability services Solaris / OpenSolaris (Project Colorado) Kernel-level components plus userland Agents monitor applications Geo Edition to provide Disaster Recovery using Replication Open Source since June 2007
http://hub.opensolaris.org/bin/view/Community+Group+ha-clusters/WebHome

http://hub.opensolaris.org/bin/view/Project+ha-mysql/WebHome

Flipper

Perl Script managing pairs of MySQL servers replicating to each other. Automatic switchover, triggered manually Enables one half of the pair to be taken offline for maintenance work while other half carries on Moves IP addresses based on a role ("readonly", "writable")
http://provenscaling.com/software/flipper/

Q&A

Questions, Comments?

Thank you!

Lenz Grimmer <lenz@grimmer.com> Twitter: @lenzgr http://lenzg.net/