Académique Documents
Professionnel Documents
Culture Documents
Neil Armitage
Global transaction ID
Multiple masters
Multiple sources
Flexible topologies
Parallel replication
Heterogeneous replication
Automated management
Automatic failover
Transparent connections
Requirements
Components
Installation
Topologies
Administration
Troubleshooting
5
5 Monday, April 22, 13
Continuent 2013
Tungsten Replicator Concepts
6
Role
service
Replicator
Master, slave, direct slave
A.k.a. "pipeline"
The replication engine
stage
extract,queue,apply
6 Monday, April 22, 13
Continuent 2013
Tungsten Replicator Components
7
THL
service schema
properties file
Transaction History Log
Makes the node crash proof
service denition
tools
Ruling from a centralized
location
7 Monday, April 22, 13
Continuent 2013
Tungsten Replicator in a nutshell
binlog
THL
slave
master
host1
host2
THL
trep_commit_seqno trep_commit_seqno trep_commit_seqno
origin seqno eventid
trep_commit_seqno trep_commit_seqno trep_commit_seqno
origin seqno eventid
global
transaction ID
8
8 Monday, April 22, 13
Continuent 2013
Planning
9
Hosts
Topology
System Requirements
Validate !rst
tools/ tungsten-installer
tools/ con!gure-service
tools/update
Run cookbook
15
15 Monday, April 22, 13
Continuent 2013
REQUIREMENTS
--master-slave
--direct
17
17 Monday, April 22, 13
Continuent 2013
binlog
THL
THL
slave
slave
master
host1
host2
host3
THL
18
master-slave
18 Monday, April 22, 13
Continuent 2013
binlog
THL
slave
master
relay log
host1
host2
host3
THL
slave
relay log
19
direct
19 Monday, April 22, 13
Continuent 2013
Overview of Virtual Machines
4 Nodes host1->host4
Staging directory:
Installation directory:
Step-by-step demo
25
25 Monday, April 22, 13
Continuent 2013
Overview of Tungsten
cookbook
26
26 Monday, April 22, 13
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cluster-home
+--/cookbook
+--/tools
+--/tungsten-replicator
27
27 Monday, April 22, 13
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_MASTER_SLAVE.sh
+--install_master_slave
+--show_cluster
+--test_cluster
...
28
28 Monday, April 22, 13
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_ALL_MASTERS.sh
+--install_all_masters
+--show_cluster
+--test_cluster
...
29
29 Monday, April 22, 13
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_STAR.sh
+--install_star
+--show_cluster
+--test_cluster
...
30
30 Monday, April 22, 13
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_FAN_IN.sh
+--install_fan_in
+--show_cluster
+--test_cluster
...
31
31 Monday, April 22, 13
Continuent 2013
tungsten cookbook
$ cat COMMON_NODES.sh
export NODE1=host1
export NODE2=host2
export NODE3=host3
export NODE4=host4
32
32 Monday, April 22, 13
Continuent 2013
tungsten cookbook
$ cat USER_VALUES.sh
# User defined values for the cluster to be
installed.
export TUNGSTEN_BASE=$HOME/installs/cookbook
export DATABASE_USER=tungsten
export BINLOG_DIRECTORY=/var/lib/mysql
export MY_CNF=/etc/my.cnf
export DATABASE_PASSWORD=secret
export DATABASE_PORT=3306
export TUNGSTEN_SERVICE=cookbook
export RMI_PORT=10000
export THL_PORT=2112
export START_OPTION=start
33
33 Monday, April 22, 13
Continuent 2013
Getting started: VALIDATE FIRST
export VERBOSE=1
./cookbook/check_cookbook
./cookbook/validate_cluster
34
34 Monday, April 22, 13
Continuent 2013
sample master-slave installation
edit cookbook/COMMON_NODES.sh
edit cookbook/USER_VALUES.sh
run cookbook/install_master_slave
and then:
run cookbook/show_cluster
run cookbook/test_cluster
35
35 Monday, April 22, 13
Continuent 2013
What does the installation do
host4
1: Validate all servers
host1 host2 host3
Report all errors
36
36 Monday, April 22, 13
Continuent 2013
What does the installation do
host4
1: (again) Validate all servers
host1 host2 host3
37
37 Monday, April 22, 13
Continuent 2013
What does the installation do
2: install Tungsten in all servers
host3
$HOME/
tinstall/
config/
releases/
relay/
thl/
tungsten/
backups/
host4
host1
host2
38
38 Monday, April 22, 13
Continuent 2013
example (from manual installation)
ssh r2 chmod 444 $HOME/tinstall
./tools/tungsten-installer \
--master-slave --master-host=r1 \
--datasource-user=tungsten \
--datasource-password=secret \
--service-name=dragon \
--home-directory=$HOME/tinstall \
--thl-directory=$HOME/tinstall/logs \
--relay-directory=$HOME/tinstall/relay \
--cluster-hosts=r1,r2,r3,r4 --start
ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable
39
39 Monday, April 22, 13
Continuent 2013
example
ssh r2 chmod 755 $HOME/tinstall
./tools/tungsten-installer \
--master-slave --master-host=r1 \
--datasource-user=tungsten \
--datasource-password=secret \
--service-name=dragon \
--home-directory=$HOME/tinstall \
--thl-directory=$HOME/tinstall/logs \
--relay-directory=$HOME/tinstall/relay \
--cluster-hosts=r1,r2,r3,r4 --start
# no errors
40
40 Monday, April 22, 13
Continuent 2013
After installation.
A tour of the cookbook
utilities
41
41 Monday, April 22, 13
Continuent 2013
General principles (1)
42
e.g. ./cookbook/help
./cookbook/install_master_slave
edit cookbook/COMMON_NODES.sh
edit cookbook/USER_VALUES.sh
cookbook/install_all_masters
68
68 Monday, April 22, 13
Continuent 2013
Multiple masters
fan-in
Steps:
or :
cookbook/install_fan_in
69
69 Monday, April 22, 13
Continuent 2013
Multiple masters
star topology
Steps:
cookbook/install_star
70
70 Monday, April 22, 13
Continuent 2013
Taking Over from Standard
Replication
cookbook/install_standard_replicaton
cookbook/takeover
71
71 Monday, April 22, 13
Continuent 2013
Replication Management
72
72 Monday, April 22, 13
Continuent 2013
Common Commands
replicator
trepctl
thl
check status
skip events
inspect internals
change roles
heartbeat
backup/restore
info
index
purge
78
78 Monday, April 22, 13
Continuent 2013
Tungsten service schema
named "tungsten_SERVICE_NAME"
What is GTID
replication break
131
131 Monday, April 22, 13
Continuent 2013
Data duplication
132
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
4 Matt 130
4 Matt 140
BREAKS
REPLICATION
132 Monday, April 22, 13
Continuent 2013
auto_increment o$sets
are not a remedy
A popular recipe
auto_increment_increment +
auto_increment_offset
resolving
avoiding
preventing
Sharded by database
De!ned dynamically
methods:
unknown shards
unwanted shards
whitelisted shards
reminder
MORE_OPTIONS='--channels=10'
./cookbook/install_master_slave
178
178 Monday, April 22, 13
Continuent 2013
Checking parallel replication
trepctl status
trepctl status -name tasks
trepctl status -name shards
trepctl status -name stores
179
179 Monday, April 22, 13
Continuent 2013
Parallel replication demo
180
180 Monday, April 22, 13
Continuent 2013
Troubleshooting
181
181 Monday, April 22, 13
Continuent 2013
Identify the Failed Component
Steps
1. trepctl services
2. trepctl -service SVC_NAME status
3. look at the logs
4. Take action
182
182 Monday, April 22, 13
Continuent 2013
reading the logs
ls $TUNGSTEN_BASE/tungsten/tungsten-replicator/logs/
trepsvc.log user.log
...
or
./cookbook/show_log
# let's see it in practice
183
183 Monday, April 22, 13
Continuent 2013
Parting thoughts
184
184 Monday, April 22, 13
Continuent 2013
Open source Tungsten
Replicator now includes
Oracle-to-MySQL and
Oracle-to-Oracle extractors
and appliers!
185
185 Monday, April 22, 13
Continuent 2012 186
Continuent Website:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://yingclusters.blogspot.com
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com
186 Monday, April 22, 13