Tungsten Installation Guide

Tungsten Replicator Installation Guide

Tungsten version 2.0.4 This document was published on 01 December 2011. www.continuent.com [http://www.continuent.com] Copyright Continuent
The trademarks, logos, and service marks in this Document are the property of Continuent or other third parties. You are not permitted to use these Marks without the prior written consent of Continuent or such appropriate third party. Continuent, Tungsten, uni/cluster, m/cluster, p/cluster, uc/connector, and the Continuent logo are trademarks or registered trademarks of Continuent in the United States, France, Finland and other countries. All Materials on this Document are (and shall continue to be) owned exclusively by Continuent or other respective third party owners and are protected under applicable copyrights, patents, trademarks, trade dress and/or other proprietary rights. Under no circumstances will you acquire any ownership rights or other interest in any Materials by or through your access or use of the Materials. All right, title and interest not expressly granted is reserved to Continuent. All rights reserved.
Tungsten Replicator Installation Guide - Document issue 2.0.4
Tungsten version 2.0.4
Table of Contents
1. Introduction ............................................................................................................................................ 1 2. Product Installation and Configuration ..................................................................................................... 2 2.1. Prerequisites ............................................................................................................................... 2 2.2. Software Download ..................................................................................................................... 2 2.3. Setting Up Master/Slave Replication ............................................................................................. 2 2.3.1. Master Replicator Configuration ......................................................................................... 2 2.3.2. Testing Replication ............................................................................................................ 3 3. Tungsten Configuration Procedures ......................................................................................................... 5 3.1. tungsten-installer ......................................................................................................................... 5 3.1.1. Getting Help ..................................................................................................................... 5 3.1.2. Replicator Topologies ........................................................................................................ 5 3.1.3. Service Name ................................................................................................................... 6 3.1.4. Home Directory ................................................................................................................ 6 3.1.5. Database Login and Passwords ........................................................................................ 6 3.1.6. Replicator Start-Up ........................................................................................................... 6 3.1.7. Installation Checks ............................................................................................................ 6 3.1.8. Advanced Property Settings .............................................................................................. 6 3.2. configure ..................................................................................................................................... 7 3.2.1. Getting Help ..................................................................................................................... 7 3.2.2. Cluster Hosts ................................................................................................................... 7 3.2.3. Home Directory ................................................................................................................ 7 3.2.4. Replicator Start-Up ........................................................................................................... 7 3.2.5. Installation Checks ............................................................................................................ 8 3.3. configure-service ......................................................................................................................... 8 3.3.1. Getting Help ..................................................................................................................... 8 3.3.2. Replicator Service Role ..................................................................................................... 8 3.3.3. Service Name ................................................................................................................... 8 3.3.4. Service Type .................................................................................................................... 8 3.3.5. Release Directory ............................................................................................................. 9 4. Configuring Specialized Replication Topologies ...................................................................................... 10 4.1. MySQL Slave Replacement with Parallel Replication ................................................................... 10 4.1.1. MySQL Replication Setup ............................................................................................... 10 4.1.2. Tungsten Configuration and Slave Takeover ..................................................................... 12 4.1.3. Re-enabling MySQL Slave SQL Thread ........................................................................... 12 4.2. Setting up Bi-Directional Replication ........................................................................................... 13 4.2.1. Unpack software ............................................................................................................. 14 4.2.2. First Master Replicator Configuration ............................................................................... 14 4.2.3. Second Master + Slave Replicator Configuration .............................................................. 14 4.2.4. Remote Slave Service Configuration ................................................................................ 14 4.3. Setting up MySQL to PostgreSQL Replication ............................................................................. 15 5. Host Prerequisites ................................................................................................................................ 16 5.1. Tungsten Login .......................................................................................................................... 16 5.1.1. Configuring Remote Host Access for Tungsten Login ........................................................ 16 5.1.2. Configuring Remote Host Access for PostgreSQL Login .................................................... 17 5.2. Operating System Prerequisites ................................................................................................. 18 5.2.1. Notes on Operating System Prerequisites ........................................................................ 18 5.2.2. Linux Prerequisites ......................................................................................................... 18 5.2.3. Mac OS X Prerequisites .................................................................................................. 18
iii Tungsten version 2.0.4
Tungsten Replicator Installation Guide 5.2.4. Solaris Prerequisites ....................................................................................................... 5.3. Java Virtual Machine ................................................................................................................. 5.4. Sudo Prerequisites .................................................................................................................... 5.4.1. Sudo Configuration Hints ................................................................................................. 5.5. Network Prerequisites ................................................................................................................ 5.6. MySQL Database Prerequisites .................................................................................................. 5.6.1. Configuring MySQL System Variables for Tungsten .......................................................... 5.6.2. Configuring XtraBackup ................................................................................................... 5.7. PostgreSQL Database Prerequisites ........................................................................................... 5.7.1. Configuring PostgreSQL Server Account Access .............................................................. 5.7.2. Preparing the Archive Folder ........................................................................................... 5.7.3. Configuring Primary and Standby Servers ........................................................................ 5.8. Oracle Database Prerequisites ................................................................................................... 5.9. Backup Prerequisites ................................................................................................................. 5.9.1. Backup Storage .............................................................................................................. 5.9.2. Backup Methods ............................................................................................................. 6. Detailed Configuration .......................................................................................................................... 6.1. Message Log Configuration ........................................................................................................ 6.1.1. Replicator Logs ............................................................................................................... 6.1.2. Introducing Log4j ............................................................................................................ 6.1.3. Log4j Configuration Settings ............................................................................................ 6.2. Java Service Wrapper Configuration ........................................................................................... 6.2.1. Introducing Service Wrapper ........................................................................................... 6.2.2. Wrapper Configuration Settings ....................................................................................... 6.3. Network Configuration ................................................................................................................ 6.3.1. Network Port Access ....................................................................................................... 6.4. JMX Configuration ..................................................................................................................... 6.4.1. Introducing JMX .............................................................................................................. 6.4.2. JMX Configuration Settings ............................................................................................. 6.4.3. JMX on NAT-enabled hosts ............................................................................................. 6.5. Backup Configuration ................................................................................................................. 6.5.1. Xtrabackup ..................................................................................................................... 6.5.2. Mysqldump ..................................................................................................................... 6.5.3. Custom Backup Methods ................................................................................................ 6.5.4. Automatic Backup ........................................................................................................... 6.5.5. Automatic Restore .......................................................................................................... 6.5.6. Configuring backup after installation ................................................................................. 19 19 20 20 21 22 23 23 24 25 26 26 27 27 27 27 29 29 29 29 29 30 30 30 30 30 31 31 31 31 31 31 32 32 32 32 33
iv Tungsten version 2.0.4
Introduction
Chapter 1. Introduction
This document contains the installation procedure for Tungsten 2.0.4. The procedure describes how to install Tungsten Replicator in the following configurations: Basic master/slave MySQL native slave replacement with parallel replication Multi-master replication There are many sources of information about Tungsten Replicator besides the product documentation. You can find useful tips and tutorials in the following locations. Tungsten Replicator project home at http://code.google.com/p/tungsten-replicator. This contains information about location of builds, mailing lists, IRC channels, and issue tracking. Blog articles. Two blogs that specialize in replication topics and especially Tungsten are the Scale-Out Blog [http://scale-out-blog/blogspot.com] and The Datacharmer [http://datacharmer.blogspot.com]. Webinars and conference presentations. Check the Continuent website at http://www.continuent.com for more information.
Note
Tungsten installation changed radically between version 2.0.3 and 2.0.4. Common installation patterns are now single-line commands. The pre-2.0.4 installations are deprecated and will be removed in the next release.
1 Tungsten version 2.0.4
Product Installation and Configuration
Chapter 2. Product Installation and Configuration

This section provides a quick introduction to download and installation in a standard master/slave configuration. Further replication topologies are covered in Chapter 4, Configuring Specialized Replication Topologies.
2.1. Prerequisites
Before installing, you must meet all prerequisites for Tungsten installation as documented in Chapter 5, Host Prerequisites. The Tungsten installer script checks these and other prerequisites and will fail if they are not met.
2.2. Software Download

Tungsten 2.0.4 downloads are available from the Tungsten project on code.google.com at the following URL: http://code.google.com/p/tungsten-replicator/downloads/list Download the latest version of the software from this location using a web browser. You can also obtain development builds. Check the Tungsten Project home page for details.
2.3. Setting Up Master/Slave Replication

Master/slave replication uses two replicators that run identically named replication services (here named logos) on each replicator. The role name is shown in parentheses. The following diagram shows the master/slave topology.
Figure 2.1. Master/slave Topology
The master database replication service operates in the "master" role, while the slave database replication service operates in the "slave" role. In this topology, replicator processes may run on the same host as the MySQL DBMS or remotely on separate hosts.
2.3.1. Master Replicator Configuration

Follow the steps shown below to install and configure the master. The sample procedure uses the service name logos. You may change this name as desired. 1. Unpack the software in a temporary directory on either the master or slave host and cd into the release directory.
tar -xvzf tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4 2. Execute the following command to set up master/slave replication. It will copy the release to the home directory you select, configure the "logos" replicator service, and start the replicator. You must supply the correct login and password for the MySQL DBMS. tools/tungsten-installer --master-slave --master-host=logos1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=logos \ --home-directory=/opt/continuent \ --cluster-hosts=logos1,logos2 \ --start-and-report \
Master/slave replication is now configured and ready for use.
2.3.2. Testing Replication

To confirm replication is correctly set up, use heartbeat events as shown below. 1. Run trepctl services on each replicator and note the current sequence number. $ trepctl -host logos1 services Processing services command... NAME VALUE -------appliedLastSeqno: 0 appliedLatency : 0.891 role : master serviceName : logos serviceType : local started : true state : ONLINE Finished services command... $ trepctl -host logos2 services Processing services command... NAME VALUE -------appliedLastSeqno: 0 appliedLatency : 4.765 role : slave serviceName : logos serviceType : local started : true state : ONLINE Finished services command...
2. 3.
Run trepctl heartbeat on the master. Run trepctl services again on each replicator and confirm the sequence number has advanced by at least one. $ trepctl -host logos1 services Processing services command... NAME VALUE -------appliedLastSeqno: 1 appliedLatency : 0.057 role : master serviceName : logos serviceType : local started : true state : ONLINE Finished services command... $ trepctl -host logos2 services Processing services command... NAME VALUE -------appliedLastSeqno: 1 appliedLatency : 0.79 role : slave serviceName : logos serviceType : local started : true state : ONLINE Finished services command...
Tungsten Configuration Procedures
Chapter 3. Tungsten Configuration Procedures

Tungsten configuration procedures are located in the tools directory of the Tungsten release directory. There are three main scripts that help with installation. configure - Sets up release directory and defines config information for hosts and database servers. configure-service - Configures and starts replication services on replicator processes. tungsten-installer - Sets up common replication topologies using a single command. This command combines the functions of configure and configure-service.
Warning
The configure and configure-service scripts located in the root Tungsten directory are now deprecated. They will be removed in a future build. We recommend at all users upgrade to the new installs as they are more powerful and much easier to use.
3.1. tungsten-installer
The tungsten-installer script installs common replication topologies such as master/slave and direct configurations in a single command. The tungsten-installer script copies code from the directory in which you unpack the Tungsten tar ball to the final release directory. It can copy and configure across multiple hosts. The following command shows a typical example of tungsten-installer usage. tools/tungsten-installer --master-slave --service-name=logos \ --master-host=logos1 \ --cluster-hosts=logos1,logos2 \ --home-directory=/opt/tungsten \ --datasource-user=tungsten \ --datasource-password=secret \ --start-and-report \
The following subsections provide tips on commonly used arguments for replicator installation.
3.1.1. Getting Help

The --help option prints basic installer help. To see a list of standard options for all topologies, use --help-all. To see an exhaustive list of all options including expert options, use --help-all -a.
3.1.2. Replicator Topologies

You must designate a replication topology using either the --master-slave or --direct options. One or the other option is required. Master/slave topology installs a master with one or more slaves. You must list the hosts in the configuration using the --cluster-hosts option and designate a master using the --master-host. 5 Tungsten version 2.0.4
Tungsten Configuration Procedures Direct topology installs a single replicator that replicates directly from a master to a slave. You should list a single host in the --cluster-hosts option.
3.1.3. Service Name

The replication service name identifies a group of services that share a common master. You can set the name with the --service-name option.
3.1.4. Home Directory

The normal way to install Tungsten is to unpack in a temporary directory on a single host and run tungsten-installer with the --home-directory option to copy files to the final install location. The home directory can have the same name as the directory into which you initially unpack files, in which case tungsten-installer will not copy files on that host.
3.1.5. Database Login and Passwords

You can specify database login and password using the --datasource-user and --datasource-password options. You can also specify credentials separately for master and slave, for example using the --master-user option.
3.1.6. Replicator Start-Up

You may choose to start replicator(s) automatically after installation. The --start starts the replicator process The --start-and-report option starts the replicator process and also fetches the service status after startup.
3.1.7. Installation Checks

tungsten-installer automatically checks a wide range of prerequisites before doing the installation. If there is an an error the installation will abort. Sometimes you may need to install in spite of a failed installation check. You can suppress individual checks by name using --skip-validation-check as in the following example. tools/tungsten-installer --master-slave \ ... \ --skip-validation-check=MySQLApplierServerIDCheck \ --start-and-report You can suppress all validation checks with the --no-validation option.
3.1.8. Advanced Property Settings

tungsten-installer allows a wide range of option settings that cover standard replication configurations. However, you may find it necessary to make arbitrary customizations of replication services. If you specify -a you may use the following options to add additional filters to replication pipelines. --svc-extractor-filters - Add one or more extra filters after extraction. --svc-thl-filters - Add one or more extra filters before placing an event in the transaction history log. --svc-applier-filters - Add one or more extra filters before applying an event to a slave DBMS. 6 Tungsten version 2.0.4
Tungsten Configuration Procedures Finally, you can also override specific properties in the replication service properties file using the -properties=name=value option, as shown in the following example. tools/tungsten-installer --master-slave -a \ ... \ --property=replicator.filter.shardfilter.unknownShardPolicy=error \ --start-and-report
3.2. configure
The configure performs base replication setup and release directory configuration. Among other things it generates a file named tungsten.cfg that contains information about hosts as well as replicator process settings. This file is used as a basis for configuring replication services using configure-service.
Caution
The tungsten.cfg format is based on JSON and is completely different from Tungsten 2.0.3 and earlier configuration files. The following example shows an example of setting up replicator code and processes on two hosts named logos1 and logos2. tools/configure --home-directory=/opt/continuent \ --cluster-hosts=logos1,logos2 \ --start-and-report=true The following subsections provide tips on commonly used arguments for configure. Generally speaking options are consistent with those used by tungsten-installer.
3.2.1. Getting Help

As with other installation commands, the --help option prints help. To see an exhaustive list of all options including expert options, use --help -a.
3.2.2. Cluster Hosts

You may install code across multiple hosts using the --cluster-hosts option. If omitted code will only be installed on the current host.
3.2.3. Home Directory

configure supports the same --home-directory option to copy files to the final install location as tungsten-installer.
3.2.4. Replicator Start-Up

You may choose to start replicator(s) automatically after installation. The --start starts the replicator process The --start-and-report option starts the replicator process and also fetches the service status after startup. Note that the replicators will not have any replication services defined when they start.
Tungsten Configuration Procedures
3.2.5. Installation Checks

configure automatically checks a wide range of prerequisites before doing the installation. As with other commands you can skip checks individually with --skip-validation-check. You can skip all checks using --no-validation.
3.3. configure-service
The configure-service script configures an individual replication service. Tungsten permits multiple replication services to run inside a single replicator process. You must first have run either tungsten-installer or configure to set up the tungsten.cfg file. configure-service usage is similar to tungsten-installer, with many of the same options. The following example shows how to define an extra slave service for multi-master replication. tools/configure-service -C --role=slave --host=logos2 \ --service-name=logos1 \ --master-host=logos1 \ --local-service-name=logos2 \ --release-directory=/opt/continuent/tungsten --service-type=remote --allow-bidi-unsafe=true --svc-start The following subsections provide tips on commonly used arguments for service configuration.
Note
To configure replication services effectively you will need to understand exactly how replication service properties files work. There are many options and not all of them play well together. Look at the Tungsten Replicator Guide and/or existing service properties files for hints on correct settings.
3.3.1. Getting Help

As with other installation commands, the --help option prints help. To see an exhaustive list of all options including expert options, use --help -a.
3.3.2. Replicator Service Role

You must designate a service role, such as "master", "slave", or "direct" using the --role option.
3.3.3. Service Name

You must specify a service name using the --service-name option.
3.3.4. Service Type

Replication services come in two types: local and remote. Local services are the normal type used in master/slave replication and do not require special configuration. There is normally only one local service per replicator process.
Tungsten Configuration Procedures Remote services are used to transfer data from one master to another in multi-master replication. Remote services are normally slaves in the current replicator implementation. There may be multiple remote services per replicator process. You can configure a remote service using the --service-type=remote option. When you do so you must also provide the name of the master service using the --local-service-name option.
3.3.5. Release Directory

You must specify the location of an existing Tungsten release created either by tungsten-installer or configure.
Configuring Specialized Replication Topologies
Chapter 4. Configuring Specialized Replication Topologies

This section contains instructions on configuration of replication topologies other than the conventional master/slave.
4.1. MySQL Slave Replacement with Parallel Replication

In this replication configuration Tungsten replaces the function of the MySQL slave, which allows users to apply Tungsten parallel replication capabilities. The following diagram shows the MySQL slave replacement topology.
Figure 4.1. MySQL Slave Replacement Topology
In this topology, a Tungsten replicator takes over the function of the MySQL IO and SQL threads. This enables use of Tungsten parallel apply, which can speed up I/O bound MySQL slaves considerably. The replicator uses a direct pipeline that logs events and then applies them to the slave.
4.1.1. MySQL Replication Setup

The first step in slave replacement is to configure native MySQL replication. MySQL replication configuration is described in MySQL documentation, for example at the following URL: http://dev.mysql.com/doc/refman/5.1/en/replication-howto.html The following procedure summarizes MySQL replication setup. Note that this is a simplified configuration that minimizes steps. It assumes that both master and slave are idle. On the MySQL master host, perform the steps shown below. 1. Edit my.cnf and set a unique value for server-id and the location and file pattern for binlogs. You may optionally set the binlog format for MySQL 5.1 and above. The following example shows typical values.
[mysqld] server-id=10 log-bin=/usr/local/mysql-5.1/data/mysql-bin binlog_format=statement 2. Restart mysqld so that any changed my.cnf values take effect. If you have not changed values, skip this step. $ sudo service mysql restart
Note
Depending on your OS and distribution you may need to use sudo restart mysql or /etc/init.d/mysqld restart. 3. Create a replication account if it does not already exist. The account must have REPLICATION SLAVE privilege at a minimum. The user name and password of this account will be used to configure the slave. mysql> GRANT REPLICATION SLAVE ON *.* to tungsten@'%' IDENTIFIED by 'secret'; 4. Execute a RESET MASTER command to delete old logs and reset binary logging. mysql> RESET MASTER; Query OK, 0 rows affected (0.74 sec) On the MySQL slave host, perform the steps shown below. 1. Edit my.cnf and set a unique value for server-id. Note also the location of the data directory, as this is where relay logs are written. [mysqld] server-id=20 2. Restart the slave mysqld so that any changed my.cnf values take effect. If you have not changed values, skip this step. sudo service mysql restart 3. Login to the MySQL server and issue a CHANGE MASTER command to enable built-in MySQL replication. This command uses the user name, password, and log file data collected on the master. Then start the slave IO and SQL threads.
mysql> CHANGE MASTER TO master_host='master_hostname', master_port=3306, master_user='tungsten', master_password='secret' mysql> START SLAVE 4. Confirm that replication is working properly by creating a table on the master and ensuring that it replicates through to the slave.
4.1.2. Tungsten Configuration and Slave Takeover

Follow the steps shown below to configure Tungsten and enable slave takeover. The sample procedure uses the service name sql_thread. You may change this or other parameters as needed. 1. 2. Unpack software on the MySQL slave host. Execute the following installation command. (Your options may differ.) tools/tungsten-installer \ --direct \ --native-slave-takeover \ --master-host=logos1 \ --master-user=repl \ --master-password=secret \ --slave-host=logos2 \ --slave-user=tungsten \ --slave-password=secret \ --service-name=sql_thread \ --home-directory=/opt/continuent \ --svc-parallelization-type=disk \ --channels=10 \ --start-and-report
Warning
Parallel replication depends on workload being distributed across independent shards. If you are unsure whether your applications meet this requirement, omit the --channels option to use a single apply thread.
4.1.3. Re-enabling MySQL Slave SQL Thread

You can stop Tungsten replication and re-enable the native SQL thread as follows. 1. Stop Tungsten sql_thread service. $ trepctl -service sql_thread offline
$ replicator stop 2. Re-enable MySQL replication. mysql> start slave; 3. Confirm that MySQL native replication is working correctly by entering one or more transactions on the master and ensuring that they replicate correctly through to the slave.
4.2. Setting up Bi-Directional Replication

Bi-directional replication uses two replicators with two services each. One service is a local master service, which means that it does not attempt to log updates to the database. The other is a remote slave service that fetches from the remote master. The following diagrams show the service names, role names, and service types in this topology. Just to add interest we have also added a replicator C which is an ordinary slave.
Figure 4.2. Bi-Directional Replication Topology
Any update on a master will replicate to that master's slaves, including the remote services. So, for example, an update on the DBMS attached to Replicator A will be picked up by Replicator A's master service logos1. The remote slave service logos1 will receive the update and apply a logged update to the DBMS attached to Replicator B. This update will in turn be read from the binlog by the logos2 master service and hence replicated to the DBMS attached to Replicator C.
Warning
In this example there are two distinct replication services. The replication service should be named the same on the master and the slave to work properly. Service names must also be unique within each instance of Tungsten Replicator and each database server.
4.2.1. Unpack software

Start by unpacking the Tungsten tarball in a temporary location and cd into the release directory. This location should be different from the final installation or you may get confused after installing to multiple hosts.
4.2.2. First Master Replicator Configuration

Follow the steps shown below to install and configure the first master. In this case we will configure Replicator A from the example shown previously. The sample procedure uses the service name logos1 for this master. tools/tungsten-installer --master-slave \ --master-host=logos1 \ --cluster-hosts=logos1 \ --service-name=logos1 \ --datasource-user=tungsten \ --datasource-password=secret \ --home-directory=/opt/continuent \ --start-and-report The first master replication service is now up and running.
4.2.3. Second Master + Slave Replicator Configuration

In this case we will use Replicator B and Replicator C from the example shown previously. The sample procedure uses the service name logos2. tools/tungsten-installer --master-slave \ --master-host=logos2 \ --cluster-hosts=logos2,logos3 \ --service-name=logos2 \ --datasource-user=tungsten \ --datasource-password=secret \ --home-directory=/opt/continuent \ --start-and-report The second master and its slave are now up and running.
4.2.4. Remote Slave Service Configuration

Remote slaves are special replication services that transfer data between masters. To complete the installation we must define a remote slave on each master. First create and start a remote slave replication service on logos1 to transfer data from the master on logos2. tools/configure-service -C --host=logos1 \ --role=slave \ --master-host=logos2 \
--service-name=logos2 \ --service-type=remote \ --local-service-name=logos1 \ --datasource=logos1 \ --release-directory=/opt/continuent/tungsten \ --allow-bidi-unsafe=true \ --svc-start Now create a remote slave replication service on logos2 to transfer data from the master on logos1. tools/configure-service -C --host=logos2 \ --role=slave \ --master-host=logos1 \ --service-name=logos1 \ --service-type=remote \ --local-service-name=logos2 \ --datasource=logos2 \ --release-directory=/opt/continuent/tungsten \ --allow-bidi-unsafe=true \ --svc-start Once these commands complete bi-directional replication is set up and working. Try entering transactions on both masters to confirm that data move in both directions as well as onto the the slave on logos3.
4.3. Setting up MySQL to PostgreSQL Replication

This section will describe how to set up replication to PostgreSQL.
Host Prerequisites
Chapter 5. Host Prerequisites

This chapter contains a list of prerequisites for hosts that participate in a Tungsten cluster. Review these carefully to avoid problems during configuration and deployment.
5.1. Tungsten Login

Tungsten can run as root but it is better to use a separate login such as 'continuent' or 'tungsten'.
Table 5.1. Tungsten Checklist

Description Ensure install locations are writable. Check Ensure the tungsten account can write to the directory where the installation is unpacked. Expected Result The directory should be writable.
Ensure release directo- Ensure locations like /opt/ ry locations are writable. continuent exist and are writable by the tungsten account.
The locations should be writable.
5.1.1. Configuring Remote Host Access for Tungsten Login

Remote host access uses certificate-based ssh access. Here are the steps to generate certificates for host access for the 'tungsten' account. Substitute the proper name if you are going to use a different system user.
Note
File and directory names may vary according to your operating system and the type of key generated. 1. Use the ssh-keygen program to generate a key. Either omit a passphrase, or use ssh-agent when installing. Here is an example of use. pasayten:~/.ssh # ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/opt/tungsten/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /opt/tungsten/.ssh/id_rsa. Your public key has been saved in /opt/tungsten/.ssh/id_rsa.pub. The key fingerprint is: 4c:cf:40:4e:1e:a3:be:91:11:f1:9a:48:4a:5d:f8:33 tungsten@pasayten 2. Copy the private key id_rsa to ~tungsten/.ssh for each host that will participate in the cluster. 3. Similarly add the public key to the authorized_hosts file using a command like the following: cat id_rsa.pub >> ~/.ssh/authorized_keys 16 Tungsten version 2.0.4
Host Prerequisites 4. Ensure the .ssh directory and files within it are readable only to the 'tungsten' account. chmod 700 ~/.ssh;chmod 600 ~/.ssh/* 5. Test ssh access by logging in between hosts. You may need to do this once for each host-to-host combination, including the local hostname, to eliminate prompts like the following:
$ ssh pasayten The authenticity of host 'pasayten (172.16.238.128)' can't be established. RSA key fingerprint is 16:54:e3:e4:e8:88:45:fc:62:07:5e:69:59:86:4c:c4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'pasayten,172.16.238.128' (RSA) to the list of known hosts.
5.1.2. Configuring Remote Host Access for PostgreSQL Login

Remote host access uses certificate-based ssh access. Here are the steps to generate certificates for host access for the 'postgres' account.
Note
File and directory names may vary according to your operating system and the type of key generated. 1. Use the ssh-keygen program to generate a key. Do not put a passphrase on the key. Here is an example of use. pasayten:~/.ssh # ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/var/lib/pgsql/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /var/lib/pgsql/.ssh/id_rsa. Your public key has been saved in /var/lib/pgsql/.ssh/id_rsa.pub. The key fingerprint is: 4c:cf:40:4e:1e:a3:be:91:11:f1:9a:48:4a:5d:f8:33 postgres@pasayten 2. Copy the private key id_rsa to ~postgres/.ssh for each host that will participate in the cluster. 3. Similarly add the public key to the authorized_hosts file using a command like the following: cat id_rsa.pub >> ~/.ssh/authorized_keys 4. Ensure the .ssh directory and files within it are readable only to the 'postgres' account. chmod 700 ~/.ssh;chmod 600 ~/.ssh/* 5. Test ssh access by logging in between hosts. You may need to do this once for each host-to-host combination, including the local hostname, to eliminate prompts like the following:
$ ssh pasayten The authenticity of host 'pasayten (172.16.238.128)' can't be established. 17 Tungsten version 2.0.4
Host Prerequisites
RSA key fingerprint is 16:54:e3:e4:e8:88:45:fc:62:07:5e:69:59:86:4c:c4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'pasayten,172.16.238.128' (RSA) to the list of known hosts.
5.2. Operating System Prerequisites

5.2.1. Notes on Operating System Prerequisites
Tungsten has minimal prerequisites for installation and configuration. The following sub-sections describe requirements for each supported operating system. Note that Ruby is required for all operating systems.
5.2.2. Linux Prerequisites

Table 5.2. Linux Checklist
Description Check for the which command. Check for the rsync command. Check Ruby release. Check (commands in this font) which ruby which rsync ruby --version Expected Result You should see the path to the Ruby command like / usr/bin/ruby You should see the path to the rsync command like / usr/bin/rsync You should see a Ruby 1.8 or later version, as in the following example: ruby 1.8.5 (2006-08-25) [i386-linux] You can install Ruby using yum install ruby for Redhat distributions or apt-get install ruby for Ubuntu. Check for the Openssl Ruby module. echo "p 'hello'" | ruby -ropenssl The command should output "hello"
5.2.3. Mac OS X Prerequisites

Table 5.3. Mac OS X Checklist
Description Check for the which command. Check for the rsync command. Check Ruby release. Check (commands in this font) which ruby which rsync ruby --version Expected Result You should see the path to the Ruby command like / usr/bin/ruby You should see the path to the rsync command like / usr/bin/rsync You should see a Ruby 1.8 or later version, as in the following example: ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] The command should output "hello"
Check for the Openssl Ruby module.
echo "p 'hello'" | ruby -ropenssl
Host Prerequisites
5.2.4. Solaris Prerequisites

Table 5.4. Solaris Checklist
Description Check for the which command. Check for the rsync command. Check Ruby release. Check (commands in this font) which ruby which rsync ruby --version Expected Result You should see the path to the Ruby command like / usr/bin/ruby You should see the path to the rsync command like / usr/bin/rsync You should see a Ruby 1.8 or later version, as in the following example: ruby 1.8.5 (2006-08-25) [i386-linux] Ruby is not installed by default on all Solaris hosts. You can obtain Ruby from http://www.sunfreeware.com. Check for the Openssl Ruby module. Check for GNU tar. echo "p 'hello'" | ruby -ropenssl [/usr/local/bin/]tar -version The command should output "hello" You should see GNU tar, for example: tar (GNU tar) 1.21 Copyright (C) 2008 Free Software Foundation, Inc. GNU tar is not installed by default on most Solaris hosts. You can obtain GNU tar from http:// www.sunfreeware.com.
5.3. Java Virtual Machine

Tungsten requires either Oracle JDK 6 or OpenJDK 6. Other Java versions will result in failures. Users can obtain the Oracle JDK from http://java.sun.com. The OpenJDK is pre-installed for many Linux distributions or can be obtained quickly using a suitable command like sudo apt-get install openjdk-6-jre. You may use either the full JDK or JRE.
Table 5.5. Java Checklist

Description Check JAVA_HOME. Check Java release. Check (commands in this font) echo $JAVA_HOME java -version Expected Result Should point to Sun JDK install location. You should see the Sun JDK, for example: java version "1.6.0_18" OpenJDK Runtime Environment (IcedTea6 1.8) \ (6b18-1.8-4ubuntu3~9.10.2) OpenJDK 64-Bit Server VM (build 16.0-b13, \ mixed mode)
Host Prerequisites
Important
Many Linux distributions ship non-Sun Java versions such as GNU Java. These versions use different command line flags and different libraries from the Sun JDK. We recommend users remove non-Sun versions if present as they may cause Tungsten processes to fail at start-up or otherwise misbehave.
Warning
Do not mix Java versions. All hosts running Tungsten must run Java 6. Running a mix of versions may result in runtime errors.
5.4. Sudo Prerequisites

Non-root accounts typically require sudo to start and stop database servers through the Tungsten as part of a backup or restore procedure. Only backup procedures such as XtraBackup that need root access require sudo privileges. Sudo access is also required to manage PostgreSQL Warm Standby replication, which requires a server restart during slave provisioning.
Note
You can disable install-time checking for sudo using --skip-validation-check=SudoCheck.
Table 5.6. Network Checklist

Description Check (commands in this font) Expected Result Must show a list of commands that this login can execute without a password. For example like (ALL) NOPASSWD: ALL. Ensure the command works. If you get a message like sudo: sorry, you must have a tty to run sudo you will need to unset requiretty as described below.
Ensure sudo is config- sudo -l ured for non-root accounts and does not require a password. Allow batch sudo commands from ssh. ssh localhost sudo -l requiretty is unset.
5.4.1. Sudo Configuration Hints

The sudo program is configured using the sudoers file. You must run visudo as root to edit this file. Recent sudo releases by default prevent commands like ssh from issuing sudo commands remotely. Tungsten needs this capability, which you can enable by commenting out requiretty. More recent versions of sudo may also require that you enable visiblepw. Actual sudo privileges must be accessible without a password or Tungsten will hang while attempting to execute commands. For MySQL users:
# Batch sudo commands do not work if requiretty is set.
Host Prerequisites
#Defaults requiretty Defaults visiblepw ... ## For MySQL users only use one of these two options ## Allow tungsten to run any command tungsten ALL=NOPASSWD: ALL For PostgreSQL users:
# Batch sudo commands do not work if requiretty is set. #Defaults requiretty Defaults visiblepw ... ## For PostgreSQL users ## Allow postgres to run any command postgres pasayten=(root) NOPASSWD: ALL
5.5. Network Prerequisites

Tungsten requires that machines have assigned host names and that all hosts used in a Tungsten cluster have names that resolve reliably to non-loopback IP address. IP addresses and ports used by Tungsten may not be blocked by firewalls.
Table 5.7. Network Checklist

Description Check hostname. Check local host IP. Check (commands in this font) uname -n hostname --ip-address Expected Result Should resolve to the unique name of host. Must resolve to a "real" IP address; not to the loopback address 127.0.0.1. Private IP addresses are accepted. All host names must resolve to a non-loopback IP address. There should be no overlaps or inconsistency in names between hosts. The ports must be visible. For test installations we recommend you remove firewalls between hosts. Also, do not forget normal database ports like 3306 for MySQL.
Ensure that all other host names in cluster resolve to their IP addresses from this host. Check visibility of TCP/ IP ports.
hostname --ip-address
Check visibility of ports from another host. You can use telnet hostname port to verify accessibility. Here are the key ports that must be open. 2112 - Port for replication THL. Must be open on any host that runs a replicator. 10000 - Port for Replicator JMX connections. Must be
Host Prerequisites
Description
Check (commands in this font) open to enable remote access to the replicator.
Expected Result
Warning
Tungsten is highly dependent on correct and consistent resolution of host names to IP addresses. Inconsistencies due to different /etc/hosts files or other name resolution issues are one of the most common causes of cluster failures. Always check host name resolution very carefully on each new host and recheck if you encounter problems in operation.
5.6. MySQL Database Prerequisites

Tungsten installations that use MySQL must be able to reach MySQL using TCP/IP. Login permissions and requirements vary by Tungsten component. The Tungsten Replicator requires an account with full administrative privileges in order to perform replication. (For details refer to Chapter 2 of Tungsten Replicator Guide.)
Table 5.8. MySQL Database Checklist

Description Check the resolution and accessibility of backend host. Check (commands in this font) ping host name or ip Expected Result Should ping successfully.
Ensure recommended Review settings in my.cnf and MySQL parameters are restart MySQL if any changes set. are made.
Settings must match or exceed required values shown in Section 5.6.1, Configuring MySQL System Variables for Tungsten.
Download the latls -al /opt/mysql/mysql-conThe JDBC jar file is readable by the tungsten usest MySQL JDBC nector-java-5.1.13/mysql-con- er. driver from http:// nector-java-5.1.13-bin.jar www.mysql.com/downloads/connector/j/ to /opt/mysql. This is required only if you need to replicate data to MySQL 4.1. Allow the tungsten us- ls -al /var/lib/mysql/ er to read the binary log files. Add the tungsten user to the mysql group in /etc/group and ensure MySQL binlogs are group readable. Check TCP/IP login of account used for replication. mysql -utungsten -h{hostname} --port=3306 -psecret The tungsten user can read binary log files from the MySQL directory.
Must be able to login using password.
Host Prerequisites
Description Check that replicator login can create a database. Check TCP/IP login of client accounts used by applications. Check ability to restart database. This is required if you use a backup method that needs to restart the DBMS.
Check (commands in this font) mysqladmin -utungsten -h{hostname} --port=3306 -psecret create test
Expected Result Must be able to create a database successfully.
Must be able to login; the password is optional.
[sudo] /etc/init.d/mysqld [start Must be able to execute commands using sudo | stop] for non-root accounts. Root does not require sudo. The boot script name may vary according to your operating system and version of MySQL. See the sudo configuration instructions in Section 5.4, Sudo Prerequisites for hints on setting up sudo.
5.6.1. Configuring MySQL System Variables for Tungsten

Tungsten requires minimal values for certain system variables that are larger than the defaults supplied by MySQL. These values are specified in /etc/my.cnf for most platforms. The following example shows my.cnf values that must be changed from their default values. [mysqld] # Master replication settings. log-bin=mysql-bin server-id=1 # Required InnoDB parameter settings for Tungsten. Buffer pool size may be # larger but should not be smaller for production deployments. innodb_buffer_pool_size = 512M # Recommended InnoDB settings for Tungsten. default-table-type=InnoDB innodb_flush_log_at_trx_commit=2 sync_binlog=0 # Recommended general settings. max_allowed_packet must be greater than # the size of the largest transaction. max_allowed_packet=48m
Note
Whenever you make changes to my.cnf values you should restart MySQL and check settings using the SHOW VARIABLES command to ensure they take effect.
5.6.2. Configuring XtraBackup

You must make sure that the innobackupex-1.5.1 program is in the path if you would like to use XtraBackup on your database server.
Host Prerequisites XtraBackup is not bundled with Tungsten Enterprise but the latest version can be downloaded from http:// www.percona.com/downloads/XtraBackup/LATEST/.
Table 5.9. MySQL XtraBackup Checklist

Description Confirm that innobackupex-1.5.1 is in the current path. A sample command is shown below. Check (commands in this font) which innobackupex-1.5.1 Expected Result Should show a path to innobackupex-1.5.1 like /usr/bin/innobackupex-1.5.1.
5.7. PostgreSQL Database Prerequisites

Tungsten installations that use PostgreSQL must be able to connect using TCP/IP without a password to both the local server as well as the master server. In addition, warm standby clusters require certificate-based ssh access in order to rsync files from the master to slave server during provisioning.
Warning
Tungsten for PostgreSQL accounts MUST use the PostgreSQL system account to run Tungsten. For standard PostgreSQL installations, you must perform the following checks while logged in as the 'postgres' user on the operating system.
Table 5.10. PostgreSQL Database Checklist

Description Check (commands in this font) Expected Result Should return version information for pg_standby, as in the following example: pg_standby (PostgreSQL) 8.4.0. Must be able to login without a password. See the note on configuring account access below if this check does not succeed.
Confirm presence of pg_standby --version pg_standby executable. Check TCP/IP access psql -hlocal_host_name to local Postgres server using PostgreSQL system account without a password. Check TCP/IP access psql -hmaster_host to the master Postgres server using PostgreSQL system account without a password. Check ability to restart database [sudo] /etc/init.d/postgresql [start | stop]
Must be able to login to master host without a password. See the note on configuring account access below if this check does not succeed.
PostgreSQL login must be able to restart the database. See sudo configuration instructions in Section 5.4, Sudo Prerequisites to set up sudo access if this test does not work. The database start script name and location may vary according to your operating system and version of PostgreSQL.
Host Prerequisites
Description Check ability to ssh to other servers in the cluster without a password. Do this for each server that participates in the cluster.
Check (commands in this font) ssh host_name ls
Expected Result Command must show files on the remote host without prompting for a password or host key authentication. If you do not have certificates configured see the note below on host access.
5.7.1. Configuring PostgreSQL Server Account Access

Tungsten depends the PostgreSQL system account being able to log in without a password to the local PostgreSQL server. Warm standby clustering must also login to the master server without a password in order to start backups. PostgreSQL supports such logins in a secure manner but needs to be configured carefully to avoid introducing security holes. For local access only we recommend configuring ident authentication using the pg_hba.conf file, which controls server access. Use an editor to make these changes. Here is an example of typical ident settings for local access that also preserves password access for logins from remote hosts.
# "local" is for Unix domain socket connections only local all all # IPv4 local connections: host all all 127.0.0.1/32 # TCP/IP connections host all all 172.16.238.1/24
ident ident md5
Note
Whenever you make changes to the pg_hba.conf file you should restart the PostgreSQL server and double check settings to ensure they work correctly. In cases where both local and remote access are required, we recommend using a .pgpass file, which contains passwords to be used for logins. The following instructions describe how to create a .pgpass file for the 'postgres' account. 1. While logged in as 'postgres', edit file .pgpass in the home directory using a command like the following: vi ~/.pgpass 2. Add a line for the 'postgres' account like the following example, where 'secret' is the password. *:*:*:postgres:secret 3. Ensure the file is readable only for the 'postgres' login using the following chmod command. chmod 600 ~/.pgpass If Streaming Replication is used, pg_hba.conf must specify a connection for replication explicitly (it is not considered a part of option all). For example: 25 Tungsten version 2.0.4
Host Prerequisites
host all all 172.16.210.0/24 md5 host replication all 172.16.210.0/24 md5 Switch and failover will not work unless you enable connections for replication on all your PostgreSQL servers.
Note
Tungsten relies on the psql console command. Tungsten does not specify a password to this command as a parameter. Instead, it relies on you to preconfigure it in the .pgpass file. Thus, if the postgres user has a password, define it in the .pgpass file and check whether a plain psql call can connect to the database.
5.7.2. Preparing the Archive Folder

Tungsten saves new WAL files into what is known as the archive folder. On a master, new WAL files stay in this folder until they are sent out to slaves, and on a slave until they are recovered and the pg_standby process decides that they are no longer needed. You must create this folder before running ./configure, which will prompt for it. The archive folder must not be a subfolder of the PostgreSQL data directory and the postgres user must own it.
Note
In the case of a lagging slave, the master's archive folder (to be more specific, the outbox folder inside of it) will accumulate WAL files, which are waiting to be sent out to the lagging node. Depending on the lag, the archive folder can drastically increase in size. Slaves should not lag in a healthy cluster slaves, but be prepared for a corner case like this by providing a couple of gigabytes of extra disk space for the archive folder. In the case of an unseen noticeable slave lag, troubleshoot and solve the issue. For more information, see chapter How Tungsten Replicator Implements PostgreSQL WAL Shipping in Tungsten Replicator Guide.
5.7.3. Configuring Primary and Standby Servers

This section provides a list of recommendations on configuring the primary and standby servers in the PostgreSQL environment: Create the primary and standby servers in a way that they are as similar as possible, at least from the perspective of the database server. The path names associated with tablespaces will be passed across unmodified. If you use tablespaces, the primary and standby servers must have the same mount paths for tablespaces. The hardware architecture on the primary and standby servers must be the same. A 32-bit and 64-bit system are not interoperable. In general, log shipping between servers running different major PostgreSQL release levels is not possible. Because PostgreSQL Global Development Group does not make changes to disk formats during minor release upgrades, it is likely that running different minor release levels on the primary and standby servers will work. However, no formal support for that is offered and we recommend that you the keep the primary and standby 26 Tungsten version 2.0.4
Host Prerequisites servers at the same release level as much as possible. When updating to a new minor release, the safest policy is to update the standby servers first - a new minor release is more likely to be able to read WAL files from a previous minor release than vice versa.
5.8. Oracle Database Prerequisites

Note
When replicating to Oracle the database user must be tungsten_ plus the service name. If you are deploying a replication service called alpha, the user would be tungsten_alpha.
Table 5.11. Oracle Database Checklist

Description Check the resolution and accessibility of backend host. Check (commands in this font) ping host name or ip Expected Result Should ping successfully.
Check database login of sqlplus tungsten_alpha/ account used for repli- secret@ORCL cation.
Must be able to login.
5.9. Backup Prerequisites

5.9.1. Backup Storage
Tungsten requires the location of shared storage that is accessible by every replicator in the cluster. This location must be large enough to store multiple copies of your dataset. During the configuration process you will determine how many copies will be kept, the recommend number is 3. The procedure for setting up this shared storage is outside the scope of this document. Of the options available, the simplest is to mount an NFS shared directory on each server in your cluster, however this creates a potential point of failure. More complex options include the Oracle Cluster File System (OCFS) or GlusterFS, both of these allow you to mirror storage across all of your servers efficiently and consistently.
5.9.2. Backup Methods

You may as part of configuration specify a backup method to use. The standard options for MySQL are mysqldump or xtrabackup. The standard option for PostgreSQL is pg_dump. You also have the option to write a custom script that will be used for backup and restore operations. For more information refer to Section 6.5.3, Custom Backup Methods.
Note
Backup method names are lower case. 27 Tungsten version 2.0.4
Host Prerequisites
5.9.2.1. XtraBackup
The XtraBackup utility creates a consistent backup of the database without blocking write access for your application or the replication process. It is distributed by Percona at http://www.percona.com/software/percona-xtrabackup/ for multiple platforms. XtraBackup outputs a MySQL data directory that is ready for operation. During the restore process you will not need to wait for a SQL script to load when restoring a backup. In order to use XtraBackup, you must install the XtraBackup package on each server and ensure that the tungsten user has full sudo access.
5.9.2.2. Mysqldump
Mysqldump is supported by default with any MySQL installation. Mysqldump creates a SQL script that may be used to restore all data on a new server. Depending on your MySQL version, the Mysqldump script may require you to take the datasource offline prior to running the backup process. Mysqldump will run the SQL script during the restore process and rebuild all databases from the original database server. For large datasets this time can be extensive, but for small datasets it might be faster than XtraBackup.
Note
Mysqldump does not drop any databases that are unique to the target server during the restore process. This can be either a benefit or a risk depending on your installation so this should be considered when you are deciding which option to use.
5.9.2.3. PG Dump
PG dump is supported by default with any Postgresql installation. PG dump creates a SQL script that may be used to restore all data on a new server. PG dump will run the SQL script during the restore process and rebuild all databases from the original database server.
5.9.2.4. Which One Is Right For You?

For MySQL, our recommendation is that XtraBackup provides the best features to ensure data accessibility and consistency. You should try it in your environment to ensure that it meets your needs and you are comfortable using it. Since the XtraBackup project is not maintained by the MySQL community, it is possible that compatibility issues could arise in future releases. If XtraBackup is not an option for you, then we recommend Mysqldump because it is important to have an integrated backup option setup. For Postgresql, we recommend pg_dump because it is the only option available out of the box. If you have another backup method that you prefer to use then refer to Section 6.5.3, Custom Backup Methods for further instructions.
Detailed Configuration
Chapter 6. Detailed Configuration

This chapter contains information about detailed configuration of replication. This is intended to help you make configuration settings above and beyond those supplied by the automated Tungsten installation scripts.
6.1. Message Log Configuration

6.1.1. Replicator Logs
Tungsten provides two message logs both of which are located by default in the tungsten-replicator/log directory. The user.log contains summary information about replicator operation including state changes as well as errors that cause replication to stop. It is useful for monitoring as well as getting a quick history of replicator status. The trepsvc.log contains detailed diagnostic messages. It is useful for diagnosing problems as well as showing exact start and stop positions of replication. Position information is often useful for recovering safely from crashes.
6.1.2. Introducing Log4j

Tungsten uses Apache log4j to log messages. Each replicator process has a log4j.properties file to control the level of log messages and whether they go to the console, a file, or to Unix syslog facility. For further information on log4j.properties, check the documentation at the Apache Log4j website, which is located at http://logging.apache.org/log4j/1.2/index.html.
6.1.3. Log4j Configuration Settings

All Tungsten services come with pre-configured log4j settings. You may alter these as you wish in order to adjust log levels or destination by editing the log4j.properties file. After changing settings, you must restart the service for new settings to take effect. Generally speaking, Tungsten services define the following output destinations. stdout - Sends output to the console. This setting is the default for all Tungsten services. The service wrappers automatically redirect this to a file. file - Send output to a log file with defined rules for roll-over. syslog - Send output to Unix syslog facility. Most Tungsten services write at the INFO level, which produces a reasonable level of notifications, warnings, and error messages. The DEBUG level may be enabled for diagnostic purposes. Examples are provided at the end of each log4j.properties file. The following example shows log4j settings to enable debug logging on Tungsten replicator. # Example of how to turn on debugging. Specify the name of a package or # a Java class. This turns on debugging for all replicator classes. log4j.logger.com.continuent.tungsten.replicator=DEBUG, Console log4j.additivity.com.continuent.tungsten.replicator=false
Note
Changing Log4j settings only affects the service whose log4j.properties you alter. Other services are not affected.
Warning
Debug logging generates voluminous output and slows replication. It should not be enabled on production systems unless required to diagnose problems. For best results, enable logging on specific classes or packages rather than all replicator code.
6.2. Java Service Wrapper Configuration

6.2.1. Introducing Service Wrapper
Tungsten uses Java Service Wrapper (http://wrapper.tanukisoftware.org/) to allow Java processes that implement cluster functions to operate as operating system services. The Java Service Wrapper uses a configuration file that specifies arguments for the Java process. From time to time it is necessary to make configuration changes to the wrapper.
6.2.2. Wrapper Configuration Settings

Tungsten services define Java Service Wrapper configuration in file wrapper.conf. To make changes you can edit the file and change settings as needed. You must restart the service for new settings to take effect. Most settings are documented with comments. The most commonly changed settings are the following. Memory - You can raise Java VM memory by editing the wrapper.java.maxmemory property. Java VM Options - You can supply additional Java VM options using wrapper.java.additional properties. Debugging - You can enable Java remote debugging by uncommenting the debug properties included in most Tungsten wrapper.conf files. These are for developer use only and should not be enabled unless needed.
6.3. Network Configuration

6.3.1. Network Port Access
Tungsten nodes require open access to the following TCP/IP ports on each host. Local environments may require adjustment of firewalls to ensure access. In Amazon cloud environments you must use a security group that opens these ports to other members of the cluster.
Note
Some ports are particular to MySQL or PostgreSQL and are so noted. Ports in general are default values; actual values may differ for particular installations. 22 - Default port for ssh and rsync access. Tungsten requires this port to manage PostgreSQL warm standby clusters. 2112 - Default port for master Tungsten Replicator to accept client connections. This must be open for any host that runs a replicator that will act as a master.
Detailed Configuration This port is not currently used for PostgreSQL clustering. 3306 - Default port for MySQL listener. Tungsten itself does not require this port to be open for management reasons but it must be open for applications to connect. 5432 - Default port for PostgreSQL listener. Tungsten requires remote access to this port in order to run backups in PostgreSQL clusters. It must also be open for applications to connect.
6.4. JMX Configuration

6.4.1. Introducing JMX
Tungsten uses JMX for management and monitoring. JMX enables remote management of Java processes. The protocol and its use are a standard feature of Java Virtual Machines. For further information on JMX, consult the documentation available at the following location: http://java.sun.com/ j2se/1.5.0/docs/guide/management. At this location you will also find definitions for property settings used to control JMX behavior.
6.4.2. JMX Configuration Settings

Generally speaking there is little need for users to make JMX configuration changes. That said, there are a number of properties that control JMX behavior. These can be set directly when starting the Java process, as in the following example. java -Dproperty=value ... You can also set multiple properties at once using a JMX properties file. A sample JMX properties file is included the conf directories of certain Tungsten services and is named sample.jmx.properties. To use this file, you must add the following property when starting Java. java -Dcom.sun.management.config.file=path/to/propfile
6.4.3. JMX on NAT-enabled hosts

JMX requires the host name used for connection to match the name of the host itself. This can be a problem in some cases where the host is accessed through NAT, in which case the host name likely resolves to a different name and IP from that used by remote JMX clients. This will result in refused connections. To cure this problem you must set the following Java property either directly when invoking Java or using the JMX properties file. You can use either the host name or the IP address to which you are binding. java.rmi.server.hostname=public.host.name For further information consult JMX documentation. This problem is also discussed in various online forums. You can search on the java.rmi.server.hostname property to find online descriptions of the problem and how to resolve it.
6.5. Backup Configuration

6.5.1. Xtrabackup
If you experienced trouble using the Xtrabackup utility then you may need to modify the standard configuration. The tungsten-replication/conf/replicator.properties file holds the configuration values. Modify the values in replicator.backup.agent.xtrabackup.options to match the requirements of your environment.
6.5.2. Mysqldump
Depending on your version of MySQL, you may have the option of using Mysqldump without having to take the datasource offline. When the configure script is complete, open the tungsten-replicator/conf/ replicator.properties file and review the options that are set for the Mysqldump command. You can find them by searching for the replicator.backup.agent.mysqldump.mysqldumpOptions setting.
6.5.3. Custom Backup Methods

The backup and restore operations allow you to write a custom shell script that will be used. You can view a sample of this script at tungsten-replicator/samples/scripts/backup/custom-backup.sh. This script shows an example of doing the backup and restore using mysqldump. You must update the replicator service properties file with the path for your script and any configuration options. There are a number of backup options with the prefix --backup available through tungsten-installer and configure-service. Check the command help for a complete list. The following example shows typical service properties configuration. # Script Agent--Executes a script to backup or restore. replicator.backup.agent.script= \ com.continuent.tungsten.replicator.backup.generic.ScriptDumpAgent # The full path to the custom script replicator.backup.agent.script.script= \ ${replicator.home.dir}/samples/scripts/ backup/custom-backup.sh # The prefix that should be added before the command? Usually 'sudo' replicator.backup.agent.script.commandPrefix= # Can this backup be run while the database is being use for reads and writes? replicator.backup.agent.script.hotBackupEnabled=false
Tip
Any line that ends with \ should be joined with the line following it.
6.5.4. Automatic Backup

You can manually configure the replicator to create a backup the next time it is started. Open the tungsten-replicator/conf/dynamic.properties file and add the following. replicator.auto_backup=true The replicator will execute a backup the next time it is started. Once complete, the replicator.auto_backup setting will be changed to false.
6.5.5. Automatic Restore

You can manually configure the replicator to restore from the latest backup the next time it is started. Open the tungsten-replicator/conf/dynamic.properties file and add the following.
replicator.auto_provision=true The replicator will execute a restore the next time it replicator.auto_provision setting will be changed to false. is started. Once complete, the
6.5.6. Configuring backup after installation

There are two options if you have already run the configure script and have an operating cluster. Run the configure script again and answer the questions related to configuring backup. You will need to manually restart the replicator once the script is complete. Update the replicator.backup entries in the tungsten-replicator/conf/replicator.properties file. You will need to restart the replicator once finished.

Tungsten Installation Guide

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Tungsten Installation Guide

Transféré par

Droits d'auteur :

Formats disponibles

Tungsten Replicator Installation Guide

Tungsten Replicator Installation Guide

Tungsten Replicator Installation Guide - Document issue 2.0.4

Tungsten version 2.0.4

Tungsten Replicator Installation Guide

Tungsten Replicator Installation Guide - Document issue 2.0.4

iii Tungsten version 2.0.4

Tungsten Replicator Installation Guide - Document issue 2.0.4

iv Tungsten version 2.0.4

Tungsten Replicator Installation Guide - Document issue 2.0.4

1 Tungsten version 2.0.4

Product Installation and Configuration

Chapter 2. Product Installation and Configuration

2.2. Software Download

2.3. Setting Up Master/Slave Replication

Figure 2.1. Master/slave Topology

2.3.1. Master Replicator Configuration

Tungsten Replicator Installation Guide - Document issue 2.0.4

2 Tungsten version 2.0.4

Product Installation and Configuration

Master/slave replication is now configured and ready for use.

2.3.2. Testing Replication

Tungsten Replicator Installation Guide - Document issue 2.0.4

3 Tungsten version 2.0.4

Product Installation and Configuration

Tungsten Replicator Installation Guide - Document issue 2.0.4

4 Tungsten version 2.0.4

Tungsten Configuration Procedures

Chapter 3. Tungsten Configuration Procedures

3.1.1. Getting Help

3.1.2. Replicator Topologies

Tungsten Replicator Installation Guide - Document issue 2.0.4

3.1.3. Service Name

3.1.4. Home Directory

3.1.5. Database Login and Passwords

3.1.6. Replicator Start-Up

3.1.7. Installation Checks

3.1.8. Advanced Property Settings

Tungsten Replicator Installation Guide - Document issue 2.0.4

3.2.1. Getting Help

3.2.2. Cluster Hosts

3.2.3. Home Directory

3.2.4. Replicator Start-Up

Tungsten Replicator Installation Guide - Document issue 2.0.4

7 Tungsten version 2.0.4

Tungsten Configuration Procedures

3.2.5. Installation Checks

3.3.1. Getting Help

3.3.2. Replicator Service Role

3.3.3. Service Name

3.3.4. Service Type

Tungsten Replicator Installation Guide - Document issue 2.0.4

8 Tungsten version 2.0.4

3.3.5. Release Directory

Tungsten Replicator Installation Guide - Document issue 2.0.4

9 Tungsten version 2.0.4

Configuring Specialized Replication Topologies

Chapter 4. Configuring Specialized Replication Topologies

4.1. MySQL Slave Replacement with Parallel Replication

Figure 4.1. MySQL Slave Replacement Topology

4.1.1. MySQL Replication Setup

Tungsten Replicator Installation Guide - Document issue 2.0.4

10 Tungsten version 2.0.4