Académique Documents
Professionnel Documents
Culture Documents
Categories
Getting Started Beginner's Guide Linode Platform Using Linux Linux Tools DNS Services LAMP Guides LEMP Guides Email Guides Security Networking Web Servers Web Applications Application Stacks Web Frameworks Server Monitoring High Availability Databases Communications Troubleshooting How to Contribute Sitemap
Linode Library Home :: Linux High Availability Guides :: Build a Highly Available HTTP Load Balancer on Ubuntu 10.04 LTS (Lucid)
Linode Home
Build a Highly Available HTTP Load Balancer on Ubuntu 10.04 LTS (Lucid)
Author: Phil Paradis Published: August 13, 2010 Revised: August 15, 2010
High availability refers to the practice of keeping online resources available through node failure or system maintenance. This guide will demonstrate a method for using two Linodes to provide HTTP load balancing services with nginx. Services will be avilable even when one load balancer node is powered off or put into standby mode. IP failover, Heartbeat 3.x, and Pacemaker 1.x will be used for this example configuration. This setup is intended to provide a load balancer that distributes inbound HTTP requests to a pool of frontend web/application servers in the same datacenter. The web servers could host mirrored content locally on each of their filesystems, or they could mount content directories over NFS and access databases hosted on another highly available server cluster. A simple network diagram illustrating such a setup might look like this:
In the diagram above, a highly available load balancer comprised of two nodes distributes inbound HTTP connections to multiple frontend web servers. These web servers connect to a highly available database and file server comprised of two nodes. While this guide only covers configuration of the HTTP load balancer portion of the diagram, you could add your own web server nodes and highly available file/database resources to form a complete highly available and load balanced network configuration. As high availability is a complex topic with many methods available for achieving various goals, it should be noted that the method discussed here may not be appropriate for some use cases. However, it should provide a good foundation for developing a customized HA solution. This guide assumes you have a minimum of four active Linodes on your account, and that two of them are freshly deployed Ubuntu 10.04 LTS
(Lucid) instances. The remaining two (or more) Linodes will serve as frontend web servers. If needed, you may add additional Linodes by clicking the "Add a Linode to this Account" link on the "Linodes" tab of the Linode Manager. Important: All your Linodes must reside in the same datacenter for IP failover (a required component of this guide) and private network communications between the nodes to work. These instructions work with the Linode platform. If you don't have a Linode yet, sign up for a Linux VPS and get started today.
Contents
Terminology Prerequisites Basic System Configuration Assign Static IP Addresses Set Up IP Failover Linkage Install Heartbeat, Pacemaker, and nginx Configure Heartbeat Configure nginx Configure Cluster Resources Monitor Cluster Resources More Information License
Terminology
Throughout this document, the following terms are used: ha1-lb - the primary Linode ha2-lb - the secondary Linode 12.34.56.78 - a static public IP address assigned to the primary Linode 98.76.54.32 - a static public IP address assigned to the secondary Linode 55.55.55.55 - a "floating" public IP address, initially assigned to the primary Linode, which may be brought up on either Linode 192.168.88.88 - a private IP address, assigned to the primary Linode 192.168.99.99 - a private IP address, assigned to the secondary Linode CHANGEME - a password or authentication key bambookites.com - an example domain name You should substitute your own values for these terms wherever they are
found.
Prerequisites
Both of your newly provisioned Linodes must have a private IP address assigned to them. In the Linode Manager, add private IPs by visiting each Linode's "Remote Access" tab. As your your primary Linode will need an additional public IP (to serve as a "floating" address), you'll need to add an additional IP from the "Extras" tab in the Linode Manager. Once your Linodes are assigned the additional IPs, reboot both of them to allow the new addresses to be properly routed. Additionally, the Linodes that will serve as frontend web/application servers must have private IP addresses assigned to them. This will allow your load balancer to communicate with them over the private network, which won't count against your traffic quota. You should configure DNS to point your highly available website's domain name (or subdomain) at the "floating" IP address. Requests made to this IP will be forwarded to your frontend web server pool.
Remember to substitute your primary and secondary Linode's IP addresses for 12.34.56.78 and 98.76.54.32, respectively, along with appropriate hostnames for each. You will find the IP addresses for your Linodes on their "Remote Access" tabs in the Linode Manager. For the sake of simplicity, it is recommended that you keep the short hostnames assigned as ha1-lb and ha2-lb. Next, issue the following commands to generate SSH keys for the root user on each VPS, synchronize their SSH host keys, set their hostnames, and allow passwordless logins from each to the other. SSH host key synchronization will prevent issues with key checking later on, which might otherwise occur should you need to perform an SSH login via a hostname pointing to a floating IP while the secondary node is serving as the primary in a failover condition. You will be prompted to assign passphrases to the SSH keys; this is optional, and you may skip this step by pressing the "Enter" key.
ssh-keygen -t rsa scp ~/.ssh/id_rsa.pub root@ha2-lb:/root/ha1_key.pub ssh root@ha2-lb "ssh-keygen -t rsa" ssh root@ha2-lb "echo \`cat ~/ha1_key.pub\` >> ~/.ssh/authorized_keys2" ssh root@ha2-lb "rm ~/ha1_key.pub" scp root@ha2-lb:/root/.ssh/id_rsa.pub /root cat ~/id_rsa.pub >> ~/.ssh/authorized_keys2 rm ~/id_rsa.pub scp /etc/ssh/ssh_host* root@ha2-lb:/etc/ssh/ rm ~/.ssh/known_hosts ssh root@ha2-lb "/etc/init.d/ssh restart" scp /etc/hosts root@ha2-lb:/etc/hosts echo "ha1-lb" > /etc/hostname hostname -F /etc/hostname ssh root@ha2-lb "echo \"ha2-lb\" > /etc/hostname" ssh root@ha2-lb "hostname -F /etc/hostname"
On the secondary Linode, edit the /etc/network/interfaces file to resemble the following, making sure the values entered match those shown on the "Remote Access" tab for the secondary Linode:
You should be able to ping each Linode's public address from the other, and you should be able to ping each Linode's private address from the other. If you can't, review your network configuration for errors.
After issuing the commands listed above, the required packages will be installed, and the nginx service will be temporarily stopped on both Linodes. Additionally, the system startup links for nginx will be removed on
both Linodes, as Pacemaker will be responsible for starting and stopping it as necessary.
Configure Heartbeat
On the primary Linode, create a file named /etc/heartbeat/ha.cf with the following contents. Replace 98.76.54.32 with the statically assigned public IP address of the secondary Linode. File: /etc/heartb eat/ha.cf (on primary Linode)
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 98.76.54.32 auto_failback on node ha1-lb node ha2-lb use_logd yes crm respawn
On the secondary Linode, create a file named /etc/heartbeat/ha.cf with the following contents. Replace 12.34.56.78 with the statically assigned public IP address of the primary Linode. File: /etc/heartb eat/ha.cf (on secondary Linode)
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 12.34.56.78 auto_failback on node ha1-lb node ha2-lb use_logd yes crm respawn
On the primary Linode, create the file /etc/heartbeat/authkeys with the following contents. Make sure to change "CHANGEME" to a strong password consisting of letters and numbers. File: /etc/heartb eat/authkeys (on primary Linode)
auth 1 1 sha1 CHANGEME
On the primary Linode, issue the following commands to set proper permissions on this file, copy it to the secondary Linode, and start the Heartbeat service on both nodes:
chmod 600 /etc/ha.d/authkeys /etc/init.d/heartbeat start scp /etc/ha.d/authkeys root@ha2-lb:/etc/ha.d/ ssh root@ha2-lb "chmod 600 /etc/ha.d/authkeys" ssh root@ha2-lb "/etc/init.d/heartbeat start"
Configure nginx
The nginx web server will perform round-robin distribution of inbound HTTP requests to your frontend web servers. On the primary Linode, edit the file /etc/nginx/nginx.conf to resemble the following, making sure to use your frontend web server private IP addresses in the "upstream" section and your domain name in the "server" section. File: /etc/nginx/nginx.conf (on primary Linode)
user www-data; worker_processes 1;
error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; sendfile on; keepalive_timeout 65; tcp_nodelay on; gzip on; gzip_disable "MSIE [1-6]\.(?!.*SV1)"; upstream webservers { server 192.168.15.15 max_fails=3 fail_timeout=5s; server 192.168.16.16 max_fails=3 fail_timeout=5s; server 192.168.17.17 max_fails=3 fail_timeout=5s; } server { server_name bambookites.com www.bambookites.com; location / { proxy_pass http://webservers; proxy_set_header X-Real-IP $remote_addr; proxy_next_upstream timeout; } }
Issue the following command to copy the nginx configuration to your secondary Linode:
scp /etc/nginx/nginx.conf root@ha2-lb:/etc/nginx/
For the purposes of these instructions, it will be assumed that you are are using vim as your editor. On the primary Linode, issue the following command to start the cluster resource manager in "edit" mode:
crm configure edit
You will be presented with information resembling the following. If you don't see anything, enter ":q" to quit the editor and wait a minute before restarting it.
node $id="1d548242-8908-49c6-bcb3-a594267b81e2" ha1-lb node $id="5a7ab511-e274-435f-8662-7dbc737a1786" ha2-lb property $id="cib-bootstrap-options" \ dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \ cluster-infrastructure="Heartbeat"
To begin editing your configuration, press the "i" key. To leave edit mode, press "Ctrl+c". To quit without saving any changes, press ":" and enter "q!". To save changes and quit, press ":" and enter "wq". Insert the following lines in between the second "node" line at the top of the configuration and the "property" line at the bottom. Important: Be sure to replace both instances of 55.55.55.55 with the "floating" public IP address.
primitive ip1 ocf:heartbeat:IPaddr2 \ params ip="55.55.55.55" nic="eth0:1" \ op monitor interval="5s" primitive ip1arp ocf:heartbeat:SendArp \ params ip="55.55.55.55" nic="eth0:1" primitive nginx ocf:heartbeat:anything \ params \ binfile="/usr/sbin/nginx" \ cmdline_options="-c /etc/nginx/nginx.conf" group HAServices ip1 ip1arp nginx \ meta target-role="Started" order ip-before-arp mandatory: ip1:start ip1arp:start order ip-before-nginx mandatory: ip1:start nginx:start
Change the "property" section to resemble the following excerpt. You'll be adding an "expected-quorum-votes" entry due to the fact that your cluster only has two nodes, as well as adding the lines for "stonith-enabled" and "no-quorum-policy". Don't forget the trailing "\" after the "clusterinfrastructure" line.
property $id="cib-bootstrap-options" \ dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \ cluster-infrastructure="Heartbeat" \ expected-quorum-votes="1" \ stonith-enabled="false" \ no-quorum-policy="ignore"
After making these changes, press "Ctrl+c" and enter ":wq" to save the configuration and exit the editor.
============ Last updated: Fri Aug 13 20:49:19 2010 Stack: Heartbeat Current DC: ha2-lb (5a7ab511-e274-435f-8662-7dbc737a1786) - partition with Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd 2 Nodes configured, 1 expected votes 1 Resources configured. ============ Online: [ ha2-lb ha1-lb ] Resource Group: HAServices ip1 (ocf::heartbeat:IPaddr2): ip1arp (ocf::heartbeat:SendArp): nginx (ocf::heartbeat:anything): Started ha1-lb Started ha1-lb Started ha1-lb
In this example, the clustered resources are started on ha1-lb. To simulate a failover situation, issue the following command to put ha1-lb into standby:
crm node standby ha1-lb
Within a few seconds, the resources will be stopped on the initial node and started on the other one. To bring ha1-lb back online, simply issue the following command:
crm node online ha1-lb
At this point, you should be able to shut down the Linode hosting your resources and watch them automatically migrate to the other Linode (provided you have crm_mon running in a terminal on the still-active Linode). Note that because "resource-stickiness" is set at "100", resources should stay wherever they are migrated they until you manually move them to another node. This can be helpful in cases where you need to perform maintenance on a node, but don't want services resuming on it until you're ready. Congratulations, you've created a highly available HTTP load balancer!
More Information
You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials. Linux-HA Information Pacemaker Homepage nginx Homepage
License
This guide is licensed under a Creative Commons Attribution-No Derivative Works 3.0 United States License . Please feel free to redistribute unmodified copies of it as long as attribution is provided, preferably via a link to this page.
Comments
Submitted by Robin Matthews on Monday, October 25 2010 at 13:42:02 GMT If you add pidfile="/var/run/nginx.pid" heartbeat/ pacemaker will also be able to start and stop your nginx load balancers as needed. Also, I'm curious why you used the the public IP addresses in your ha.cf. I've used the private IP addresses which seems to be working. What was your thinking there? Cheers Robin
Got a comment?
BBCode formatting is allowed. Email addresses are confidential, and are only used for gravatars and sending document/comment updates if requested. Please refer to our privacy policy. All comments are moderated and may take some time to appear on this page. Your Name (optional): Your Email (optional): Notify on update? Comment:
Preview
Submit