Académique Documents
Professionnel Documents
Culture Documents
Notices
The information in this document is subject to change without notice. While every effort has been made to ensure that all information in this document is accurate, the Authors accept no liability for any errors that may arise. No part of this document may be transmitted or copied in any form, or by any means, for any purpose, without the written permission of the Authors. Issue 1.5 November 7, 2008 Part No. 0084354-06A
Acknowledgments Linux is a trademark of Linus Torvalds. Red Hat is a trademark of Red Hat, Inc. SUSE is a trademark of Novell, Inc. All names, brands, products or services are trademarks or registered trademarks of their respective companies.
ii
Contents
Contents
Preface ................................................................................................................................................... v Revision History ...................................................................................................................................... v 1 Introduction ..................................................................................................................................... 1.1 Introduction .................................................................................................................................. 1.2 System Requirements ................................................................................................................. 1.2.1 Hardware Requirements .................................................................................................... 1.2.2 Supported Operating Systems ........................................................................................... 1.2.3 Supported Architectures ..................................................................................................... 1.2.4 Software ............................................................................................................................. 1 1 1 1 2 2 2
Installation ....................................................................................................................................... 3 2.1 Introduction .................................................................................................................................. 3 2.2 Red Hat Enterprise Linux 4 ......................................................................................................... 3 2.2.1 Check the operating system level ...................................................................................... 3 2.2.2 Install device mapper and device-mapper-multipath .......................................................... 3 2.2.3 Install StorView Path Manager for Linux ............................................................................ 4 2.3 Red Hat Enterprise Linux 5 ......................................................................................................... 5 2.3.1 Check the operating system level ...................................................................................... 5 2.3.2 Install device mapper and device-mapper-multipath .......................................................... 5 2.3.3 Install StorView Path Manager for Linux ............................................................................ 6 2.4 SUSE Linux Enterprise Server 9 ................................................................................................. 7 2.4.1 Checking the operating system level .................................................................................. 7 2.4.2 Install device-mapper and multipath-tools .......................................................................... 7 2.4.3 Install StorView Path Manager for Linux ............................................................................ 7 2.5 SUSE Linux Enterprise Server 10 ............................................................................................... 9 2.5.1 Checking the operating system level .................................................................................. 9 2.5.2 Install device-mapper and multipath-tools .......................................................................... 9 2.5.3 Install StorView Path Manager for Linux ............................................................................ 9 2.6 Multipathing on the Boot Device ................................................................................................ 10 2.6.1 How to Install RHEL 5.1 with Boot Disk Multipathing ....................................................... 11 2.7 Configuring HBAs for multipath use .......................................................................................... 13 2.8 Verifying the Installation ............................................................................................................ 13 2.9 Uninstallation ............................................................................................................................. 13 Overview ........................................................................................................................................ 3.1 Paths ......................................................................................................................................... 3.2 Software Capabilities ................................................................................................................. 3.2.1 svpm Man Page ............................................................................................................... 3.2.2 svpm-dm Man Page ......................................................................................................... 3.2.3 mpath_prio-svpm Man Page ............................................................................................ How To ....................................................................................................................................... 4.1 How to Get Information on Multipath Devices ........................................................................... 4.2 How to Determine Which Path is in Use ................................................................................... 4.3 How to Interpret SVPM Path Priorities ...................................................................................... 15 15 15 16 16 16 17 17 18 19
iii
4.4 How to Set Path Priorities ......................................................................................................... 19 4.5 How to Blacklist local-attached SCSI devices ........................................................................... 20 4.6 How to Automount a Multipath Device ...................................................................................... 20 5 6 Known Issues / Limitations ......................................................................................................... 23 Troubleshooting ........................................................................................................................... 6.1 Installation fails .......................................................................................................................... 6.2 Failback doesn't occur .............................................................................................................. 6.2.1 The multipathd daemon has failed ................................................................................... 6.2.2 The device is not mounted properly ................................................................................. 6.2.3 All required services have not been started ..................................................................... 6.2.4 Multipath.conf entries are not valid .................................................................................. 6.2.5 No LUNs are available ..................................................................................................... 6.2.6 The system needs to be rebooted ................................................................................... 6.3 Other problems ......................................................................................................................... 25 25 25 25 25 26 26 26 26 26
Glossary ............................................................................................................................................... 27
iv
Preface
Preface
What is in this guide This user guide gives you an overview of how to use the Xyratex StorView Path Manager for Linux to configure multipathing in a RAID system. Who should use this guide This user guide is for those needing to set up a multipath solution in the Linux environment. About this guide This user guide provides the following information: Chapter 1, Introduction, on page 1 provides an introduction and system requirements. Chapter 2, Installation, on page 3 gives details on how to set up the software, how configure the system for multipath use, and how to uninstall. Chapter 3, Overview, on page 15 describes multipathing and outlines the softwares capabilities. Chapter 4, How To , on page 17 provides step-by-step instructions for a number of common tasks. Chapter 5, Known Issues / Limitations, on page 23 gives warning of any limitations in the software. Chapter 6, Troubleshooting, on page 25 provides resolutions for some common problems. The Glossary, on page 27, defines some of the terms used throughout this user guide.
Revision History
Version 1.0 1.1 1.2 1.3 1.4 1.5 Date June 14, 2007 Description of Change Initial Release.
September 19, 2007 Support for EM64T architetures. March 12, 2008 July 9, 2008 August 22, 2008 November 7, 2008 Support for the RS-1220-F4-6412E storage system (F6412E). Support for RHEL5 and IA64; multipath booting on RHEL5.1; update to output from svpm -l command. Minor updates. Support for F6500E controllers.
vi
Introduction
Chapter 1
Introduction
1.1 Introduction
StorView Path Manager for Linux (svpm) works in conjunction with the Linux device-mapper and multipath tools to provide multipath capability for the F5402E, E5402E, F5404E, F5412E, E5412E, F6412E and F6500E controllers. Multipathing is the use of multiple physical components (such as host bus adapters, cables and switches) to provide two or more data paths to a storage device and hence avoid a single point of failure. When a component in one path fails, a different path to the device will still be available for data transfer. Multiple paths can be used simultaneously if the storage device supports it. StorView Path Manager for Linux provides failover/failback, path priorities and load balancing features.
RS-1220-F4-6412E (F6412E) HB-2435-F8-6500E (F6500E) HB-2435-E311-XPN (F6500E) A storage configuration where there are at least two paths from the host to the storage.
1.2.4 Software
Device Mapper (included with RHEL 4, RHEL 5, SLES 9, SLES 10). Multipath tools (included with RHEL 4, RHEL 5, SLES 9, SLES 10). For more information, see Installation beginning on page 3.
Installation
Chapter 2
Installation
2.1 Introduction
The pre-requisites for installation are included in the installation instructions that follow.
You must be running the Enterprise version of Red Hat Linux, version ES or AS, release 4, with update 4 or greater.
Note
The only supported versions of device-mapper and device-mapper-multipath are those that shipped with the operating system distribution.
If the commands return nothing, the components are not installed. The RPMs can be found on the installation media for your operating system: [CD2]/RedHat/RPMS/device-mapper-1.02.07-4.0.RHEL4.i386.rpm [CD4]/RedHat/RPMS/device-mapper-multipath-0.4.5-16.1.RHEL4.i386.rpm where [CD2] and [CD4] are the mount directories of the second and fourth installation CDs.
Note 4
The location of the RPMs are current at the time of the printing of this document. Change to the directory containing the RPMs and install them:
cd [CD2]/RedHat/RPMS rpm -ivh device-mapper-1.02.07-4.0.RHEL4.i386.rpm cd [CD4]/RedHat/RPMS rpm -ivh device-mapper-multipath-0.4.5-16.1.RHEL4.i386.rpm
If a version of svpm not is installed on the system you should get a message: package svpm-dm is not installed. If a version of svpm is installed on the system you should get a message: svpm-dm-<version>.rhel4.xyr and you must uninstall the svpm rpm before continuing (see 2.9, Uninstallation, on page 13). 2 Type the following commands:
cd [CD path]/drivers/svpm/linux rpm -ihv svpm-dm-<version>.rhel4.xyr.i386.rpm
Determine if the rpm installation was successful by typing the following command:
rpm -q svpm-dm
If the installation was successful the command should return svpm-dm-<version>.rhel4.xyr. If the command returns nothing, the component was not installed. 4 Important Using a text editor, merge the contents from /etc/multipath.conf.svpm (the section devices {}) into /etc/multipath.conf. If /etc/multipath.conf is empty, also merge the section defaults {}. If the step above is not performed, svpm-dm will not operate correctly and results will be unpredictable. You will need to make sure the multipath and svpm daemons are started when the system boots:
/sbin/chkconfig multipathd on
If nothing is returned, the daemons have started successfully. 7 Follow the instructions in 2.7, Configuring HBAs for multipath use, on page 13.
Installation
You must be running the Enterprise version of Red Hat Linux, version ES or AS, release 5 with update 1.
Note
The only supported versions of device-mapper and device-mapper-multipath are those that shipped with the operating system distribution. If the commands return nothing, the components are not installed. The RPMs can be found on the installation media for your operating system: [CD1]/Server/device-mapper-1.02.20-1.el5.i386.rpm [CD3]/Server/device-mapper-multipath-0.4.7-12.el5.i386.rpm where [CD1] and [CD3] are the mount directories of the second and fourth installation CDs. Or [DVD]/Server/device-mapper-1.02.20-1.el5.i386.rpm [DVD]/Server/device-mapper-multipath-0.4.7-12.el5.i386.rpm where [DVD] is the mount directory of the installation DVD.
Note
The location of the RPMs are current at the time of the printing of this document.
Or
cd [DVD]/Server rpm -ivh device-mapper-1.02.20-1.el5.i386.rpm rpm -ivh device-mapper-multipath-0.4.7-12.el5.i386.rpm
If a version of svpm not is installed on the system you should get a message: package svpm-dm is not installed. If a version of svpm is installed on the system you should get a message: svpm-dm-<version>.rhel5.xyr and you must uninstall the svpm rpm before continuing (see 2.9, Uninstallation, on page 13). 2 Type the following commands:
cd [CD path]/drivers/svpm/linux rpm -ihv svpm-dm-<version>.rhel5.xyr.i386.rpm
Determine if the rpm installation was successful by typing the following command:
rpm -q svpm-dm
If the installation was successful the command should return svpm-dm-<version>.rhel5.xyr. If the command returns nothing, the component was not installed. 4 Important Using a text editor, merge the contents from /etc/multipath.conf.svpm (the section devices {}) into /etc/multipath.conf. If /etc/multipath.conf is empty, also merge the section defaults {}. If the step above is not performed, svpm-dm will not operate correctly and results will be unpredictable. You will need to make sure the multipath and svpm daemons are started when the system boots:
/sbin/chkconfig multipathd on
If nothing is returned, the daemons have started successfully. 7 Follow the instructions in 2.7, Configuring HBAs for multipath use, on page 13.
Installation
You must be running the Enterprise version of SUSE Linux, version 9, with patch level 4.
Note
The only supported versions of device-mapper and device-mapper-multipath are those that shipped with the operating system distribution. If the commands return nothing, the components are not installed. The RPMs can be found on the installation media for your operating system. [CD1]/suse/i586/device-mapper-1.01.01-1.6.i586.rpm [CD1]/suse/i586/device-mapper-multipath-0.4.7-75.7.i586.rpm where [CD1] is the mount directories of the first installation CDs.
Note 4
The location of the RPMs are current at the time of the printing of this document. Change to the directory containing the RPMs and install them:
cd [CD1]/suse/i586 rpm -ivh device-mapper-1.01.01-1.6.i586.rpm cd [CD1]/suse/i586 rpm -ivh multipath-tools-0.4.7-75.7.i586.rpm
If a version of svpm not is installed on the system you should get a message: package svpm-dm is not installed. If a version of svpm is installed on the system you should get a message: svpm-dm-<version>.sles9.xyr and you must uninstall the svpm rpm before continuing (see 2.9, Uninstallation, on page 13). 2 Type the following commands:
cd [CD path]/drivers/svpm/linux rpm -ihv svpm-dm-<version>.sles9.xyr.i586.rpm
Determine if the rpm installation was successful by typing the following command:
rpm -q svpm-dm
If the installation was successful the command should return svpm-dm-<version>.sles9.xyr. If the command returns nothing, the component was not installed. 4 Important Using a text editor, merge the contents from /etc/multipath.conf.svpm (the section devices {}) into /etc/multipath.conf. If /etc/multipath.conf is empty, also merge the section defaults {}. If the step above is not performed, svpm-dm will not operate correctly and results will be unpredictable. You will need to make sure the multipath and svpm daemons are started when the system boots:
/sbin/chkconfig /sbin/multipathd on
If nothing is returned, the daemons have started successfully. 7 Use YaST - SYSTEMS / Runlevel Editor / expert mode to enable the following modules: boot.multipath level B boot.device-mapper level B boot.udev level B multipathd level 3 and 5 8 9 Follow the instructions in 2.7, Configuring HBAs for multipath use, on page 13. The system must be rebooted before newly installed devices and LVM entries will be recognized.
Installation
You must be running the Enterprise version of SUSE Linux, version 10, with patch level 1.
Note
The only supported versions of device-mapper and device-mapper-multipath are those that shipped with the operating system distribution. If the commands return nothing, the components are not installed. The RPMs can be found on the installation media for your operating system. [CD1]/suse/i586/device-mapper-1.02.13-6.9.i586.rpm [CD1]/suse/i586/multipath-tools-0.4.7-34.18.i586.rpm where [CD1] is the mount directories of the first installation CDs.
Note 4
The location of the RPMs are current at the time of the printing of this document. Change to the directory containing the RPMs and install them:
cd [CD1]/suse/i586 rpm -ivh device-mapper-1.02.13-6.9.i586.rpm cd [CD1]/suse/i586 rpm -ivh multipath-tools-0.4.7-34.18.i586.rpm
If a version of svpm not is installed on the system you should get a message: package svpm-dm is not installed. If a version of svpm is installed on the system you should get a message: svpm-dm<version>.sles10.xyr and you must uninstall the svpm rpm before continuing (see 2.9, Uninstallation, on page 13). 2 Type the following commands:
cd [CD path]/drivers/svpm/linux rpm -ihv svpm-dm-<version>.sles10.xyr.i586.rpm
Determine if the rpm installation was successful by typing the following command:
rpm -q svpm-dm
If the installation was successful the command should return svpm-dm-<version>.sles10.xyr. If the command returns nothing, the component was not installed. 4 Important Using a text editor, merge the contents from /etc/multipath.conf.svpm (the section devices {}) into / etc/multipath.conf. If /etc/multipath.conf is empty, also merge the section defaults {}. If the step above is not performed, svpm-dm will not operate correctly and results will be unpredictable. You will need to make sure the multipath and svpm daemons are started when the system boots:
/sbin/chkconfig /sbin/multipathd on
If nothing is returned, the daemons have started successfully. 7 Use YaST - SYSTEMS / system services / expert mode to enable the following modules: boot.multipath level B boot.device-mapper level B boot.udev level B multipathd level 3 and 5 8 9 Follow the instructions in 2.7, Configuring HBAs for multipath use, on page 13. The system must be rebooted before newly installed devices and LVM entries will be recognized.
10
Installation
2.6.1.1
1 2 3
Pre-Installation Setup
Perform the following steps before installing RHEL 5.1: Using StorView, configure the LUN you wish to use for your operating system partition. Using StorView SAN Mapping or Port Mapping, ensure that the LUN has 2 or more paths from the host. It is recommended that only one LUN is visible to the host before performing the install. This will simplify your install process. Additionally, any LUNs that can be seen during install will not have their aliases reflected by svpm-dm. To make the non-OS LUNs invisible to the host, use StorView SAN Mapping or Port Mapping. Configure your HBA to allow booting from the SAN.
2.6.1.2
1 2
3 4 5 6 7
2.6.1.3
1 2 3 4
Linux Installation
Insert your operating system CD or DVD and boot from the disc. From the boot: prompt, enter linux mpath. Follow the prompts given by the Red Hat installer to choose your language and keyboard setup. Choose the option to Install Red Hat Enterprise Linux Server.
11
You will then see a list of available partitions. Make sure your install partition is listed as mapper/mpath#. If the partition does not have a mapper/mpath listing, make sure the LUN is visible by more than one path, and restart the process. If you are installing to mapper/mpath0, choose the correct partition and click Next. If more than one LUN is visible to the host and/or you are not installing to mapper/mpath0, you must perform the following additional steps or the system will not be bootable: 1 Note which path is being used for the installation. For example: mapper/mpath1. 2 Check Review and modify partitioning layout and click Next. 3 On the screen that shows the drive layout, click Next. 4 The top line on the screen will indicate where the GRUB boot loader will be installed. By default, this will be /dev/mapper/mpath0. You will need to change this to install the boot loader onto the drive being used for the installation, as follows: 5 Check Configure advanced boot loader options and click Next. 6 Click Change Drive Order. 7 Arrange the drives so that the one being installed is at the top of the list. For example, if you are installing to mapper/mpath1 move /dev/mapper/mpath1 to the top of the list. Click OK. 8 Confirm that the next screen indicates that the boot loader will be installed to the correct device.
6 Note
7 8 9 10 11
Choose your network settings, timezone, and root password as normal. When prompted to customize the software selection, choose Customize now. Select Base System, then highlight Base and click Optional packages. Make sure that device-mapper-multipath is selected. Continue with the install process and reboot when prompted.
2.6.1.4
1 2
12
Installation
/etc/modprobe.conf /etc/modprobe.conf.local
See sections 4.2, How to Determine Which Path is in Use and 4.3, How to Interpret SVPM Path Priorities for the correct output.
2.9 Uninstallation
To uninstall StorView Path Manager for Linux: 1 Using a text editor, the /etc/multipath.conf file should be edited to remove the lines added during installation from /etc/multipath.conf.svpm Type the following:
rpm -e svpm-dm
13
14
Overview
Chapter 3
Overview
3.1 Paths
A path is defined as the complete route from host to controller, including the HBA, cabling and possibly a switch. A multipath solution requires that there is more than one of these paths. The number of paths is easy to see if direct connections are made (it is simply the number of physical cables), but less obvious if a switch is used. If, for example, there are 3 HBAs connected to a switch, and the switch is connected to both ports of a controller, this results in a total of 6 paths. When using Device Mapper each path is given a standard linux block device name (/dev/sda, /dev/sdb, and so on), but these are also merged into a single multipath block device in the /dev/mapper directory. The name of the device is the SCSI ID which will be in one of the following forms: /dev/mapper/36000bb500020c5f20000000000000009 /dev/mapper/mpath1 This device behaves much like a standard Linux block device. It can be mounted, or checked using the fsck command. In order to use multipath correctly, you must mount the mapper device (/dev/mapper/ 36000bb500020c5f20000000000000009 or /dev/mapper/mpath1) and not one of the path block devices (/dev/sdb). Note The selection of SCSI ID or mpath device name is controlled by the setting of user-friendly-name in the defaults section of the multipath.conf file.
15
Allow the system to automatically select the path priorities (svpm -F). Determine the alias of a multipath device (svpm --alias --get). Additional information is available in the man pages once the RPM is installed. Below is a list of svpm man pages. See the man pages and help for additional command explanation and examples.
16
How To ...
Chapter 4
How To
4.1 How to Get Information on Multipath Devices
Type the following command:
multipath -ll
The output will be something like the following for each multipath device:
mpath1 (36000bb500020c5f20000000000000009) dm-5 XYRATEX,F5412E [size=23G][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=3][active] \_ 2:0:1:1 sdc 8:32 [active][ready] \_ round-robin 0 [prio=2][enabled] \_ 2:0:0:1 sda 8:0 [active][ready]
In this example, the multipath device with SCSI ID 36000bb500020c5f20000000000000009 and alias mpath1 consists of two path groups containing the Linux devices /dev/sdc and /dev/sda. Linux device / dev/sdc has the higher prio value and therefore has priority. Broken down into parts, the most useful information is as labeled:
Alias (friendly name) SCSI ID mpath1 (36000bb500020c5f20000000000000009) [size=23 G][features=1 queue_if_no_path][hwhandler=0]
17
Device priority, set using svpm --prio Path Group State \_ round-robin 0 [prio=3][active] \_ 2:0:0:0 sdb 8:32 [active][ready]
Path Condition Path State Linux block device major:minor number Linux device (for example, /dev/sdb) SCSI Port:Bus:Target:LUN
Path Group States: [active] - The path group is receiving I/O requests. [enabled] - The path group will be used if the [active] path group has no paths in the [ready] state. [disabled] - The path group will be used if the [active] path group and all [enabled] path groups have no paths in the [ready] state. Path States: [ready] - The path is ready to deal with I/O requests. [active] - The path is up and is the currently active path. [faulty] - The path is not ready to deal with I/O requests. [failed] - The path is not ready to deal with I/O requests.
The output will be something like the following for each multipath device:
mpath1 (36000bb500020c5f20000000000000009) dm-5 XYRATEX,F5412E [size=23G][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=3][active] \_ 2:0:1:1 sdc 8:32 [active][ready] \_ round-robin 0 [prio=2][enabled] \_ 2:0:0:1 sda 8:0 [active][ready]
The path that is currently in use is the one marked [active] at every stage of the tree view. In this case, /dev/sdc is being used to route I/O.
18
How To ...
The output will be something like the following for each multipath device:
Alias set when logical drive was created SCSI ID LD_1 (36000bb500020c5f20000000000000009) \_ C0P1 [2:0:0:1] [priority=2] [/dev/sda] \_ C1P0 [2:0:1:1] [priority=3] [/dev/sdc]
Linux block device Current priority setting SCSI Port:Bus:Target:LUN Controller port number Controller number
Alternatively, you can set a path priority yourself with the following command:
svpm --prio --set <path> <preference>
where: <path> is the linux block device path (for example, /dev/sdb). <preference> is the priority number (5-8) where 8 is the highest priority. StorView Path Manager for Linux automatically assigns the values 1-4, so any value set using this method will override the system's automatic path choice. If you are using dissimilar hardware for different paths, you may want to assign a high priority to: The path with the highest quality components, which may be more reliable. The fastest path. A high performance LUN so that it has exclusive use of a path.
19
Warning
If you do not verify that you have your host disk blacklisted you are taking a risk in accidentally formatting your host drive containing your operating system.
It is recommended that the final value (filesystem checking) is set to 2 or higher to ensure the multipath device is checked after the root filesystem(s).
20
How To ...
Check the mounted mpath after reboot. If you have difficulties booting with the specified mount, either use a rescue disk or reboot into single user mode and comment out the selection with a # to permit debugging.
21
22
Chapter 5
23
24
Troubleshooting
Chapter 6
Troubleshooting
6.1 Installation fails
If a message stating Error: failed dependencies is given, make sure the stated software has been installed and meets the minimum version requirements. For more details, see Installation beginning on page 3.
If multipathd not listed in the output, the daemon has failed. To restart it, type: On Red Hat Enterprise Linux:
/sbin/service multipathd restart
25
26
Glossary
Glossary
DM DM (Device-Mapper) is a block system device introduced in the 2.6 version of the Linux kernel. It handles logical volume management. References to DM in this document concern dm-multipath - the DM subsystem that handles multipath I/O. Failover/Failback Failover occurs when a path becomes unavailable and another path is used to route I/O. Failback occurs when a path with a higher priority than the currently active path is restored. In this case I/O will fail back to the higher priority path once it is available again. This type of multipath behavior can be set using the multipath -p failover command. I/O must be running to allow the Failover/Failback paths to reflect the active state. Load Balancing The process where I/O to a device uses all available paths to that device. If a path is made unavailable (by component failure, for example), I/O is re-directed down the remaining paths. If paths are later added, they will be automatically detected and used. This type of multipath behavior can be set using the multipath -p multibus command. Multipath Tools A user-space application for creating, configuring, manipulating and maintaining DM multipath devices. It consists of the multipath command and the multipathd daemon. Documentation can be found at the following web page: http://christophe.varoqui.free.fr/ Path Group A path group consists of one or more paths and has a single device entry in the /dev/ mapper directory. By default, the name of the path group is the SCSI ID of the device in hex, but this can be changed to something more meaningful by assigning the path group an alias. Path Priority The path with the highest priority value (as reported by the multipath -ll command) will be used in preference to other paths. Upon failure, the path with the next highest priority will be used. If a higher priority path is later restored, I/O will switch back to it. The operating system uses priorities 1-4 automatically, leaving priorities 5-8 for the user to specify.
27
28