Académique Documents
Professionnel Documents
Culture Documents
1
Install, configure, and troubleshoot VxVM
Shashidhar Soppin (shsoppin@in.ibm.com) Staff Software Engineer IBM 24 June 2008
Learn to install, configure, and troubleshoot Veritas (Symantec) 5.0 Volume Manager (VxVM) on IBM AIX 5L and AIX 6 machines using command line options. The article includes a brief explanation of VxVM basic concepts, troubleshooting, and easy fixes for a few sundry problems.
Introduction
In the world of UNIX storage management, there are two primary leaders: IBM and Veritas (now Symantec). Both companies offer products that help UNIX system administrators manage storage with very flexible methods. Veritas offers the Veritas Volume Manager (VxVM), which is either packaged as a standalone add-on, or part of a larger package, such as the Veritas On-Line Storage Manager. VxVM is a storage management subsystem that lets you manage physical disks as logical devices. (Logical devices do not have the limitations of physical disks.)
developerWorks
ibm.com/developerWorks/
Physical disks
A physical disk is the basic storage device, or media, where the data is ultimately stored. You can access the data on a physical disk by using a device name to locate the disk. The physical disk device name varies with the computer system you use. Not all parameters are used on all systems. Typical device names are of the form hdisk(number), where (number) specifies a unique disk number starting at 0.
Disk arrays
Performing I/O to disks is relatively slow because disks are physical devices that require time to move the heads to the correct position before reading or writing. If all of the read or write operations are done to individual disks, one at a time, the read-write time can become unmanageable. Performing these operations on multiple disks can help reduce the problem.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1 Page 2 of 22
ibm.com/developerWorks/
developerWorks
A disk array is a collection of physical disks that VxVM can represent to the operating system as one or more virtual disks or volumes. To the operating system, the volumes created by VxVM look and act like physical disks. Applications that interact with volumes should work the same way they do with physical disks. Figure 2 shows how VxVM presents the disks as several volumes, in a disk array, to the operating system.
Data can be spread across several disks within an array to distribute or balance I/O operations across the disks. Using parallel I/O across multiple disks improves I/O performance by increasing data transfer speed and overall throughput for the array.
Device discovery
Device discovery is the process of discovering the disks that are attached to a host. This feature is important for dynamic multipathing (DMP) because it needs to support a growing number of disk arrays from several vendors. In conjunction with the ability to discover devices attached to a host, the device discovery service lets you add support dynamically for new disk arrays. The operation, which uses a facility called the device discovery layer (DDL), does not require a reboot.
Virtual objects
Virtual objects in VxVM include:
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1 Page 3 of 22
developerWorks Disk groups Volume manager (VM) disks Subdisks Plexes Volumes
ibm.com/developerWorks/
The connection between physical objects and VxVM objects is made when you place a physical disk under VxVM control. After installing VxVM on a host system, you must bring the contents of physical disks under VxVM control by collecting the VM disks into disk groups and allocating the disk group space to create logical volumes. To bring the physical disk under VxVM control, the disk must not be under LVM control. VxVM must take control of the physical disks; the disk cannot be under the control of another storage manager (such as LVM). Listing 1 shows an example of using the vxdisk command to list the disk status.
VxVM creates virtual objects and makes logical connections between the objects. The virtual objects are then used by VxVM to do storage management tasks.
Disk groups
A disk group is a collection of disks that shares a common configuration, and is managed by VxVM. A disk group configuration is a set of records with detailed information about related VxVM objects, their attributes, and their connections. A disk group name can be up to 31 characters long. You can create additional disk groups when you need them. Disk groups allow you to group disks into logical collections. A disk group and its components can be moved as a unit from one host machine to another.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1 Page 4 of 22
ibm.com/developerWorks/
developerWorks
VM disks
When you place a physical disk under VxVM control, a VM disk is assigned to the physical disk. A VM disk is under VxVM control and is usually in a disk group. Each VM disk corresponds to one physical disk. VxVM allocates storage from a contiguous area of VxVM disk space. A VM disk typically includes a public region (allocated storage) and a small private region where VxVM internal configuration information is stored. Each VM disk has a unique disk media name, called a virtual disk name. You can either define a disk name of up to 31 characters, or allow VxVM to assign a default name of diskgroup(number), where diskgroup is the name of the disk group to which the disk belongs. Figure 3 shows an example.
Figure 3. VM Disk
Subdisks
A subdisk is a set of contiguous disk blocks. A block is a unit of space on the disk. VxVM allocates disk space using subdisks. A VM disk can be divided into one or more subdisks. Each subdisk represents a specific portion of a VM disk, which is mapped to a specific region of a physical disk. The default name for a VM disk is diskgroup(number). The default name for a subdisk is diskgroup(number-number), as shown in Figure 4, where diskgroup is the name of the disk group to which the disk belongs.
Figure 4. Subdisk
A VM disk can contain multiple subdisks, but subdisks cannot overlap or share the same portions of a VM disk. Figure 5 shows a VM disk with three subdisks. The VM disk is assigned to one physical disk.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 5 of 22
developerWorks
ibm.com/developerWorks/
Plexes
VxVM uses subdisks to build virtual objects called plexes. A plex consists of one or more subdisks located on one or more physical disks. For example, the plex vol01-01 is shown in Figure 6.
Volumes
A volume is a virtual disk device that appears to applications, databases, and file systems like a physical disk device, but does not have the physical limitations of a physical disk device. A volume consists of one or more plexes, each holding a copy of the selected data in the volume. Due to its virtual nature, a volume is not restricted to a particular disk or a specific area of a disk. You can change the configuration of a volume by using VxVM user interfaces. Making configuration changes will not cause disruption to applications or file systems that are using the volume. For example, a volume can be mirrored on separate disks or moved to use different disk storage. VxVM uses the default naming conventions of vol(number) for volumes and vol(number-number) for plexes in a volume. For ease of administration, you can choose to select more meaningful names for the volumes that you create.
ibm.com/developerWorks/
developerWorks
vxiod VxVM I/O kernel threads provide extended I/O operations without blocking calling processes. By default, 16 I/O threads are started at boot time, and at least one I/O thread must continue to run at all times. vxrelocd The hot-relocation daemon monitors VxVM for events that affect redundancy, and performs hot-relocation to restore redundancy. Before you start installation, make sure that the PATH variable is set properly because most of the commands used in the installation are in the /sbin or /usr/sbin directory. Add the directories to your PATH environment variable, as shown in Listing 2.
Prerequisites
The following Veritas Storage Foundation packages are required or optional (as noted) for installation.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1 Page 7 of 22
developerWorks
ibm.com/developerWorks/
VRTSperl PERL language VRTSspt VERITAS support tools VRTSvlic VERITAS licensing utilities VRTSvmdoc VERITAS Volume Manager online guides VRTSvmman VERITAS Volume Manager manual pages VRTSvmpro VERITAS Volume Manager Provider VRTSvxvm VERITAS Volume Manager VRTSveki VERITAS Kernel Interface component VRTSvxfs VERITAS File System VRTSddlpr VERITAS Device Discovery Layer Provider VRTSfspro VERITAS File System Provider VRTSob VERITAS Enterprise Administrator Service VRTSobgui VERITAS Enterprise Administrator VRTSFman VERITAS File System manual pages VRTSFsdoc VERITAS File System online guides
Required Required Required Optional Optional Required for VEA and ISP Required Required Required Required Required for VEA, even if you are not installing the VERITAS File System software Required for VEA and ISP Required for VEA client Optional Optional
Listing 4. installp
Note:- Check the size of the file by using cksum command as below, # cksum VRTSvxvm.bff 1823433071 126310400 VRTSvxvm.bff installp command is used as below, installp -acXd VRTSvmdoc.bff VRTSvmdoc installp -acXd VRTSvmman.bff VRTSvmman installp -acXd VRTSveki.bff VRTSveki installp -acXd VRTSspt.bff VRTSspt installp -acXd VRTSvxvm.bff VRTSvxvm installp -acXd VRTSvlic.bff VRTSvlic installp -acXd VRTSvxfs.bff VRTSvxfs installp -acXd VRTSperl.bff VRTSperl Note:- All these commands can be put inside a ksh script and
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 8 of 22
ibm.com/developerWorks/
developerWorks
Use the lslpp command to verify the complete installation of Veritas VxVM, as shown in Listing 5.
2. To install from a CD-ROM, mount a CD-ROM and begin installation as follows. 1. Log in as superuser. 2. Place the Veritas software disc in a CD-ROM drive connected to your system. 3. Mount the CD manually:
# mkdir -p /mnt/cdrom # mount -V cdrfs -o ro /dev/cdN /mnt/cdrom
where N in /dev/cdN is a number for the CD device configured on the target system. Usually the value for N is 0. Check the CD device nodes on your system in the /dev directory, or use the method below and follow Step 4 onward. To invoke the common installer, run the install command on the CD as shown below.
# /mnt/cdrom/CD_NAME/installer
4. If the VRTSvlic licensing package is not installed, a message asks if you want to install VRTSvlic. Follow the instructions to install the package. 5. If the VRTSvlic licensing package is installed, the Product Status page displays: Products available for installation. Products currently installed. Products that are licensed. Options for operations you can initiate. Use the /opt/VRTS/bin/vxlicrep command to view a report of the license type for each product 6. At the prompt on the Product Status page, enter L to add a license key and click Return to begin. 7. At the next prompt, enter the license key and click Return. 8. At the Product Status page, enter I for the product installer and click Return. The product installer is displayed. 9. At the Veritas product installer, enter the number of the product you want to install and click Return. The product installation begins automatically. At the end of the process, the Veritas product installer is displayed again.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 9 of 22
developerWorks
ibm.com/developerWorks/
10. You are prompted to enter the systems' names on which the software is to be installed. When prompted, click Enter to continue.
Veritas licensing
Install the license key using the vxlicinst command, as shown in Listing 6. (The license key below is an example only. For actual installation, the key is obtained from Symantec.)
There is more information about the vxconfigd daemon and vxdctl later in this section. To check the Storage Foundation processes and confirm successful Veritas File System installation, use the following command:
# lsvfs vxfs vxfs 32 /sbin/helpers/vxfs /sbin/helpers/vxfs
To confirm that the Volume Manager processes (vxconfigd, vxnotify, and vxrelocd) are running, use:
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1 Page 10 of 22
ibm.com/developerWorks/
developerWorks
# ps -e |grep vx
Now the disks have to be configured as a Veritas Volume disk group (vxvmdg) by initializing and adding disks. There are two levels of initialization for disks in the Volume Manager: Formatting of the disk media itself. This must be done outside of the Volume Manager. Storing identification and configuration information on the disk for use by the Volume Manager. Volume Manager interfaces are provided to step you through this level of disk initialization. A fully initialized disk can be added to a disk group and used to replace a previously failed disk, or to create a new disk group.
developerWorks
ibm.com/developerWorks/
The disk listing sometimes displays in the format hdisk(number). Use the format accordingly while configuring with the vxdg command.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 12 of 22
ibm.com/developerWorks/
developerWorks
Disk_24 Disk_25
auto auto
online LVM
hdisk28 hdisk26
In the above listing, Disk_20 is in error state and needs to be fixed to continue VxVM operation. To see the list of hdisks with an error state:
Try to use the vxdisksetup and vxdiskunsetup commands to format the hdisk, as shown in Listing 13.
Since both the commands are failing to clear the error state, see the detailed list using the vxdisk command for error one and proper one, as shown in Listing 14.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 13 of 22
developerWorks
# vxdisk list Disk_20 Device: Disk_20 devicetag: Disk_20 type: auto info: format=cdsdisk,privoffset=256 flags: online error private autoconfig pubpaths: block=/dev/vx/dmp/Disk_20 char=/dev/vx/rdmp/Disk_20 guid: {5d5aa3b8-1dd2-11b2-aeae-2a5848ed1fb8} udid: AIX%5FVDASD%5FDISKS%5F600507630EFFFD670000000000000115 site: errno: Disk is not useable, bad format Multipathing information: numpaths: 1 hdisk32 state=enabled b) Proper one # vxdisk list Disk_7 Device: Disk_7 devicetag: Disk_7 type: auto info: format=none flags: online ready private autoconfig invalid pubpaths: block=/dev/vx/dmp/Disk_7 char=/dev/vx/rdmp/Disk_7 guid: udid: AIX%5FVDASD%5FDISKS%5F600507630EFFFD670000000000000122 site: Multipathing information: numpaths: 1 hdisk42 state=enabled
ibm.com/developerWorks/
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 14 of 22
ibm.com/developerWorks/
developerWorks
# lspv hdisk26 hdisk27 hdisk28 hdisk29 hdisk30 hdisk31 hdisk32 hdisk33 hdisk34 hdisk35 hdisk36 hdisk37 hdisk38 hdisk39 hdisk40 hdisk41 hdisk42 hdisk43 hdisk44 hdisk45 hdisk46 hdisk47 hdisk48 hdisk49 hdisk50 hdisk51
00cc17cec566971d 00cc17ceebb54c8c none none none none none none none 00cc17ce59d7d248 none none none none none none none none none none none 00cc17cee3638f87 00cc17cee2ea1ed0 00cc17cee2ea1f4d 00cc17cee2ea1fe1 00cc17cec8c84486
rootvg rootvg None VeritasVolumes None None VeritasVolumes None None None None VeritasVolumes None VeritasVolumes None None None VeritasVolumes VeritasVolumes None None None None None None None
active active
Even after executing the chpv command, the Veritas volumes tag is not removed. Try enabling and disabling with the vxconfigd daemon.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 15 of 22
developerWorks The tag is still not removed, and the disk is still in the error state.
ibm.com/developerWorks/
Run the set of commands below for up to seek=3 or 4, and then run the other set of commands. The dd command clears the bad blocks, if any, and the error state is removed.
Listing 17. dd
# dd if=/dev/zero of=/dev/hdisk37 bs=512 seek=0 count=10 10+0 records in 10+0 records out # dd if=/dev/zero of=/dev/hdisk37 bs=512 seek=1 count=10 10+0 records in 10+0 records out # dd if=/dev/zero of=/dev/hdisk32 bs=512 seek=0 count=10 10+0 records in 10+0 records out
Now run the vxdisksetup command, and it will clear the error state of the disk.
Note:- Here numbers 7,8,9,.... etc indicates hdisk numbers. Now vxdisk command output displays the added disks as vxvmdg disks. # vxdisk list DEVICE TYPE hdisk0 auto:LVM hdisk1 auto:cdsdisk hdisk2 auto:cdsdisk hdisk3 auto:none hdisk4 auto:cdsdisk hdisk5 auto:none hdisk6 auto:aixdisk hdisk7 auto:cdsdisk hdisk8 auto:cdsdisk hdisk9 auto:cdsdisk hdisk10 auto:cdsdisk hdisk11 auto:cdsdisk hdisk12 auto:aixdisk hdisk13 auto:aixdisk DISK vxvmdg1 vxvmdg2 vxvmdg3 vxvmdg4 vxvmdg5 vxvmdg6 vxvmdg7 GROUP vxvmdg vxvmdg vxvmdg vxvmdg vxvmdg vxvmdg vxvmdg STATUS LVM online online online invalid online online invalid online online online online online online online online
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 16 of 22
ibm.com/developerWorks/
developerWorks
hdisk14
auto:cdsdisk
vxvmdg8
vxvmdg
online
The Volume Manager Support Operations menu interface, which provides a menu of disk operations. Each entry in the main menu leads you through a particular operation by providing information and asking questions. Default answers are provided for many questions, so common answers can be selected quickly.
vxdiskadd
Use this utility to add standard disks to the Volume Manager. vxdiskadd leads you through initializing a new disk by displaying information and asking questions.
vxdisk
A command-line utility for administering disk devices. You can use vxdisk to define special disk devices, to initialize information stored on disks that the Volume Manager uses to identify and manage disks, and to perform additional special operations.
vxdg
The command-line utility for operating on disk groups. You can use it to create new disk groups, to add and remove disks from disk groups, and to enable (import) or disable (deport) access to disk groups.
vxdisk list
Lists disk information and displays spare disks with a spare flag.
vxprint
Lists disk and other information, and displays spare disks with a SPARE flag. Listing 20 shows an example of using the vxdiskadm command to control various Volume Manager operations. Any of these options can be selected for the required operation to be performed.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 17 of 22
developerWorks
20 21 22 list ? ?? q Mark a disk as allocator-reserved for a disk group Turn off the allocator-reserved flag on a disk Change the disk naming scheme List disk information Display help about menu Display help about the menuing system Exit from menus
ibm.com/developerWorks/
Select an operation to perform: Below is the example for creating volume group vxvmdg and adding a disk to it. Select an operation to perform: 1 Add or initialize disks Menu: VolumeManager/Disk/AddDisks Use this operation to add one or more disks to a disk group. You can add the selected disks to an existing disk group or to a new disk group that will be created as a part of the operation. The selected disks may also be added to a disk group as spares. Or they may be added as nohotuses to be excluded from hot-relocation use. The selected disks may also be initialized without adding them to a disk group leaving the disks available for use as replacement disks. More than one disk or pattern may be entered at the prompt. some disk selection examples: all: hdisk8: scsi3 : fscsi6 : xyz_0 : xyz_ : Here are
all disks a single disk (in the hdisk# naming scheme) all disks on scsi controller 3 all disks on fiber controller 6 a single disk (in the enclosure based naming scheme) all disks on the enclosure whose name is xyz
Select disk devices to add: [(pattern-list),all,list,q,?] hdisk2 Here is the disk selected. hdisk2 Continue operation? [y,n,q,?] (default: y) y You can choose to add this disk to an existing disk group, a new disk group, or leave the disk available for use by future add or replacement operations. To create a new disk group, select a disk group name that does not yet exist. To leave the disk available for future use, specify a disk group name of "none". Which disk group [<group>,none,list,q,?] (default: none) vxvmdg Create a new group named vxvmdg? [y,n,q,?] (default: y) y Create the disk group as a CDS disk group? [y,n,q,?] (default: y) y Use a default disk name for the disk? [y,n,q,?] (default: y) y Add disk as a spare disk for vxvmdg? [y,n,q,?] (default: n) y Add site tag to disk? [y,n,q,?] (default: n) y A new disk group will be created named vxvmdg and the selected disks will be added as spares to the disk group with default Output format: [Device_Name]
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 18 of 22
ibm.com/developerWorks/
developerWorks
Now you can use lspv or vxdisk to see that hdisk1 has been added to the vxvmdg group. Listing 21 shows how to use vxdg to destroy any unwanted disk group entries.
# vxdisk -e list DEVICE TYPE Disk_0 auto Disk_1 auto Disk_2 auto Disk_3 auto Disk_4 auto Disk_5 auto Disk_6 auto Disk_7 auto Disk_8 auto Disk_9 auto Disk_10 auto Disk_11 auto Disk_12 auto Disk_13 auto Disk_14 auto Disk_15 auto
DISK -
GROUP -
STATUS online online online online online online online online online online online online LVM online LVM online
OS_NATIVE_NAME hdisk15 hdisk14 hdisk13 hdisk12 hdisk10 hdisk9 hdisk11 hdisk8 hdisk7 hdisk6 hdisk5 hdisk4 hdisk0 hdisk1 hdisk3 hdisk2
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 19 of 22
developerWorks
# for i in 7 6 5 4 3 2;do > vxdisksetup -if Disk_$i format=aixdisk > done # for i in 7 6 5 4 3 2;do > vxdisksetup -if Disk_$i format=cdsdisk > done # vxdg -g vxvmdg adddisk vxvmdg2=hdisk5 VxVM vxdg ERROR V-5-1-6478 Device hdisk5 cannot be added to a CDS disk group [root@ortega01] /ezcfg/dat # vxdisk list DEVICE TYPE hdisk0 auto:LVM hdisk1 auto:cdsdisk hdisk2 auto:cdsdisk hdisk3 auto:none hdisk4 auto:cdsdisk hdisk5 auto:aixdisk
ibm.com/developerWorks/
DISK vxvmdg1 -
GROUP vxvmdg -
If you get the above type of error, fix it using: #/etc/vx/bin/vxdisksetup -if hdisk5 format=cdsdisk Now run: # vxdg -g vxvmdg adddisk vxvmdg2=hdisk5 # vxdisk list DEVICE TYPE hdisk0 auto:LVM hdisk1 auto:cdsdisk hdisk2 auto:cdsdisk hdisk3 auto:none hdisk4 auto:cdsdisk hdisk5 auto:cdsdisk DISK vxvmdg1 vxvmdg2 GROUP vxvmdg vxvmdg STATUS LVM error error online invalid online online
Summary
This article provided a brief overview of VxVM. You also learned about installation, configuration, administration, troubleshooting, and some command-line utilities. Though you now have a better understanding of VxVM, it is recommended you go to the Veritas Symantec Web site for more information (see Resources).
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 20 of 22
ibm.com/developerWorks/
developerWorks
Resources
Learn For a comparison of AIX and Veritas Volume Manager concepts, read the White Paper "AIX Logical Volume Manager and Veritas Volume Manager quick reference." See the Veritas Volume Manager 3.2 User's Guide - Veritas Enterprise Administrator. Read the IBM Redbooks: AIX Logical Volume Manager, from A to Z Introduction and Concepts AIX Logical Volume Manager, from A to Z Troubleshooting and Commands Refer to the Red Paper Quick Reference: AIX Logical Volume Manager and Veritas Volume Manager. "AIX 5L improvements in volume management performance" (developerWorks, Sep 2003) discusses optimizations in AIX. Browse the technology bookstore for books on these and other technical topics. Get products and technologies Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2, Lotus, Rational, Tivoli, and WebSphere. Discuss Information on various Veritas (Symantec) products and documents Veritas (Symantec)Volume Manager for Unix/Linux products AIX Wiki AIX UNIX forums Check out developerWorks blogs and get involved in the developerWorks community.
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 21 of 22
developerWorks
ibm.com/developerWorks/
Manage your logical volumes with Veritas Volume Manager (VxVM) V5 for AIX 5.3 and 6.1
Page 22 of 22