AIX SAN boot considerations

There are significant advantages when booting from SAN (installing AIX rootvg on LUNs):
1. advantages conferred by a disk storage subsystem:
1. better I/O performance due to caching and striping across multiple spindles,
2. ability to redeploy disk space when a server is retired from service, and
3. option to use FlashCopy to capture a rootvg backup (but mind the caveats) and
2. option to move an AIX image from one physical server/LPAR to another.

There are, however, some disadvantages. AIX sysadmins who are aware of the disadvantages, mitigate
them, and find disadvantage acceptable and can boot from SAN with confidence.

1. If a SAN hardware problem or an AIX defect causes intermittent loss of access to the SAN,
there is no way of capturing a dump to determine what went wrong. The AIX error log and
the AIX dump logical volume are in rootvg. If rootvg is on LUNs and AIX can not access
LUNs, there is no way to write a dump to the SAN.

Option to mitigate:

Configure dump space on a SCSI hdisk dedicated to the LPAR (or on a vSCSI disk which is
mapped to an internal SCSI disk allocated to a VIO Server LPAR). Extend rootvg onto the
SCSI (or vSCSI) hdisk and configure dump space on it. Because this hdisk is not on the SAN,
AIX can write a dump even if access to the SAN is lost. And the /var/adm/ras directory
(which contains the AIX errlog) can also be allocated on a SCSI (or vSCSI) hdisk, although
this seems less important than configuring non-SAN dump space.

2. It is difficult to update the Multipath Subsystem Device Driver (SDD or SDDPCM) or other
Fibre Channel multipath I/O support (eg, EMC PowerPath or HDS HDLM) when rootvg is on
hdisks accessed via multiple Fibre Channel paths, assuming such multi-path access is
supported for rootvg hdisks. For example, when Veritas DMP is used for the AIX rootvg, one
must disable DMP before updating it, which requires an AIX reboot. See the Veritas Volume
Manager: Configuring DMP on AIX for SAN Booting Symantech technical note for more
information.

Option to mitigate:

For help upgrading AIX when SDDPCM is installed, please see Revised Scripts are
Available. Please contact Technical Support if you are booting AIX rootvg from SAN devices
and want to migrate SDD to SDDPCM or upgrade AIX OS when SDDPCM is
installed Technote.

No mitigation is required when using SDDPCM with AIX MPIO. A new version of SDDPCM
can be installed while the current version remains in use. An AIX reboot is required to enable
use of the new SDDPCM version for rootvg hdisks. For more information, see the "Updating
SDDPCM" section of the Multipath Subsystem Device Driver User's Guide (downloadable
from the Latest Multipath Subsystem Device Driver User's Guide web page. ( As with any
system update, it is very important to preserve the option to fall back to a working version
should an update render the AIX image unusable. See a note on the AIX backup and
restore web page for methods of updating AIX while preserving an option to fall back.)
 When using SDD, boot from a LUN accessed via a single path (hdisk) rather than multiple
paths (vpath). Mirror AIX on two such LUNs (accessed via different Fibre adapters) so that
AIX can survive the failure of one of the Fibre adapters. This mitigation is not optional. As
stated in the Multipath Subsystem Device Driver User's Guide:

SDD does not support:

o Multipathing to a system boot device
o Placing system primary paging devices (for example, /dev/hd6) on an SDD vpath
device
o Configuring SDD vpath devices as system primary or secondary dump devices

When SDD is configured without multipathing to system boot devices, there is no difficulty
updating SDD software to a new level. It seems almost certain that the mitigation appropriate
for SDD will work equally well for EMC PowerPath, HDS HDLM, and HP AutoPath. (It is
likely that, like SDD, third-party multipath device drivers do not support multipathing to
rootvg hdisks.)

3. Running AIX on LUNs can be the source of some very mysterious AIX behavior if the SAN
occasionally injects delays into I/O operations, particularly paging operations. (According to
a comment by Jim Carstensen, updating SAN zoning will inject delays in some SANs.) And it
is certainly the case that AIX hangs lasting several minutes are a big concern in a cluster
(HACMP, VCS, etc), where, if a node hangs for a long time and suddenly wakes up, there is
the risk of data corruption. It seems imprudent to boot from SAN (or allocate paging space on
SAN for) any cluster node unless the cluster's shared volume groups are protected by disk
reservation locks. (In this context, booting from vSCSI disks mapped to LUNs is equivalent
to booting from SAN.)

Please note that vSCSI disks don't currently support SCSI-3 persistent reserves, so it is
currently impossible to protect (with disk reservation locks) a cluster's shared volume group if
that volume group resides on vSCSI disks. However, SCSI-3 persistent reserves are supported
with N Port ID Virtualization (NPIV), which is available with 8 GB Fibre Channel PCIe
adapters and PowerVM Standard Edition, assuming minimum VIOS level (V2.1) and other
requirements are met. NPIV for Power Systems is announced in IBM US Software
Announcement 208-341 dated October 7, 2008.

If a SAN delay persists long enough that a write I/O request times out and fails and AIX does
not crash as a result, there should be concern regarding data and filesystem integrity. While
AIX is designed to handle write I/O request failures properly, it is not possible to inject every
possible write I/O error in a test environment. Because every write I/O failure scenario can
not possibly be tested, there is the potential that an undiscovered AIX software defect will
impact data and filesystem integrity when a write I/O failure occurs. Therefore, even if write
I/O failures do not cause AIX crashes, such failures must be treated as very serious SAN
problems which deserve the greatest possible effort to diagnose and resolve.

4. Accidentally installing AIX on the wrong LUNs or booting a system from the wrong LUNs.
These risks can generally be avoided with prudent SAN administration. For more
information, please see the Working with SAN Boot and Multi-path IO on AIX ATS Techtalk.

Please note that rootvg can be placed on a vSCSI disk mapped to a LUN without concern for
disadvantage #2. That's because the VIO client (to which the rootvg belongs) does not use (nor need)
SDD for multipathing. MPIO with the AIX default PCM is used instead. And please note that
regarding the algorithm attribute of vSCSI hdisks, the IBM PowerVM Best Practices Redbook says,
"A virtual SCSI disk only supports fail_over mode." So algorithm=round_robin (for example) is not
supported on vSCSI hdisks.
Please note that APAR IZ72620 in AIX 6.1 TL6 adds AIX Boot Path Enablement enhancements,
which (among other things) add support for a pathid option on the AIX bootlist command. For more
information, please see the AIX Boot Path Enablement Technote. The enhancements are also available
in AIX V7.1.

Moving an AIX rootvg from one physical server/LPAR to

another

Note

Moving an AIX rootvg from one server/LPAR to another isn't supported but might work
provided the CPU architectures of the source and target are the same and the source and target
have identical PCI adapter configurations.
However, according to anonymous sources in IBM AIX Development, even after a LUN has
booted successfully on a server/LPAR other than the one from which it was installed/built,
there is no guarantee that the LUN will continue to boot successfully. According to the
anonymous sources, the only methods supported by IBM for copying an AIX rootvg from one
server/LPAR to another are to use the AIX mksysb, mkcd, or mkdvd command, to use the
AIX alt_disk_copy command with the -O flag, or to use the bare metal restore option of IBM
Tivoli Storage Manager for System Backup and Recovery (formerly known as SysBack). See
the AIX backup and restore wiki page for more information about options for cloning an AIX
image.
The AIX higher availability using SAN services article documents alternatives for moving an
AIX rootvg from one server/LPAR to another when the rootvg resides on virtual disks. Near
the bottom, the article lists known potential issues which might be encountered when
attempting to boot up a rootvg in an LPAR other than the one in which it was last running. It is
possible that the issues discussed in the artcle can be addressed using the
AIX devreset command. ( Warning: Do not use devreset without first reviewing
information about it in file /usr/lpp/bos/README.PARTITION_INSTALL!) Options for
cloning an AIX image are discussed in the AIX higher availability using SAN services article,
as well. Furthermore, as of 6/7/2013, the article says:
Many clients would like to be able to use a binary copy of rootvg from one CEC to boot an
operating system image from either a different LPAR or a different CEC. This is especially
true when designing disaster recovery processes with limitations on the available time to
restore a large quantity of AIX rootvg images. Although AIX cannot issue a blanket statement
of support for booting a binary copy of rootvg because of the sheer number of possible
permutations, you may be able to design workable procedures for your environment.
The sys0 ghostdev attribute was added to facilitate booting an AIX image from either a
different LPAR or different CEC. By running 'chdev –a ghostdev=1 –l sys0' prior to replicating
the AIX image, the AIX ghostdev feature will be enabled. When ghostdev is enabled, it will
trigger during AIX boot if the AIX image is detected as booting from either a different LPAR
 or CEC. When the ghostdev feature is triggered, the customized devices and customized
attributes in ODM will be deleted, and new customized devices will be created. The new
customized devices will be created with the default attribute values, and may have different
logical device names. Enabling the ghostdev feature prior to booting the AIX from a different
LPAR or CEC will reduce complexity by removing the Defined and Missing customized
devices.
LPM operations will not trigger the ghostdev feature. Middleware or applications that have a
dependency on tracking the hardware on the system may need to be configured again after
moving the operating system image to a different CEC or LPAR. If you are using multi-path
software besides AIX MPIO, then it is recommended you contact your multi-path software
vendor prior to using ghostdev attribute. The ghostdev attribute requires a minimum AIX level
of 6.1 TL 7 or 7.1 TL 1. If AIX 6.1 TL 7 is used, then it is recommended that APAR IV11039
is installed. If AIX 7.1 TL 1 is used, then it is recommended that APAR IV10982 is installed.
AIX levels later than AIX 6.1 TL 7 and AIX 7.1 TL 1 do not require any additional APARs.

When using either the ghostdev attribute or the devreset command, please note that Ethernet IP
interfaces might be lost and/or moved to Ethernet adapters other than those with which the
interfaces were associated before devreset was run or the ghostdev feature was triggered. Plan to
login to AIX via the system console when AIX first boots in a new LPAR (or reboots after
running devreset) and to confirm that the TCP/IP configuration (interfaces and routes) is correct.
Please note that if AIX is shut down on one server/LPAR and booted up on another, AIX will come up
using Fibre Channel adapters which have different WWPNs than those on which it ran earlier. LUN
masking (and probably SAN zoning) must be changed to accommodate the new WWPNs. The
Ethernet MAC addresses will change, too. And unless the source and target LPARs have identical PCI
adapter configurations, AIX may have difficulty mapping existing IP addresses to Ethernet adapters in
the target server/LPAR.
Please note that if an AIX rootvg is moved from an LPAR on one server to an LPAR on another server,
difficulties with DLPAR might be seen in the new server, as described in the After alternate disk
install cloning, improper hostname resolution may cause HMC RSCT errors Redbooks tip.