AIX6.

1 -
Alternate
Disk
Migration
from AIX
5.3
Date: 03/10/2010
Version: 1.0

ToCC UNIX (Chase Card Services)

03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Pre-Implementation:

Note: Can be performed one day prior to the migration

1: Login to the target AIX5.3 server and note the machine type and firmware version using
below commands.

/usr/sbin/prtconf

/usr/sbin/lsmcode

2: Go to the below IBM website to use FLRT (Fix Level Recommendation Tool).

http://www14.software.ibm.com/webapp/set2/flrt/

Input the fields as shown in the screen shot and click submit.

Fields that needs to be selected

AIX
System Model
Processor Clock Speed
AIX version -> 6100-02-04

Status should be green as shown in the below screen shot. If not contact IBM support and
make sure you can proceed with migration or not.

2
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

3
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

3: Check for latest frame microcode update available in IBM fix central website.
Follow: http://www-933.ibm.com/support/fixcentral/

Make the selections as shown in the below image and Click Continue:

Machine Type-Model -> (From prtconf output)

Select: System firmware. Obtain system firmware only. The power subsystem firmware
will be included if applicable and click Continue.

4
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Select: I know what I want. I am looking for a specific firmware level & click Continue.

From lsmcode output select the “The current temporary system firmware image” and
click Continue.

Click Continue and download the latest available firmware.

Select: System Firmware and click Continue.

5
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Click Continue.

Select “Download using Bulk FTP” and click Continue.

6
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

4: FTP Download instructions:

1: Login to pddm2
2: cd /install/aix/firmware
3: Check if the firmware is already downloaded by someone for the same machine
type.
4: If not: mkdir –p /install/aix/firmware/<Machine-Type>
5: Ex: /install/aix/firmware/8204-E8A
6: cd /install/aix/firmware/8204-E8A
7: ftp ftpproxy.jpmchase.net 8021
Connected to ftpproxy.dmzglb.jpmchase.net.
Name (ftpproxy.jpmchase.net:syshej): anonymous@delivery04-
mul.dhe.ibm.com
331 Guest login ok, send any password.
Password: <Your Email Address>
cd <Directory from the IBM Bulk Download Page>
bin
hash
prompt
mget *

7
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Implementation:

Note: Check with Application team and make sure all applications are stopped.

1: Upgrade the Frame Firmware during the outage time using the upgrade procedure
document located at:

G:\WIL\ISOperations\CustomerCare\Unix\Mid-Range Servers\AIX\ Power5

Firmware_Update-short.doc

2: Schedule Host and NIM server maintenance at http://hubeq.chase.com

3: Run mksysb script on target host as user root (use sudo su – to switch user) and verify that
it completes successfully. Mksysb script can be found in the crontab. Run mksysb with same
syntax as in the crontab.

4: Collect and save information about the server configuration

Copy scripts from qaapp01:/exports/scripts to your home directory on target server.

backup-files.ksh

capture_info.ksh

5: Copy the pre_migration and post_migration scripts from the NIM server to target host:

Location: qaapp01:/export/scripts/

6: Execute pre_migration script on the target host as root.

This script (provided by IBM) will perform various upgrade checks and save the output under:
/home/pre_migration.TIMESTAMP

At least the following data will be generated:

configuration_files_being_merged
configuration_files_missing_from_system
configuration_files_not_merged
device_software_being_removed
other_software_being_removed
saved_configuration_files
software_checksum_verification
software_consistency
software_file_existence_check
software_installed_before_migration
software_link_existence_check

Review both the script output and the files above for any issues.

Note: Due to a bug in the pre-migration scripts – no data in the output directory, if no data
in the /home/migration.date directory, skip and continue the tasks.

7: Verify software consistency:

/usr/bin/lppchk –v

8
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

8: Check the AIX logs for any hardware or software issues with the server that may affect the
upgrade, for example:

/usr/bin/errpt
/usr/bin/tail /var/adm/messages
/usr/bin/alog -f /var/adm/ras/conslog –o

9: Verify the Major number for /dev/rootvg and /dev/hd5 is 10.

ls -la /dev/rootvg /dev/hd5

brw-rw---- 1 root system 10, 1 Mar 6 09:00 /dev/hd5
crw-rw---- 1 root system 10, 0 Jun 11 2008 /dev/rootvg

Note: If the Major number is not 10 on your system, contact IBM Tech Support.

10: Check Boot Logical Volume.

The hd5 LV holds Server’s Boot Image. There are a few checks to make against the
boot LV before starting your update_all process. First, find out which disk contains your
hd5 logical volume.

lslv -m hd5
hd5:N/A
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0001 hdisk0 0001 hdisk1

The listing under the “PV1” column indicates on which disk hd5 is located. Other
entries under “PV2”... etc, represent mirrored copies.

11: Verify Server’s Boot Image can be recreated successfully using the hdisk# from above:

bosboot -ad /dev/hdisk0

bosboot: Boot image is 35795 512 byte blocks.

bosboot -ad /dev/ipldevice

bosboot: Boot image is 35795 512 byte blocks.

If any of these commands fail for any reason, please call the IBM support center for
resolution before proceeding.

lsvg -l rootvg |grep hd5

hd5 boot 1 2 2 closed/syncd N/A

lsvg rootvg |grep SIZE

VG STATE: active PP SIZE: XX megabyte(s)

This information tells us that hd5 is 1 partition, and that the partition size is XX MB.
This should be enough to contain the boot image.

Your hd5 partitions also do need to be contiguous partitions. Check this by running the
following command:

lspv -M hdisk0 |grep hd5

hdisk0:1 hd5:1

9
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

hdisk0:2 hd5:2
hdisk0:3 hd5:3

You can see in this example that the hd5 logical volume covers the first 3 partitions on
the disk and they are all contiguous. If your partitions are not contiguous, or are not
covered on the first partitions of the disk please call the software support center for
assistance with correcting this. Again, you may only have 1 partition that is large
enough to handle the boot image, or you may have multiple smaller partitions. Either is
fine.

12: Check ipldevice Minor number and make sure it matches with hdisk0 (or Primary Boot
Disk). Example:

ls -l /dev/hdisk0 /dev/ipldevice /dev/hdisk1

brw------- 1 root system 18, 0 Jun 2 2009 /dev/hdisk0
crw------- 2 root system 18, 0 Jun 11 2008 /dev/ipldevice
brw------- 1 root system 18, 1 Jun 2 2009 /dev/hdisk1

13: Make sure there is ample free space on rootvg filesystems

14: Disable TCB from ODM by executing:

odmget -q attribute=TCB_STATE PdAt | sed 's/tcb_enabled/tcb_disabled/' |
odmchange -o PdAt -q attribute=TCB_STATE

15: Save server’s current language and locale information:

/usr/bin/locale
/usr/bin/smitty mle_cc_cust_hdr

This will typically be either C (Posix) or en_US (US English):

x ISO8859-1 C (POSIX) [C] x
x ISO8859-1 English (United States) [en_US] x

16: Verify that the root user has a primary authentication method of SYSTEM.

/usr/sbin/lsuser -a auth1 root

17: Verify the location of server, if it is in SESF you need to make sure the NIM server
qaapp01 will be able to communicate over maint IP. So accordingly you can change default
gateway on the target host during outage window to be able to migrate it to AIX6.1.

Note: Only on SESF Servers execute the following Network/Gateway tasks.

A: Login to the Server’s Console as root.

B: netstat –rn > netstat-rn.out
C: mailx –s “netstat-nr Output from <Hostname> <your Email> < netstat-rn.out
D: execute netstat –i.

10
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

E: Note down the interface where maint is configured and check subnet mask with
below command
F: Delete the default route using: smitty route

Example:

Hit ENTER

G: After removing the default route of main interface, Add new default route so that
maint network is pingable from NIM master.

11
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

smitty route

Example:

Hit ENTER.

18: Login to qaapp01 and try to ping the server with <hostname-maint> DNS name and you
should be able to replies to ICMP packets.

19: Make sure and verify the maint interface was changed by CDC support to 1000/full before
proceeding to alternate disk migration. Re-Configure the Maint interface to use 1000FD.

20: On NIM server when you define machine, please specify the <hostname>–maint as
machine name.

21: All other communications from NIM master to Client will be using the <hostname-maint>.
Please use hostname-maint in this document if you are working on a SESF server.

12
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

22: Find which disks are Primary and Secondary Boot disks in the rootvg. We will be
performing ALT DISK Migration on the Second disk in the order.

In the above example, hdisk1 is the second disk in rootvg

22: Un-Mirror Root Volume Group.

/usr/sbin/unmirrorvg rootvg hdisk1

Get the dumplv name that needs to be migrated onto primary disk

/etc/lspv –l hdisk1
/etc/migratepv -l <Dump LV name> hdisk1 hdisk0
/etc/reducevg rootvg hdisk1
/etc/chpv -c hdisk1

/usr/bin/bosboot -ad /dev/hdisk0

/usr/bin/bootlist -m normal hdisk0
/usr/bin/bootlist -m normal -o (Verify bootlist shows hdisk0 with blv=hd5)
hdisk0 blv=hd5

23: Verify hdisk1 is not owned by any VG.

/etc/lspv | more

24: Verify that all software filesets are committed:

/usr/bin/lslpp -l | grep -i applied

If not, commit all applied software filesets using: smitty commit

25: mv /etc/niminfo /etc/niminfo-ORIG

26: installp –u bos.sysmgt.nim.master

13
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

27: Enable (Uncomment) shell, login and exec services in /etc/inetd.conf file and change tcp6
to tcp like below.

28: Refresh inetd daemon as follows:

refresh -s inetd

29: Create .rhosts file in root’s home directory on the Client Machine.

cd /home/root/
cat .rhosts
qaapp01.card.jpmchase.net

30: Start NFS daemons on the Client Machine: startsrc –g nfs

31: Verify TCB is disabled on the Client Machine using the following command:
/usr/bin/o dmget -q attribute=TCB_STATE PdAt

32: Login to NIM server qaapp01 and verify you are able to execute remote command using
rsh
rsh <client hostname> date

33: On the NIM server, add the Client’s IP address information to /etc/hosts file.

34: NIM Configuration

/usr/bin/smitty nim

14
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Configure the client as shown in the below screen shot:

Hit ENTER to continue.

15
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

35: Login to target host and run the below command to configure niminfo file.
smitty niminit

Hit ENTER to complete the configuration.

36: Login into NIM server qaapp01 as root and run the below command to verify nim is able
to communicate properly.
nim –o lslpp

37: On the NIM Server – qaapp01:

mkdir /tmp/<Client-hostname>
cd /tmp/<Client-hostname>

nohup nimadm -l lpp_source_6100-02-04-0920 -s spot_6100-02-04-0920 -c

<Client hostname> -d hdisk1 -Y -V -B &

tail –f nohup.out

16
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

After you see the above messages in nohup.out alt disk migration is completed.

Note: - Migration may take up to 3 to 4 hours

38: Reboot the client host after adding hdisk1 to the bootlist.
bootlist –m normal hdisk1
bootlist –m normal –o (Should display hdisk1)
shutdown –Fr now

39: Node should boot back up using AIX 6.1 kernel. Verify and fix filesets after node boots up
using AIX6.1
lppchk –v

40: If you get error about bos.dlc.qllc fileset, copy the fileset from qaapp01 and install it.

qaapp01: rcp /export/lpp_source/AIX6100-02-BASE/bos.dlc.6.1.0.0.I <hostname>:/tmp

cd /tmp; smitty installp

17
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

41: Check the New OS version:

oslevel –s
oslevel –rq
oslevel –rl 6100-02
oslevel –s should show 6100-02-04-0920

42: Check if there are any filesets missing to match 6100-02 oslevel. If any missing, Install
them by copying the base filesets from qaapp01:/export/lpp_source/AIX6100-02-BASE.

Back-Out Steps:

1: Mirror back hdisk1 with hdisk0 that has AIX5.3 image. No outage is required if alt disk
migration fails.

2: If required we can use backout using mksysb image, follow Maint_boot-NIM.doc.

G:\WIL\ISOperations\CustomerCare\Unix\Mid-Range Servers\AIX\ Maint_boot-NIM.doc

Note: Alt disk migration does not require the system to be shutdown and AIX5.3 instance will
be up and running on primary hdisk0 during the migration process and we usually do not have
to reimage using mksysb image.

Post-Migration Configuration

1: Enable TCB:
odmget -q attribute=TCB_STATE PdAt | sed 's/tcb_disabled/tcb_enabled/' |
odmchange -o PdAt -q attribute=TCB_STATE

odmget PdAt | grep -p TCB_STATE (Verify TCB_STATE is enabled)

2: If you encounter issue in adding accounts via useradd command and addUser.pl script:
useradd test
No such file or directory
getDefaultHomeDir[23]: /usr/lib/security/mkuser.default: cannot open
No default group.
Check "/usr/lib/security/mkuser.default" file.

Fix by executing the commands below

rm /usr/lib/security/mkuser.default /usr/lib/security/mkuser.sys
Copy /usr/lib/security/mkuser.default /usr/lib/security/mkuser.sys from qamq27

chown root:security /usr/lib/security/mkuser.default /usr/lib/security/mkuser.sys

ln -s /usr/lib/security/mkuser.default /etc/security/mkuser.default
ln -s /usr/lib/security/mkuser.sys /etc/security/mkuser.sys

Validate:

useradd testuser
userdel testuser

18
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

3: Fileset Validations:
lppchk –v
- Should not return anything.

/usr/sbin/instfix -i | /usr/bin/grep ML
All filesets for 6.1.0.0_AIX_ML were found.
All filesets for 6100-00_AIX_ML were found.
All filesets for 6100-01_AIX_ML were found.
All filesets for 6100-02_AIX_ML were found.

instfix -ciqk 6100-02_AIX_ML |grep ":-:"

- Should not return anything.

4: Check the error logs:

/usr/bin/errpt
/usr/bin/tail /var/adm/messages
/usr/bin/alog -f /var/adm/ras/conslog -o

5: Run the post_migration script:

/usr/lpp/bos/post_migration

This script will perform various upgrade checks, compare pre- and port-upgrade file contents,
and save the output under:

/home/post_migration.TIMESTAMP

At least the following data will be generated:

compared_saved_configuration_files
filesets_to_be_updated
filesets_updated
software_checksum_verification
software_consistency
software_file_existence_check
software_installed_after_migration
software_link_existence_check

6: Check free space for filesystems in rootvg:

/usr/bin/df -k

7: Check the date and timezone. The timezone will be changed to the timezone that the NIM
server is, so, you may be required to change that back to what it was earlier on your lpar.

/usr/bin/date
/usr/bin/grep TZ /etc/environment

8: Optionally upgrade any AIX filesets & System utilities which are not part of the standard
LPP_SOURCE or ML, such as bash, xlC.*, vac.*, Java SDK, any RPM software, SSH, sudo, etc.

19
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

9: Re-enable anything commented out above in /etc/inittab, /etc/rc.shutdown, /etc/rc.d, etc.,

as well as any disabled cron jobs.

10: Go through the security hardening procedure to ensure that the upgraded server is
compliant. Remember to turn off rsh and rlogin and remove /.rhosts. Remove the NIM server's
entry on /etc/hosts file.

11: Check the saved files for any changes to be rolled back.

/etc/hosts, /etc/inittab, /etc/motd, ifconfig -a, df -k, netstat -rn, netstat -in, /etc/inetd.conf,
/etc/mail/sendmail.cf, /etc/ntp.conf, /etc/rc.net, /etc/rc.nfs, /etc/rc.tcpip, /etc/services,
/etc/syslog.conf etc.

12: Comment the following daemons in /etc/inetd.conf after the migration, if they were not
running on AIX 5.3.
time
daytime
cmsd
wsmserver
xmquery
ttdbserver
shell
login
exec

Restart inetd: refresh -s inetd

13: Verify & fix the routing table, if needed.

14: /etc/rc.nfs usually has 'nfsrgyd' enabled after the migration. Disable it if it was not
running earlier.

15: /etc/rc.tcpip is found with portmap and muxatmd enabled after the migration. Please
disable those servcies if they weren't running earlier.

16: Restore /etc/motd back to how it looked earlier.

17: /etc/inittab is found with nfs and piobe included as part of startup. Remove them from
/etc/inittab if they weren't enabled earlier.

18: Reboot server for a final time and Start up and test any Applications and Databases.

19: Notify developer’s teams to check their applications

20: Initiate the mksysb script. Mksysb script can be found in the crontab.

21: check the following Services are started after migration:

SSH, cron,
Tivoli endpoint (lcfd)
ESM
Netbackup or TSM process
TWS (On Batch Servers)

20
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

22: Check for the community string on /etc/snmpd.conf and make sure the public is
commented out and vm61bo5 is enabled.
grep –i community /etc/snmpd.conf
#community public
community vm61bo5

23: Copy /etc/motd from other server to keep the standard.

24: Update relay host in /etc/mail/sendmail.cf file.

25: Disable the tftp and bootp on the nim server once the upgrade is completed. Please make
sure no one in the team performing the migration. Comment the following lines in
/etc/inetd.conf:

#tftp dgram udp6 SRC nobody /usr/sbin/tftpd tftpd -n

#bootps dgram udp wait root /usr/sbin/bootpd bootpd /etc/bootptab

refresh -s inetd

26: Fix the following to avoid ESM violation:

chown -R snapp:snapp /usr/sbin/snapp
chown -R ipsec:ipsec /etc/ipsec

27: Fix the syslogd:

chssys -s syslogd -a "-r"
stopsrc -s syslogd
startsrc -s syslogd

28: Fix the src daemon error in error report:

/usr/sbin/rsct/bin/rmcctrl –z (stops rsct)
cd /var/ct
ls -l -> IW (check if it is a link to a number)
/usr/sbin/rsct/install/bin/recfgct -s
/usr/sbin/rsct/bin/rmcctrl -p

29: mkdir /tmp/perf_logs

30: Login to console and verify you are able to access the server via console.

31: If needed, Upgrade Powerpath to 5.3 SP1

21
03/10/2010 AIX6.1 - Alternate Disk Migration from AIX 5.3

Final Tasks:

Note: Wait for few days and check with application team before mirroring rootvg.

1: Execute the following tasks as ROOT (sudo su – or From Console as Root)

alt_rootvg_op –X old_rootvg
extendvg -f rootvg hdisk0
mirrorvg –S rootvg hdisk0
migratepv -l dump01 hdisk1 hdisk0
bosboot -ad /dev/hdisk0
bootlist –m normal hdisk0 blv=hd5 hdisk1 blv=hd5
bootlist –m normal –o

2: If bootlist command throws an error when run using below syntax after migration.
bootlist -m normal hdisk0 hdisk1
0514-229 bootlist: Multiple boot logical volumes found on disk 'hdisk1'.
Use the 'blv' attribute to specify the one from which to boot.

Note: IBM support confirmed that this is a known issue after the alternate disk migration as
bootlist gets confused of having multiple boot logical volumes on hdisk1 and does not create
any issues to the server when you update bootlist with the below command which I
documented in AIX6.1 migration document.

3: To clear the above message we need to follow the below procedure after we mirror the
rootvg with hdisk1 which is AIX6.1 disk and hdisk0.

rmlvcopy hd5 1 hdisk1

chpv -c hdisk1
mklvcopy hd5 2 hdisk1
savebase –v
syncvg -l hd5
bosboot -ad /dev/hdisk1
bootlist –m normal hdisk0 hdisk1
bootlist –m normal –o
ipl_varyon –i

Verify Boot Device Flag is set to YES for both the disks under rootvg.

------------------------------------------- Happy Ending ---------------------------------------

22