27/01/2015

AIX configuration recommendations for Oracle RAC

Version
Version Date
0.1
1st December
2009
0.2
1st December
2009
0.3
2nd December
2009
0.4
7th December
2009

Author
Gnanasambandam,
Swaminathan
Ben Wright
Ben Wright
Ben Wright

Comments
Initial Draft
Tidy up and create script out
of instructions
Addition of extra stuff for
configuration
Read through tidy up and
check

dd if=/dev/zero of=/dev/emcasm001 bs=125829120 count=1

Introduction
This document goes through the configuration of an AIX Server for installing Oracle
RAC.
It first goes through the User Requirements for the Oracle Cluster and Oracle User
and then goes through some best practises identified by the RAC Assurance Team
relevant to AIX specific RAC implementations.
The link below provides a base point for the rest of the document.
http://download.oracle.com/docs/cd/B28359_01/install.111/b28258/preaix.htm#BAB
FDGHJ
The server containing these scripts is sinupe02.infra.go2uti.com (10.240.12.103) and
they are located in /data/software/scripts/admin/aix/oracle.
This can be mounted by issuing the commands:
mkdir /tmp/net
mount 10.240.12.103:/data/software /tmp/net
The following software should be installed:
The following software packs should be installed on the server.
bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
rsct.basic.rte
rsct.compat.clients.rte
Ben Wright
bwright@go2uti.com

27/01/2015

xlC.aix61.rte:9.0.0.1 (or later)

xlC.rte:9.0.0.1 (or later)
gcc
jdk
Note: Following command lists the existing installed packages information
lslpp l
lslpp a (for all options)
Other than gcc and jdk the other Software Sets should be able to be installed from
the NIM Server (for installation instructions please see AIX configuration after NIM
installation they are found in the Base Installation Software Package).
For GCC, it needs to be installed from the rpm package provided in
10.240.12.103:/data/software/aix
To install the rpm package run the command:
rpm i /tmp/net/aix/gcc-4.2.0-3.aix6.1.ppc.rpm
For JDK, this comes installed with the OS, different versions can be found on the
NIM Server.
Creating the required Groups and Users
We need to create following users along with following groups.
1. User - Oracle:pry grp : oinstall Sec grp:dba
2. User - asm: pry grp : oinstall Sec grp : asm, dba, asmdba
3. User - crs: pry grp : oinstall Sec grp : crs, dba
Example of Creating the Oracle Clusterware User and OraInventory Path - new
The following is an example of how to create the Oracle Clusterware software owner
(in this case, crs), and a path compliant with OFA structure with correct permissions
for the oraInventory directory. This example also shows how to create separate Oracle
Database and Oracle ASM homes with correct ownership and permissions:
# mkdir /u01/app/crs
# chown crs:oinstall /u01/app
# mkdir /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/app/
# mkdir /u01/app/asm
# chown asm:oinstall /u01/app/asm
At the end of this procedure, you will have the following:
/u01 owned by root.
/u01/app owned by crs:oinstall with 775 permissions. This ownership and
Ben Wright
bwright@go2uti.com

27/01/2015

permissions enables OUI to create the oraInventory directory, in the path

/u01/app/oraInventory.
/u01/app/crs owned by crs:oinstall with 775 permissions. These
permissions are required for installation, and are changed during the installation
process.
/u01/app/oracle owned by oracle:oinstall with 775 permissions.
/u01/app/asm owned by asm:oinstall with 775 permissions.
In the .profile of all user crs,asm,orcale
test -t 0
if ($status == 0) then
stty intr ^C
endif
umask 022
aix_thread =s

the group dba will OSDBA(Database DBA),the group asm will be(asmdba),the group
crs will be crs
The plan is to have separate oracle_base for Clusterware and owned by crs
user.The asm and db will share the same oracle_base but separate oracle_home.
Note: Edit .profile of the oracle, asm and crs users
.profile must be contains the following entries and remaining lines should be
commented before installing softwares
Umask 022
TMP, TMPDIR location and export the same
ORACLE_BASE
AIXTHREAD_SCOPE=S
After installation complete uncomment all the entries and add required entries.
The Groups and Users can be created using the following script:
AIX-Unix-Ora-RAC-User-Create.sh
The scripts should perform all the tasks required by the documentation any
additional changes should be reviewed and added below (preference is to modify the
script for future rollouts).
After the above script has been run the /etc/security/limits file needs to be modified so
that the default is set to:
Ben Wright
bwright@go2uti.com

27/01/2015

default:
fsize = -1
core = -1
cpu = -1
data = 512000
rss = 512000
stack = 512000
nofiles = 2000
To set the Maximum Number of User Processes to 2176 or above
To set the System Block Size Allocation to 128 or above
Then run the script AIX-Set-Limits.sh
Tune Virtual Memory Parameters
The Virtual Memory Parameters can be set by running the script:
AIX-Set-VMM.sh
This will tune the following Virtual Memory Parameters:
minperm%=3
maxperm%=90
maxclient%-90
lru_file_repage=0
strict_maxperm=0
strict_maxclient=1
page_steal_method=1
And set the AIXTHREAD_SCOPE value to S in the /etc/environment file
And to disable the Processor Folding for Oracle Environments (vpm_xvcpus=-1)
Network requirements
Network parameters on the server need to be adjusted to improve the performance of
the Oracle Cluster. Run the script:
AIX-Set-NetworkParam.sh
This will set the following network parameters:
udp_sendspace=65536
udp_recvspace=655360
tcp_sendspace=65536
tcp_recvspace=65536
rfc1323=1
sb_max=2*655360
ipqmaxlen=512
$ ./rda.sh -T hcve
Ben Wright
bwright@go2uti.com

27/01/2015

Processing HCVE tests ...

Available Pre-Installation Rule Sets:
1. Oracle Database 10g R1 (10.1.0) PreInstall (AIX)
2. Oracle Database 10g R2 (10.2.0) PreInstall (AIX)
3. Oracle Database 11g R1 (11.1.0) PreInstall (AIX)
4. Oracle Application Server 10g (9.0.4) PreInstall (AIX)
5. Oracle Fusion Middleware 11g R1 (11.1.1) PreInstall (AIX)
6. Oracle Portal PreInstall (Generic)
Available Post-Installation Rule Sets:
7. Oracle Portal PostInstall (generic)
8. Data Guard PostInstall (Generic)
Enter the HCVE rule set number
Hit 'Return' to accept the default (1)
>3
Enter value for < Planned ORACLE_HOME location >
Hit 'Return' to accept the default (/opt/app/oracle/asm/product/11.1.0/asm)
>
Enter value for < JDK Home >
> /usr/java5_64
Test "Oracle Database 11g R1 (11.1.0) PreInstall (AIX)" executed at Fri Dec 18
07:57:58 2009
Test Results
~~~~~~~~~~~~
ID NAME
RESULT VALUE
====== ==================== =======
==========================================
A00010 OS Certified?
PASSED Certified with Oracle Database 11g R1
A00020 User in /etc/passwd? PASSED userOK
A00040 Group in /etc/group? PASSED GroupOK
A00050 Input ORACLE_HOME RECORD
/opt/app/oracle/asm/product/11.1.0/asm
A00060 ORACLE_HOME Valid? PASSED OHexists
A00070 O_H Permissions OK? PASSED CorrectPerms
A00080 oraInventory Permiss PASSED oraInventoryOK
A00090 Got ld,nm,ar,make? PASSED ld_nm_ar_make_found
A00100 Umask Set to 022? PASSED UmaskOK
A00110 ulimits OK?
FAILED DataTooSmall StackTooSmall
A00140 LDLIBRARYPATH Unset? FAILED IsSet
A00160 LIBPATH Unset?
FAILED IsSet
A00190 JDK Home
RECORD /usr/java5_64
A00200 JDK Version
PASSED Adequate
A00210 Other O_Hs in PATH? FAILED OratabEntryInPath
A00220 Other OUI Up?
PASSED NoOtherOUI
Ben Wright
bwright@go2uti.com

27/01/2015

A00230 /tmp Adequate?

PASSED TempSpaceOK
A00240 Disk Space OK?
PASSED DiskSpaceOK
A00250 Swap (in MB)
RECORD 8192
A00260 RAM (in MB)
PASSED 4096
A00270 SwapToRAM OK?
PASSED SwapToRAMOK
A00290 IP Address
RECORD 10.240.8.117
A00300 Domain Name
RECORD devtst.go2uti.com
A00310 DNS Lookup
FAILED Host not correctly registered in DNS
A00320 /etc/hosts Format FAILED Missing host.domain
A00340 AIXTHREAD_SCOPE=S? PASSED AIXTHREAD_SCOPEOK
A00350 LINK_CNTRL is Unset? PASSED LINK_CNTRLunset
A00410 Got OS Patches?
PASSED NotChecked
A00430 Got OS Packages? PASSED All required OS packages are installed
Result file: /tmp/rda/output/RDA_HCVE_A200DB11R1_aix_res.htm
Network configuration
Each node in the cluster must have at least two network adapters together with the
names associated with the network adapters being the same across all nodes.
Thus for two nodes enN must be used for the same purpose public or private.
Using the following commands change the values of the following parameters(in
AIX):
# no -o tcp_sendspace=262144
# no -o tcp_recvspace=262144
# no -o udp_sendspace=65536
# no -o udp_recvspace=262144
Also the parameter rfc1323 needs to be set to 1 for any TCP/ UDP parameters
settings of over 64K.
no -o rfc1323=1
Adding these entries to /etc/rc.net will ensure that the parameter take effect on
every reboot.

IP Address Requirements
Before starting the installation, fill in the table below for the Oracle DBAs (below is
an example from the Test RAC Installation.
Node Name
Sinutn02

Interface
Name
Sinutn02

Ben Wright
bwright@go2uti.com

Type

IP Address

Public

10.240.8.117

Registered
in
DNS

Device
En?
6

27/01/2015

Sinutn02vip
Sinutn02priv
Sinutn03
Sinutn03vip
Sinutn03priv

Sinutn02vip
Sinutn02priv
Sinutn03
Sinutn03vip
Sinutn03priv

Public

10.240.8.201

DNS

En?

Private

10.240.14.117 Hosts

En?

Public
Public

10.240.8.125
10.240.8.202

DNS
DNS

En?
En?

Private

10.240.14.12
5

Hosts

En?

Notes:

An IP address with an associated network name registered in the domain name

service (DNS) for the public interface. If you do not have an available DNS,
then record the network name and IP address in the system hosts
file, /etc/hosts.
One virtual IP (VIP) address with an associated network name registered in
DNS. If you do not have an available DNS, then record the network name and
VIP address in the system hosts file, /etc/hosts. Select an address for your VIP
that meets the following requirements:
o The IP address and network name are currently unused
o The VIP is on the same subnet as your public interface
Before installation, check that the default gateway can be accessed by
a ping command. During installation, OUI uses the ping command to ensure
that the VIP is reachable. To find the default gateway, use the route command,
as described in your operating system's help utility. After installation,
configure clients to use either the VIP address, or the network name associated
with the VIP. If a node fails, then the node's virtual IP address fails over to
another node.
A private IP address with a host name for each private interface

Oracle recommends that you use private network IP addresses for these interfaces (for
example: 10.*.*.* or 192.168.*.*). Use the /etc/hosts file on each node to associate
private network names with private IP addresses.
A useful command to see the Network Interfaces and their IP Addresses use:
# netstat -i
Note: /etc/hosts file entries should be like the following example
Cat /etc/hosts
127.0.0.1
10.240.8.130
10.240.8.210
10.240.14.130
10.240.8.131
10.240.8.211

loopback localhost

# loopback (lo0) name/address

sinuqn01.qc.go2uti.com
sinuqn01-vip.qc.go2uti.com
sinuqn01-priv.qc.go2uti.com
sinuqn02.qc.go2uti.com
sinuqn02-vip.qc.go2uti.com

Ben Wright
bwright@go2uti.com

sinuqn01
sinuqn01-vip
sinuqn01-priv
sinuqn02
sinuqn02-vip
7

27/01/2015

10.240.14.131

sinuqn02-priv.qc.go2uti.com

sinuqn02-priv

Setting up the I/O Disk configuration

Oracle make the following recommendation however as we are using LPARs this is
set on the VIO Server layer:
Set the Fibre Channel SCSI I/O Controller Protocol Device event error recovery
policy to fast_fail for each fibre-channel adapter:
chdev -l fscsi0 -a fc_err_recov=fast_fail -P
Creating the RAW Devices for Oracle Cluster
When implementing RAC, you must take care with the SAN storage chosen. The
SAN Storage must be capable of read/write concurrency (writing at the same time
from any member of the RAC cluster) through its drivers. This means that
reserve_policy attribute from the discovered disks (hdisk, hdiskpower, dlmfdrv, etc
) must be capable of handling settings with: no_reserve or no_lock values.
To configure Oracle on multiple servers within a cluster it is beneficial to create all
the disk devices with the same name, by this we mean access name.
The standards are for normal DATA or LOG Disk Groups the raw disk device files are
given the alias:
/dev/emcasm00N
/dev/emcasm00N+1
The standards for the Oracle Voting and Cluster Disks are:
/dev/ora_vot_raw
/dev/ora_ocr_raw
To create these alias files create the directory: /dev/rdsk.
The command ls alF /dev/*hdisk* will display the Hard Disk Files currently known
by the Operating System:
brw------- 1 root system 24,8192 Dec 05 2001 /dev/hdiskn
crw------- 1 root system 24,8192 Dec 05 2001 /dev/rhdiskn
In this example, the device file /dev/rhdiskn represents the character raw device, 24 is
the device major number, and 8192 is the device minor number.
Enter a command similar to the following to create the new device file, specifying the
new device file name and the device major and minor numbers that you identified in
the previous step:
Note:
In the following example, you must specify the character c to create a character raw
device file.
mknod /dev/rdsk/ora_ocr_raw c 24 8192
Enter commands similar to the following to change the owner, group, and permissions
on the character raw device file for the disk:
Ben Wright
bwright@go2uti.com

27/01/2015

OCR:
chown root:oinstall /dev/rdsk/ora_ocr_raw
chmod 640 /dev/rdsk/ora_ocr_raw
Oracle Clusterware voting disk:
chown oracle:dba /dev/rdsk/ora_vote_raw
chmod 660 /dev/rdsk/ora_vote_raw
ASM disks:
Chown asm:dba /dev/emcasm00N
Chmod 660 /dev/emcasm00N
Enter a command similar to the following to verify that you have created the new
device file successfully:
ls -alF /dev/rdsk | grep "24,8192"
The output should be similar to the following:
brw------- 1 root system 24,8192 Dec 05 2001 /dev/hdiskn
crw-r----- 1 root oinstall 24,8192 Dec 05 2001 /dev/ora_ocr_raw
crw------- 1 root system 24,8192 Dec 05 2001 /dev/rhdiskn
To enable simultaneous access to a disk device from multiple nodes, you must set the
appropriate Object Data Manager (ODM) attribute, depending on the type of reserve
attribute used by your disks. The following section describes how to perform this task
using hdisk logical names. Refer to your operating system documentation to find
logical device names.
To determine the reserve setting your disks use, enter the following command,
where n is the hdisk device number:
lsattr -E -l hdiskn | grep reserve_
The response is either a reserve_lock setting, or a reserve_policy setting.
If the attribute is reserve_lock, then ensure that the setting is reserve_lock = no.
If the attribute is reserve_policy, then ensure that the setting is reserve_policy =
no_reserve.
If necessary, change the setting with the chdev command using the following syntax,
where n is the hdisk device number:
chdev -l hdiskn -a [ reserve_lock=no | reserve_policy=no_reserve ]
For example, to change a setting for the
device hdisk4 from reserve_lock=yes to reserve_lock=no, enter the following
command:
# chdev -l hdisk4 -a reserve_lock=no
To verify that the setting is correct on all disk devices, enter the following command:
# lsattr -El hdiskn | grep reserve
Ben Wright
bwright@go2uti.com

27/01/2015

Enter commands similar to the following on any node to clear the PVID from
each disk device that you want to use:
# chdev -l hdiskn -a pv=clear
When you are installing Oracle Clusterware, you must enter the paths to the
appropriate device files when prompted for the path of the OCR and Oracle
Clusterware voting disk, for example:
/dev/rdsk/ora_vot_raw
/dev/rdsk/ora_crs_raw
For DATA & LOG Disk group change the ownership of alias that has been created for
this to asm:oinstall
Ourcase
Emcasm001
Emcasm002
Note: Both oracle voting and cluster disks are must have the following owner and
permissions
/dev/ora_vot_raw oracle:dba
/dev/ora_ocr_raw root:dba
And for ASM disk owner should be like asm:dba
Configuring SSH on Cluster Member Nodes
To configure SSH, you must first create RSA or DSA keys on each cluster node, and
then copy all the keys generated on all cluster node members into an authorized keys
file that is identical on each node. Note that the SSH files must be readable only by
root and by the software installation user (oracle, crs, asm), as SSH ignores a private
key file if it is accessible by others. When this is done, then start the SSH agent to
load keys into memory. In the examples that follow, the RSA key is used.
You must configure SSH separately for each Oracle software installation owner that
you intend to use for installation.
To configure SSH, complete the following:
Create .SSH, and Create RSA Keys On Each Node
Complete the following steps on each node:
1. Log in as the software owner (in this example, the crs user).
2. To ensure that you are logged in as the Oracle user, and that the user ID
matches the expected user ID you have assigned to the Oracle user, enter the
commands id and id oracle. Ensure that Oracle user group and user and the
terminal window process group and user IDs are identical. For example:
3. $ id
4. uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)
5. $ id crs
6. uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)
Ben Wright
bwright@go2uti.com

10

27/01/2015

7. If necessary, create the .ssh directory in the crs user's home directory, and set
permissions on it to ensure that only the oracle user has read and write
permissions:
8. $ mkdir ~/.ssh
9. $ chmod 700 ~/.ssh
10. Enter the following command:
11. $ /usr/bin/ssh-keygen -t rsa
12. Important : ssh users home directory permisson must be 750.
At the prompts:
o Accept the default location for the key file (press Enter).
o Enter and confirm a pass phrase unique for this installation user.
This command writes the RSA public key to the ~/.ssh/id_rsa.pub file and the private
key to the ~/.ssh/id_rsa file.
Never distribute the private key to anyone not authorized to perform Oracle software
installations.
13. Repeat steps 1 through 4 on each node that you intend to make a member of
the cluster.
13.Repeat the steps for asm & oracle user.
Add All Keys to a Common authorized_keys File
Complete the following steps:
1. On the local node, change directories to the .ssh directory in the Oracle
Clusterware owner's home directory (typically, either crs or oracle).
Then, add the RSA key to the authorized_keys file using the following commands:
$ cat id_rsa.pub >> authorized_keys
$ ls
In the .ssh directory, you should see the id_rsa.pub keys that you have created, and the
file authorized_keys.
2. On the local node, use SCP (Secure Copy) or SFTP (Secure FTP) to copy
the authorized_keys file to the oracle user .ssh directory on a remote node. The
following example is with SCP, on a node called node2, with the Oracle
Clusterware owner crs, where the crs user path is /home/crs:
3. [crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/
You are prompted to accept an RSA key. Enter Yes, and you see that the node you are
copying to is added to the known_hosts file.
When prompted, provide the password for the crs user, which should be the same on
all nodes in the cluster. The authorized_keys file is copied to the remote node.
Your output should be similar to the following, where xxx represents parts of a valid
IP address:
[crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/
The authenticity of host 'node2 (xxx.xxx.173.152) can't be established.
RSA key fingerprint is 7e:60:60:ae:40:40:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,xxx.xxx.173.152' (RSA) to the list
of known hosts
crs@node2's password:
authorized_keys
100%
828
7.5MB/s
00:00
Ben Wright
bwright@go2uti.com

11

27/01/2015

4. Using SSH, log in to the node where you copied the authorized_keys file,
using the pass phrase you created. Then change to the .ssh directory, and using
the cat command, add the RSA keys for the second node to
the authorized_keys file:
5. [crs@node1 .ssh]$ ssh node2
6. The authenticity of host node2 (xxx.xxx.100.102) can't be established. RSA
key fingerprint is z3:z3:33:z3:z3:33:zz:76:z3:z3:z3.
7. Are you sure you want to continue connecting? (yes/no)? yes
8. Enter passphrase for key '/opt/oracle/.ssh/id_rsa':
9. [crs@node2 crs]$ cd .ssh
10. [crs@node2 ssh]$ cat id_rsa.pub >> authorized_keys
11. Make the authorized_keys permission to 600
Repeat steps 2 and 3 from each node to each other member node in the cluster.
When you have added keys from each cluster node member to
the authorized_keys file on the last node you want to have as a cluster node member,
then use scp to copy theauthorized_keys file with the keys from all nodes back to each
cluster node member, overwriting the existing version on the other nodes.
If you want to confirm that you have all nodes in the authorized_keys file, enter the
command more authorized_keys, and check to see that there is an RSA key for each
member node. The file lists the type of key (ssh-rsa), followed by the key, and then
followed by the user and server. For example:
ssh-rsa AAAABBBB . . . = crs@node1
Note:
The crs user's /.ssh/authorized_keys file on every node must contain the contents from
all of the /.ssh/id_rsa.pub files that you generated on all cluster nodes.
After you have copied the authorized_keys file that contains all keys to each node in
the cluster, complete the following procedure, in the order listed. In this example, the
Oracle Clusterware software owner is named crs:
1. On the system where you want to run OUI, log in as the crs user.
2. Use the following command syntax, where hostname1, hostname2, and so on,
are the public hostnames (alias and fully qualified domain name) of nodes in
the cluster to run SSH from the local node to each node, including from the
local node to itself, and from each node to each other node:
3. On each node, enter the following commands to start the SSH agent, and to load
the SSH keys into memory:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
At the prompt, enter the pass phrase for each key that you generated.
For example:
[crs@node1 .ssh]$ exec /usr/bin/ssh-agent $SHELL
[crs@node1 .ssh]$ ssh-add
Enter passphrase for /home/crs/.ssh/id_rsa
Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)
These commands start the ssh-agent on the node, and load the RSA keys into
memory so that you are not prompted to use pass phrases when issuing SSH
commands
If you have configured SSH correctly, then you can now use the ssh or scp commands
without being prompted for a password or a pass phrase. For example:
Ben Wright
bwright@go2uti.com

12

27/01/2015

3. [crs@nodename]$ ssh hostname1 date

4. [crs@nodename]$ ssh hostname2 date
5.
.
6.
.
7.
.
For example:
[crs@node1 crs]$ ssh node1 date
The authenticity of host 'node1 (xxx.xxx.100.101)' can't be established.
RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,xxx.xxx.100.101' (RSA) to the list of
known hosts.
Enter passphrase for key '/home/crs/.ssh/id_rsa':
Mon Dec 4 11:08:13 PST 2006
[crs@node1 crs]$ ssh node1.example.com date
The authenticity of host 'node1.example.com (xxx.xxx.100.101)' can't be
established.
RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1.example.com,xxx.xxx.100.101' (RSA) to the
list of known hosts.
Enter passphrase for key '/home/crs/.ssh/id_rsa':
Mon Dec 4 11:08:13 PST 2006
[crs@node1 crs]$ ssh node2 date
Enter passphrase for key '/home/crs/.ssh/id_rsa':
Mon Dec 4 11:08:35 PST 2006
.
.
.
At the end of this process, the public hostname for each member node should be
registered in the known_hosts file for all other cluster member nodes. Check the ssh
connectivity for the localnode too.
If you are using a remote client to connect to the local node, and you see a message
similar to "Warning: No xauth data; using fake authentication data for X11
forwarding," then this means that your authorized keys file is configured correctly, but
your ssh configuration has X11 forwarding enabled. To correct this issue, proceed
to "Setting Display and X11 Forwarding Configuration".
8. Repeat step 2 on each cluster node member.
9. On each node, enter the following commands to start the SSH agent, and to
load the SSH keys into memory:
10. $ exec /usr/bin/ssh-agent $SHELL
11. $ /usr/bin/ssh-add
At the prompt, enter the pass phrase for each key that you generated.
For example:
[crs@node1 .ssh]$ exec /usr/bin/ssh-agent $SHELL
[crs@node1 .ssh]$ ssh-add
Enter passphrase for /home/crs/.ssh/id_rsa
Ben Wright
bwright@go2uti.com

13

27/01/2015

Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)

These commands start the ssh-agent on the node, and load the RSA keys into memory
so that you are not prompted to use pass phrases when issuing SSH commands
If you have configured SSH correctly, then you can now use the ssh or scp commands
without being prompted for a password or a pass phrase. For example:
[crs@node1 ~]$ ssh node2 date
Mon Feb 26 23:34:42 UTC 2007
[crs@node1 ~]$ ssh node1 date
Mon Feb 26 23:34:48 UTC 2007
[crs@node1 ~]$ ssh node2
If any node prompts for a password or pass phrase, then verify that
the ~/.ssh/authorized_keys file on that node contains the correct public keys, and that
you have created an Oracle software owner with identical group membership and IDs.
Time Synchronization, xntpd, Mandatory
Time Synchronization should have been configured post AIX installation from the
NIM Server however it can be checked as below:
There is a general requirement for Oracle RAC that the time on all the nodes be the
same. Typically this can be achieved using Network Time Server protocol.
Oracle recommends that you use xntpd for this purpose. xntpd is a complete
implementation of the NTP version 3 satndard and is more accurate than "timed".
To configure xntpd, follow these steps on each cluster node create each of the
configuration files:
# touch /etc/ntp.drift /etc/ntp.trace /etc/ntp.conf
Then ensure that the following entries are in the /etc/ntp.conf file :
driftfile /etc/ntp.drift
tracefile /etc/ntp.trace
peer 10.240.4.100
peer 10.254.4.120
peer 10.254.4.125
2.6.2. Increasing System Block Size Allocation
Oracle recommends that you increase the space allocated for ARG/ENV list to 128.
The size is specified by number of 4K blocks.
For example:
chdev -l sys0 -a ncargs='128' Not done
Run runclufy for hwos stage check ( note in 11g it is new and different from clufy.sh).
/tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runcluvfy.sh stage -post hwos -n
sinutn02 verbose
Crs .profile changed as below,

Ben Wright
bwright@go2uti.com

14

27/01/2015

export ORACLE_BASE=/opt/app/crs/product
export AIXTHREAD_SCOPE=S #(S for system-wide thread scope)
export TEMP=/tmp//tmpexport
TMP=/tmp
export TMPDIR=/tmp
umask 022
if [ -t 0 ]; then
stty intr ^C
fi
Give the inventory location as /opt/app/crs/product/orainventory
This orainventory directory will be created by oui

$ /tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runcluvfy.sh comp ssa -n all

-verbose

export CRS_HOME=/crs/11.1.0
export ORACLE_CRS_HOME=$CRS_HOME
export ORACLE_HOME=$ORA_CRS_HOME
export LD_LIBRARY_PATH=$CRS_HOME/lib:$CRS_HOME/lib32
export LIBPATH=$LD_LIBRARY_PATH
export PATH=$CRS_HOME/bin:$PATH
Notes Specify Home Details
Specify an ORACLE_HOME name and destination directory for the CRS installation.
The destination directory should not be within the $ORACLE_BASE subtree.
Next we will install the Oracle Clusterware layer.
Oracle Clusterware is an essential component of the Oracle RAC database
infrastructure.
Oracle Clusterware does not require any other clustering software.
Oracle Clusterware will be installed in /crs/11.1.0 ($CRS_HOME,
$ORA_CRS_HOME) on each node.
But we have used
/opt/app/crs/product/11.1.0/crs as crs_home
You may install the clusterware where you please with these exceptions.
Do not install the clusterware in a directory which is sub-directory of either the
intended install location
for ASM or the Database.
Ben Wright
bwright@go2uti.com

15

27/01/2015

Failure to observe this rule could well mean that other people when removing an ASM
or Oracle home will
destroy your clusterware installation.
Vipca called maulally and OS bug issue solved by oracle by changing the racgvip
script
Modify the script "racgvip" in <CRS-HOME>/bin to pick up the value of the 6th
column ($6) instead of the 5th column ($5) - replace $5 with $6 Clusterware script
"racgvip" lines #263 and #275;
from
_O1=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"`
and
_O2=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"`
to
_O1=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$6; exit}}"`
and
_O2=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$6; exit}}"`
Install patch 11.1.0.7
/tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runclufy stage -post crsinst -n
sinutn02
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
6.1. CVU check before creating ASM instance
Verify the cluster is configured correctly for an instance creation,as asm user
/tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runclufy stage -pre dbinst -n
node1,node2 -r 11gR1 -verbose
Show CVU -pre dbinst output Hide CVU output
6. Oracle ASM Home Software Install
In this chapter we are going to install the asm software (only). Creating the asm
storage is in the next chapter.
The software will be installed in a directory separate from any crs or database home,
preferably in the $ORACLE_BASE tree and not as a subtree in the database home.
Create group Asm for asm user
Create a asm user - primary group is oinstall and secondary group is asm & asmdba

Ben Wright
bwright@go2uti.com

16

27/01/2015

/opt/app/oracle ownership has been changed to oracle:oinstall

Create a asm & oracle base directory as /opt/app/oracle/product as oracle user
Change the product directory permission 777
Asm home is /opt/app/oracle/product/11.1.0/asm
Oracle base is /opt/app/oracle/product
In the asm & rbdms directory
/oracle/asm
/oracle/rdbms
Login as the asm owner (asm in our case). In vnc
Start the vncserver and login to gui.
Setup and export your DISPLAY, TMP and TEMP variables With /tmp or other
destination having enough
free space, about 500Mb on each node.
Please note the actual display may be different. For instance if you are running from
the console this will
typically be :0.0 The display setting and the xhost setting can be checked by typing
'xclock' if you see the
clock then you are good to go.
{node1:asm} export DISPLAY=node1:1
{node1:asm} export TMP=/tmp
{node1:asm} TEMP=/tmp
{node1:asm} export TMPDIR=/tmp
/tmp/net/oracle/5L/11G/rdbms_11gr1/11gR1_2of2/database
Call .runintsaller
Once software installation finshed. Then change the asm profile as below.
vi $HOME/.profile file in the asm users home directory.
Add the entries in bold.
PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.
export PATH
if [ -s "$MAIL" ] # This is at Shell startup. In normal
then echo "$MAILMSG" # operation, the Shell checks
fi # periodically.
Ben Wright
bwright@go2uti.com

17

27/01/2015

ENV=$HOME/.kshrc
export ENV
#The following line is added by License Use Management installation
export PATH=$PATH:/usr/opt/ifor/ls/os/aix/bin
export PATH=$PATH:/usr/java14/bin
export MANPATH=$MANPATH:/usr/local/man
export ORACLE_BASE=/opt/app/oracle
export AIXTHREAD_SCOPE=S
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
umask 022
export CRS_HOME=/opt/app/crs/product/11.1.0/crs
#note for future installation limit the home to 11.1.0
export ORACLE_CRS_HOME=$CRS_HOME
export ORACLE_HOME=$ORACLE_BASE/asm/product/11.1.0/asm
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$CRS_HOME/lib:
$ORACLE_HOME/lib32:$CRS_HOME/export LIBPATH=$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORACLE_SID=+ASM1
similarly in other nodes.note change the sid accordinly.
reload profile
8. Oracle RAC Database Listener Creation
The Oracle network listeners traditionally run from the ASM home. Here we are
going to create the listeners
using netca from the ASM home.
8.1. Create Node specific network listeners
As asm user.
Before you start the Oracle Universal Installer you need to execute the following
command in the same session as in which you will start the OUI.
On the system where you want to install the software log in as the user oracle.
Enter the following commands:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

After the prompt you need to add the passphrase for each key you generated.
As we have not given passphrase during ssh generation you need not enter any thing.
Just press enter you will get the message that identity added.

Ben Wright
bwright@go2uti.com

18

27/01/2015

Now you must be able to create a ssh connection without being prompted for a
password
Test this before you start the OUI.
$ssh nodename date
If you don't get prompted for a password your ssh is working correctly.
cd $ORACLE_HOME/bin asms oracle home
./netca a
Note: Check if both the nodes listeners are up and running fine or not. If not then add
listener which is not running by using srvctl and start the same.
Call dbca from ASMs oracle_home for ASM Instance installation.
As asm user
cd $ORACLE_HOME/bin asms oracle home.
./dbca

Oracle 10G software installation for DBs

{node1:root}/ # chown rdbms:oinstall /opt/app/oracle/rdbms
{node1:root}/ # chmod 665 /oracle/rdbms
{node1:root}/
{node2:root}/ # chown oracle:oinstall /opt/app/oracle/rdbms
{node2:root}/ # chmod 765 /opt/app/oracle/rdbms
{node2:root}/ #
Setup and export your DISPLAY, TMP and TEMP variables
With /tmp or other destination having enough free space, about 500Mb on each node.
In the oracle profile put the below entry.
$ vi .profile
export TMP=/tmp
export TMPDIR=/tmp
umask 022
if [ -t 0 ]; then
stty intr ^C
fi
umask 022

Ben Wright
bwright@go2uti.com

19

27/01/2015

export CRS_HOME=/opt/app/crs/product/11.1.0/crs
#note for future installation limit the home to 11.1.0
export ORACLE_CRS_HOME=$CRS_HOME
#export ORACLE_HOME=$ORACLE_BASE/11.1.0/asm
export PATH=$CRS_HOME/bin:$PATH
#export TNS_ADMIN=/opt/app/oracle/product/11.1.0/asm/network/admin
As a oracle user give crs_stat see asm,listener are running
./runcluvfy.sh stage -pre dbinst -n sinutn02 -r 10gR2 -verbose

Run /tmp/net/oracle/5L/10G/extra/p6613550_10203_AIX64-5L/rootpre.sh
/tmp/net/oracle/5L/10G/rmdb_10gr2/Disk./runinstaller
Choose enterprise edition
In the specify home details

9.2. Completed creation of ASM Instance and add of the DATA_DG

diskgroup
Congratulations you have installed ASM, Network listeners and created an ASM
diskgroup.
10. Oracle RAC Database Home Software Install
10.1. CVU check -pre Database Instance Install
Before installing the RAC Database software you should use the CVU utility to
confirm the status of the
cluster.
/tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runcluvfy.sh stage -pre dbinst -n
stnsp001,stnsp002 -r 11gR1 verbose
For our case
/tmp/net/oracle/5L/11G/cluster_11gr1/clusterware/runcluvfy.sh stage -pre dbinst -n
sinutn02 -r 10gR2 verbose
12. Oracle RAC Database Creation
Ben Wright
bwright@go2uti.com

20

27/01/2015

We are nearly ready to use the Database Configuration Assistant (dbca) to create the
Oracle RAC instances on
all the nodes.
MANDATORY: Change permissions to allow the rdbms user to write to directories
owned by the asm user,
from node1 execute:
# chmod -R g+w /oracle/asm/11.1.0/network
# chmod -R g+w /oracle/cfgtoollogs
# chmod -R g+w /oracle/admin
# chmod -R g+w /oracle/diag
# rsh node2 chmod -R g+w /oracle/asm/11.1.0/network
# rsh node2 chmod -R g+w /oracle/cfgtoollogs
# rsh node2 chmod -R g+w /oracle/admin
# rsh node2 chmod -R g+w /oracle/diag
For our case use the below.
$ chmod -R g+w /opt/oracle/product/11.1.0/asm/network
$ ssh sinuqn02 chmod -R g+w /opt/oracle/product/11.1.0/asm/network
chmod -R g+w /opt/oracle/product/diag
ssh sinuqn02 chmod -R g+w /opt/oracle/product/diag
chmod -R g+w /opt/oracle/product/admin
ssh sinuqn02 chmod -R g+w /opt/oracle/product/admin
Note: If you executed the CVU -pre dbinst command during the ASM instance
installation, you do
not have the execute the command again.
10.2. Database Software Install
On each node
Run the AIX command "/usr/sbin/slibclean" as "root" to clean all unreferenced
libraries from memory.
{node1:root}/ # /usr/sbin/slibclean
{node2:root}/ # /usr/sbin/slibclean
9.1. Verify ASM instances are online
As rdbms user from node1 :
{node1:rdbms} $CRS_HOME/bin/crs_stat -t
Name Type Target State Host
-----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE ONLINE node1
ora.node1.gsd application ONLINE ONLINE node1
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip application ONLINE ONLINE node1
Ben Wright
bwright@go2uti.com

21

27/01/2015

ora....SM2.asm application ONLINE ONLINE node2

ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application ONLINE ONLINE node2
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip application ONLINE ONLINE node2

Show CVU -pre dbinst output Hide CVU output

The oui will give error for Clusterware version,just check mark it in oui.
In oui choose not to upgrade
Select software only
Once software installation done.
10.3.Symbolic links creation for listener.ora, tnsnames.ora and
sqlnet.ora
Create the symbolic links, make sure that the files 'listener.ora', 'tnsnames.ora' and
'sqlnet.ora' exist in
/oracle/asm/11.1.0/network/admin/listener.ora on both nodes.
As rdbms user on each node :
{node1:oracle}/ # ln -s /opt/oracle/product/11.1.0/asm/network/admin/listener.ora
/opt/oracle/product/10.2/db_1/network/admin/listener.ora
{node1:oracle}/ # ln -s /opt/oracle/product/11.1.0/asm/network/admin/tnsnames.ora
/opt/oracle/product/10.2/db_1/network/admin/tnsnames.ora
{node1:oracle}/ # ln -s /opt/oracle/product/11.1.0/asm/network/admin/sqlnet.ora
/opt/oracle/product/10.2/db_1/network/admin/sqlnet.ora
On both the nodes create this links
Check the same using ls l /opt/oracle/product/11.1.0/asm/network/admin/*.ora
Now modify /add the following to oracles .profile.
ORACLE_HOME=/opt/app/oracle/product/10.2/db_1
PATH=$PATH:/usr/sbin:/usr/bin:/usr/local/bin:$ORACLE_HOME/bin:
$CRS_HOME/bin:.
PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.
export PATH
if [ -s "$MAIL" ] # This is at Shell startup. In normal
then echo "$MAILMSG" # operation, the Shell checks
fi # periodically.
ENV=$HOME/.kshrc
export ENV
Ben Wright
bwright@go2uti.com

22

27/01/2015

#The following line is added by License Use Management installation

export PATH=$PATH:/usr/opt/ifor/ls/os/aix/bin
export PATH=$PATH:/usr/java14/bin
export MANPATH=$MANPATH:/usr/local/man
export ORACLE_BASE=/opt/oracle/product
export AIXTHREAD_SCOPE=S
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
umask 022
export CRS_HOME=/opt/oracle/crs/product/11.1.0
export ORA_CRS_HOME=$CRS_HOME
export ASM_HOME=$ORACLE_BASE/11.1.0/asm
export ORA_ASM_HOME=$ASM_HOME
export ORACLE_HOME=$ORACLE_BASE/10.2/db_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$CRS_HOME/lib:
$ORACLE_HOME/lib32:$CRS_HOME/export LIBPATH=$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH
export TNS_ADMIN=$ASM_HOME/network/admin

Database Servers Connectivity Details:

SNo

DB Name

1
2

SINOQ01
SINOQ02

Service
Name
SINO_Q01
SINO_Q02

Server Name

IP address

Port Number

sinuqn01-vip
sinuqn02-vip

10.240.8.210
10.240.8.211

1521
1521

How to Connect
a) Using TNS entry:
For to connect SINOQ01 Database
SINO_Q01 =
(DESCRIPTION =
(ADDRESSLIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn01-vip)(PORT = 1521)
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn02-vip)(PORT = 1521)))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = SINO_Q01.qc.go2uti.com)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
Ben Wright
bwright@go2uti.com

23

27/01/2015

(RETRIES = 180)
(DELAY = 5)
)
)
)
For example
sqlplus DDMYUTIOWNER/xxxx@SINO_Q01

For to connect SINOQ02 database

SINO_Q02 =
(DESCRIPTION =
(ADDRESSLIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn01-vip)(PORT = 1521)
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn02-vip)(PORT= 1521)))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = SINO_Q02.qc.go2uti.com)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)
For example
sqlplus lportal/xxxx@SINO_Q02
b) JDBC connectivity:
For to connect SINOQ01 database
sqlplus
vip:1521/SINO_Q01.qc.go2uti.com

DDMYUTIOWNER/xxxx@sinuqn01-

url="jdbc:oracle:thin:@(description=
(load_balance = off)
(ADDRESSLIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn02-vip)(PORT = 1521)
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn01-vip)(PORT = 1521)))
(connect_data = (service_name = SINO_Q01.qc.go2uti.com)))"
For to connect SINOQ02 database
sqlplus lportal/xxxx@sinuqn02-vip:1521/SINO_Q02.qc.go2uti.com
Ben Wright
bwright@go2uti.com

24

27/01/2015

url="jdbc:oracle:thin:@(description=
(load_balance = off)
(ADDRESSLIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =sinuqn02-vip)(PORT=1521)
(ADDRESS = (PROTOCOL = TCP)(HOST = sinuqn01-vip)(PORT = 1521)))
(connect_data = (service_name = SINO_Q02.qc.go2uti.com)))"

Ben Wright
bwright@go2uti.com

25